# `TypeSafeAPI.Test`
[🔗](https://github.com/typesend/typesafe_ai/blob/v0.1.0-alpha.3/lib/typesafe_api/test.ex#L1)

Stub the TypeSafe API in your tests without fixtures or a network.

Add `plug` to your test dependencies (`{:plug, "~> 1.16", only: :test}`),
then build a test client and describe the answers you want by question id:

    # test/test_helper.exs
    ExUnit.start()

    # in a test
    setup :typesafe_stubs

    test "routes billing tickets" do
      client = TypeSafeAPI.Test.client()

      TypeSafeAPI.Test.stub(client,
        dept: {:choice, :billing, 0.9},
        urgent: {:noul, 0.3},
        anger: {:score, 1, 0.8}
      )

      assert {:ok, result} = MyApp.Triage.run(client, "Where is my refund?")
      assert result.answers.dept.choice == :billing
    end

The stub reads the questions in each request and builds a response the way
the API would, so the decoded structs are identical to those from a real
call: probabilities are spread over exactly the options or levels you sent,
they sum to one, the winning option or level is the argmax, Score answers
carry `level`, `label` and a `legend` echoing your own labels, and keys come
back as atoms or strings exactly as you sent them.

## Answer specs

  * `{:noul, probability}` - `probability` is in `0.0..1.0`
  * `{:choice, option, confidence}` - `option` is one of the question's keys;
    the remaining probability is spread evenly across the other options
  * `{:score, level, confidence}` - `level` is a 0-based index; the remaining
    probability is spread evenly across the other levels

`confidence` has to beat the uniform baseline, which is `1 / n` for a
question with `n` options or levels. Below it the option you named would not
be the argmax and the answer would contradict itself, so the stub rejects it
with a message naming the minimum for that question. A shape that is wrong
whatever the question (a confidence above `1.0`, a negative level) is
rejected by `stub/2` itself, before any request is made.

A request that asks a question you did not stub is answered with HTTP 422 and
a message naming the question, so `TypeSafeAPI.evaluate/4` returns
`{:error, %TypeSafeAPI.Error{type: :validation}}` and a test cannot silently
pass on a default answer. It is an error response rather than a raise so that
a missing stub under `TypeSafeAPI.evaluate_many/4` surfaces as one error per
state instead of an exception inside a task that takes the test process with
it.

## Composing stubs

Every helper adds to the same stub, so a client can serve several endpoints
at once and a later call refines an earlier one:

    client =
      TypeSafeAPI.Test.client()
      |> TypeSafeAPI.Test.stub_models([%{name: "jev-1.13.0"}])
      |> TypeSafeAPI.Test.stub(urgent: {:noul, 0.3})
      |> TypeSafeAPI.Test.stub(dept: {:choice, :billing, 0.9})

Requests are dispatched on method and path: `GET /v1/models` is served by
`stub_models/2` and `POST /v1/systemone` by `stub/2`. Anything else gets a
422 naming the route.

## Errors

`stub_error/4` queues an error response. By default it answers every request
from then on; `times:` limits it to the next call or calls, which is how you
test a retry or a recovery path:

    client
    |> TypeSafeAPI.Test.stub(urgent: {:noul, 0.3})
    |> TypeSafeAPI.Test.stub_error(429, %{"error" => "slow down"},
      headers: [{"retry-after", "1"}],
      times: 1
    )

The first call gets the 429; the retry gets the stubbed answers.

## Recording fixtures

When the answers you want are the ones the real model gives, record them once
and replay them forever after. `record/2` wraps a client so every response is
appended to a fixture file; `replay/2` turns that file back into a client:

    # once, against the real API
    TypeSafeAPI.new()
    |> TypeSafeAPI.Test.record("test/fixtures/triage.jsonl")
    |> MyApp.Triage.run("Where is my refund?")

    # in every test run afterwards, with no network
    test "routes billing tickets" do
      client = TypeSafeAPI.Test.replay("test/fixtures/triage.jsonl")

      assert {:ok, result} = MyApp.Triage.run(client, "Where is my refund?")
      assert result.answers.dept.choice == :billing
    end

The replayed `TypeSafeAPI.Result` is the one the API sent, decoded by the same
code, so it is identical to the recorded run. Requests are matched by method,
path and body, and the same request twice gets the two recordings in order;
pass `match: [:method, :path]` to ignore the body. A request that matches
nothing raises with the request, the closest unserved recording and the first
field they differ on.

### Fixture format

A fixture is [JSON Lines](https://jsonlines.org): one JSON object per line,
in the order the responses arrived, so appending one entry is one write and a
recorded diff reads one response per line. Each line is

    {"request":{"method":...,"path":...,"body":...},
     "response":{"status":...,"headers":...,"body":...}}

Only the request method, path and JSON body are written, never request
headers, so an API key cannot reach a fixture that you commit. Of the
response headers only `content-type` and `x-typesafe-request-id` are kept.
The format is what `record/2` writes and `replay/2` reads; treat a fixture as
generated output and re-record it rather than editing it by hand. A fixture
written as a whole-file JSON array by an earlier version still loads.

Every response with an HTTP status is recorded, including 4xx and 5xx, so a
test can replay the error path too. A transport failure never reached the
server and is not recorded. A run that was retried records each attempt, so
replaying it reproduces that same sequence: give the replay client a retry
policy (`replay(path, retry: [max_retries: 2])`) when you want the retry to
be replayed rather than surfaced.

### Ordering

Entries are appended as responses arrive. Under `TypeSafeAPI.evaluate_many/4`
that order is the order the concurrent calls finished in, not the order of
the input states, and it differs from run to run. Body matching (the default)
is unaffected, because each request finds its own recording wherever it sits.
`match: [:method, :path]` pairs requests to recordings by position instead, so
use it only for fixtures recorded by sequential calls.

Unlike a missing stub, a request that matches no recording raises rather than
answering 422, because the message is the whole value of a fixture mismatch.
Under `TypeSafeAPI.evaluate_many/4` that raise happens inside a task, so
replay a fan-out only from a fixture that holds a recording for every state.

## Concurrency

Stubs use `Req.Test`, which follows the ownership model of `Mox`: call
`Req.Test.set_req_test_from_context/1` in `setup` (or the `typesafe_stubs/1`
helper here) and stubs are private to each async test. The stubs a client
accumulates are tracked per process, so build them from the test process;
the tasks `TypeSafeAPI.evaluate_many/4` starts reach them through `$callers`
the same way `Req.Test` ownership does.

# `answer_spec`

```elixir
@type answer_spec() ::
  {:noul, number()}
  | {:choice, atom() | String.t(), number()}
  | {:score, non_neg_integer(), number()}
```

How to answer one question.

# `error_option`

```elixir
@type error_option() ::
  {:headers, [{String.t(), String.t()}]}
  | {:times, pos_integer() | :infinity}
  | {:path, String.t()}
```

Options for `stub_error/4`.

# `client`

```elixir
@spec client(keyword()) :: TypeSafeAPI.Client.t()
```

Builds a client whose requests are served by this module's stubs.

Options are passed to `TypeSafeAPI.new/1`; `api_key` defaults to `"test-key"`
and retries are disabled unless you set `:retry`. Pass `:name` to use a
custom `Req.Test` stub name.

# `json`

```elixir
@spec json(Plug.Conn.t(), pos_integer(), term(), [{String.t(), String.t()}]) ::
  Plug.Conn.t()
```

Sends a JSON response from inside a custom `Req.Test` stub, for cases the
built-in stubs do not cover:

    Req.Test.stub(TypeSafeAPI.Test, fn conn ->
      TypeSafeAPI.Test.json(conn, 200, %{"models" => []})
    end)

# `record`

```elixir
@spec record(TypeSafeAPI.Client.t(), Path.t()) :: TypeSafeAPI.Client.t()
```

Wraps `client` so every response is appended to the fixture at `path`, and
returns it.

Run it once against the real API (or any stub), then hand the file to
`replay/2` and your tests run offline. Each entry holds the request method,
path and decoded JSON body, and the response status, decoded JSON body and
the `content-type` and `x-typesafe-request-id` headers. Request headers are
never written, so the `Authorization` header and your API key stay out of the
fixture. See the "Fixture format" section above for the file layout and what
is and is not recorded.

# `replay`

```elixir
@spec replay(Path.t(), keyword()) :: TypeSafeAPI.Client.t()
```

Builds a client that answers from the recordings in `path`.

Requests are matched against the fixture by method, path and body; repeated
matches are served in the order they were recorded. A request that matches no
remaining recording raises `ArgumentError` naming the request, the closest
unserved recording and the first field they differ on.

## Options

  * `:match` - which parts of a request must be equal, any of `:method`,
    `:path` and `:body`. Defaults to all three; `match: [:method, :path]`
    ignores the body, which is what you want when the state text varies.
    It pairs requests to recordings by position, so use it only on a fixture
    recorded by sequential calls
  * `:name` - a custom `Req.Test` stub name; one is generated per call
  * anything else is passed to `client/1`, including `:retry`, which is what
    you want when the fixture holds a retried sequence

# `stub`

```elixir
@spec stub(
  TypeSafeAPI.Client.t(),
  keyword() | %{optional(atom() | String.t()) => answer_spec()}
) ::
  TypeSafeAPI.Client.t()
```

Stubs the evaluation endpoint with one answer per question id.

Ids given here are merged into whatever this client already answers, so
several calls compose and the last spec for an id wins. Returns the client so
the call can be piped.

Raises `ArgumentError` for a spec that cannot be right for any question: a
probability or confidence outside `0.0..1.0`, a negative level, or a shape
that is not one of the three in `t:answer_spec/0`. A spec that only the
question can judge - an option the question does not have, a level past its
last one, a confidence under that question's uniform baseline - comes back
as an HTTP 422 when the request is made.

# `stub_error`

```elixir
@spec stub_error(TypeSafeAPI.Client.t(), pos_integer(), term(), [error_option()]) ::
  TypeSafeAPI.Client.t()
```

Queues an HTTP error response.

It takes priority over the other stubs on this client, so a client can hold
both an error and the answers that follow it.

## Options

  * `:headers` - response headers, as `{name, value}` pairs. Defaults to none
  * `:times` - how many requests this error answers, a positive integer or
    `:infinity`. Defaults to `:infinity`, which is every request from now on
  * `:path` - only answer requests whose path ends with this string.
    Defaults to answering every path

## Examples

    TypeSafeAPI.Test.stub_error(client, 429, %{"error" => "slow down"},
      headers: [{"retry-after", "1"}],
      times: 1
    )

# `stub_models`

```elixir
@spec stub_models(TypeSafeAPI.Client.t(), [map()]) :: TypeSafeAPI.Client.t()
```

Stubs the models endpoint. Each entry needs a `:name`; `:description` and
`:release_date` are optional.

Replaces whatever list this client served before, and leaves its other stubs
alone.

# `typesafe_stubs`

```elixir
@spec typesafe_stubs(map()) :: :ok
```

An ExUnit setup callback: `setup :typesafe_stubs`.

Makes stubs private to async tests and shared otherwise.

---

*Consult [api-reference.md](api-reference.md) for complete listing*