Configuration and concurrency

How client settings are resolved, what every client and call option does, and how to run many evaluations at once without exhausting your connection pool.

Option precedence

Every TypeSafeAPI.Client setting is resolved in this order, first match wins:

1. the option passed to TypeSafeAPI.new/1
2. application config: config :typesafe_api, api_key: "...", base_url: "...", model: "..."
3. an environment variable, for the three settings that have one: TYPESAFE_API_KEY, TYPESAFE_BASE_URL, TYPESAFE_DEFAULT_MODEL
4. the built-in default

timeout and connect_timeout have no environment variable; they fall back straight from the explicit option (or app config, config :typesafe_api, timeout: ...) to their default. TYPESAFE_LOG_LEVEL is separate from client configuration: it is read by TypeSafeAPI.Telemetry.attach_logger/1, not by TypeSafeAPI.new/1, and only when no :level option is given to attach_logger/1.

An empty-string environment variable or app config value is treated as unset and falls through to the next source, so an env var left blank in a shell profile does not shadow app config underneath it.

Client options

TypeSafeAPI.new/1 accepts:

• api_key — string, no default; new/1 raises ArgumentError when none is found anywhere in the precedence chain.
• base_url — string, defaults to https://api.typesafe.ai.
• model — string, defaults to jev-latest, a pinned alias the API resolves to a versioned id. Call TypeSafeAPI.models/1 for the ids your account can pin to instead.
• timeout — positive integer milliseconds, default 10_000. Sets receive_timeout for every call made with this client, unless overridden per call.
• connect_timeout — positive integer milliseconds, default 5_000. How long to wait for the TCP/TLS handshake. Client-level only — see "Finch pools" below for why.
• retry — keyword list, default [], passed to TypeSafeAPI.Retry.new/1.
• req_options — keyword list, default []. Escape hatch merged into the underlying Req.Request; see "Sharing your existing Finch pool" below.

Per-call options on TypeSafeAPI.evaluate/4 (and evaluate_prepared/4) are a subset: model, timeout, retry, req_options, telemetry. Each overrides the client's value for that one call; req_options merges on top of the client's req_options rather than replacing it, so a per-call option wins only for the keys it sets.

Finch pools and connection reuse

Req selects (or starts) a Finch connection pool by the connection settings of a request. Two requests whose connect_options differ do not share a pool, so varying the wrong option per call quietly throws away connection reuse.

What selects a pool:

• the client's finch: option — names a pool you start and supervise yourself.
• req_options carrying connect_options: — a different value is a different pool. What this library sets as the connect timeout is merged under yours, so connect_timeout survives unless you set timeout: inside connect_options.
• the client's connect_timeout, which is exactly the connect_options: [timeout: ...] this library sets; it is fixed for the life of a client.

What does not select a pool: the client's timeout and the per-call :timeout, which means the same thing in evaluate_many/4 — they only set receive_timeout; also :retry, :telemetry, headers, the body, and the path. Vary :timeout per call as much as you like; set connect_options once, on the client.

evaluate_many/4: batching many states

TypeSafeAPI.evaluate_many(client, states, questions,
  max_concurrency: 8,
  task_timeout: 40_000,
  timeout: 5_000,
  ordered: true,
  on_error: :collect
)

The question set is validated and encoded once, then each state runs as its own task under Task.Supervisor.async_stream_nolink/6, on the supervisor this library's application starts. The tasks are deliberately not linked to the caller: a task that raises becomes one failed outcome instead of taking the calling process, and every other in-flight request, down with it. Options:

• max_concurrency — positive integer, default 8. A guess: TypeSafe has not published rate limits. Raise it if your account allows, and watch for :rate_limited errors, which the retry policy already absorbs short bursts of.
• task_timeout — the cap in milliseconds on one state's whole task, covering every retry and every backoff delay. Defaults to the retry budget plus one attempt timeout (40_000 ms with every other default), or 60_000 when the budget is disabled (retry: [budget: nil]).
• ordered — boolean, default true. false yields outcomes as they finish rather than in input order.
• on_error — :collect (default) returns {:error, _} in place for a failed state without failing the batch, or :raise to raise the first error once every task has been collected. Under ordered: false that "first" error is the first task to finish, not the earliest state in the input.

Every other option (model, timeout, retry, req_options, telemetry) passes through to each individual evaluate/4 call and means exactly what it means there. :timeout in particular is still one HTTP attempt, not the task.

An invalid question set is the one failure with no per-state slot to sit in, so it comes back as a bare {:error, error} rather than a list. Under on_error: :raise it raises like any other error. Pass a %TypeSafeAPI.SystemOne.Prepared{} from TypeSafeAPI.SystemOne.prepare/1 instead of a question set to skip that path.

timeout vs task_timeout, concretely

The task timeout must always be large enough to let one call exhaust its retry budget; if it is not, the stream kills a task that was still legitimately retrying. That is why the default task_timeout is derived from the retry budget:

default task_timeout = retry.budget + timeout   # 40_000 ms with defaults
                     = 30_000       + 10_000

If you shorten timeout to fail fast on one slow question, either leave task_timeout alone (it still generously covers the shorter attempts) or shorten both:

TypeSafeAPI.evaluate_many(client, states, questions,
  timeout: 3_000,        # give up on a single slow attempt sooner
  task_timeout: 15_000   # and don't let the task run much longer than that
)

The Task.async_stream timeout gotcha

If you roll your own fan-out instead of evaluate_many/4 — say, to interleave other work between calls — remember that Task.async_stream/3's :timeout option is a per-task wall-clock deadline, not a per-HTTP-attempt one. A task that is still inside its second or third retry when the deadline hits is killed outright (on_timeout: :kill_task), and the caller sees {:exit, :timeout}, not an ordinary TypeSafeAPI.Error. Size that timeout the same way evaluate_many/4 does: at least the retry budget plus one attempt timeout, never just the attempt timeout alone. evaluate_many/4's own unwrap/1 turns that {:exit, :timeout} into %TypeSafeAPI.Error{type: :timeout} for you; a hand-rolled Task.async_stream/3 call gets the raw exit tuple and has to handle it itself.

Sharing your existing Finch pool

Shops that already run Finch (directly, or through another Req- or Tesla-based client) can point this library at that same pool instead of letting it start its own, with the finch: client option. It is also the only way to control pool size:

# application.ex
children = [
  {Finch, name: MyApp.Finch, pools: %{default: [size: 50, count: 2]}}
  # ... your other children
]

# wherever you build the client
TypeSafeAPI.new(
  api_key: System.fetch_env!("TYPESAFE_API_KEY"),
  finch: MyApp.Finch
)

connect_timeout does not apply to a pool you own: Req refuses :finch and :connect_options together, so a client with finch: sends neither. Connection settings for MyApp.Finch belong in its own child spec, which is where you were already configuring them. Setting finch: alongside a connect_options in req_options raises at TypeSafeAPI.new/1 rather than on every request.

:finch in req_options is rejected outright, with a message naming the client option — as are :retry, :auth and :base_url, each of which has one.

Proxy and TLS options

A proxy or custom TLS configuration goes through req_options' connect_options, same as any other Req client:

TypeSafeAPI.new(
  api_key: api_key,
  req_options: [
    connect_options: [
      timeout: 5_000,
      proxy: {:http, "proxy.internal", 8080, []},
      transport_opts: [cacertfile: "/etc/ssl/certs/internal-ca.pem"]
    ]
  ]
)

Set this once, on the client. A connect_options that differs between calls (for example, passed per-call instead of on the client) starts a new Finch pool for every distinct value, defeating connection reuse — the same reason connect_timeout is a client-only setting.