Making Calls¶

Suspending calls¶

JevClient implements the JevApi interface. Its calls are suspend functions, so call them from a coroutine:

// JevApi calls are suspend functions: call them from a coroutine.
val result = jev.ask(Triage, state = "Our dashboard has been down for an hour.")

evaluate and models also take per-call options.

query and ask are extension functions on JevApi. Import them with import com.pambrose.jev4k.query and import com.pambrose.jev4k.ask; IDEs add these automatically.

// evaluate is the one network call everything else builds on.
val result = jev.evaluate(JsonPrimitive("Refund me now!"), Triage.questions, model = null)

Blocking calls¶

On the JVM, jev.blocking mirrors every call without coroutines, for scripts, main, tests and Java callers:

// Outside coroutines (scripts, main, Java callers), use the blocking mirror.
JevClient().use { jev ->
    val result = jev.blocking.ask(Triage, state = "Our dashboard has been down for an hour.")
    println(result[Triage.team].choice)
    println(jev.blocking.models().map { it.name })
}

Any other JevApi, such as a fake or a per-call options view, has the same blocking calls through blocking():

// Code handed a JevApi, such as a fake in a test, gets the same blocking calls from blocking().
val result = api.blocking().ask(Triage, state = "Our dashboard has been down for an hour.")

Blocking calls block the calling thread. Don't use them from inside a coroutine. An interrupted thread gets an InterruptedException, which every blocking call declares, so Java code can catch it. They exist on the JVM only; on every other platform BlockingJev has no members. Kotlin/JS and Kotlin/Wasm can't block a thread at all, and Kotlin/Native code that needs to can wrap the suspend calls in runBlocking itself.

Choosing a model¶

Every call takes an optional model, which overrides the client's defaultModel for that request:

// Override the model per call. Pin a versioned model once you've tuned thresholds against it,
// because an alias such as jev-latest can move to a newer release.
val pinned = jev.ask(Triage, state = "The API returns 500s.", model = "jev-1.14.0")
println("answered by ${pinned.model}")

// List the models this account can use. The list also carries the call's request id.
val models = jev.models()
models.forEach { println("${it.name} (${it.releaseDate}): ${it.description}") }
println("request ${models.requestId}")

jev-latest always points at the newest stable model, so its answers can shift when TypeSafe ships a release. Once you've tuned thresholds against a model, pin its version (for example jev-1.14.0) and move to a new one deliberately.

Per-call options¶

The client's timeout, retry policy and headers suit most calls. JevCallOptions changes them for the calls that need something else, as both official SDKs allow, without a second client:

// A tight budget for the interactive path, on the same client as everything else.
val interactive =
    JevCallOptions {
        timeout = 2.seconds             // per attempt, instead of the client's
        retry = RetryPolicy.NONE        // replaces the client's whole policy
        headers["X-Trace-Id"] = traceId // over the client's and the built-in headers
    }
val urgent = jev.withOptions(interactive).ask(Triage, state = "Checkout fails for every customer.")

// Or for one evaluate call, adding a top-level body field that jev4k doesn't model.
val tuned = JevCallOptions { extraBody["beam_width"] = JsonPrimitive(4) }
val result = jev.evaluate(JsonPrimitive("Refund me now!"), Triage.questions, model = null, options = tuned)

A setting left unset keeps the client's value. jev.withOptions(options) returns a JevApi that applies them to every call, query, ask and blocking() included; options passed to a call on it are merged over its own. Options are checked when they're built, and a problem is a JevConfigException. Headers follow the client's rules: names are HTTP tokens and values ASCII, and Content-Length, Content-Type, Transfer-Encoding and Upgrade can't be set. Each extraBody field may nest at most 64 levels. extraBody is an escape hatch for API parameters jev4k doesn't model yet; whether the service accepts a field is up to TypeSafe.

Long context¶

A request has to fit two context limits: 64k tokens for the state plus every question, and 32k for the state plus any one question. TypeSafe's TypeSafe-Danger-Zone-Long-Context: true header raises both to 262k tokens, on jev-1.13.0 and jev-1.14.0, at the same price per token. A long-context request is slower and less accurate, so send the header only on the calls that need it:

// Both context limits rise to 262k tokens for the calls made through longContext.
val longContext = JevCallOptions { headers["TypeSafe-Danger-Zone-Long-Context"] = "true" }
val result = jev.withOptions(longContext).ask(Triage, state = transcript)

Concurrency¶

Suspending calls make it easy to process many states at once. Bound the parallelism, though: TypeSafe's examples found that about 8 concurrent requests on a shared key already hit rate limits.

// Run many requests concurrently, but bound the fan-out: shared keys hit rate limits at around 8 at once.
suspend fun triageBatch(
    jev: JevApi,
    tickets: List<String>,
    parallelism: Int = 4,
): List<Team> {
    val permits = Semaphore(parallelism)
    return coroutineScope {
        tickets
            .map { ticket -> async { permits.withPermit { jev.ask(Triage, state = ticket)[Triage.team].choice } } }
            .awaitAll()
    }
}

Rate-limited requests are retried automatically (see Retries & Errors), but staying under the limit is faster than being retried.

Before reaching for concurrency, check whether the questions could go in one request instead. Questions about the same state are answered in parallel for the cost of one request; see Speculative Fan-Out.

Designing for testability¶

Depend on the JevApi interface rather than JevClient, and pass it in:

// Depend on the JevApi interface, not JevClient, and pass it in.
class TicketRouter(
    private val jev: JevApi,
) {
    suspend fun queueFor(ticket: String): String =
        when (jev.ask(Triage, state = ticket)[Triage.team].choice) {
            Team.BILLING -> "billing"
            Team.TECHNICAL -> "engineering"
            Team.SALES -> "sales"
        }
}

// In production, wire in the real client; elsewhere, any JevApi implementation will do.
fun productionRouter(): TicketRouter = TicketRouter(JevClient())

query and ask both go through JevApi.evaluate, so a fake or mock that implements evaluate covers all three, without a network. models() is the interface's other network call; stub it too if your code lists models, returning a ModelList. The members that take JevCallOptions have default implementations that ignore the options and call the plain ones, so a fake needn't implement them. A MockK mock does need them stubbed if the code under test passes options or uses withOptions: coEvery { jev.evaluate(any(), any(), any(), any()) }. To exercise the real client without a network, give it a Ktor MockEngine through the engine setting.

Build what the fake returns with jevResult(body, questions), which maps a response body exactly as the client does (a leading byte-order mark included), and jevApiException(status) for the error path, which reads the retry hint and the x-typesafe-request-id from its headers as the client does. Recording a real response and replaying it keeps a fixture honest, and a malformed one, or text that isn't a JSON object at all, still raises JevResponseValidationException. A QuestionSet prints its ids, so a mock's "no answer found" message shows which set it received.

A result answers only the handles of the question set it was built for. Code that builds its questions on every call (inline handles, or a JevQuery class) sends a new set each time, so a MockK mock should build the result from the set it received, answers { jevResult(body, secondArg()) }, rather than return one built in advance.