Routing¶

A fast, cheap classification in front of expensive resources: deterministic code handles what it can, specialist LLMs get only what they need, and people see the cases that deserve them.

Intent routing¶

Classify the intent and how complex the request is:

enum class CustomerIntent(
    override val description: String,
) : JevOption {
    ORDER_STATUS("Asking about an existing order"),
    PRODUCT_QUESTION("Asking about a product before buying"),
    RETURN_EXCHANGE("Wants to return or exchange something"),
    COMPLAINT("Unhappy with the experience, wants resolution"),
}

object IntentRouting : JevQuery() {
    val intent by choice<CustomerIntent>("What is the primary intent of this customer message?")
    val complexity by score("How complex is this request to resolve?") {
        level("Simple lookup or standard procedure")
        level("Requires some judgment or a multi-step process")
        level("Unusual situation, edge case, or escalation needed")
    }
}

Then pick a handler:

sealed interface Handler {
    data object Deterministic : Handler

    data class SpecialistLlm(
        val specialty: String,
    ) : Handler

    data object Human : Handler
}

// One fast, cheap call picks the handler; expensive resources are used only when needed.
suspend fun chooseHandler(
    jev: JevApi,
    message: String,
): Handler {
    val result = jev.ask(IntentRouting, state = message)
    val intent = result[IntentRouting.intent]
    val complexity = result[IntentRouting.complexity]

    if (intent.confidence < 0.5) return Handler.Human

    return when (intent.choice) {
        CustomerIntent.ORDER_STATUS -> {
            Handler.Deterministic
        }

        CustomerIntent.PRODUCT_QUESTION -> {
            Handler.SpecialistLlm("product catalog")
        }

        CustomerIntent.RETURN_EXCHANGE -> {
            Handler.SpecialistLlm("returns policy")
        }

        CustomerIntent.COMPLAINT -> {
            val tooHardToAutomate = complexity.score > 1 || complexity.confidence < 0.5
            if (tooHardToAutomate) Handler.Human else Handler.SpecialistLlm("complaints")
        }
    }
}

One intent goes to deterministic code with no LLM at all. Two go to different specialist LLMs, each loaded with its own context. Complaints use the complexity Score, and its confidence, to decide between an LLM and a person.

Confidence-gated routing¶

The answer says what; confidence says whether to act. Riskier actions deserve a higher bar:

enum class BankAction(
    override val description: String,
) : JevOption {
    CHECK_BALANCE("Check the balance of an account"),
    APPROVE_TRANSFER("Approve the pending transfer request"),
    OTHER("Something else"),
}

object VoiceCommand : JevQuery() {
    val action by choice<BankAction>("What action is the user requesting?")
}

// The bar for acting rises with the cost of being wrong.
suspend fun handleVoiceCommand(
    jev: JevApi,
    utterance: String,
): String {
    val action = jev.ask(VoiceCommand, state = utterance)[VoiceCommand.action]
    return when {
        action.confidence < 0.6 -> "route to a support agent"

        // Cheap to get wrong: the floor is enough.
        action.choice == BankAction.CHECK_BALANCE -> "read the balance"

        // Costly to get wrong: act alone only when very sure.
        action.choice == BankAction.APPROVE_TRANSFER && action.confidence > 0.85 -> "approve the transfer"

        action.choice == BankAction.APPROVE_TRANSFER -> "ask the user to confirm first"

        else -> "route to a support agent"
    }
}

Checking a balance at 0.6 confidence is fine: at worst, the user hears the wrong screen read out. Approving a transfer on the same evidence isn't. See Confidence & Thresholds.

Model routing¶

The same idea chooses which LLM handles a prompt, sending easy prompts to cheap models:

enum class ModelTier { SMALL, LARGE, REASONING }

object PromptDifficulty : JevQuery() {
    val difficulty by score("How much reasoning does answering this prompt require?") {
        level("A lookup, rewrite, or short factual answer")
        level("Several steps, or combining a few pieces of information")
        level("Long multi-step reasoning, math, or planning")
    }
    val needsTools by noul("Does answering require acting on files, accounts, or external services?")
}

// Route each prompt to the cheapest model that can handle it.
suspend fun pickModel(
    jev: JevApi,
    prompt: String,
): ModelTier {
    val result = jev.ask(PromptDifficulty, state = prompt)
    val difficulty = result[PromptDifficulty.difficulty]
    return when {
        // Unsure: don't under-provision.
        difficulty.confidence < 0.5 -> ModelTier.LARGE

        result[PromptDifficulty.needsTools].isTrue() || difficulty.score > 1.5 -> ModelTier.REASONING

        difficulty.score > 0.5 -> ModelTier.LARGE

        else -> ModelTier.SMALL
    }
}

When the difficulty estimate itself is uncertain, the router errs toward the more capable model.