Speculative Fan-Out¶

Questions in one request are evaluated in parallel, so asking more of them barely changes latency, and costs only their input tokens. Put every question your code might need into one request, including ones that only matter on some code paths, and let code decide which answers to use.

TypeSafe measured this on a 13-question compliance briefing. One batched request was 12.2 times cheaper and 10 times faster than 13 separate requests, with the same answers: the state is sent and read once instead of 13 times.

Support triage¶

Classify the ticket, and ask the bug- and billing-specific questions up front:

enum class TicketCategory(
    override val description: String,
) : JevOption {
    BUG_REPORT("The user is reporting something that is broken or producing errors"),
    BILLING("Charges, invoices, refunds, subscriptions"),
    FEATURE_REQUEST("The user is requesting new functionality"),
    ACCOUNT("Login, permissions, profile, security"),
}

object SupportFanOut : JevQuery() {
    val category by choice<TicketCategory>("Determine the broad category of this support ticket")

    // Speculative: only matter for bug reports.
    val bugSeverity by score("How severe is the reported issue?") {
        levels("Cosmetic; no impact to functionality", "Broken; workaround exists", "Blocking; no workaround")
    }
    val hasReproSteps by noul("Does the user describe specific steps to reproduce the issue?")

    // Speculative: only matters for billing.
    val refundRequested by noul("Is the user explicitly asking for a refund or credit?")

    // Useful for every category.
    val frustration by score("How frustrated does the user appear?") {
        levels("Calm, matter-of-fact", "Frustrated but civil", "Very angry")
    }
}

Then route in code, reading only the answers that apply:

// One request answers every question; code decides which answers matter for this ticket.
suspend fun handleTicket(
    jev: JevApi,
    ticket: String,
) {
    val result = jev.ask(SupportFanOut, state = ticket)

    when (result[SupportFanOut.category].choice) {
        TicketCategory.BUG_REPORT -> {
            val blocking =
                result[SupportFanOut.bugSeverity].score > 1.5 &&
                    result[SupportFanOut.hasReproSteps].isTrue(0.6)
            routeTo(if (blocking) "engineering-urgent" else "bug-backlog", ticket)
        }

        TicketCategory.BILLING -> {
            routeTo(if (result[SupportFanOut.refundRequested].isTrue(0.7)) "billing-refunds" else "billing", ticket)
        }

        TicketCategory.FEATURE_REQUEST -> {
            routeTo("product-feedback", ticket)
        }

        TicketCategory.ACCOUNT -> {
            routeTo("account-support", ticket)
        }
    }

    if (result[SupportFanOut.frustration].score > 1.5) routeTo("priority-response", ticket)
}

If the ticket turns out to be a feature request, the bug-severity answer is simply ignored. Its uncertainty doesn't matter either.

A smart-home assistant¶

Asking sequentially ("is this a command?", then "which device?", then "what should the lights do?") costs a round trip per step. Speculative questions answer them all at once:

enum class Room { LIVING_ROOM, KITCHEN, BEDROOM, WHOLE_HOUSE, NONE }

enum class Device { LIGHTS, THERMOSTAT, DOOR_LOCK, SPEAKER, NONE }

object HomeCommand : JevQuery() {
    val isCommand by noul("Is the user asking the assistant to change something in the home?")
    val multipleActions by noul("Does the request ask for more than one distinct action?")
    val room by choice<Room>("Which room is the request about?")
    val device by choice<Device>("Which device is the request about?")

    // Speculative: asked before we know the device is the lights.
    val lightAction by choice("If the request is about lights, what should happen to them?") {
        options("turn_on", "turn_off", "dim", "brighten", "none")
    }

    // Speculative: asked before we know the device is the thermostat.
    val temperatureChange by choice("If the request is about temperature, which way should it go?") {
        options("warmer", "cooler", "none")
    }
}

suspend fun handleUtterance(
    jev: JevApi,
    utterance: String,
): String {
    val result = jev.ask(HomeCommand, state = utterance)
    // Every branch's answer arrives in the one request, so reading them all costs nothing extra.
    val device = result[HomeCommand.device].choice
    val lightAction = result[HomeCommand.lightAction].choice
    val room = result[HomeCommand.room].choice
    return when {
        !result[HomeCommand.isCommand].isTrue() -> "hand off to a conversational LLM"
        result[HomeCommand.multipleActions].isTrue() -> "split into single commands, then ask again for each"
        device == Device.LIGHTS -> "lights $lightAction in $room"
        device == Device.THERMOSTAT -> "make it ${result[HomeCommand.temperatureChange].choice}"
        else -> "ask the user to clarify"
    }
}

The assistant pairs Jev with a generative LLM only where text is actually needed:

Compound requests. When the "multiple actions" Noul fires, an LLM splits the request into single commands, and each is evaluated again.
General questions and small talk. When the request isn't a command, it goes to a conversational LLM.

Jev's response is so fast that it adds little latency in front of the LLM.

When a second request is right¶

Answers are independent: one question never sees another's answer. Make a second request only when you can't build it without the first answer, for example to:

fetch new evidence for the state, such as the full text of the top three matches
construct state that didn't exist before, such as blocks assembled from the first pass
choose the next options, such as the children of the chosen category in a taxonomy

Classification has examples of each.