Skip to content

Classification

Small and medium option sets

One Choice handles up to 255 options, reliably up to about 240. For most classification tasks, a single Choice plus a confidence gate is all you need; see Choice and the parent-label fallback.

Walking a taxonomy

For deep hierarchies (patent classes, product catalogs, medical subject headings), ask one Choice per level. Each option's description is its subtree, so the model sees what lives under a branch before committing to it:

// A tree of categories: each node maps child names to their own subtrees (empty for leaves).
data class Node(
    val children: Map<String, Node> = emptyMap(),
)

// Walk the taxonomy one level per request. Each option carries its subtree, so the model can see what
// lives under a branch before committing to it. Stop early when a level is too uncertain.
suspend fun classifyPath(
    jev: JevApi,
    item: String,
    root: Node,
    minConfidence: Double = 0.5,
): List<String> {
    val path = mutableListOf<String>()
    var node = root
    while (node.children.isNotEmpty()) {
        val next = pickChild(jev, item, node, minConfidence) ?: break // stop at the deepest confident ancestor
        path += next
        node = node.children.getValue(next)
    }
    return path
}

private suspend fun pickChild(
    jev: JevApi,
    item: String,
    node: Node,
    minConfidence: Double,
): String? {
    node.children.keys.singleOrNull()?.let { return it } // a single child needs no question
    val level =
        jev
            .query(state = item) {
                choice("child", "Which direct child category best matches this item?") {
                    node.children.forEach { (name, child) -> name means jsonOf(child.children.keys.toList()) }
                }
            }.choice("child")
    return level.choice.takeIf { level.confidence >= minConfidence }
}
  • Stop at the deepest confident level. Returning a correct parent is more useful than a guessed leaf.
  • Beware catch-all nodes. "Other" nodes and near-synonym siblings can trap a greedy walk early. TypeSafe's beam search, which keeps the best 3 paths at each level scored by the geometric mean of their probabilities, classified 4 of 4 test documents correctly, against 2 of 4 for the greedy walk.
  • Freeze the taxonomy. Option order is part of the question, so walk a fixed snapshot.

Large option sets in two stages

When the options are many and their short descriptions look alike, rank them all cheaply, then judge the top few against their full details:

data class Skill(
    val name: String,
    val summary: String,
    val details: String,
)

// For a large roster: rank everything cheaply, then judge the top few against their full details.
suspend fun suggestSkill(
    jev: JevApi,
    request: String,
    skills: List<Skill>,
): Skill? {
    val wide =
        jev.query(state = request) {
            choice("which", "Which of these skills, if any, is the right one for the user's request?") {
                skills.forEach { it.name means it.summary }
            }
            noul("needs_action", "Is the assistant being asked to act on the user's files, accounts, or services?")
        }
    if (!wide.noul("needs_action").isTrue(0.3)) return null

    val shortlist = wide.choice("which").ranked().take(3).map { (name, _) -> skills.first { it.name == name } }
    val detailed =
        jev.query(state = request) {
            choice("which", "Exactly one of these skills fits the request. Which one? Read what each actually does.") {
                shortlist.forEach { it.name means "${it.summary}${it.details}" }
            }
            shortlist.forEach { skill ->
                noul("fits_${skill.name}", "Does the skill '${skill.name}' do what the user asks? It: ${skill.summary}")
            }
        }
    // The Choice picks which skill; the independent Nouls decide whether to suggest one at all.
    if (detailed.nouls.values.none { it.isTrue(0.3) }) return null
    return shortlist.first { it.name == detailed.choice("which").choice }
}
  • The Choice and the Nouls do different jobs. The Choice settles which option; the independent "fits" Nouls decide whether to suggest one at all, and they can all come back low.
  • The gate asks about action, not topic. Subject-matter questions can't tell "explain what a monad is" from a request to run a tool.
  • The second stage can only reject. It can't recover an option the first stage left out.

In TypeSafe's example, picking one of 182 agent skills this way more than halved both wrong loads (16.8% to 7.3%) and needless loads (9.8% to 4.0%).