Composite Scoring¶

Asking one question for a complex judgment ("rate this candidate") hides several judgments behind one number. Instead, score each dimension separately and combine them with weights in code. You can see exactly how every result was computed, and when the ranking doesn't match your team's judgment, you change a weight, not a prompt.

Score each dimension¶

object ResumeScreen : JevQuery() {
    val pythonDepth by score("How much depth of Python experience does this candidate have, based on the resume?") {
        level("No Python experience mentioned")
        level("Mentioned but no detail")
        level("Used in projects, some specifics")
        level("Primary language, multiple projects")
        level("Deep expertise: architecture, performance, libraries")
    }
    val leadership by score("How much experience does this candidate have managing or leading engineering teams?") {
        level("No management experience mentioned")
        level("Informal mentorship or tech lead role")
        level("Led a small team or project")
        level("Managed a team with direct reports")
        level("Managed multiple teams or an engineering org")
    }
    val systemDesign by score("How much experience does this candidate have designing large-scale systems?") {
        level("No architecture work mentioned")
        level("Contributed to design discussions")
        level("Designed components of a larger system")
        level("Owned the architecture of a significant system")
        level("Designed systems at scale across multiple domains")
    }
}

Combine with weights¶

normalized puts every Score on 0–1, whatever its number of levels, so weights mean what they say:

data class RoleWeights(
    val python: Double,
    val leadership: Double,
    val systemDesign: Double,
)

val seniorEngineer = RoleWeights(python = 0.45, leadership = 0.10, systemDesign = 0.45)
val engineeringManager = RoleWeights(python = 0.15, leadership = 0.55, systemDesign = 0.30)

// The same answers rank candidates for different roles; only the weights change.
fun fit(
    result: JevResult,
    weights: RoleWeights,
): Double =
    weights.python * result[ResumeScreen.pythonDepth].normalized +
        weights.leadership * result[ResumeScreen.leadership].normalized +
        weights.systemDesign * result[ResumeScreen.systemDesign].normalized

Rank without new requests¶

The answers are reusable data. Re-weighting, re-ranking or filtering reads the stored results; it never calls Jev again:

suspend fun shortlist(
    jev: JevApi,
    resumes: Map<String, String>,
    weights: RoleWeights,
    top: Int = 5,
): List<Pair<String, Double>> =
    coroutineScope {
        resumes
            .map { (name, resume) -> async { name to jev.ask(ResumeScreen, state = resume) } }
            .awaitAll()
            .map { (name, result) -> name to fit(result, weights) } // re-weighting needs no new requests
            .sortedByDescending { it.second }
            .take(top)
    }

Weighted yes/no signals¶

The same approach works with Nouls. Here several independent spam signals replace one vague "is this spam?":

// Independent yes/no signals combined with weights, instead of one vague "is this spam?".
suspend fun spamRisk(
    jev: JevApi,
    email: String,
): Double {
    val result =
        jev.query(state = email) {
            noul("credentials", "Does the message ask the recipient for a password or other login credential?")
            noul("reward", "Does the message claim the recipient received an unexpected prize, payment, or reward?")
            noul("pressure", "Does the message pressure the recipient to act quickly?")
            noul("sender_mismatch", "Does the sender's named organization conflict with their email domain?")
        }
    return 0.40 * result.noul("credentials").noul +
        0.25 * result.noul("sender_mismatch").noul +
        0.20 * result.noul("reward").noul +
        0.15 * result.noul("pressure").noul
}

A weighted sum suits signals that compensate for each other. For "any serious violation" logic, where one signal alone should trigger, use separate conditions or a maximum instead; see Verification.

Learned weights¶

With labeled outcomes, Jev's probabilities become features for a classical model such as gradient boosting. TypeSafe's feature-discovery cookbook predicted wine critics' scores from tasting notes this way. An LLM proposed questions, Jev answered them for every note, and a CatBoost model learned the weights. With 38 questions, that beat asking Jev for the score directly: an RMSE of 1.77 against 2.15.