Writing Good Questions¶

Question design matters more than anything else in a Jev integration. These guidelines are condensed from TypeSafe's documentation and cookbooks.

One snap judgment per question¶

Broad questions hide several judgments behind one answer:

// Too broad: one answer hides several judgments you can't inspect or tune.
object SpamBroad : JevQuery() {
    val spam by noul("Is this message spam?")
}

Atomic questions expose them, so you can inspect, tune and combine each one:

// Atomic: each question is a snap judgment; code combines them.
object SpamSignals : JevQuery() {
    val requestsCredentials by noul("Does `message.body` ask the recipient for a password or other login credential?")
    val unexpectedReward by noul("Does `message.body` claim the recipient received an unexpected prize or payment?")
    val timePressure by noul("Does `message.subject` or `message.body` pressure the recipient to act quickly?")
    val senderMismatch by noul(
        "Does the organization in `message.sender.display_name` conflict with the domain in `message.sender.email`?",
    )
}

Decomposing doesn't add round trips: questions about the same state run in parallel in one request.

Say exactly what you mean¶

Jev answers the question you wrote, not the one you meant. Scoping words, negations and implied conditions are read at face value:

object RefundLiteral : JevQuery() {
    // Read literally, "Is this about refunds?" is also true for "What's your refund policy?".
    val aboutRefunds by noul("Is this message about refunds?")

    // State the exact condition you mean.
    val requestsRefund by noul("Does the customer explicitly ask for their own money back for a specific purchase?")
}

suspend fun literalCheck(
    jev: JevApi,
    message: String,
) = jev.ask(RefundLiteral, state = message)[RefundLiteral.requestsRefund].isTrue()

When you look at a wrong answer and find yourself explaining what you really meant, that explanation is the missing half of the instruction. Avoid double negatives and multi-hop phrasing ("a property of a property"); name the part of the state you mean with a backticked path instead.

Describe levels as situations¶

object LevelWording : JevQuery() {
    // Bad: numbers and degrees give the model nothing to match against.
    val vague by score("Rate severity from 0 to 2, where 2 is worst") {
        levels("0", "1", "2")
    }

    // Good: each level describes a situation that can be recognized on its own.
    val concrete by score("How severe is the reported issue?") {
        level("Cosmetic; no impact to functionality")
        level("Broken or degraded feature, but a workaround exists")
        level("Blocking issue; no workaround exists")
    }
}

Use the right type for degree¶

object PythonSkill : JevQuery() {
    // A Noul answers "is this true?": 0.5 means "equally likely", not "medium skill".
    val usedAtWork by noul("Does the resume state that the candidate has used Python at work?")

    // A Score measures degree along described levels.
    val depth by score("How much Python experience does the resume show?") {
        levels("No experience", "Some familiarity", "Daily use", "Deep expertise")
    }
}

Ask narrowly for the deciding fact¶

In TypeSafe's text-reformatting example, "are these two lines part of the same paragraph?" merged unrelated list items, because the topic carried over. "Does this line pick up mid-sentence?" asked for the one fact that decides the question, and worked.

Give options an escape¶

Add other or none when the options might not cover every input. Pair a "which one?" Choice with an "is there any?" Noul, because Choice probabilities always sum to 1.

Keep computation in code¶

• Counting. Jev doesn't count reliably. Ask one Noul per item and count in code.
• Arithmetic and magnitudes. Let Jev identify values; compute with them in code. Don't interpolate exact numbers from a Score.
• Dates. Jev reads dates as text. Extract the parts and compare them in code.
• Numeric representations. It judges "red" better than #FF0000. Convert to a name or a bucket first.

Send focused state¶

Accuracy falls as the state fills with detail the question doesn't need. Retrieve and filter in code, or filter with Nouls, before asking.

Keep instructions and criteria aligned¶

Treat criteria as an extension of the instruction. A Noul whose whenTrue describes a "no", or options that contradict the instruction, lowers accuracy.

Keep questions and thresholds reviewable¶

The questions and the threshold constants are what people need to review. Keep them together in a small number of JevQuery objects and constants, not scattered through the code.

Validate on your own data¶

Typed output guarantees the shape of an answer, not its truth. Calibration holds across many answers, not for any single one. Before relying on thresholds:

• label a representative sample of your own inputs
• compare answers and confidence against the labels
• re-check when you move to a new model version

Known limits of jev-1.13¶