Skip to content

Extraction

Jev chooses; it doesn't generate. That's an advantage for extraction: when the model picks from candidates your code found, it can't invent a value or transpose a digit.

Pick from candidate spans

Find candidates with a regex (or a parser, a roster, or an NER model), offer them as Choice options, and copy the chosen span verbatim:

private val EMAIL = Regex("""[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}""")

// Code finds the candidates; Jev picks the one the question means. The answer is always a verbatim span.
suspend fun receiptAddress(
    jev: JevApi,
    email: String,
): String? {
    val candidates = EMAIL.findAll(email).map { it.value }.distinct().toList()
    if (candidates.isEmpty()) return null

    val pick =
        jev
            .query(state = email) {
                choice("address", "Which email address does the sender want their receipt sent to?") {
                    candidates.forEach { option(it) }
                    "none" means "None of these is the requested address"
                }
            }.choice("address")

    if (pick.choice == "none" || pick.confidence < 0.6) return null
    return pick.choice.lowercase()
}
  • Finding candidates is the hard part. Regexes should over-find: Jev handles the disambiguation. For example, the sender wants the receipt at their personal address, not the one in the From: line.
  • Always include a none option, and check for it before using the value.
  • Stay under 255 candidates. For more, narrow in two stages: pick a section first, then a span inside it.

Dates: read the parts, compute in code

Jev reads dates as text, not as ordered quantities, so comparing dates or counting days in the model is unreliable. Instead, ask which date parts the text states, each from a closed set, and do the calendar math in code:

enum class DateMode(
    override val description: String,
) : JevOption {
    ABSOLUTE("A calendar date naming a month, e.g. 'August 14'"),
    RELATIVE("Relative to today: today, tomorrow, or a named weekday"),
    NONE("The document does not state this date"),
}

enum class RelativeDay { TODAY, TOMORROW, DAY_AFTER, NONE }

// The model reads which date parts the text names; code does all of the calendar math.
class DueDateQuery(
    role: String,
) : JevQuery() {
    val mode by choice<DateMode>("How is $role written?")
    val month by choice<Month>("If $role is an absolute date, which month is it in?")
    val day by choice("If $role is an absolute date, which day of the month is it?") {
        (1..31).forEach { option(it.toString()) }
        "none" means "The document does not state a day"
    }
    val relative by choice<RelativeDay>("If $role is relative to today, which day is it?")
}
data class DueDate(
    val date: LocalDate?,
    val needsReview: Boolean,
)

suspend fun dueDate(
    jev: JevApi,
    document: String,
    today: LocalDate,
): DueDate {
    val dueDateQuery = DueDateQuery("the deadline to return the form")
    val result = jev.ask(dueDateQuery, state = document)
    val mode = result[dueDateQuery.mode]

    val (date, confidence) =
        when (mode.choice) {
            DateMode.ABSOLUTE -> {
                val month = result[dueDateQuery.month]
                val day = result[dueDateQuery.day]
                nextDate(month.choice, day.choice, today) to minOf(mode.confidence, month.confidence, day.confidence)
            }

            DateMode.RELATIVE -> {
                val relative = result[dueDateQuery.relative]
                val offset =
                    when (relative.choice) {
                        RelativeDay.TODAY -> 0L
                        RelativeDay.TOMORROW -> 1L
                        RelativeDay.DAY_AFTER -> 2L
                        RelativeDay.NONE -> null
                    }
                offset?.let(today::plusDays) to minOf(mode.confidence, relative.confidence)
            }

            DateMode.NONE -> {
                null to mode.confidence
            }
        }
    // Gate on the weakest part that was used.
    return DueDate(date, needsReview = date == null || confidence < 0.6)
}

// The next occurrence of month/day on or after today; null for "none" or an impossible date like February 30.
private fun nextDate(
    month: Month,
    day: String,
    today: LocalDate,
): LocalDate? {
    val dayOfMonth = day.toIntOrNull() ?: return null
    return try {
        LocalDate.of(today.year, month, dayOfMonth).let { if (it < today) it.plusYears(1) else it }
    } catch (e: DateTimeException) {
        println("impossible date: ${e.message}")
        null
    }
}

Validate in code, too. In TypeSafe's example, a document containing a different date made the mode come back as "absolute" with no month. Only the incomplete parts and the low minimum confidence (0.46) caught it.

Function calling

Map a natural-language command onto an ordinary typed function:

  • a Choice picks the function
  • a Choice per closed-set argument fills in its value
  • a "stated" Noul per optional argument decides whether the user specified it at all
enum class ChartStyle { LINE, CANDLES }

enum class Window { ONE_DAY, ONE_WEEK, ONE_MONTH, THREE_MONTHS }

fun plotPrice(
    symbol: String,
    style: ChartStyle = ChartStyle.LINE,
    window: Window = Window.ONE_MONTH,
) = println("plot $symbol as $style over $window")

// One request routes the call and fills its arguments; "stated" Nouls leave unstated arguments at their defaults.
object PlotCommand : JevQuery() {
    val symbol by choice("Which stock is the user asking about?") {
        "NVDA" means "Nvidia, often written NVDA"
        "AAPL" means "Apple, often written AAPL"
        "TSLA" means "Tesla, often written TSLA"
    }
    val style by choice<ChartStyle>("Does the user want a plain line or candles?")
    val styleStated by noul("Does the user say how the chart should be drawn, such as a line or candles?")
    val window by choice<Window>("How far back is the user asking about?")
    val windowStated by noul("Does the user say how far back to look, such as today, this week, or this quarter?")
}

suspend fun runPlotCommand(
    jev: JevApi,
    command: String,
) {
    val result = jev.ask(PlotCommand, state = command)
    plotPrice(
        symbol = result[PlotCommand.symbol].choice,
        style = if (result[PlotCommand.styleStated].isTrue()) result[PlotCommand.style].choice else ChartStyle.LINE,
        window = if (result[PlotCommand.windowStated].isTrue()) result[PlotCommand.window].choice else Window.ONE_MONTH,
    )
}

Without the "stated" Nouls, a Choice would confidently fill in a window the user never mentioned.

Some tips for argument questions:

  • Write each question about the idea, not the user's likely words.
  • Don't name a question after its parameter.
  • When two arguments share options, spell out each one's role: "the stock being measured, named first" versus "the one it's compared against".

Keep arithmetic in code

The same rule applies to amounts: let Jev identify which number is the total, then compute with it in code:

// Jev reads meaning, not numbers: let it pick out the value, then do the arithmetic in code.
suspend fun overBudget(
    jev: JevApi,
    invoice: String,
    budgetUsd: Double,
): Boolean {
    val amounts = Regex("""\$([\d,]+\.\d{2})""").findAll(invoice).map { it.groupValues[1] }.distinct().toList()
    if (amounts.isEmpty()) return false
    val total =
        jev
            .query(state = invoice) {
                choice("total", "Which amount is the total the customer must pay?") { amounts.forEach { option(it) } }
            }.choice("total")
            .choice
    return total.replace(",", "").toDouble() > budgetUsd
}