Images & Box¶

Jev reads PNG and JPEG images as well as text. An image can go anywhere text can: in the state, in a question's instructions, and in its criteria. The Box question type finds where something is in an image.

Experimental

TypeSafe calls images and Box "Dreamer-ready": the request format, the answers and the limits can change between its releases, and so can this part of jev4k. Use them for prototypes, and see TypeSafe's Images and Box pages.

Turning images on¶

Everything here is marked @ExperimentalJevImages. Opt in with @OptIn(ExperimentalJevImages::class) on the code that uses it, or for a whole module with optIn.add("com.pambrose.jev4k.ExperimentalJevImages") in Gradle's compilerOptions. jev4k then sends TypeSafe's TypeSafe-Danger-Zone-Images: true header with every request that carries an image (attached, or hand-built in extraBody["assets"]) or a Box question. A header of the same name in the client's or the call's headers replaces it.

Placing and attaching¶

A JevImage comes from bytes (JevImage.of), base64 text (JevImage.fromBase64), or a file (JevImage.read, with a kotlinx-io Path anywhere, or a java.nio.file.Path or File on the JVM). Each one has a reference, a string that stands for the image wherever it is placed. Its toString() is that reference, so string templates, jsonOf, entry and @Serializable fields all place it. The API replaces every occurrence of a reference with the image.

Placing an image doesn't send it. attach it in the query block, or in a JevQuery, so its bytes go with the questions. jev4k refuses to send a request that attaches an image nothing references, or that places a generated reference without attaching its image.

@OptIn(ExperimentalJevImages::class)
suspend fun checkDelivery(jev: JevApi) {
    val door = JevImage.read(Path.of("door-camera.jpg"))       // read now, with a generated reference
    val state = jsonOf(mapOf("door" to door))                  // placed in the state as its reference
    val result =
        jev.query(state) {
            attach(door)                                       // and attached, so its bytes are sent
            noul("person", "Is there a person in `door`?")
            choice("carrying", "What is the person in `door` carrying?") {
                "package" means "A box or parcel."
                "bag" means "A bag of groceries or shopping."
                "nothing" means "Empty hands."
            }
        }
    println(result.noul("person").noul)
}

Images in instructions¶

An image only one question needs can go straight into its instructions. An image several questions use belongs in the state, where it is sent once; every place a reference appears adds another copy of the image to what the model reads.

@OptIn(ExperimentalJevImages::class)
suspend fun sameProduct(
    jev: JevApi,
    upload: JevImage,
    catalog: JevImage,
): Boolean {
    val result =
        jev.query("A seller uploaded $upload for a listing whose catalog photo is $catalog.") {
            attach(upload, catalog)
            noul("same", "Does $upload show the same product as $catalog?")
        }
    return result.noul("same").isTrue()
}

Images in criteria¶

Choice options, Score levels and Noul definitions can be pictures, or descriptions with example pictures. Option names are never replaced, so the picture goes in the option's description. A JevQuery attaches its own pictures, which travel with every request that asks it; ask(query, state, images...) adds the call's own images after them.

@OptIn(ExperimentalJevImages::class)
object HandGestures : JevQuery() {
    private val thumb = attach(JevImage.read(Path.of("gestures/thumb.png")))
    private val palm1 = attach(JevImage.read(Path.of("gestures/palm1.png")))
    private val palm2 = attach(JevImage.read(Path.of("gestures/palm2.png")))

    val gesture by choice("Which gesture is the hand in `camera` making?") {
        "thumbs_up" means thumb                                  // an option that is a picture
        "open_palm" means
            entry(
                "description" to "A hand held up with the palm facing the camera.",
                "examples" to listOf(palm1, palm2),              // or a description with example pictures
            )
        "none" means "No hand, or a hand that matches none of the examples."
    }
}

@OptIn(ExperimentalJevImages::class)
suspend fun readGesture(
    jev: JevApi,
    frameBytes: ByteArray,
) {
    val frame = JevImage.of(frameBytes)
    val result = jev.ask(HandGestures, jsonOf(mapOf("camera" to frame)), frame)   // the query's pictures, then this one
    println(result[HandGestures.gesture].choice)
}

JevImage.read in an object's initializer, as above, reads the file there, and a missing file fails the object's first use with an ExceptionInInitializerError. When that matters, read the bytes elsewhere and use JevImage.of.

A @Serializable field of type JevImage is written as its reference:

@OptIn(ExperimentalJevImages::class)
@Serializable
data class ReturnClaim(
    val claim: String,
    val photo: JevImage,                                         // written as its reference
)

@OptIn(ExperimentalJevImages::class)
suspend fun checkClaim(
    jev: JevApi,
    claim: ReturnClaim,
) {
    val result =
        jev.query(claim) {
            attach(claim.photo)
            noul("damaged", "Is the product in `photo` damaged?")
            score("severity", "How severe is the damage in `photo`?") { levels("None", "Cosmetic", "Unusable") }
        }
    println(result.score("severity").nearestLevel)
}

Your own references¶

A generated reference is jev4k-image- followed by a UUID. You can choose your own, for prompts kept in configuration, but the API matches references as plain text anywhere inside a string: a reference cat also matches inside category. Use delimiters that can't occur in your text.

@OptIn(ExperimentalJevImages::class)
suspend fun matchesCatalog(
    jev: JevApi,
    delivered: ByteArray,
    catalog: ByteArray,
): Double {
    val result =
        jev.query("A customer returned <delivered>. The catalog photo is <catalog>.") {
            attach(JevImage.of(delivered, reference = "<delivered>"), JevImage.of(catalog, reference = "<catalog>"))
            noul("matches", "Is <delivered> the product shown in <catalog>?")
        }
    return result.noul("matches").noul
}

Box¶

A Box question finds one target and answers with the box around it. Each edge is a fraction, from 0 to 1, of the image's height (top, bottom) or width (left, right), measured from the top-left corner. A Box always returns a box, even when the target isn't there, so ask a Noul in the same request. Box needs Jev 1.14 (jev-preview or jev-1.14.0).

@OptIn(ExperimentalJevImages::class)
suspend fun findMug(
    jev: JevApi,
    photo: JevImage,
) {
    val result =
        jev.query(jsonOf(mapOf("photo" to photo)), model = "jev-preview") {
            attach(photo)
            noul("mug_visible", "Is there a coffee mug in `photo`?")  // a Box always returns a box, so ask
            box("mug", "the coffee mug in `photo`")
        }
    if (result.noul("mug_visible").isTrue()) {
        val box = result.box("mug").box                          // edges as fractions of the height and width
        println("mug across ${box.left..box.right} of the width, ${box.top..box.bottom} of the height")
    }
}

To crop to the box, multiply each edge by the image's size in pixels:

@OptIn(ExperimentalJevImages::class)
fun cropToBox(
    image: BufferedImage,
    box: BoundingBox,
): BufferedImage {
    val x = (box.left * image.width).toInt().coerceIn(0, image.width - 1)
    val y = (box.top * image.height).toInt().coerceIn(0, image.height - 1)
    val width = ((box.right - box.left) * image.width).roundToInt().coerceIn(1, image.width - x)
    val height = ((box.bottom - box.top) * image.height).roundToInt().coerceIn(1, image.height - y)
    return image.getSubimage(x, y, width, height)
}

What jev4k checks¶

Before sending, jev4k reports every problem at once, as a JevValidationException:

an image that is empty, isn't valid base64, or is neither a PNG nor a JPEG
a reference that is blank or longer than 1,024 bytes, two images sharing a reference, or references over 64 KiB in total
an attached image that no string references, or a generated reference placed without its image attached
a Box question with no image, attached or in extraBody["assets"], or extraBody["assets"] on a request that attaches images

The API enforces its own size and pixel limits (each side 256 to 1568 pixels, 16 MiB per file, 32 MiB per request), answering with a 400 or a 413. Resize and rotate images before sending them; the API strips EXIF data and doesn't apply its orientation.

A request with images is sent as multipart/form-data, the JSON body first and then one part per image, as the official SDKs send it. Images count as input tokens at the text price.

If you cache results on the model, the state and the questions, include each attached image's bytes in the key too: QuestionSet.toJson() holds only the references.

Example: hot dog or not¶

A runnable example asks about two photos, one request each: a dog panting in the sun on a hot day, and an actual hot dog. The first photo is hot and shows a dog, but it isn't a hot dog, and the questions keep those ideas apart: a Noul for each, and a Choice for what the photo shows.

// What a photo mainly shows.
enum class Subject(
    override val description: String,
) : JevOption {
    HOT_DOG("A hot dog: a cooked sausage served in a sliced bun"),
    DOG("A live dog"),
    SOMETHING_ELSE("Anything else"),
}

object HotDogCheck : JevQuery() {
    val hotDog by noul("Is `photo` a picture of a hot dog?") {
        whenTrue("The food: a cooked sausage served in a sliced bun")
        whenFalse("Anything else, including a live dog, however hot the weather")
    }
    val hotDay by noul("Does `photo` look like it was taken outdoors on a hot, sunny day?")
    val subject by choice<Subject>("What is the main subject of `photo`?")
}

Each photo is attached to its request and placed in the state as photo:

val photo = JevImage.of(photoBytes(name))
val result = jev.blocking.ask(HotDogCheck, jsonOf(mapOf("photo" to photo)), photo)
val hotDog = result[HotDogCheck.hotDog]
val verdict = if (hotDog.isTrue()) "Hotdog!" else "Not hotdog"
val hotDay = result[HotDogCheck.hotDay].noul
val subject = result[HotDogCheck.subject].choice
println("$name: $verdict (hot dog %.2f, hot day %.2f, looks like $subject)".format(hotDog.noul, hotDay))

Run it from a checkout:

make hotdog-example                # needs TYPESAFE_API_KEY

One run printed this. Jev isn't fully deterministic, so the numbers can move a little from run to run:

hot-day-dog.jpg: Not hotdog (hot dog 0.01, hot day 0.96, looks like DOG)
hot-dog.jpg: Hotdog! (hot dog 1.00, hot day 0.15, looks like HOT_DOG)

Both photos are CC0, from Wikimedia Commons: Panting-dog-No.3 by Raffael Herrmann, and Hot Dogs by Alabama Extension.

From Java¶

Java ignores the opt-in annotation. Build an image with JevImage.of(Files.readAllBytes(path)):

static BoundingBox findMug(BlockingJev blocking, byte[] photoBytes) throws InterruptedException {
    JevImage photo = JevImage.of(photoBytes);
    JevResult r = blocking.query("The photo: " + photo.getReference(), qb -> {
        qb.attach(photo);
        qb.noul("mug_visible", "Is there a coffee mug in the photo?");
        qb.box("mug", "the coffee mug in the photo");
        return Unit.INSTANCE;
    });
    return r.box("mug").getBox();
}