What We Learned Teaching Jev: One Word, 24 Messages, Three Languages

We have a habit at Cocoon. When a new kind of AI appears, we do not write a slide about it. We build a lab where people can break it.

Micro LLM came from that habit: a tiny language model you build in your browser from text your room types in, so people can see how a chatbot picks its next word. When TypeSafe released Jev, a model that decides instead of writing, it needed a lab of its own. So we built Jev Lab, and in building it we ran a lot of small experiments. These are the ones that changed how we teach.

Finding 1: one word moved the answer from 28% to 94%

Same message, same model, one question asked three ways. The message was a customer charged twice, wanting it sorted before Friday or they would go to their bank.

A repeat run later the same day gave 27%, 77% and 94%. So the model is consistent. What moved was the question. The customer never literally asked for a refund, so all three answers are defensible, and each one answers a different business question.

What we changed: stage four of the lab now starts with a single question and two buttons, "remove explicitly" and "change to want", with a history of every run. It is the fastest way we have found to make people take the wording of a question seriously. With Jev, the question is the whole interface. We explain why in How Jev Works.

Finding 2: on 24 messages, the numbers were honest

We wrote 24 short customer messages, half that ask for money back and half that do not, including some deliberately awkward ones: a customer who prefers a replacement, a thank-you for a refund already received, a question about refund policy. We labelled each by hand, then recorded Jev's answer once.

Twenty-four messages cannot prove calibration across the scale, and we say so every time we show it. But the shape is what a careful person would want: sure when it should be, unsure where it should be. We explain why that matters in System One Models Explained.

Finding 3: the chatbot was certain and wrong

The one message Jev was unsure about was "Do you offer refunds if the workshop is cancelled? Asking for my manager." It mentions refunds, but it asks about policy. It does not ask for money back. Jev said 61%.

We asked the chatbot the same question and told it to give a confidence number from 0 to 1. It said yes, with confidence 1.0.

A confident wrong answer is the dangerous kind, because nothing tells you to check it. We saw the same pattern when we asked the chatbot simply to deal with the double-charge message. It replied that it had "located your order" and would refund it by Friday. It cannot see any order system. It wrote what a helpful agent would say, and it sounded completely sure.

What we changed: stage two of the lab now shows this recorded evidence right under the explanation of what a probability means, and the Teach mode turns the 24 messages into a duel where a room competes against Jev and the chatbot on Brier score.

Finding 4: Sinhala and Tamil scored like English

Micro LLM cannot read Sinhala or Tamil at all: its simple tokenizer only knows the Latin alphabet. So we were curious. We sent the same angry refund message in three languages, with the questions in English.

TypeSafe says English is primary and other languages have variable accuracy, and one clear message is not a language study. But for teams here that answer customers in all three languages, it is an encouraging start. The one real difference was cost: the Sinhala and Tamil versions needed 14% and 25% more tokens to say the same thing.

Finding 5: three small questions beat one big one

The final stage of the lab asks learners to sort six messages into auto-reply, human queue or manager, and get all six right. We tested the intended solution against the live model: three yes/no questions (money back, anger, time pressure), each with a threshold of 0.6.

Six out of six. Notice how far from the threshold almost every number sits. That gap is what lets a business trust the rule. A single broad question like "is this a priority?" would give one number and no way to see why. The three small questions show their working. More on choosing question types in Noul, Choice and Score.

Finding 6: it is fast and cheap enough to check everything

Measured from our own server, with the network round trip included, single-question calls took between 0.26 and 0.69 seconds. A message with three questions used about 512 input tokens, which at TypeSafe's list price of $42 per billion input tokens is roughly $21 for a million messages. That changes what is worth automating: you can check every message, not a sample.

What did building the lab teach us about teaching?

Our first version showed everything at once, and people got lost. So the Learn mode became ten short stages, each with one idea, one thing to try, and one check question. A guide, Coco the caterpillar from our Kids section, explains each idea with an everyday comparison: a talent-show judge, a weather forecast, a pass mark, and keeps a to-do list that ticks itself off. It now works for a 15-year-old learning alone, and for a room of managers in Teach mode.

The deeper lesson is the one in all our labs, from RAG Lab to Feedback Lab: people understand AI when they can change something and watch what happens. A slide that says "wording matters" is forgotten by lunch. Pressing a button and watching 28% become 94% is not.

Frequently asked questions