Two-stage routing

The first version sent the whole knowledge prompt to the model on every step: tens of thousands of input tokens, about 50,000 per request when I first measured it. Cutting that with word rules helped, but word rules read words, not meaning: they sent project cards for "what are his rates for a freelance project?", the contact card for an email mentioned inside a project, and the whole skill list for a skill they did not recognise.

I added a small routing model, Jev (TypeSafe's decision model, reached through OpenRouter), as the first stage. It classifies each question into one of a set of topics, each described in a sentence, and says whether the answer needs detail from inside a project write-up. A topic it is confident about decides which parts of the notes the main model gets; otherwise the word rules decide. I ran it as a proof of concept against the same questions as the word rules before adopting it, and shipped it behind a switch with a timeout and a hard fallback to the word rules. Jev suits the job: in one cheap call it returns a topic with a confidence score and a yes/no on detail, so there is no free text to parse, and its output is free.

A small router model (Jev) decides which part of the notes each question needs, so a typical question sends about 1,000 tokens instead of the whole knowledge base. The same step keeps answers focused on the part of the notes that matters.

• It adds about half a second to a question on average, and in one test run the service stalled past 15 seconds, which is why the timeout and fallback are required.
• A second company sees visitor questions, and it is an early-access, single-vendor service.
• At the current main-model price it is about break-even on tokens, so the case for it is accuracy and precise context, not savings.

The context the main model sees was 46% smaller on average, and answer quality held (one more question passed than with the word rules, within noise). It routes by meaning, so the rules I had to keep patching are now only the fallback.