Overview
Most LLM applications send every query to the most expensive model available, including “hi” and “what is 2+2”. The dollar cost is obvious; the latency cost is the one people miss.
This system decides per query through a three-layer cascade. It was evaluated on 1,500 labeled cases across two eval sets, and on the 1,000-case set it reaches 92.6% routing accuracy at a 14.7% cascade rate.
The three layers
- Layer 1: local ML router. TF-IDF over a tag-augmented encoding of the query, classified by logistic regression. No network call, no added latency, and it resolves 85.3% of traffic on its own.
- Layer 2: LLM fallback classifier. Only low-confidence queries reach a GPT-4o-mini JSON classifier, and they reach it before any answer is generated, so a routing decision never wastes a completion.
- Layer 3: answer. GPT-4o-mini answers simple queries; only complex ones escalate to GPT-4o.
Feature engineering
The router does not vectorize the raw query. Each query is first rewritten into a tag-augmented string: length bands (LEN_SHORT at ≤8 words, LEN_MEDIUM, LEN_LONG above 40), a code-presence flag, and related signals. Explicit categorical features are what let a model as small as TF-IDF plus logistic regression reach 92.6%: the hard part of the decision is encoded, not learned.
Personalization on both paths
- Persistent user profiles in SQLite store expertise level, tone, domain, interests, topics already discussed, and preferred response format.
- Sliding-window memory keeps the last three turn-pairs verbatim. Older turns are compressed into a rolling summary capped at 80 tokens, so context stays bounded without losing the thread.
- A self-improving loop logs every routing decision for retraining, and a Streamlit UI shows each choice so routing is inspectable.
Measured results
- 92.6% routing accuracy on the 1,000-case eval, up from an 88.2% baseline.
- Precision 0.885, recall 0.963, F1 0.922 (TP 438, FN 17, FP 57, TN 488).
- Cascade rate cut from 30.8% to 14.7%.
- Mean routing confidence up from 0.706 to 0.764.
- 85.3% of queries handled by the local layer alone, at zero API cost.