In a voice turn the main LLM reads the question, decides it needs a tool, calls it, waits for the result and only then answers. A shopping question often takes two or three of these round trips, and each one delays the whole answer. A small model could guess the lookups first and start them while the rest of the turn is being prepared. Then the results would already be in the LLM’s prompt. A wrong guess would cost one wasted API call, never a wrong answer, because the LLM still calls tools itself when it needs them.
I tried the cheap options first. Regex rules fired on only 10 of 49 turns. GLiNER 2.5 is a 194M-parameter model that reads a list of tool definitions and a message in one pass, and it was the option worth fine-tuning. Out of the box it gave nothing useful. In Echo’s real pipeline it injected zero tool results over 50 turns, and on the gold benchmark it got the exact tools on 0 of 50 turns.
My first fine-tune failed too. The frozen four-arm benchmark compared no prediction, regex, the released model and the fine-tune: 196 matched turns, graded blind with a second reviewer. The fine-tune made zero usable calls and added about 0.8 s of model time per turn. I set it aside and kept the checkpoint only as a diagnostic: it had trained for only 40 updates, and its dev split overlapped its training data.
Then I rebuilt each part. The new dataset has 7,830 training examples. About 62% are synthetic turns for the store’s seven tools, and the rest are public tool-calling data. It is split so that no message family appears in both training and test. The pipeline was giving up after 1.5 s while the model was allowed 4 s, so it threw correct answers away; I made the two limits match. And I moved the model from a CPU to a GPU.
On the frozen benchmark, exact picks rose to 18/50. A second checkpoint scored 19/50; it trained on 10,164 examples, adding turns where a result is already in context. In one six-turn conversation, replayed twice each way, the main LLM made 3 tool calls instead of 9. The whole conversation took 28% less time, but the first word came no sooner.
The limits are plain. The gain rests on one replayed conversation. A GPU kept warm costs about $580–1,020 a month. And in the first end-to-end run on the hosted GPU, 3 of 4 planned calls were dropped as ambiguous: working out which product “it” means is still unsolved.
