The store’s shopping assistant answers by voice and text on every page. In the company’s internal operations dashboard, the team could see only a list of conversations with an AI summary. Nothing showed how many visitors opened the assistant, whether asking led to a cart or an order, or whether its answers were right. The business had to decide whether to push the assistant, leave it, or change direction, and it had no evidence.
The funnel
Every visit moves through six steps: site visit, product view, opened the assistant, asked a question, added to cart after, and purchased after, with revenue per currency. The website sends each event to Mixpanel with the visit, device, country, entry point and who started the chat: the shopper or a proactive nudge. Each message also records how it was sent: typed, tapped, a voice note or hands-free. A job copies these events into the operations database every 4 hours. The order service writes the conversation ID onto each successful payment, so “purchased after” counts real orders linked to a chat.
Eight cards break the funnel down by week, device, entry point, who started it, first message, input type and country. A rate with a base under 5 is greyed out as a small sample.
The agentic pipeline
When a chat ends, the backend fetches its transcript from the voice-agent platform. One LLM call writes the summary and the shopper’s intent. A second call, the judge, reads each shopper question with fresh catalogue data for the products the assistant looked up. It returns one of seven verdicts: correct, partly, wrong, not answered, handed off, not a question, or couldn’t verify. Each verdict has a reason and a line of evidence. If a step fails, the chat goes back in the queue with back-off. The model is a setting: DeepSeek by default, or any OpenAI-compatible endpoint.
The Conversations tab is the explorer. It has filters for verdict, reason, intent and turns, kept in the address so that a view can be shared. Each row shows its turns as squares coloured by verdict, and a drawer shows the transcript with the judge’s verdict under every answer.
How I built it
I split the work into instrumentation, judge and API, and the dashboard, and built all three end to end, then verified the whole pipeline with the real model against the catalogue. Before that I had added the conversation list’s country filter, stats cards and spoken-or-typed tags. The pictures are redrawn with illustrative values, because the recorded numbers include test visits.
