Your Agent Writes Perfect SQL. It Still Gets the Answer Wrong.

I was running an analytics modernization program at a large national telco when someone asked how many connections we had. It was a complex organization with many teams and legacy warehouses, and delivery split between onshore and offshore. We had made progress on the engineering: a standardized way of working, a consolidated tech stack, and a first move from traditional BI toward machine learning.

Sales meant a Revenue Generating Unit: a paying customer with internet access. Infrastructure meant a live technical connection, something they could measure a signal on, copper or fiber. Finance meant a customer who had paid a first invoice. Three teams, three numbers, and every one of them correct inside its own domain.

We had metadata management and a business glossary. Having those tools had not settled which definition someone should use outside their own department. There was a reason for each team’s measure, but the word alone did not tell you which one a question referred to.

I thought about that program again while reading MotherDuck’s Guide to BI in the Agentic Era  . It describes agents using the Model Context Protocol (MCP) to reach live data and answer questions with SQL. An agent can inspect a schema, run a query, and explain the result without the user finding a suitable dashboard first.

That is useful, particularly for a question nobody anticipated when building the reports. It also makes the telco’s ambiguity harder to contain. With separate departmental dashboards, the report itself can signal which definition applies. A general chat interface has to establish that context some other way.

Imagine someone in finance asking an agent connected to that warehouse how many connections we have. If it selects the table of live technical connections, it can join and filter it correctly and still return a number finance did not ask for. Checking the SQL against the schema would not reveal the mistake. We would also have to check why the agent chose that measure.

An agent could ask a clarifying question. To ask a useful one, it needs to know that competing definitions exist and when to use each. We cannot expect it to recover that agreement from table names if the organization has never made the agreement explicit.

MotherDuck’s guide reports an experiment across five tiers of preparation on DABstep, a benchmark of 460 payment-processing questions. Accuracy was 29.8% with a raw schema and 30.1% with column comments. Pre-computed views raised it to 86.6%; a simple prompt with views and macros reached 93.2%. That is a 63.4-percentage-point improvement between the raw-schema and highest-scoring setups.

These are vendor-reported results on one benchmark. The experiment changes the preparation and the instructions available to the agent, so I would not attribute the whole gain to the underlying data alone. What I take from it is that substantial query logic had to be made available in a form the agent could use. Comments by themselves made little difference in this test.

That does not show that data preparation can resolve the disagreement we had at the telco. Before we could put the definitions into views or instructions, someone still had to decide what they meant. An agent might execute the resulting logic well. The benchmark cannot make that business decision for us.

Semantic layers give those decisions somewhere to live. Apache Ossie (Incubating)  , previously Open Semantic Interchange, defines a vendor-neutral format for exchanging metrics, dimensions, and their relationships. Making a definition portable across tools is useful. At the telco, we would first have needed to agree on qualified names for our different connection counts and the circumstances in which each applied.

The definitions could remain different. Finance’s first-invoice condition serves a different purpose from infrastructure’s signal measurement. A shared interface needs enough information to choose between them or ask the user. Collapsing them into a single unqualified metric would hide the disagreement without resolving it.

A definition also needs data that supports its calculation. If finance counts a customer after the first paid invoice, the model needs to preserve the evidence for that event. It needs a clear rule for multiple invoices and for the reporting date. This is where apparently separate work on Type Casting and Warehouse History becomes relevant: the meaning has to survive the implementation.

Keeping that definition current takes ownership too. A pipeline change can leave a semantic description out of step with the deployed model, just as it can leave Source Documentation Out of Date . Generating the technical facts helps, but it cannot tell us whether finance has changed its policy on when a customer should count.

For an agent-facing interface, I would want an answer to identify the measure it used, the reporting period, and where the data came from. If the question leaves more than one reasonable measure, asking the user should be expected behavior. Showing the generated SQL can help an analyst review the calculation; a business user also needs the definition in language they recognize.

I would test that behavior with questions that use the organization’s ambiguous terms, rather than only questions with a known SQL translation. In our case, “How many connections do we have?” would have been an important test precisely because a single confident number was insufficient. A correct response might begin by asking whether the user meant paying customers or live technical connections.

The conversation I would want to have again at that telco is about who owns each definition and how someone outside the department should recognize it. A better query engine would not have settled that for us. It would have made reaching an agreement, and recording it clearly, more urgent.

We’re building Amfelt around shared metadata and pipeline definitions, with that groundwork in mind: Join the Waitlist .