AI analytics writes correct SQL 83% of the time; humans manage 93%

01Three years took text-to-SQL from 39% to 83%

BIRD is the public benchmark most vendors and labs quote for turning English into SQL: thousands of questions over real databases, scored on whether the generated query returns the right rows. Its leaderboard keeps the 2023 baselines alongside the current entries, which makes the trend easy to read.¹ ChatGPT, given the database and hints, scored 39.30% in March 2023. GPT-4 scored 54.89% that July. The best entry dated on or before 30 September 2026, a Purdue University system submitted on 26 September, scored 82.95%.

Two caveats keep this honest. A benchmark database is cleaner than yours: tidy names, documented columns, no fifteen-year-old table called CUST_MSTR_NEW2. And the leaderboard systems are research pipelines tuned for the test, not the assistant bundled into a BI licence. Read 83% as a ceiling for well-described data, not a promise for your ERP export.

02The business definitions are the multiplier, not the model

The most useful experiment on enterprise data is still a 2023 one. Juan Sequeda, Dean Allemang and Bryon Jacob at data.world took an insurance-industry schema, wrote 43 business questions, and asked GPT-4 to answer each one two ways: by writing SQL straight against the tables, and by querying a knowledge graph that described what the tables meant. Every question ran repeatedly between September and October 2023.²

Models have improved since, as section 1 shows, but the direction of the finding has not changed and the vendors now say it themselves. Microsoft’s own Copilot documentation tells customers to prepare the semantic model first, because without it Copilot “can misinterpret the data and return generic or inaccurate results”.³ ThoughtSpot sells Spotter on the same argument: answers grounded in a governed semantic model, and a separate agent, SpotterModel, whose job is to help build that model.⁵

03What it takes to switch it on in Power BI

Our Power BI page records it as the de facto BI default in Indian GCC analytics teams, and Pro is often already paid for inside Microsoft 365 E5, so its rules are the practical ones.⁴ Microsoft’s requirements, as documented in August 2026, have four gates, and a Pro licence passes none of them on its own.³

The third gate is the one Indian buyers miss. An Indian Fabric tenant sits outside both named boundaries, so Copilot stays off until someone with authority accepts that prompts and the data in them may be processed in another country. Microsoft also lists India West among the regions where the standalone and app-scoped Copilot experiences were not yet available.³ That is a data-protection decision, not an IT toggle, and it should be written down by whoever signs for personal data.

The bill also behaves differently from a seat. Microsoft says token consumption drives Copilot billing, that Copilot’s use is measured in the capacity’s own compute units and processed as background operations, and that a newly bought or resized capacity can take up to 24 hours before Copilot recognises it.³ In our reading, a popular Copilot competes with the reports that share its capacity; watch the capacity metrics in the first month before deciding the F2 was enough.

04What the 2023 promise looks like now

The original piece promised speed, accuracy greater than manual methods, and predictive insight for every team. Set against the evidence above, the claims that held are narrower and more operational.

The practical consequence is that you can measure your own version of the BIRD number before you buy, and you should. It takes a week, not a project:

1. Write down twenty to thirty questions your business users already ask, each with the answer finance or sales would sign off: last quarter’s revenue by region, open orders over ninety days, churned customers by plan.
2. Ask each one three times in the tool, in the words a user would type, and score it the way BIRD does: does the result match the signed-off answer, yes or no.
3. Add the definitions: table and column descriptions, synonyms, the three or four measures everyone argues about. Then run the same questions again.
4. Compare the two scores. The gap is what the semantic model is worth to you, and it is usually the better investment than a pricier licence.