---
title: "AI analytics writes correct SQL 83% of the time; humans manage 93%"
summary: "What the BIRD text-to-SQL leaderboard, a 2023 knowledge-graph benchmark and Microsoft's Copilot rules show about asking data questions in English, as of September 2026."
url: https://www.thetechbag.com/blog/ai-analytics-future-data
format: Analysis
desk: "AI in the enterprise"
published: 2025-09-16
facts_checked: 2026-09-30
author: TechBag (The TechBag Journal)
tags: ["AI analytics","text-to-SQL","BIRD benchmark","semantic model","Power BI Copilot","Microsoft Fabric","ThoughtSpot Spotter","DPDP","data residency"]
cover: https://www.thetechbag.com/blog/covers/ai-analytics-future-data.png
---

# AI analytics writes correct SQL 83% of the time; humans manage 93%

*What the BIRD text-to-SQL leaderboard, a 2023 knowledge-graph benchmark and Microsoft's Copilot rules show about asking data questions in English, as of September 2026.*

Analysis · AI in the enterprise · By TechBag · Published 2025-09-16 · Facts checked 2026-09-30 · Canonical: https://www.thetechbag.com/blog/ai-analytics-future-data

The short version

- The best system on the BIRD text-to-SQL benchmark answered 82.95% of questions correctly by 30 September 2026; BIRD’s human baseline is 92.96%. AI analytics is useful and still wrong about one time in six.[1]
- The lever is not the model but the business definitions underneath it. In one benchmark, GPT-4 went from 16.7% to 54.2% correct when it queried a knowledge graph instead of raw tables.[2]
- Switching on Copilot in Power BI needs paid capacity from F2 (about ₹21,750 a month before GST) and, for an Indian tenant, an admin decision to let prompts be processed outside India.[3][4]

In 2023 this blog described AI analytics as software that finds patterns faster than people can, sorted into descriptive, predictive and prescriptive kinds. Three years on, the product that Indian buyers are actually offered is narrower and more useful to judge: type a business question in English, and the software writes the query, runs it against your data and draws the chart. The question for a CIO or a head of finance is whether to trust those answers enough to let business users ask on their own, and what has to be in place first.

## Three years took text-to-SQL from 39% to 83%

BIRD is the public benchmark most vendors and labs quote for turning English into SQL: thousands of questions over real databases, scored on whether the generated query returns the right rows. Its leaderboard keeps the 2023 baselines alongside the current entries, which makes the trend easy to read.[1] ChatGPT, given the database and hints, scored 39.30% in March 2023. GPT-4 scored 54.89% that July. The best entry dated on or before 30 September 2026, a Purdue University system submitted on 26 September, scored 82.95%.

**Figure: The best text-to-SQL score more than doubled since 2023, but is still ten points short of people**

*BIRD test-set execution accuracy, %, selected entries by submission date, March 2023 to September 2026*

ChatGPT baseline

Mar 2023

39.3%

GPT-4 baseline

Jul 2023

54.9%

Best entry of 2025

Ant Group, Sep 2025

81.7%

Best entry to date

Purdue, Sep 2026

83%

Human baseline

93%

|  | Value (%) |
| --- | --- |
| ChatGPT baseline | 39.3 |
| GPT-4 baseline | 54.9 |
| Best entry of 2025 | 81.7 |
| Best entry to date | 83 |
| Human baseline | 93 |

Source: BIRD-SQL leaderboard, entries dated on or before 26 September 2026 · as of 30 September 2026

TechBag countTechBag count

Of the eight highest-scoring BIRD entries we read, the highest dated in 2025 is Ant Group’s at 81.67%. The top score therefore rose 1.28 points in the twelve months to September 2026, against 27 points between the GPT-4 baseline and that 2025 entry.[1] The easy gains are taken; the remaining errors are the hard ones.

Two caveats keep this honest. A benchmark database is cleaner than yours: tidy names, documented columns, no fifteen-year-old table called `CUST_MSTR_NEW2`. And the leaderboard systems are research pipelines tuned for the test, not the assistant bundled into a BI licence. Read 83% as a ceiling for well-described data, not a promise for your ERP export.

## The business definitions are the multiplier, not the model

The most useful experiment on enterprise data is still a 2023 one. Juan Sequeda, Dean Allemang and Bryon Jacob at data.world took an insurance-industry schema, wrote 43 business questions, and asked GPT-4 to answer each one two ways: by writing SQL straight against the tables, and by querying a knowledge graph that described what the tables meant. Every question ran repeatedly between September and October 2023.[2]

**Figure: Given a model of what the tables mean, GPT-4 answered three times as many questions correctly**

*Average execution accuracy, %, GPT-4 on 43 insurance questions, SQL on raw tables vs a knowledge graph, by question and schema complexity, 2023*

|  | SQL on raw tables | With a knowledge graph | Change |
| --- | --- | --- | --- |
| All questions | 16.7 | 54.2 | 37.5 pts |
| Simple both | 25.5 | 71.1 | 45.6 pts |
| Hard question | 37.4 | 66.9 | 29.5 pts |
| Hard schema | 0 | 35.7 | 35.7 pts |
| Hard both | 0 | 38.5 | 38.5 pts |

Source: Sequeda, Allemang and Jacob, arXiv 2311.07509, Table 1 · as of 13 November 2023 · simple and hard refer to question and schema complexity as the authors defined them

ReportedReported, 2023

On questions that touched the more complex part of the schema, SQL written directly against the tables answered none correctly. With the knowledge graph, accuracy on those questions was 35.7% and 38.5%.[2]

Models have improved since, as section 1 shows, but the direction of the finding has not changed and the vendors now say it themselves. Microsoft’s own Copilot documentation tells customers to prepare the semantic model first, because without it Copilot “can misinterpret the data and return generic or inaccurate results”.[3] ThoughtSpot sells Spotter on the same argument: answers grounded in a governed semantic model, and a separate agent, SpotterModel, whose job is to help build that model.[5]

## What it takes to switch it on in Power BI

Our Power BI page records it as the de facto BI default in Indian GCC analytics teams, and Pro is often already paid for inside Microsoft 365 E5, so its rules are the practical ones.[4] Microsoft’s requirements, as documented in August 2026, have four gates, and a Pro licence passes none of them on its own.[3]

**Figure: Four gates stand between a Power BI Pro licence and a working Copilot**

*Requirements for Copilot in Power BI, per Microsoft's documentation, August 2026*

1. 01Paid capacityFabric F2 or higher, or Premium P1. “A Power BI Pro or Premium Per User license alone isn’t sufficient.” F2 lists from about US$262 a month.
2. 02Admin switchThe tenant setting that lets users use Copilot and Azure OpenAI features must be on. It is on by default, but an admin may have turned it off.
3. 03Cross-region consentOutside the US and EU data boundary, Copilot is off by default until an admin allows data sent to Azure OpenAI to be processed outside the tenant’s region.
4. 04A prepared modelDescriptions, synonyms and verified answers on the semantic model. Without them Copilot falls back on generic answers.

Source: Microsoft Learn, Copilot for Power BI overview; TechBag Power BI intel page for the F2 price · as of 30 September 2026

The third gate is the one Indian buyers miss. An Indian Fabric tenant sits outside both named boundaries, so Copilot stays off until someone with authority accepts that prompts and the data in them may be processed in another country. Microsoft also lists India West among the regions where the standalone and app-scoped Copilot experiences were not yet available.[3] That is a data-protection decision, not an IT toggle, and it should be written down by whoever signs for personal data.

The bill also behaves differently from a seat. Microsoft says token consumption drives Copilot billing, that Copilot’s use is measured in the capacity’s own compute units and processed as background operations, and that a newly bought or resized capacity can take up to 24 hours before Copilot recognises it.[3] In our reading, a popular Copilot competes with the reports that share its capacity; watch the capacity metrics in the first month before deciding the F2 was enough.

## What the 2023 promise looks like now

The original piece promised speed, accuracy greater than manual methods, and predictive insight for every team. Set against the evidence above, the claims that held are narrower and more operational.

**Figure: AI analytics turned out to be a query writer that depends on your definitions, not an oracle**

*The 2023 framing of this blog against what the benchmarks and vendor documentation show, as of September 2026*

What we said in 2023

- AI analytics is more accurate than manual analysis
- Descriptive, predictive and prescriptive are the categories to buy
- The main barrier is specialist skills
- Adopt it and insight follows

What the evidence says in 2026

- The best benchmark system is right 83% of the time; people 93%
- The product is an English question turned into a governed query
- The main barrier is an undocumented data model
- Adopt it after the definitions, the capacity and the data-transfer decision

Source: BIRD leaderboard; Sequeda et al. 2023; Microsoft Learn · as of 30 September 2026

The practical consequence is that you can measure your own version of the BIRD number before you buy, and you should. It takes a week, not a project:

1. **Write down twenty to thirty questions** your business users already ask, each with the answer finance or sales would sign off: last quarter’s revenue by region, open orders over ninety days, churned customers by plan.
2. **Ask each one three times** in the tool, in the words a user would type, and score it the way BIRD does: does the result match the signed-off answer, yes or no.
3. **Add the definitions**: table and column descriptions, synonyms, the three or four measures everyone argues about. Then run the same questions again.
4. **Compare the two scores.** The gap is what the semantic model is worth to you, and it is usually the better investment than a pricier licence.

Our view

Our view: roll AI analytics out first to the twenty questions your finance and sales teams already ask every week, each with a verified answer you can check it against. Widen access only when it gets those right. A tool that is wrong one time in six is a fine analyst’s assistant and a poor board-pack author.

If you are buying this in India

In rupees, the entry ticket for Copilot in Power BI is the capacity, not the seat. TechBag’s Power BI page lists Pro at US$14, about ₹1,165 a user a month, and Fabric F2 from about US$262, about ₹21,750 a month; both attract 18% GST, which takes F2 to roughly ₹25,665.[4] ThoughtSpot, whose founders are of Indian origin and which has run a Bengaluru engineering centre since 2017, is quote-priced in US dollars and invoiced in INR through a reseller.[5] TechBag quotes both in INR; ask for the capacity sized against your real query volume, not the list.

The regulation that bites is the cross-region gate. The DPDP Rules were notified on 13 November 2025 and bring the security-safeguard and breach-notice duties into force on 13 May 2027.[6] For banks, the RBI’s Cybersecurity, Technology Risk and Resilience Framework Directions of 31 July 2026 add vendor audit and inspection rights to any contract that processes their data.[7] Before an admin ticks the box that lets prompts leave India, record which data the semantic model exposes, and keep personal data out of the tables Copilot can see.

How we did this

Figures as of 30 September 2026

BIRD scores are test-set execution accuracy as shown on the public leaderboard; we read the top eight entries and the 2023 baselines and excluded any entry dated after 30 September 2026. The knowledge-graph figures are Table 1 of the Sequeda, Allemang and Jacob preprint; the quadrant labels are shortened. The Copilot requirements are quoted from Microsoft Learn, page dated 24 August 2026. INR prices are from TechBag’s Power BI intel page as committed on 30 September 2026, converted there at its stated rate; the GST-inclusive F2 figure is our arithmetic. The 2023 original quoted Forbes Insights, Gartner and PwC forecasts; we did not repeat them because they were forecasts for 2023 and we could not trace the primary documents.

Spotted an error? Write to us; corrections are published at the foot of the piece.

Sources

1. [BIRD-SQL, A Big Bench for Large-Scale Database Grounded Text-to-SQLs, leaderboard](https://bird-bench.github.io/) — entries dated March 2023 to 26 September 2026
2. [Sequeda, Allemang and Jacob, A Benchmark to Understand the Role of Knowledge Graphs on LLM Accuracy for Question Answering on Enterprise SQL Databases (arXiv 2311.07509)](https://arxiv.org/abs/2311.07509) — 13 November 2023
3. [Microsoft Learn, Copilot for Power BI overview (requirements, regions, data preparation)](https://learn.microsoft.com/en-us/power-bi/create-reports/copilot-introduction) — 24 August 2026
4. [TechBag, Microsoft Power BI intel page (Pro, PPU and Fabric F-SKU prices in USD and INR)](https://www.thetechbag.com/microsoft/microsoft-power-bi) — as committed 30 September 2026
5. [TechBag, ThoughtSpot Spotter intel page (semantic-model grounding, SpotterModel, India R&D)](https://www.thetechbag.com/thoughtspot/thoughtspot-spotter) — as committed 30 September 2026
6. [DPDP Rules 2025, commencement timeline](https://legismith.com/dpdp-rules-2025-timeline/) — notified 13 November 2025
7. [RBI, Cybersecurity, Technology Risk and Resilience Framework Directions, 2026, via practitioner summary](https://taxguru.in/rbi/rbi-issues-commercial-banks-cybersecurity-technology-risk-resilience-framework-directions-2026.html) — 31 July 2026 RBI PDF not fetched.

Related on TechBag: the [ThoughtSpot Spotter page](https://www.thetechbag.com/thoughtspot/thoughtspot-spotter) covers the agentic analyst and its semantic model; the [Power BI page](https://www.thetechbag.com/microsoft/microsoft-power-bi) prices Pro, PPU and Fabric capacity in INR. For the seat-versus-capacity arithmetic, see our note on [when Power BI F64 beats Pro seats](https://www.thetechbag.com/blog/impact-data-analytics-decision-making).

**Disclosure.** TechBag is a software reseller: it earns a margin when a reader buys through it, including from vendors named here. The analysis is TechBag’s own; no vendor paid for or saw this piece before publication. Facts are labelled reported, TechBag count or our view.

Go deeper on TechBag

- [ThoughtSpot Spotter](https://www.thetechbag.com/thoughtspot/thoughtspot-spotter) — the agentic analyst, grounded in a governed semantic model
- [Microsoft Power BI](https://www.thetechbag.com/microsoft/microsoft-power-bi) — Pro, PPU and Fabric capacity priced in INR
- [Databases & data tools guide](https://www.thetechbag.com/browse/devops/databases-data-tools) — the databases the semantic model sits on