AttestQL

a static viewer over the JSON an audit writes

Audit text-to-SQL gold and predictions on PostgreSQL with typed replay evidence.

Ten minutes

Two commands, no server and no account. The second audits the sandbox the package carries and writes a directory of JSON, one per question.

uv tool install attestql        # or: pip install attestql
attestql demo --out demo

The output below is that command's own, printed while this page was built. The run it names is this build's: run audit-5ac1b5ac-ceba-418a-841e-577381666b11, data as of 2026-09-08T04:11:01.285823+00:00. Yours will carry your own.

q1029 european_football_2 R-ORD  NOT_EQUAL  smells=none  demo/audit/q1029/
q879  formula_1   R-ORD  NOT_EQUAL  smells=ordering-over-numeric-text  demo/audit/q879/
q207  toxicology  R-SET  NOT_EQUAL  smells=none  demo/audit/q207/
q900001 synthetic   R-ORD  GOLD-ONLY  smells=not-a-function-of-the-data  demo/audit/q900001/
q900002 synthetic   R-ORD  GOLD-ONLY  smells=arbitrary-cut,not-a-function-of-the-data  demo/audit/q900002/
q900005 synthetic   R-ORD  EQUAL      smells=ordering-over-numeric-text  demo/audit/q900005/
6 questions: 3 NOT_EQUAL, 5 smells fired, 0 credited by BIRD but NOT_EQUAL (0 multiplicity, 0 type, 0 order, 0 truncation), 0 timed out (0 gold, 0 prediction)
rerun: attestql audit --engine sqlite --dsn demo/fixture.sqlite --questions demo/questions.json --predictions demo/predictions.json --out demo/audit

One of them, in full

NOT_EQUAL states that these two statements disagree on this data under this rule. It does not state which of them is wrong.

in gold, not in the prediction, 1 row

from counterexample.json, 1 row, up to 25 shown per side

side nationalityTEXT
gold Norwegian

in the prediction, not in gold, 1 row

from counterexample.json, 1 row, up to 25 shown per side

side nationalityTEXT
second Peruvian

NOT_EQUAL never means the gold is wrong. It means these two statements disagree on this data under this rule; the rows above are the disagreement; a maintainer decides.

The whole of that question, with both statements, both evidence records, the probes and the JSON each part was rendered from.

What the published runs found

The three headline numbers are rendered from the aggregate the next phase writes. It is not in this build, so no number is shown: a number typed here by hand would be the one thing this site is against.

The runs

Each row is one CLI run, and each links to the page rendered from its own summary.json. Every benchmark is indexed on its own page.

sandbox

run date engine question set question set digest
demo 2026-09-08T04:11:01.285823+00:00 sqlite demo/questions.json sha256:7ad58addc070b425cabfc9d67bbd9c95840253160cf3b73d672e9aeb4897f5ff

Read on