attestql run audit-ceef14ee-7aab-4fc2-a493-981dcc90708e

The whole of this run is a release asset

This site holds 0 of the 5 question directories this run wrote; the archive holds every one:

archive: minidev-sqlite-mistralai-mixtral-8x7b-instru-4.tar.gz
size: 1,075,349 bytes
sha256: 7f34c38f4a2fbc7c9cfd7720ee1dcc8af8c27748e47e2bdd6ebe556f7e3703ea

What it counted

what	how many
questions audited	52
ERROR	35
EQUAL	12
NOT_EQUAL	5
probes fired	1

Of the questions this run audited: 35 ERROR, 12 EQUAL, 5 NOT_EQUAL

the probes

probe	fired on
ordering-over-numeric-text	0
arbitrary-cut	1
not-a-function-of-the-data	0
float-aggregate-order	0
direction-against-question	0

Of 52 golds this run read, the probes fired on: arbitrary-cut on 1.

credited by BIRD's own check and NOT_EQUAL here

what	how many
credited by BIRD and NOT_EQUAL here	0
multiplicity	0
type	0
order	0
truncation	0
other	0
the test-suite check answered 1	0
the test-suite check answered 0	0

What the run was made of

run: audit-ceef14ee-7aab-4fc2-a493-981dcc90708e
summary format: attestql/audit/summary/2
question file: data/questions/mini_dev_sqlite.json sha256:4ba5fa8de55856222f484d380d2ba872b380bf79d825de70478e2120cb0fc43b | https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_sqlite.json, 2024-06-19
questions in the file: 500
prediction file: data/preds-sqlite/predict_mini_dev_mistralai-mixtral-8x7b-instru-4_sqlite.json sha256:53cdf9142e3f2e49941eb7b50ebbea90361b2c5b46de37d5ba1c244187d643a2 | https://raw.githubusercontent.com/bird-bench/mini_dev/b3d4bcbbae9a96934ad812551eb400c7a3b23c12/llm/exp_result/sql_output_kg/predict_mini_dev_mistralai-mixtral-8x7b-instru-4_sqlite.json, 2024-06-19
statements read: 500
keyed by: position
data file: data/minidev/dev_databases/superhero/superhero.sqlite sha256:75e94a2c3236ee3bb2c01fb97a1c4b4c1c269bcefd4eab1d04be323d2d0825b1 | minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f), member minidev/MINIDEV/dev_databases, superhero/superhero.sqlite, 2024-06-13
server: SQLite 3.53.4 | file=/private/tmp/attestql-runs/data/minidev/dev_databases/superhero/superhero.sqlite | size=237568
database role: file
engine: sqlite
parser: validator audit:sqlglot-sqlite-parse, sqlglot 30.18.0, dialect sqlite
serialization: attestql/audit/2
statement timeout: 30 s
fixture digest depth: counts
schema digest: sha256:921b79dd088877e3b6090a31ab919676ab6b45adf398a468edc025b74a77f750
data as of: 2026-09-08T05:04:59.960649+00:00 the instant the run started
shuffled copies: prepared
shuffle seed: 1
experimental probe: off

The session the run was made in

what the engine reported, 9 settings

sqlite_version: 3.53.4
encoding: UTF-8
reverse_unordered_selects: 0
query_only: 1
journal_mode: delete
data_version: 1
compile_options: ATOMIC_INTRINSICS=1,COMPILER=clang-21.0.0,DEFAULT_AUTOVACUUM,DEFAULT_CACHE_SIZE=-2000,DEFAULT_FILE_FORMAT=4,DEFAULT_JOURNAL_SIZE_LIMIT=-1,DEFAULT_MMAP_SIZE=0,DEFAULT_PAGE_SIZE=4096,DEFAULT_PCACHE_INITSZ=20,DEFAULT_RECURSIVE_TRIGGERS,DEFAULT_SECTOR_SIZE=4096,DEFAULT_SYNCHRONOUS=2,DEFAULT_WAL_AUTOCHECKPOINT=1000,DEFAULT_WAL_SYNCHRONOUS=2,DEFAULT_WORKER_THREADS=0,DIRECT_OVERFLOW_READ,ENABLE_API_ARMOR,ENABLE_COLUMN_METADATA,ENABLE_DBSTAT_VTAB,ENABLE_FTS3,ENABLE_FTS3_PARENTHESIS,ENABLE_FTS5,ENABLE_GEOPOLY,ENABLE_MATH_FUNCTIONS,ENABLE_MEMORY_MANAGEMENT,ENABLE_PERCENTILE,ENABLE_PREUPDATE_HOOK,ENABLE_RTREE,ENABLE_SESSION,ENABLE_STAT4,ENABLE_UNLOCK_NOTIFY,MALLOC_SOFT_LIMIT=1024,MAX_ATTACHED=10,MAX_COLUMN=2000,MAX_COMPOUND_SELECT=500,MAX_DEFAULT_PAGE_SIZE=8192,MAX_EXPR_DEPTH=1000,MAX_FUNCTION_ARG=1000,MAX_LENGTH=1000000000,MAX_LIKE_PATTERN_LENGTH=50000,MAX_MMAP_SIZE=0x7fff0000,MAX_PAGE_COUNT=0xfffffffe,MAX_PAGE_SIZE=65536,MAX_SQL_LENGTH=1000000000,MAX_TRIGGER_DEPTH=1000,MAX_VARIABLE_NUMBER=250000,MAX_VDBE_OP=250000000,MAX_WORKER_THREADS=8,MUTEX_PTHREADS,SYSTEM_MALLOC,TEMP_STORE=1,THREADSAFE=1,USE_URI
collation_list: RTRIM,NOCASE,BINARY
case_sensitive_like: 0

What the run states about itself

ids the question file states twice: 137, 138
prediction positions not compared: 487, 488

The questions

question	verdict	what was asked
q717	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 1, Col: 16. SELECT sp.power[4m\[0m_name FROM superhero sh JOIN hero\_power hp ON sh.id = hp.hero\_id JOIN superpower sp ON hp.power\_i
q719	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 5, Col: 15. o s JOIN hero_power hp ON s.id = hp.hero_id JOIN superpower sp ON hp.power_id = sp.id WHERE sp.power[4m\[0m_name = 'Super Strength' AND s.height\_cm > 200;
q723	ERROR	prediction: statement: the text does not parse: Error tokenizing 'power.id AND superpower.power_name = 'Agility'; ``'
q728	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 1, Col: 11. SELECT eye[4m\[0m_colour\_id, COUNT(*) as eye\_color\_count FROM superhero WHERE publisher\_id = (SELECT id FROM publ
q730	ERROR	prediction: statement: the text does not parse: Error tokenizing 'ND publisher.publisher\_name = 'Marvel Comics'; ``'
q732	ERROR	prediction: execute: execute: no such column: publisher_name
q733	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 5, Col: 53. lisher ON superhero.publisher_id = publisher.id WHERE colour.colour = 'Gold' AND publisher.publisher[4m\[0m_name = 'Marvel Comics';
q736	ERROR	prediction: execute: execute: no such column: hero_attribute.attribute_name
q737	ERROR	prediction: statement: the text does not parse: Error tokenizing 'id = race.id WHERE superhero_name = 'Copycat'; ``'
q739	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 5, Col: 23. d = hero_power.hero_id JOIN superpower ON hero_power.power_id = superpower.id WHERE superpower.power[4m\[0m_name = 'Death Touch';
q743	ERROR	prediction: execute: execute: no such column: alignment
q744	ERROR	prediction: statement: 2 statements in one text; an audit runs one
q745	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 3, Col: 16. SELECT id FROM publisher WHERE publisher[4m\[0m_name = 'Star Trek'; ``` Note: The above SQL query will return the publisher ID for 'Star Trek'
q751	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 1, Col: 16. SELECT sp.power[4m\[0m_name FROM superhero sh JOIN hero\_power hp ON sh.id = hp.hero\_id JOIN superpower sp ON hp.power\_i
q760	ERROR	prediction: execute: execute: incomplete input
q761	ERROR	prediction: statement: the text does not parse: Expecting ). Line 4, Col: 69. uperhero.gender_id = gender.id WHERE gender.gender = 'Male' AND weight_kg > (SELECT AVG(weight_kg) [4m\[0m* 0.79 FROM superhero);
q764	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 1, Col: 13. SELECT power[4m\[0m_name FROM superpower JOIN hero\_power ON superpower.id = hero\_power.power\_id WHERE hero\_power.he
q766	ERROR	prediction: execute: execute: no such column: ha.attribute_name
q769	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 1, Col: 19. SELECT s.superhero[4m\[0m_name FROM superhero s JOIN hero\_attribute ha ON s.id = ha.hero\_id JOIN attribute a ON ha.attribut
q772	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 1, Col: 20. SELECT s.eye_colour[4m\[0m_id AS eyes, s.hair\_colour\_id AS hair, s.skin\_colour\_id AS skin\_colour FROM superhero s JOIN ge
q773	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 1, Col: 19. SELECT s.superhero[4m\[0m_name, s.publisher\_id, c1.colour as eye\_colour, c2.colour as hair\_colour, c3.colour as skin\_colo
q775	ERROR	prediction: execute: execute: incomplete input
q781	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 1, Col: 14. SELECT height[4m\[0m_cm FROM superhero JOIN colour ON superhero.eye\_colour\_id = colour.id WHERE colour.colour = 'Am
q782	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 3, Col: 10. SELECT superhero_name FROM superhero WHERE eye[4m\[0m_colour\_id = (SELECT id FROM colour WHERE colour = 'Black') AND hair\_colour\_id = (SELECT id FROM
q785	ERROR	prediction: statement: the text does not parse: Error tokenizing 'id FROM alignment WHERE alignment = 'Neutral'); ``'
q788	ERROR	prediction: execute: execute: incomplete input
q792	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 1, Col: 16. SELECT sp.power[4m\[0m_name FROM superhero sh JOIN hero\_power hp ON sh.id = hp.hero\_id JOIN superpower sp ON hp.power\_i
q796	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 1, Col: 19. SELECT a.attribute[4m\[0m_name, ha.attribute\_value FROM attribute a JOIN hero\_attribute ha ON a.id = ha.attribute\_id JOIN
q800	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 1, Col: 23. Here is the SQLite [4mcode[0m to calculate the percentage of superheroes with blue eyes: ```sql SELECT ROUND(100.0 *
q801	ERROR	prediction: execute: execute: no such function: DIVIDE
q806	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 4, Col: 21. colour.color FROM superhero JOIN colour ON superhero.eye_colour_id = colour.id WHERE superhero.full[4m\[0m_name = 'Karen Beecher-Duncan';
q819	ERROR	prediction: execute: execute: no such function: SUBTRACT
q824	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 5, Col: 23. = hero_power.hero_id JOIN superpower ON hero_power.power_id = superpower.id WHERE superpower.power[4m\[0m_name = 'Wind Control' ORDER BY superhero\_name;
q825	ERROR	prediction: statement: the text does not parse: Invalid expression / Unexpected token. Line 7, Col: 15. ero_id JOIN superpower sp ON hp.power_id = sp.id WHERE sp.power_name = 'Phoenix Force'; This query [4mwill[0m return the gender\_id of the superhero who has the ability of Phoenix Force. If you need to get the
q829	ERROR	prediction: execute: execute: no such column: publisher_name

A question whose statement could not be run wrote no directory, so it is a row here and has no page: the summary holds the side that stopped, the step it stopped at and the engine's own message.

The JSON this page was rendered from

summary.json attestql/audit/summary/2