attestql run audit-df9bb5f6-31b9-42a7-b84e-4f7361da1168

The whole of this run is a release asset

This site holds 0 of the 13 question directories this run wrote; the archive holds every one:

archive: minidev-sqlite-meta-llama-3-8b-instruct-2.tar.gz
size: 13,154,426 bytes
sha256: 606473169f3dbe0d3f73dd20ddb40ee7085c571d289f2fe20ecd0878a5b7eb29

What it counted

what	how many
questions audited	30
ERROR	16
NOT_EQUAL	13
EQUAL	1
probes fired	2

Of the questions this run audited: 16 ERROR, 13 NOT_EQUAL, 1 EQUAL

the probes

probe	fired on
ordering-over-numeric-text	0
arbitrary-cut	1
not-a-function-of-the-data	1
float-aggregate-order	0
direction-against-question	0

Of 30 golds this run read, the probes fired on: arbitrary-cut on 1; not-a-function-of-the-data on 1.

credited by BIRD's own check and NOT_EQUAL here

what	how many
credited by BIRD and NOT_EQUAL here	0
multiplicity	0
type	0
order	0
truncation	0
other	0
the test-suite check answered 1	0
the test-suite check answered 0	0

What the run was made of

run: audit-df9bb5f6-31b9-42a7-b84e-4f7361da1168
summary format: attestql/audit/summary/2
question file: data/questions/mini_dev_sqlite.json sha256:4ba5fa8de55856222f484d380d2ba872b380bf79d825de70478e2120cb0fc43b | https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_sqlite.json, 2024-06-19
questions in the file: 500
prediction file: data/preds-sqlite/predict_mini_dev_meta-llama-3-8b-instruct-2_sqlite.json sha256:e1bd066794d120ff766d010a5cf4e4118e48404c4b0168fce4dfaf362a975301 | https://raw.githubusercontent.com/bird-bench/mini_dev/b3d4bcbbae9a96934ad812551eb400c7a3b23c12/llm/exp_result/sql_output_kg/predict_mini_dev_meta-llama-3-8b-instruct-2_sqlite.json, 2024-06-19
statements read: 500
keyed by: position
data file: data/minidev/dev_databases/california_schools/california_schools.sqlite sha256:c0903eec662e63068fd1d14403d3d6c1d473287fc10c4356333ea58f878db983 | minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f), member minidev/MINIDEV/dev_databases, california_schools/california_schools.sqlite, 2024-06-13
server: SQLite 3.53.4 | file=/private/tmp/attestql-runs/data/minidev/dev_databases/california_schools/california_schools.sqlite | size=11116544
database role: file
engine: sqlite
parser: validator audit:sqlglot-sqlite-parse, sqlglot 30.18.0, dialect sqlite
serialization: attestql/audit/2
statement timeout: 30 s
fixture digest depth: counts
schema digest: sha256:0906a33c0fa9248160fb7cc746501120c03ae1250fe39bda04cf8640bdbbd18d
data as of: 2026-09-08T05:03:49.045099+00:00 the instant the run started
shuffled copies: prepared
shuffle seed: 1
experimental probe: off

The session the run was made in

what the engine reported, 9 settings

sqlite_version: 3.53.4
encoding: UTF-8
reverse_unordered_selects: 0
query_only: 1
journal_mode: delete
data_version: 1
compile_options: ATOMIC_INTRINSICS=1,COMPILER=clang-21.0.0,DEFAULT_AUTOVACUUM,DEFAULT_CACHE_SIZE=-2000,DEFAULT_FILE_FORMAT=4,DEFAULT_JOURNAL_SIZE_LIMIT=-1,DEFAULT_MMAP_SIZE=0,DEFAULT_PAGE_SIZE=4096,DEFAULT_PCACHE_INITSZ=20,DEFAULT_RECURSIVE_TRIGGERS,DEFAULT_SECTOR_SIZE=4096,DEFAULT_SYNCHRONOUS=2,DEFAULT_WAL_AUTOCHECKPOINT=1000,DEFAULT_WAL_SYNCHRONOUS=2,DEFAULT_WORKER_THREADS=0,DIRECT_OVERFLOW_READ,ENABLE_API_ARMOR,ENABLE_COLUMN_METADATA,ENABLE_DBSTAT_VTAB,ENABLE_FTS3,ENABLE_FTS3_PARENTHESIS,ENABLE_FTS5,ENABLE_GEOPOLY,ENABLE_MATH_FUNCTIONS,ENABLE_MEMORY_MANAGEMENT,ENABLE_PERCENTILE,ENABLE_PREUPDATE_HOOK,ENABLE_RTREE,ENABLE_SESSION,ENABLE_STAT4,ENABLE_UNLOCK_NOTIFY,MALLOC_SOFT_LIMIT=1024,MAX_ATTACHED=10,MAX_COLUMN=2000,MAX_COMPOUND_SELECT=500,MAX_DEFAULT_PAGE_SIZE=8192,MAX_EXPR_DEPTH=1000,MAX_FUNCTION_ARG=1000,MAX_LENGTH=1000000000,MAX_LIKE_PATTERN_LENGTH=50000,MAX_MMAP_SIZE=0x7fff0000,MAX_PAGE_COUNT=0xfffffffe,MAX_PAGE_SIZE=65536,MAX_SQL_LENGTH=1000000000,MAX_TRIGGER_DEPTH=1000,MAX_VARIABLE_NUMBER=250000,MAX_VDBE_OP=250000000,MAX_WORKER_THREADS=8,MUTEX_PTHREADS,SYSTEM_MALLOC,TEMP_STORE=1,THREADSAFE=1,USE_URI
collation_list: RTRIM,NOCASE,BINARY
case_sensitive_like: 0

What the run states about itself

ids the question file states twice: 137, 138
prediction positions not compared: 487, 488

The questions

question	verdict	what was asked
q5	ERROR	prediction: execute: execute: no such column: Virtual
q11	ERROR	prediction: statement: the text does not parse: Expecting ). Line 3, Col: 42. SELECT CDSCode FROM frpm WHERE Enrollment(K-12) + Enrollment(Ages [4m5[0m-17) > 500;
q17	ERROR	prediction: execute: execute: no such column: s.CDSCode
q23	ERROR	prediction: execute: execute: no such column: s.Enrollment (K-12)
q24	ERROR	prediction: statement: the text does not parse: Required keyword: 'this' missing for <class 'sqlglot.expressions.core.Mod'>. Line 1, Col: 30. SELECT s.School, f.Percent (%[4m)[0m Eligible Free (K-12) FROM frpm f JOIN schools s ON f.CDSCode = s.CDSCode WHERE f.Percent (%) Eli
q26	ERROR	prediction: execute: execute: no such column: f.FRPM Count (Ages 15-17)
q27	ERROR	prediction: execute: execute: no such column: s.School
q28	ERROR	prediction: statement: the text does not parse: Expecting ). Line 8, Col: 63. FROM frpm GROUP BY frpm.CDSCode HAVING AVG(frpm.Enrollment (K-12) - frpm.Enrollment (Ages [4m5[0m-17)) > ( SELECT AVG(frpm.Enrollment (K-12) - frpm.Enrollment (Ages 5-17)) FROM frpm
q31	ERROR	prediction: execute: execute: no such column: s1.FreeMealCount_K_12
q32	ERROR	prediction: execute: execute: no such column: frpm.FRPM_Count_K_12
q37	ERROR	prediction: execute: execute: no such column: sc.PctGE1500
q45	ERROR	prediction: execute: execute: no such column: s.School
q62	ERROR	prediction: execute: execute: no such column: CountyCode
q72	ERROR	prediction: execute: execute: no such column: Enrollment_Ages_5_17
q77	ERROR	prediction: execute: execute: no such column: frpm.FRPM_Count_Ages_5_17
q83	ERROR	prediction: statement: the text does not parse: Expecting ). Line 3, Col: 33. SELECT COUNT() FROM schools WHERE Magnet = 1 AND (Low Grade [4m=[0m 'K' AND High Grade = '8'); SELECT City, COUNT() AS SchoolCount FROM schools WHERE Low Grade = 'K'

A question whose statement could not be run wrote no directory, so it is a row here and has no page: the summary holds the side that stopped, the step it stopped at and the engine's own message.

The JSON this page was rendered from

summary.json attestql/audit/summary/2