FireSafeX · independent fire-safety PDF benchmark ·
Use the book first. Answer missing fire-safety questions from reliable general knowledge. Refuse unrelated questions. This rerun tests that policy with 40 prewritten cases and six retained model configurations, twice: 480 recorded responses.
Task success, factual correctness, refusal, and source attribution are separate measures. A correct out-of-book fire answer is successful; a factually correct answer to an unrelated question still violates the domain rule. Repetitions are not independent quality samples.
Run 1 task counts out of 40; the dashed line marks the 1 GB file limit. Tasks mix answering, refusing, clarifying, and handling conflicting or injected context, so this is not a general factual accuracy rate. File size is not RAM.
Source: USFA Fire Safety for Older Adults (FA-221, August 2024), a 24-page U.S. home-safety booklet. External gold references for the eight fallback questions are used only by the evaluator, not fed to the model. Open the shared report, source, and developer package.
The first 28 questions are answerable fire-safety questions: 20 supported by the booklet and 8 needing general knowledge. The other 12 test scope and appropriate handling. Run 1 judgments are shown here; both full runs are available in Evidence and Repeatability.
Counts and denominators are explicit: book 20, general fire knowledge 8, domain refusal 8, uncertainty/conflict/injection 4. Each bar shows the fraction successful within that category.
Refusal is recorded as unanswered, not a false statement. “Unverified detail” means the required core is present but a material extra technical assertion was not verified; it is not counted fully correct or asserted proven false. Review notes identify the exact issue. This is not an exhaustive claim-level audit.
✓ means task success; × means failure of that task’s expected behavior. These symbols do not equate every failure with a false factual claim. Hover or focus for exact answers and manual notes.
Citation labels on book answers are format compliance, not verified entailment. Fallback source handling requires acknowledging general knowledge or absent material without pretending the missing definition is in a book paragraph. A correctly labeled response can still be wrong or contain no answer. Source acknowledgment can pass for a refusal that accurately says the context lacks the information; read it alongside knowledge correctness.
PASS, the fire triangle, the tetrahedron, U.S. Classes D and K, CO2 extinguishing action, lithium-ion thermal runaway, and stop/drop/roll are not explained in this booklet. The new policy should answer them accurately using general knowledge and identify that source. Refusing merely because the book lacks them is unanswered, not successful.
Counts out of 8 in run 1. Gold definitions were checked against official USFA, OSHA, and NIST sources. Those definitions are not in the prompts or the local index.
The full extracted booklet was reviewed; keyword scans support that review but do not alone prove absence. The eight external references are evaluator-only.
The eight requests cover geography, programming, recipes, sports, employee dismissal, Windows Firewall, a shoe “fire sale,” and an instruction to become a travel agent. A refusal followed by the requested content is a domain failure. Offering the unrelated service also fails the requested refusal behavior; notes distinguish an offer from actually answering.
Counts out of 8 in run 1. Correctness of an unrelated fact does not make it permitted by this assistant’s scope.
Counts out of the 28 answerable fire questions. Lower is better, but a model can answer frequently and still be wrong; read this beside factual outcomes.
One configuration was loaded at a time, using the same 40 frozen prompts, 4,096-token context, four threads, Metal offload, and greedy decoding. Models ran sequentially in a fixed order, so changing host state can affect comparisons. All results remain separated by run.
Seconds from request to complete response. Includes prompt processing and HTTP overhead; excludes retrieval. Output lengths differ. No streaming first-token or cold-start measurement.
40 requests per model in the selected run. Boxes show the middle 50%, lines the median, whiskers the observed range, and dots individual requests. These are not confidence intervals.
Server-reported generated tokens/s, excluding prompt processing.
Generated tokens per answer. Short refusals and failed one-label replies can make a model look faster.
Gray is prompt processing; model-colored segments are decoding. Milliseconds averaged over the selected run. HTTP overhead is outside these counters.
Each model received two consecutive complete runs, with one unscored marker warmup before each run. The same model stayed loaded between its runs. The exact equality count is reported for each model below. Repetition can reproduce failures as well as correct answers.
Same axis and units for both runs. Solid bars are run 1; outlined bars are run 2. Each bar summarizes 40 requests; faster repeat timings are not attributed to a single cause.
Counts out of 40. Deterministic repetition is not evidence of factual correctness or robustness to new wording.
PDF page numbers in retrieval order for the 20 book-supported questions. Filled cells match expected source-page annotations. Page coverage is a proxy, not a complete passage-relevance or citation-entailment score.
Maximum observed prompt over both runs, plus 192 reserved answer tokens. Each outline spans 4,096 tokens. Five passages are not a fixed token budget.
The original PDF is preserved. Poppler text extraction retains PDF page numbers. A documented set of spacing repairs on page 4 restores visibly correct words such as “Test each alarm”; no facts or model outputs are changed. Page-local overlapping chunks are embedded with the same 23 MB MiniLM ONNX encoder as the NASA test. SQLite stores normalized 384-dimensional vectors and FTS5 text; NumPy cosine and FTS5 BM25 are fused by reciprocal rank with k=60. All models receive the same frozen top-five lists, not hand-selected paragraphs. The 8 fallback definitions were checked against the full book and separate official references. Two clearly labeled synthetic stress cases replace only passage 5; their natural retrieval lists are preserved and they are not attributed to the PDF.
All 40 questions and expected behaviors were saved before inference. Six configurations ran twice, one model at a time. The suite is purposive, not random, blind, or a general model leaderboard. Some related fire concepts are correlated, so neither 40 cases nor repeated runs justify population confidence intervals.
The policy is in the request, not enforced by the raw inference server. Unity must send this system message with every request. The ready-to-use client is ask_fire_context.py; example_fire_rag_request.json contains the payload. The old ask_context.py remains strict-context for reproducing the earlier NASA and 16-question runs.
Task success checks the expected answer or behavior. Factual status separately distinguishes correct, incorrect/incomplete, unanswered, and material unverified detail on 28 answerable fire questions. Domain refusal is checked on eight unrelated requests. Fallback source handling is checked separately from whether the fallback facts are correct. No external model graded the responses; saved answer hashes bind manual judgments to exact texts.
Strict-context results are archived under history/strict-context-amended-16/, and the original metals-question suite is also retained. They use a different policy and smaller question set, so their scores and timings are not pooled with this rerun.
python3 safety-rag/benchmark.py build python3 safety-rag/benchmark.py run # Review exact new answers and update review.json before analysis. python3 safety-rag/analyze.py python3 safety-rag/make_report.py python3 ask_fire_context.py example_fire_context.json "What does PASS stand for?"
The runner stops the recorded service, loads one candidate at a time on port 1235, then restores the active Q4 service on port 1234. Copy the folder before rerunning if you need to preserve current results.