PocketGround 0.2.0 · Fire Safety v1 dataset
How well do small models answer from documents?
We give each model the same question and five passages from a fire-safety booklet. Compare its answers, off-topic refusals and handling of missing or conflicting information. Read every response behind the scores.
Test dates and settings
Core results
Figure 1Core score, out of 32 questions
Scorecard
Scroll the table horizontally for model size, file size and timing.
Read these results with the following in mind
Terms used in this report
- RAG (retrieval-augmented generation)
- An application first searches your documents for relevant passages, then asks a language model to answer using them. Here the search step is fixed, so every model receives identical passages and only the answering is compared.
- Parameters, in billions (0.8, 1.2, 2)
- The size of the model as published by its developer. “0.8B” means about 0.8 billion parameters. Larger models are often more capable and usually slower. LFM3-Omni-2B is named after the published speech, vision and text model. Only its text part was loaded here, and that part has fewer parameters than the full model. We do not state an exact figure for it.
- Quantization (Q4_K_M, Q8_0)
- How the model’s numbers are compressed in the GGUF file format used by local runtimes. Q4_K_M stores weights at roughly 4 bits each and Q8_0 at 8 bits, instead of the usual 16. Fewer bits give a smaller file and can cost some quality. Qwen3.5-0.8B was tested at both levels.
- Core score
- The number of the 32 core questions handled as expected: 20 answered from the booklet, 8 off-topic requests refused and 4 tricky inputs handled. Each question type has its own total, so compare “7/8” with other “/8” values, not with “19/20”. The total is a convenience, not a general accuracy percentage.
- File size (MB)
- The size of the downloaded model file in decimal megabytes (1 MB = 1,000,000 bytes). Running a model needs more memory than its file size, for the conversation context and the runtime itself. Peak memory was not measured.
- Run 1 and Run 2
- Each configuration answered all 40 questions twice, with identical settings. Repeating checks consistency. It does not double the number of independent questions.
- Test date and configuration
- Tests took place in two groups, on 27 and 28 October 2026, with different builds of the local inference runtime. A configuration is one model file run with one runtime build and one set of settings. The Qwen3.5-0.8B Q4_K_M file was tested on both dates, so nine configurations cover eight files and seven models. Scores can be compared across dates. Answer times can only be compared within one date.
Why are 8 of the 40 questions scored separately?
These eight ask about fire safety but need facts that are not in the booklet. They test what the model knows on its own, rather than how it uses the supplied passages. Their results are under “Beyond the booklet”. This split was chosen after the original 40-question run. No prompt, answer or judgment was changed.
A model can answer well from the document and still answer questions it should refuse.
Each question type below has its own total. Compare models within a type.
Figure 2Success by question type
Bar length is the share passed. Labels show passed out of tested.
TableEvery core question, every configuration
✓ passed · × failed. Select any mark to read the exact answer and the reviewer’s note.
Figure 3Fact check on the 20 document questions
Reviewed separately from pass or fail. “Adds unverified detail” means the answer added a material claim that the booklet and reference sources do not establish. It is not automatically false.
Figure 4Document answers that cite a passage
Answers out of 20 containing at least one paragraph marker such as [P2]. This counts formatting only. It does not check that each citation supports the claim next to it.
Read the exact answer behind any score.
File size, answer time and answer length
All times were measured on one Apple M3 Pro laptop with 18 GB of unified memory, using Metal graphics acceleration. They show relative speed on that machine, not speed on phones or other devices. When both test dates are shown, timings are drawn separately for each date because the runtime builds differ.
Figure 5Model file size
Downloaded GGUF file in decimal megabytes. Memory use while running is higher and was not measured. For LFM3-Omni-2B this is the text model file only. Its vision projector and audio codec were not loaded, so this is not its full multimodal footprint.
Figure 6Time to receive the complete answer
Seconds from request to full answer, with the answer returned in one piece rather than streamed. Includes reading the prompt and writing the answer. Excludes retrieval and model loading. The thin line spans fastest to slowest; the box covers the middle half of calls; the dark mark is the median. This spread is not a confidence interval.
Figure 7Typical answer length
Median output tokens for the same questions. A token is roughly three-quarters of an English word. Short refusals can be fast without being useful.
Figure 8Median answer time in Run 1 and Run 2
Shown side by side, never averaged together. The model stayed loaded for both runs. Reuse of cached prompts, answer length and the fixed order of models can affect these values.
Timings for a subset are taken from the calls in the full 40-question schedule. They are not separately executed runs. First-token delay, peak memory, battery use, heat and on-device speed were not measured.
Can the model answer correctly when the booklet is silent?
These eight questions are about fire safety, but the booklet does not contain the answer. The original instructions allowed reliable general knowledge in this case, provided the answer says where it came from. They are left out of the 32 core questions and kept here so the complete result remains visible.
Figure 9Correct answers beyond the booklet
Factually correct answers out of 8. Saying where the answer came from is scored separately in Figure 10.
Figure 10Answers that are honest about their source
Out of 8: the answer says it relies on general knowledge, or that the booklet does not cover the question, and does not invent a booklet citation. A wrong answer, or an honest “I can’t find this”, can still pass this check. Read it together with Figure 9.
One question, the same five passages, every model.
- Fix the evidence. Five passages were retrieved once for each question. Every model receives identical system and user messages.
- Generate an answer. Each pinned model file runs with its own chat template, with reasoning (“thinking”) and tool use switched off.
- Review the answer. A reviewer compares it with the expected behaviour and the source text, keeps the exact answer, and records a reason for every failure.
- Report each result separately. Document answers, off-topic refusals, tricky inputs, citations, file size and speed.
What this release measures
PocketGround is an exploratory benchmark of how small local models answer from retrieved passages. The passages are fixed in advance, so it compares models, not search systems, and it does not test every part of a deployed RAG application. LFM3-Omni-2B was tested with text input and text output only; its speech and vision abilities were not evaluated.
Tested configurations
Settings
Exact system prompt
Source document and retrieval
How grading works
Pass or fail follows each question’s expected behaviour. Document questions need the requested facts without material unreliable extra advice. Off-topic questions need a refusal. Tricky inputs need the specific behaviour asked for: saying the answer is unknown, asking for missing details, pointing out conflicting passages, or ignoring an instruction planted in a passage. These are different tasks; the combined core score is a convenience, not a general accuracy figure.
Document answers also receive a fact label: correct, wrong or incomplete, not answered, or adds unverified detail. Off-topic and tricky questions are scored on behaviour instead. Source disclosure for questions beyond the booklet is scored separately. Paragraph citations are detected automatically; whether each citation supports its claim has not been audited.
Each distinct answer was reviewed by hand. A judgment was reused only when the answer text was exactly identical. No independent expert review or agreement study between reviewers has been done. Answers that reach the 192-token output limit stay visible and are scored as returned.
Limits and what should come next
The 40 questions were chosen on purpose around one U.S. home-safety booklet and are related to each other. They are not a random sample, a held-out test across many subjects, or a production certification. The 32/8 reporting split was chosen after collection, and the instructions still allow reliable general knowledge when the booklet has a gap.
Future releases should add new held-out documents and subjects, more off-topic and planted-instruction tests, independent scoring and measurements on real devices. Any change to the prompt, template, model, retrieval or grading defines a different configuration and should be kept and labelled separately.
Related work
SLM RAG Arena focuses on models under 5 billion parameters and blind side-by-side preference. MiniRAG / LiHua-World addresses lightweight on-device RAG and evidence-backed tasks. SLM-RAG-EVAL provides a study of compact models generating answers and a public reproduction package.
PocketGround emphasises small local model files, exact fixed inputs, separate behavioural checks and visible grading evidence. This describes its scope; it is not a claim to be the first benchmark with these features. Scores from other projects are not directly comparable.
Reproduce it without installing a Python package
python3 pocketground.py verify python3 pocketground.py inspect python3 pocketground.py run --model quest-chat --label my-model --out runs/my-model.json python3 pocketground.py review-sheet runs/my-model.json --out runs/my-model.review.json # Complete the manual judgments, then: python3 pocketground.py score runs/my-model.json runs/my-model.review.json
Needs Python 3.10 or later and a separately configured local chat-completions server for inference. Start with README.md and docs/REPRODUCE.md in the package. This report works offline.
Open the preserved detailed 40-question report and its 17 graphs →