Start with a real local Qwen answer: three findings, a comparison and quotations from two selected Apple disclosure paragraphs. Run the supplied checker, inspect one claim against its evidence, and record what you would keep or change. No model installation, API key or extra Python package is needed.
Run the checks
Extract the download and open a terminal inside the ai-extraction folder. Use Python 3.9 or later:
python3 audit.py --response qwen/response.json --output outputs/qwen-original.json
Use python instead if that is your environment's command. If setup fails, save the exact command and error; you can still inspect the sources and JSON in a text editor.
The original Qwen answer should pass the structure, quotation and coverage checks, including all five quotations. Read semantic_support: not_evaluated and human_support_review_required: true: a mechanical pass leaves the interpretation for you to judge. Internal source-hash consistency also does not establish source authenticity.
Make one support decision
Open qwen/response.json beside P01 and P02 in the source selections. Choose its first finding or its comparison. Record keep, revise, or reject, and explain your choice with an exact source quotation. Ask:
- Does the whole claim preserve the source's qualifications, such as “may” or “not always”?
- Does it connect two circumstances without evidence of that relationship?
- Does a statement about selected paragraphs become a claim about the entire filing?
For finding 1, examine whether possibility is preserved. For the comparison, examine the 2024 no-assurance qualification and the distinction between licensing limitations and AI-related allegation risk.
Inspect the editorial revision
After making your decision, compare qwen/editorial-response.json with the unchanged original:
python3 audit.py --response qwen/editorial-response.json --output outputs/qwen-editorial.json
This also passes, now with six quotations because the revision adds the AI sentence to the comparison evidence. It is an assistant-proposed editorial revision, not a second Qwen response or an independently human-approved answer. qwen/editorial-changes.json records the three field-level edits. Passing both files does not choose between their interpretations.
Stop when you have the original check output, one source-supported decision and any replacement wording. No extra report or fresh model generation is required.
What is in the download
| Files | What they contain |
|---|---|
source.json, SOURCES.md, prompt.txt, schema.json |
Shared source selections, instruction and response contract |
qwen/response.json, qwen/run.json |
Unchanged local Qwen output and selected run metadata |
qwen/editorial-response.json, qwen/editorial-changes.json |
Separate proposed editorial revision and its change log |
qwen/check-original.json, qwen/check-editorial.json |
Saved mechanical audit results |
audit.py, tests/ |
Offline checker and its tests |
response.json, corrected-response.json, run.json, REVIEW.md |
A different, native-assistant run and its one generated correction |
controls/, controls.json, audit-notes.md |
Constructed controls and worksheet for that separate worked run |
Exact settings and execution limits remain available in the source and execution notes. The two runs are not a controlled model comparison.
Another worked run and three controls
For a second investigation, inspect the native-assistant worked review. Its four findings and comparison involve a different review question: the granularity of a claim and its assertion label. Use the worksheet for that exercise. Its original and one generated correction are separate from the Qwen files:
python3 audit.py --response response.json --output outputs/initial-check.json
python3 audit.py --response corrected-response.json --output outputs/corrected-check.json
The following controls were deliberately constructed from that earlier example. They are not observed Qwen mistakes:
python3 audit.py --response controls/missing_field.json --output outputs/missing-field.json
python3 audit.py --response controls/invented_quote.json --output outputs/invented-quote.json
python3 audit.py --response controls/unsupported_claim.json --output outputs/unsupported-claim.json
The first two should fail and exit nonzero. The third should pass. Inspect its first claim and explain why a genuine quotation does not support the added conclusion. Keep that decision in the worksheet if you take this extension.
To inspect the checker's own tests:
python3 -m unittest discover -s tests -v
Optional fresh generation
Give prompt.txt and source.json to an assistant you already have, save its untouched answer under a new filename, and run the same checker. Record the displayed model/interface, settings you know, prompt, output and any corrections. Running Qwen locally requires a suitable model and hardware setup; the saved-output exercise above does not. See the dated model setup and memory notes before considering a local rerun. The reported model size is not a total-memory requirement.
For a new attempt, set a stopping rule before generation: one targeted correction at most, then keep, qualify, remove or manually revise unresolved claims. Preserve each original. This is a fixed model-assisted workflow; no autonomous agent or investment signal is implemented here.