AI question classification for Examiar: what we learned beyond accuracy

A Physics pilot comparing DeepSeek, ChatGPT and a custom Qwen-based classifier—and examining whether their answers were usable, repeatable and efficient.

By Faraz Ahmad · Benchmark completed 12 October 2026 UTC · Research pilot

A Physics question can contain an equation without naming the concept it tests. A familiar concept can also appear in different places in different books. For Examiar, both details matter: the task is to place a question in the right chapter of a particular book.

That makes question classification a product problem as well as an AI problem. A plausible label is useful only if it belongs to the correct catalog, survives the application’s output checks, and can be produced reliably at the workload the product needs.

We evaluated three approaches on the same 100-question test set. The pilot produced promising chapter agreement, but it also exposed issues that an accuracy-only comparison would have missed: repeated input context, formatting failures, different measurement boundaries, and an unsupported batch interface.

1. Define the decision before choosing the model

The immediate task was deliberately narrow: classify English Physics questions by chapter. Topic classification is part of the broader product direction, but it was outside this evaluation.

There are two decisions inside that seemingly simple task. First, identify what the question asks the learner to do. Second, map that concept or operation to the relevant chapter in the selected book.

Consider an illustrative example: one question asks a learner to calculate acceleration using F = ma; another asks whether the same equation is dimensionally consistent. The equation is identical. The tested operation is different. A classifier that responds only to familiar terms or formulas can miss that distinction.

This also explains why general Physics knowledge is insufficient on its own. A model may recognize a concept while still needing the book’s chapter structure to choose the correct label.

2. Evaluate approaches without assuming that fine-tuning is necessary

Large language models are a form of machine learning. Their advantage here is that pretraining can supply useful language and subject knowledge before we provide task-specific examples. That does not establish that a conventional text classifier, an embedding-based approach or a fine-tuned model would perform worse.

Those are legitimate alternatives. This pilot compared three available systems, rather than attempting to settle every possible architecture:

• DeepSeek V4.1 Flash.
• GPT-6.1 Sol Light.
• Examiar’s custom Qwen3-4B-based classifier.

The local run had no fine-tuned adapter active. Its result therefore describes the custom Qwen-based system as deployed; it is not evidence of an accuracy gain from fine-tuning. We have kept its implementation private.

The hosted systems received a compact chapter catalog and the question content, including available choices and context. We did not send the entire book on every request.

That distinction matters when considering efficiency. Reusing task context, batching questions and changing the model are different interventions. Their benefits need to be measured separately.

3. Make the evaluation difficult to accidentally flatter

We prepared separate sets of 115 training candidates, 100 validation questions and 100 test questions. Preparing training candidates did not mean training a new model during this pilot. Validation was available for configuration checks; a small subset was used to check the local input format.

Known historical overlaps were screened, and the test inputs and provisional reference labels were frozen for evaluation. Screening reduces a known source of leakage, but it cannot prove that every question is semantically novel or that a pretrained model has never encountered related material.

All systems were evaluated against the same 100 unique test questions. We tested one, ten and fifty questions per prompt where the interface supported independent answers for each question.

DeepSeek completed three repetitions at each batch size. ChatGPT completed one full single-question pass and three repetitions of the larger batches. The local system completed one native single-question pass, followed by batch capability checks.

Repetition tests consistency. It does not turn 100 unique questions into 300 independent examples. Keeping that distinction explicit prevents a small evaluation from appearing larger than it is.

4. Read the accuracy results in context

These are chapter-agreement scores from the matched first run. For that metric, we removed enclosing Markdown code fences before reading otherwise valid JSON; we did not repair labels or missing answers.

The local classifier’s 95% was encouraging, but it does not establish a general advantage over the hosted systems. We compared different deployed workflows, used a small test set, and worked with provisional references that have not been independently teacher-certified. Some chapters had only two or three test examples.

In the all-run analysis, the local single-question estimate had a 95% bootstrap interval of 91–99%; ChatGPT’s fifty-question result had an interval of 89–98%. Those intervals overlap. The study also compares different workflows and is a pilot, not a general provider ranking.

The unsupported local batches were also a result. Its interface returned one chapter decision for a combined input, rather than one decision for every question. We marked those configurations unsupported instead of assigning a misleading accuracy score.

5. A correct chapter is not always a usable response

Across DeepSeek’s three single-question runs, normalized chapter agreement was 88.3%. Correct answers that also satisfied the strict required output format accounted for 82.7% of predictions. Twenty of the 300 responses wrapped JSON in Markdown fences, causing the strict format check to fail.

A consumer can choose to accept such fences, but that is an application decision. It should not disappear inside the accuracy calculation.

ChatGPT’s single-question pass also encountered one application stream failure. It was included as an unsuccessful prediction, with no retry. A question that receives no usable answer is part of the operating outcome.

This led to a practical evaluation rule: record chapter agreement, format compliance, abstentions and application failures separately. A single percentage cannot describe all four.

6. Batching reduced repeated input substantially

When each question is sent separately, instructions and chapter context are repeated. Larger batches share that context across more questions.

Ten-question batches reduced measured input tokens by 47.0%; fifty-question batches reduced them by 51.2%, relative to single-question prompts. Output tokens remained close to 3,000 per 100 questions because the system still needed to return a label for each question.

The fifty-question configuration used the fewest input tokens, but DeepSeek’s chapter agreement was 92%, compared with 93% for ten-question batches in both the first-run and aggregate results. In this pilot, the additional input reduction therefore came with a small observed difference in agreement. We would test that trade-off on a larger set before treating it as stable.

These are token reductions, not measured reductions in total operating cost. Pricing, cache treatment, local compute and human review all affect that calculation.

DeepSeek exposed actual provider usage counts. Exact backend token counts were unavailable for the other two models. Visible-text estimates were kept separate and were not presented as billing or inference-workload measurements.

7. Measure the workflow the user actually receives

Recorded durations used different measurement boundaries and can include different delivery, startup and observation overheads.

We recorded the available timings, but did not use them to declare one underlying model faster than another. The relevant production measurement would be time from question submission to a validated, usable classification under the intended workload.

Likewise, the local result belongs to the complete custom classifier. Determining how much each component contributes would require controlled component tests. The current numbers do not isolate those contributions.

8. What the pilot changes about the next decision

The results support continuing the investigation. They do not support claiming that the problem is solved, that one model is universally best, or that fine-tuning has already delivered a gain.

The next evaluation should strengthen the reference labels through independent subject-expert review, increase coverage of sparsely represented chapters, and examine questions that plausibly span several chapters. It should also compare a conventional text baseline and an embedding-based baseline, then test proposed system changes individually.

Topic classification, additional languages and image-dependent questions need their own evaluations. Their performance cannot be inferred from an English chapter-classification pilot. Batch order sensitivity and reviewer time also remain unmeasured.

For product design, this points toward a useful decision boundary: automate classifications that meet the application’s acceptance criteria, and make uncertain or invalid outputs visible for review. Defining and validating that boundary is part of the next phase, rather than a demonstrated outcome of this pilot.

For teams building educational products, that is the question worth carrying forward: what evidence would make a classification safe and useful enough for the next step in your product?

Evidence and disclosure. Results come from the Examiar internal pilot completed on 12 October 2026 UTC. References are provisional; this is not an independently certified benchmark. The public methodology documents the scope, repetitions and measurement limits. Raw questions, book identifiers, prompts and implementation details remain private. This disclosure supports inspection of the reported method and aggregates, but does not enable full independent replication.

Study methodology and supporting measurements

Physics chapter-classification pilot, completed 12 October 2026 UTC. This page discloses the evaluation method and aggregate measurements while keeping book identity, source questions and implementation private.

Study design

• 115 training candidates prepared; no new model training performed.
• 100 validation questions prepared; a six-question local input-format diagnostic was used. This was not a full validation-set benchmark.
• 100 unique English test questions with frozen provisional chapter references; references have not been independently teacher-certified.
• DeepSeek: three repetitions each at 1, 10 and 50 questions per prompt.
• GPT-6.1 Sol Light: one full singles run and three repetitions each of batches of 10 and 50.
• Local classifier: one native singles run; checks at 10 and 50 established that independent per-question batches were unsupported.
• Hosted systems shared task instructions and a compact chapter catalog. Local native input differed. The study compares deployed workflows, not isolated model weights.
• Known historical overlaps screened; independent novelty cannot be guaranteed.

Matched first-run comparison

All-run aggregate measurements

These values combine the repetitions actually performed. They are distinct from the matched first-run table. Repeated predictions remain grouped by unique question when estimating uncertainty.

4,000 bootstrap replicates resampled unique questions, retaining repeated runs within each question. The intervals reflect sampling variability against the provisional references; they do not account for incorrect references or establish a causal provider ranking. Native contracts differ by system.

Measurement definitions

• Chapter agreement: predicted chapter matches the frozen provisional reference, after removing enclosing Markdown JSON fences where present. No missing answer or label repair.
• Correct + native contract compliant: the answer is correct and conforms to that interface’s required response format. This is not the fraction that is merely valid JSON.
• Macro F1: the unweighted average of per-chapter F1 scores, retained to show performance beyond the overall correct fraction.
• Unsupported: no independent label returned for every question. Not counted as zero classification accuracy.
• Failure handling: ChatGPT had one singles stream failure, counted as unsuccessful, without retry. DeepSeek singles had 20 fenced outputs across 300 predictions; strict checks treated them as failures.
• Tokens: actual provider-reported usage available for DeepSeek only. Mean usage is normalized to one 100-question pass. Visible-text estimates for the other models are not comparable usage measurements.
• Timing: Recorded durations used different measurement boundaries. No isolated model-speed ranking is reported.
• Local provenance: no fine-tuned adapter active in the tested local run. Its scores describe the custom Qwen-based deployed system.

Provider-reported token usage

Unresolved questions

Independent subject-expert reference review; stronger coverage of rare chapters; conventional text and embedding baselines; component ablations; performance of a genuinely fine-tuned deployment; topic classification; additional languages; image-dependent questions; batch order sensitivity; human review time; and production economics remain future evaluation work.

For technical context, scikit-learn’s text-classification example illustrates conventional baselines; Sentence-BERT provides a research basis for sentence embeddings; and LoRA describes adapting model weights. These sources explain candidate approaches. They do not establish performance on Examiar’s test set.

Source: Examiar internal benchmark. Full independent replication is limited by the withheld inputs and implementation. No production savings, learner outcomes or teacher-certified accuracy are asserted.