Experiments · Oct 19, 2026
Experiment: can a small model learn Oracle better than a large one?
We trained Qwen3-14B on public Oracle documentation and tested it against Claude Opus 4.8 on 379 questions neither had seen. Ours came out ahead on 289. Here is how we ran it, and what it does not show.
Hypothesis
An open model of moderate size, trained on one business system, will name that system's reports, tables, columns and settings more accurately than a large model that knows it only in general.
Method
- Model: Qwen3-14B, tuned with QLoRA in ForgeLab, our model lab,. Run 11: rank 32, one epoch, 1,291 steps, final training loss 0.760, 40 minutes 49 seconds on one RTX 5090.
- Data: 10,325 questions and answers generated from public Oracle documentation. We split by report, so no test report appears in training.
- Test: 379 questions the model never saw. 330 are about Oracle E-Business Suite and 49 about Oracle Fusion Cloud REST.
- Compared with: the same 14B before tuning and Qwen3-32B, both in 4-bit with thinking off. Also Claude Opus 4.8 (
claude-opus-4-8) through its API, with no tools or retrieval and the same answer-length limits. - Scoring: our own scorer. Table, column and SQL answers are scored on the names they give. Other answers are scored on word overlap with the reference. We average the score per answer.
Results
| Model | Average score |
|---|---|
| Qwen3-14B, tuned | 0.344 |
| Claude Opus 4.8 | 0.151 |
| Qwen3-32B | 0.137 |
| Qwen3-14B, before tuning | 0.112 |
Question by question, the tuned model led by 0.19 on average (95% range: 0.17 to 0.22). It won 289 questions, Opus 4.8 won 53, and 37 were ties.
What it means
A small model trained on one field can know that field better than a much larger general one, on our measure. Next: more domains, expert-judged answers, and large models given retrieval over the same documentation.