ForgeLab · Oct 19, 2026
Our tuned 14B model beat Claude Opus 4.8 on 289 of 379 Oracle questions
We trained Qwen3-14B on public Oracle documentation and put it head to head with Claude Opus 4.8 on 379 questions neither had seen. Ours came out ahead on 289 of them. Opus won 53, and 37 were ties.
- Questions where our tuned model came out ahead
- 289 of 379
- Oracle evaluation, run 11
- Questions where Claude Opus 4.8 did better (37 were ties)
- 53
- Oracle evaluation, run 11
- Average score per answer, ours vs Opus 4.8 (0 to 1)
- 0.34 vs 0.15
- 379 paired answers
- Test questions: 330 Oracle EBS, 49 Oracle Fusion Cloud
- 379
- Test set file
The question
Oracle E-Business Suite has thousands of reports, tables, columns and settings. Can a mid-sized open model, trained on that material, answer questions about it better than a large general model?
Method
- Training. We tuned Qwen3-14B in ForgeLab, our model lab, (run 11) with QLoRA: rank 32, learning rate 2e-4, one epoch, sequences of 4,096 tokens, 4-bit, seed 3407. It took 1,291 steps and 40 minutes 49 seconds on one RTX 5090. Final training loss was 0.760.
- Comparison. Four models answered the same 379 test questions: the tuned 14B, the same 14B before tuning, Qwen3-32B, and Claude Opus 4.8 (API model
claude-opus-4-8). The open models ran in 4-bit, with thinking off and no sampling randomness. Opus ran through its API with no tools and no retrieval, under the same answer-length limits as the 14B. - Scoring. Answers are scored automatically. For table and column questions, it checks how many of the right names an answer gives, and how many wrong ones (an F1 score). For settings and descriptions, it measures word overlap with the reference answer. SQL questions use a mix of both. The headline number is the average score across all 379 questions, not a count of correct answers.
Data
- Training data: 10,325 questions and answers generated from public Oracle documentation, mostly docs.oracle.com plus the Enginatics report library. No customer data.
- Test data: 379 questions the model never saw in training: 116 case studies, 49 REST, 44 settings, 44 tables, 43 descriptions, 39 columns, 34 SQL, 10 SQL outlines. The 49 REST questions are about Oracle Fusion Cloud, not E-Business Suite.
- Keeping test and training apart: we split by report. No report named in a test question appears in any training question. Still, 15 test answers share text with training answers, because related reports reuse the same documentation. Without those 15, the result is 0.341 against 0.148.
Results
| Model | Average score (379 questions) |
|---|---|
| Qwen3-14B, tuned by us | 0.344 |
| Claude Opus 4.8 | 0.151 |
| Qwen3-32B (4-bit) | 0.137 |
| Qwen3-14B before tuning (4-bit) | 0.112 |
The tuned model scored higher on 289 questions. Opus 4.8 scored higher on 53. The other 37 were ties.
A model that knows the system by heart answers every question. Opus declined 106 of them.
Tuning tripled the model's score. On SQL questions the two were close, 0.12 against 0.15 for Opus.
About the score
The scorer rewards answers that name the right tables, columns and settings in the reference format. It is the right measure for a model meant to know a system by heart, and it is why we report questions won alongside the average score. We will repeat the run on other domains and have experts judge the answers.