Which open language model should a company choose?
The best model is not the largest, it is the smallest that answers your questions correctly. Here is the method for deciding on your own data rather than on leaderboards.
Choose the smallest model that answers your real questions correctly. Public leaderboards measure general tests, not your line of work. A model twice the size costs twice the hardware for a gain that is often nil on management questions.
Why do public leaderboards mislead?
Because they measure general capabilities — reasoning, mathematics, programming — on standardised tests. Your requirement is different: finding a piece of information in your procedures and returning it in correct prose.
A model that ranks poorly may handle that task perfectly well, and an excellent model may disappoint on your own domain vocabulary.
Which criteria really matter?
Language. Open models do not all handle every language equally well. Test on your own texts, not on translated English examples.
Licence. "Open" does not mean "free for commercial use". Some licences impose restrictions according to company size or use. It is the first point to check, and the most often overlooked.
Size. It determines the hardware, and therefore the budget. See how to size the machine.
Context window. How much text does the model accept at once? Decisive if you feed it long documents.
Stability. A model you install does not change. That is an advantage over online services, whose behaviour evolves without notice.
How do you decide between two candidates?
| Step | What you do | What you watch |
|---|---|---|
| 1. Build the test | 30 real business questions | Do they cover your uses? |
| 2. Write the expected answers | With the people who know | Serves as the reference |
| 3. Query each model | Same questions, same instruction | Accuracy, not elegance |
| 4. Measure speed | On the target hardware | Time before the first word |
| 5. Choose | The smallest that passes | The surplus serves no purpose |
This protocol takes a day and will stop you paying for hardware to support a capability you will never use.
Should you change model often?
No. A model that answers your questions correctly has no reason to be replaced because a new one has come out. The cost of a change — new tests, new configuration, new validation — almost always exceeds the gain.
Rerun the test once a year, with the same questions. It is the only useful judge.
Updated Aug. 17, 2026