What is a language model?
A language model is a program that has learned the patterns of a language by reading a great deal of text. Give it the start of a sentence and it predicts which word is likely to come next, then the next, and so on. That is really all "generating text" is: very good next-word guessing, one small piece at a time.
It has no database of facts and no understanding in the human sense. It has absorbed the statistical habits of a language - which words tend to follow which - and replays them.
How language models are usually built
The famous models - the ones behind the popular chat assistants - are built at enormous scale, in roughly three moves:
1. Gather text
Hundreds of billions, even trillions, of words scraped from the web, books, and code.
2. Learn the pieces
The text is chopped into small sub-word units (called tokens) that the model works with instead of whole words.
3. Pretraining
The model reads all that text over and over, nudging billions of internal numbers (its parameters) until it predicts the next piece well. This runs on thousands of specialized chips for weeks and can cost millions of dollars. A final polishing step then makes the model pleasant and safe to chat with.
The result is powerful and general - but expensive, mostly closed, and English-first. Slovene is a rounding error in that training data.
How CankarGPT is built
This project uses the same core idea - learn a language by predicting the next piece - but deliberately tiny, cheap, and from scratch, in Slovene:
- The text is public-domain Slovene literature plus Slovenian Wikipedia (about 73 million words) - not the whole web.
- The tokenizer is trained from scratch on Slovene, so the pieces fit Slovene words, not English ones.
- The model has 26 million parameters - thousands of times smaller than frontier models.
- Pretraining ran on one desktop graphics card in under half an hour, for essentially no money.
- A short specialization pass then taught it the prose voice of one author, Ivan Cankar.
Because it is small, it does not reason or hold facts well - but it convincingly learns a voice. The next stage turns it into a styler: you write plain modern Slovene, and it rewrites the same meaning in Cankar's register.
The two approaches, side by side
| Aspect | Typical large LLM | CankarGPT |
|---|---|---|
| Training text | hundreds of billions of words (the web) | ~73 million words (public-domain Slovene) |
| Tokenizer | English-centric | trained from scratch on Slovene |
| Parameters | tens-hundreds of billions | 26 million |
| Hardware | thousands of GPUs, weeks | one desktop GPU, ~30 minutes |
| Cost | millions of dollars | about €0 |
| Goal | be generally capable | learn one Slovene literary voice |
Why build it the small way?
- To understand the whole pipeline by building every piece yourself, not by calling someone else's API.
- To show that meaningful, measurable results are possible on a hobby budget.
- To give Slovene - and one of its greatest prose writers - a small model of its own.
Everything here is measured against a frozen test set, open-source, and public from the first commit. If a claim does not carry a number, it does not go on the page.