Same two words. Two different minds.
Prompt given to two checkpoints of the same 26M model: Mati je ("The mother is...")
Mati je žensko osebno ime. Izvor imena: ime Mati je različica ženskega osebnega imena Matina. Pogostost imena: po podatkih Statističnega urada Republike Slovenije je bilo na dan 31. decembra 2007 ...
Mati je bila tesno skupaj, s prstom je sijala po sobi. Na desni je držala v roki roko, ležala mu je na postelji, na vzglavju je držal s tresočo roko ter se je ozrl doli po sobi. „Veselo je že, Mana.“
The base model, having read Wikipedia, writes an encyclopedia entry. After a short specialization pass on Cankar alone, the very same architecture stops reporting and starts grieving. That register shift is the whole project.
From token soup to a voice
Three checkpoints, one prompt - Bilo je ("It was..."). Each is scored on the same 50 held-out Cankar works it never saw, in bits per byte (lower is better).
TinyCankar / 15M / Cankar-only, overfit
Bilo je čisto vse. Zdelo se mi je, da živi nekje med mrtvimi in veselimirami, da mu je nemarna groza v srcu ... goloslužen močevečer na prejšnjem, čisto neznanem ...
base / 26M / full corpus
Bilo je jasno, da je bila stavba zgrajena med letoma 1911 in 1912. ... Leta 1912 je ameriški odred napovedal, da bo v nedeljo, 1. maja ...
CankarGPT v1 / 26M / base + Cankar
Bilo je že pozno, ko je posetil kozolec po klancu, in ko je bil v krčmi, je šel, in je šel, zakaj nihče bi ne bil šel ponj? Bilo mu je pri srcu, ko mi je prišel do srca, da bi videl, kako je bilo.
More data cut the error 32%; specialization cut another 3.6% without memorizing the held-out works. Note the word klanec (the hillside slope) surfacing in v1 - the title of one of Cankar's novels.
Every claim carries a measurement
Held-out set: 50 whole Cankar prose works (146,268 tokens, 5% of the Cankar corpus), frozen and containment-checked so no fragment leaks into training. The style classifier's signal is punctuation rhythm and function words - voice, not topic (audited).
How it has improved, release by release
This page tracks the model as it grows. Every release is scored on the same frozen held-out set, so the numbers are comparable across stages - a new row lands here each time a stage ships.
| Release | Model | Params | Held-out BPB | What changed |
|---|---|---|---|---|
| v0.1 | TinyCankar | 15M | 2.2227 | First from-scratch proof of life: Cankar-only, deliberately overfit - the charming "before". |
| v0.2 | CankarGPT v1 | 26.3M | 1.4508 | Full-corpus base model, then specialized on Cankar. A 35% lower BPB than v0.1, without overfitting. |
| next | The styler | 26.3M | tbd | Plain modern Slovene in, the same meaning back in Cankar's register (phases 5-6). |
Unconditional continuations
Prompt Na cesti je ("On the road there is..."), temperature 0.8, nothing cherry-picked across seeds.
Na cesti je pozdravil Milena. Obraz je bil zelo droben in zelo gladek. ... „Ali ste blagi, milan? Milan, prijatelj moj, ali ste blagi?“
Na cesti je bil lep večer, tudi že se je svetila v neskončnost; zmirom sem ji pogledal v obraz in že je bil žalosten. „Kaj pa ti, Tone?“
Na cesti je bila palica, na kateri so se raztresene kapljice zravnale. Mislili so, da je to švigalo ... »Duhovit grad! Ali ni bila na deželi?«
Na cesti je prihajala ženska, na stolu, vse tri panove. Njen triletni obraz je resen, kakor starikav, krepak previden, krepkost otroški obrazi ...
Where this is going
The foundation
done - phases 0-4
- Corpus - public-domain literature plus Slovenian Wikipedia, cleaned, deduplicated and merged (~73M words).
- Tokenizer - a custom Slovene sub-word vocabulary, trained from scratch on that corpus.
- Eval harness - a frozen held-out set and a style classifier, so quality is measured, never claimed.
- TinyCankar -> base -> v1 - a tiny proof of life, then general Slovene, then Cankar's voice.
The product
next - phases 5-6
- Style pairs - synthetic plain <-> Cankar sentence pairs, generated with a large model.
- The styler - fine-tune on those pairs so you feed in plain modern Slovene and get the same meaning back in Cankar's register.
- This is where a tiny model earns its keep: it supplies the voice, the meaning comes from the input.
The reach
later - phases 7-9
- Serving - a free CPU endpoint, with a stretch goal of running the model fully in the browser.
- Two-tier demo - "write like Cankar" plus "ask Cankar", the second pairing a knowledge model with the styler.
- GaMS v2 - a fine-tune of a larger Slovene base model, as an honest quality yardstick.
The current model continues a prompt in Cankar's voice. The next stages turn it into a styler: give it a plain modern-Slovene draft, get back the same meaning in Cankar's register. That is where a tiny model earns its keep - it supplies the voice, while the meaning comes from the input.
Technical overview
- Corpus
- 126,355 documents, 72.3M words - Slovenian Wikipedia + 15 public-domain authors; the Cankar slice is 2.92M tokens
- Tokenizer
- custom Slovene BPE, 8,192 vocab - 1.77 tokens/word on Cankar; trained from scratch on the merged corpus
- Model
- 26.3M-param GPT - 6 layers, 384-dim, 6 heads, 1,024 context; bf16, flash attention
- Training budget
- 142.77M tokens - base: 3 epochs, loss 9.0 -> 2.8; specialization: ~3 gentle epochs on Cankar
- Hardware
- one RTX 4070 Ti Super (16GB) - 264k tokens/sec; base pretrain ~27 minutes; run locally, €0
- Evaluation
- held-out BPB + style classifier - frozen 50-work holdout, containment-checked; confound-audited AUC 0.993
What the terms mean
- Bits per byte (BPB)
- How surprised the model is by real Cankar text. Fewer bits = better prediction. Lower is better.
- Token / BPE
- The model reads text in sub-word pieces called tokens; a BPE tokenizer learns those pieces from the corpus itself.
- Parameters
- The numbers the model learns (its weights). 26 million here - tiny; frontier models have 100,000x more.
- Epoch
- One full pass over the training text. This model saw its corpus about three times.
- Held-out set
- Works deliberately kept out of training, used only to test - so a good score cannot be memorization.
- Specialization
- Extra training of the general model on Cankar alone, to shift its register toward his voice.
- Temperature
- How much randomness the sampler allows. Higher is more varied and riskier; lower is safer and flatter.
- Style classifier
- A separate detector that tells Cankar's prose from 14 other authors', used to measure how "Cankar" the voice is.
What this is, and what it is not
What it is
- A from-scratch demonstration that a tiny model can learn a distinct Slovene literary voice from ~3M tokens.
- Locally fluent: grammar, morphology, and Cankar's melancholic cadence are convincingly there.
- Fully measured - every number here comes from a frozen eval harness, not from vibes.
What it is not
- Not logically coherent across a paragraph - at 26M parameters that is a scale floor, expected, not a bug.
- Not the styler yet: v1 continues a prompt; the plain->Cankar rewrite comes in a later stage.
- Not affiliated with Ivan Cankar or any institution - this is an AI pastiche, built for learning.