Home / Docs / Training history

Training history

The complete base-model lineage, failure analysis and v0.6 clean restart.

Work in progress: this page describes MoroLM-56M v0.6 and will change as the project moves.

Version lineage#

v0.1 — first base#

Custom 56M architecture trained for 6,000 updates. Loss decreased, but raw generation remained poor. Later discovered to use the wrong two-token-ahead objective.

v0.2 — Arabizi pilot#

Added Morisco and synthetic Latin-Darija data. Held-out metrics improved under the old objective, but generation still failed.

v0.3 — MVP candidate#

Expanded Darija/Arabic mix and replay strategy. Still trained with the hidden alignment bug.

v0.4 — strengthen run#

More replay lowered the historical loss without solving language generation.

v0.5 — fresh 100M-token run#

Major fresh-data expansion. The model still generated loops and punctuation despite substantial training, triggering a deep pipeline audit.

Alignment audit#

The base dataset was found to pre-shift labels while the model also shifted them internally. Effective target: t+2. Dataset fixed to return x, x.

500-step overfit proof#

A random-weight 56M model trained on a tiny sample reached validation loss 0.6584 / perplexity 1.93 and reproduced the exact expected continuation with rank #1 probabilities around 97–99%.

v0.6 — cleanstart#

First clean lineage. Random initialization, corrected objective, 12,500 updates and approximately one 102.4M-token pass.

The double-shift bug#

Old TokenBlockDataset#

x = tokens[i : i+seq]
y = tokens[i+1 : i+seq+1]

Because the model also shifted labels internally, the network learned A→C instead of A→B.

Fixed dataset#

x = tokens[i : i+seq]
y = x
return x, y

The model now performs the only shift, producing the intended next-token objective.

v0.6 training configuration#

Setting	Value
Sequence length	512
Batch size	2
Gradient accumulation	8
Tokens/update	8,192
Peak LR	4e-4
Warmup	300 updates
Optimizer updates	12,500
Weight decay	0.1
Precision	BF16

Validation trend#

Update	Loss	PPL
500	7.2534	1412.89
1,000	6.7544	857.80
1,500	6.4553	636.08
2,500	6.0883	440.67
3,500	5.8417	344.36
10,000	5.2623	192.93
11,000	5.2406	188.78
12,500	5.2239	185.65

Important: historical v0.1–v0.5 base losses were measured under the same faulty two-token-ahead pipeline. They are preserved as research artifacts but must not be compared as ordinary next-token perplexities against v0.6.