Home / Docs / Architecture

Architecture

The 56.6M Llama-like model architecture and the planned local inference stack.

Work in progress: this page describes MoroLM-56M v0.6 and will change as the project moves.

Model architecture#

Component	Value
Architecture	Llama-compatible decoder-only Transformer
Parameters	56,607,232
Vocabulary	24,000
Hidden size	512
Intermediate size	1,376
Layers	16
Attention heads	8
KV heads	2 (GQA)
Maximum positions	4,096
Position encoding	RoPE, θ = 10,000
Normalization	RMSNorm, ε = 1e-5
MLP	SwiGLU / SiLU
Embeddings	Tied input/output embeddings
Dropout	0.0

Attention path#

The implementation uses grouped-query attention: eight query heads and two key/value heads, with KV repetition across query groups. PyTorch scaled dot-product attention runs with causal masking enabled.

End-to-end system map#

Browser UIchat + tools

Go backendAPI + memory

llama.cpplocal server

GGUFquantized model

MoroLMlocal inference

SQLiteapp memory

Browserstreamed output

Training loss alignment#

def forward(self, input_ids, labels=None):
    logits = self.lm_head(self.model(input_ids))
    if labels is not None:
        loss = F.cross_entropy(
            logits[:, :-1].reshape(-1, logits.size(-1)),
            labels[:, 1:].reshape(-1),
            ignore_index=-100,
        )

The model performs the causal shift internally. Therefore base-training datasets must return labels aligned to the same token positions as the input.