Hand-written WGSL kernels reading model.safetensors straight off the wire — the tensor bytes go into GPU buffers directly, no export step, no runtime graph compiler.
Fetches config.json, tokenizer.json, and model.safetensors from the Hugging Face CDN, parses the safetensors header by hand, and uploads all 92 tensors straight into GPUBuffers.
Base model — it continues text, it doesn't follow instructions. Try a story opener or a sentence fragment and let it run.
| Architecture | Dense decoder-only, 10 layers, hidden 384, 6 heads (head_dim 64), FFN inner 1024, tied embeddings |
| Positional encoding | RoPE, θ=10000, applied as complex-plane rotation on interleaved (x0,x1),(x2,x3)… pairs — matched exactly, not the "rotate-half" GPT-NeoX layout |
| Normalization | RMSNorm, pre-norm, eps=1e-5, computed in f32 |
| Attention | Full scaled dot-product attention, single flash-style kernel per head with an online softmax and a causal early-exit on the key loop |
| Feedforward | SwiGLU: down_proj(silu(gate_proj(x)) ⊙ up_proj(x)), inner dim 1024 |
| Sampling | Temperature scaling + top-k, categorical draw via cumulative sum on CPU over the top-k slice returned from GPU |
| Weight loading | Raw fetch() of model.safetensors, JSON header parsed by hand, each tensor's byte range sliced from the ArrayBuffer and copied into a GPUBuffer via writeBuffer — no ONNX export, no ORT runtime, no transformers.js |
context_len; no paging, no batching, single sequence onlytokenizer.json's vocab and merges exactly, tested against the reference tokenizer output