Sattaya.AI

← Blog

Reading a model's architecture from its safetensors header, without downloading the weights

How I recover a neural network's anatomy from tensor names and shapes alone: 96.8% role accuracy on 510 architectures, from about 46 KB of header. The method behind AI Model Explorer.

Sattaya Singkul · 7 min read

You can learn most of a model’s architecture from a file header of a few tens of kilobytes. A safetensors file starts with a small JSON list of every tensor’s name, shape and storage type. In my new paper, Reading the Architecture off the Weights, I show that this list alone gives you the blocks, the role of each part, the data path and an exact parameter count. I built the idea into AI Model Explorer, a free tool that opens any Hugging Face repository in your browser without downloading the weights.

สรุปภาษาไทย: ผมสร้าง AI Model Explorer เป็นเครื่องมือเปิดดูสถาปัตยกรรมโมเดล AI จาก Hugging Face ได้โดยไม่ต้องดาวน์โหลด weights และเขียน paper อธิบายวิธีอ่านโครงสร้างโมเดลจาก header ของไฟล์ safetensors ขนาดเพียงหลักสิบกิโลไบต์ ผลทดสอบกับ 510 สถาปัตยกรรมระบุหน้าที่ของแต่ละ tensor ถูกต้อง 96.8% เครื่องมือนี้ทำขึ้นสำหรับ workshop “Squeeze Until It Breaks” ที่งาน AI Expo Thailand 2026

Why I built it

I started this for my workshop at AI Expo Thailand 2026, Squeeze Until It Breaks: A Hands-On Edge Inference Contest (Saturday 7 November, 14:00–15:00, Builder Stage). Participants had to understand a speech model before they compressed it. If you don’t know where the weights sit, you can’t tell which layer to squeeze first or which one will break.

So I built a small page that drew one model. Then I wanted it to work on the next model, and the one after that. Every new checkpoint raises the same questions. How many layers are there, and are they all alike? Is it a mixture of experts? What runs once per input, and what runs once per generated token?

For a well-known family, config.json answers these. For anything new, it often doesn’t. Keys differ between families, multimodal models nest several schemas, and some fields only make sense after you read the modeling code. Running that code through trust_remote_code is a known risk on public hubs. Graph viewers need an exported graph, and in practice the whole file.

The header is tiny, and it’s nearly always there

A safetensors file begins with an 8-byte length and a JSON header. Two HTTP range requests are enough to fetch it. Across the 510 architectures I studied, the median header is 46 KB, about 6×10−56 \times 10^{-5} of the size of the weights in BF16. PyTorch zip checkpoints keep the same information in their data.pkl record, which can be read without touching tensor data.

The header carries two kinds of evidence:

  • The name tree shows structure. Integer-indexed lists are stacks of blocks. Identical subtrees are identical block types. The module that owns a stack marks a component.
  • The shapes show function. All blocks in a stack share a residual width dd. An attention region is a set of projections that leave dd, plus one that returns to it. Two components connect where the output width of one equals the input width of the other.

That second point is what surprised me. Names help to break ties, but most of the work survives without them.

How the recovery works

The pipeline: tensor manifest, storage normalisation, module tree, shape algebra and width matching, with config.json filling the gaps and being checked against the weights Only descriptive bytes are fetched. Everything else runs in the browser.

1. Normalise storage. Quantized checkpoints store one logical weight as several tensors. NVFP4, for example, keeps a U8 tensor of shape [o,i/2][o, i/2] plus an FP8 scale tensor of shape [o,i/16][o, i/16]. I recognise these layouts from dtypes and shape ratios, so the logical shape [o,i][o, i] comes back before any name is read.

2. Find the blocks. Names form a tree. Lists of composite items become block lists, and each list gets a structural signature so that identical blocks group together. A hybrid stack that interleaves state-space, mixture-of-experts and attention layers comes out as three block types in the right order.

3. Assign roles by shape. One representative block of each type is classified by the algebra of its matrices. A feed-forward region has one output [d,f][d, f] and one or two inputs [f,d][f, d]. A mixture of experts has stacked tensors [E,⋅,⋅][E, \cdot, \cdot] next to a router [E′,d][E', d].

4. Link components. A chain of matrices whose input width matches one component and whose output width matches another becomes an adapter, which gives you the data path from, say, a vision tower to a language backbone.

5. Fuse the configuration, and check it. Some things leave no trace in shapes. A projection Wq∈RHh×dW_q \in \mathbb{R}^{Hh \times d} with HH heads of size hh has the same shape as one with H/2H/2 heads of size 2h2h; only the product HhHh is visible. The number of experts is visible, but not how many are used per token. So when config.json exists, I read those fields from it and check them against the weights:

H⋅h=x,Hkv⋅h=xkvH \cdot h = x, \qquad H_{kv} \cdot h = x_{kv}

A mismatch is reported, not resolved. Every value in the explorer says where it came from: measured, read from the config, or estimated.

The results

I evaluated on 510 architectures from the transformers library, using the Python class of each module as ground truth. The algorithm never sees those classes.

ConditionRole accuracy (macro)Block lists foundExact parameter counts
Names and shapes96.8%98.6%510/510
Shapes only, names hashed95.0%98.6%510/510
Best name-rule baseline82.3%60.7%n/a
llama.cpp name table29.7%60.7%n/a

Two results here matter most to me. Replacing every name with a random token costs only 1.8 points, so most of the work is done by structure and shapes. And parameter totals stay exact under NVFP4, FP8 and GPTQ storage.

The corpus uses default configurations from one library, so I also tested real files. I read 191 popular Hugging Face checkpoints, 12.1 TB of weights, by transferring 161 MB. Hub model pages count parameters from safetensors metadata, which gives an independent check. The totals matched for 163 of the 170 comparable repositories, and the seven differences are explained in the paper. Four of them are 4-bit releases where the Hub counts packed bytes: an NVFP4 DeepSeek-R1 is shown as 396.8 B when its logical size is 684.5 B.

Head geometry shows the honest limit. From shapes alone, the head count is right about 70–77% of the time. With configuration fusion, it’s 96–98%. The more interesting finding is the consistency check. Of 636 matched components with attention, 31 have config fields that don’t multiply out to the projection widths in the weights. None of those configs is wrong. In those families head_dim means something else, and a tool that trusted the config blindly would report the wrong KV-cache size for exactly those models.

What it tells you in practice

Take nvidia/parakeet-tdt-0.6b-v3, a speech model you might want to run on a phone. Paste the repo id and the explorer reads a 63 KB header from a 2.5 GB file. You see a 24-block conformer encoder holding 97% of the parameters, a 12 M prediction network and a 6 M joint network. Feed-forward modules hold two thirds of each block, so that’s where pruning or low-bit storage pays off. The encoder runs once per utterance while the two small networks run once per emitted label, so streaming latency depends on the small networks and not on the encoder.

Or take a 121.5 B hybrid model, Nemotron 3.5 Super VL. Experts hold 112.7 B of those parameters, and with its top-22 routing only about 13.6 B are active per token. Just 8 of its 88 layers keep a KV cache, so the cache costs 8 KB per token in BF16, an eleventh of what it would be if every layer used the same attention. That’s the kind of number you want before you buy hardware.

What it can’t do yet

Ten of the 510 models score below 50%, mostly because they depart from the block conventions the rules assume: ALBERT shares one layer across depth and stores no list at all, and PoolFormer has blocks with no attention or feed-forward signature. Output heads (F1 54) and adapters (F1 69) are the weakest roles. The eight 2026 case studies in the paper weren’t held out from development, so the unbiased accuracy estimate is the corpus result.

GGUF and ONNX aren’t handled yet, and old non-zip PyTorch pickles are rejected on purpose. Masking is invisible, because a decoder-only model with tied embeddings stores the same tensors as a bidirectional encoder of the same size. Activation functions and the order inside a block stay unknown, and compute figures are estimates that assume standard computation per role.

Try it

Open AI Model Explorer and paste any repository id. Files you drop in are parsed in your browser, and nothing is uploaded. The source and evaluation scripts are MIT-licensed, and every checkpoint result is on the benchmark page. If you use it in research, please cite the paper.

References