Make local models fit the machine.

Why 4-bit weights?

Large language models are mostly large arrays of numbers. Storing selected weights in four bits uses less disk space and less memory bandwidth, making local inference more practical on ordinary hardware.

JQ4 is designed for Reckoning’s tensor engines and keeps the surrounding Hugging Face-style model files in place. It is a quantized layout for the JVM, not a replacement for the model’s configuration or tokenizer.

One weight, two pieces

A quantized weight is stored as packed 4-bit values plus a matching scale sidecar. The scales let the runtime reconstruct an approximation of the original floating-point values during matrix operations:

model.layers.0.self_attn.q_proj.weight
  packed 4-bit values

model.layers.0.self_attn.q_proj.weight.qb
  per-block scale data

Keep the model directory familiar

A JQ4 model still looks like a normal model directory:

config.json
tokenizer.json
tokenizer_config.json
model.safetensors
model.safetensors.index.json
README.md

# generated during quantization
deliverance-quantization.json
.finished

The manifest records the source and output paths, dtype changes, generated sidecars, and shape transforms so the conversion remains inspectable.

Quantize the expensive parts

JQ4’s default policy focuses on the large attention and MLP projection matrices:

• Query, key, value, and output attention projections
• MLP gate, up, and down projections
• Large matrix-shaped weights that dominate storage and memory traffic

Embeddings, normalization weights, small vectors, and other sensitive tensors remain dense by default. This is a deliberate balance: capture most of the size and bandwidth benefit without making every tensor lossy.

See what the runtime used

Profile counters show which dtype path was actually used during a run:

[profile-counter] sampler.output_weight_Q4 count=256
[profile-counter] mlpblock.down_input_F32 count=9216