Models · KOLDOS

Model explorer

Pick a model, a quantization and a context

The spec sheet shows what has been published about each model and estimates the memory it needs. Anything KOLDOS has not published yet is marked Pending.

Model

Quantization

4-bit, about 4.85 bits per weight (Q4_K_M-style). The usual starting point for running models locally.

Context

Tokens the model keeps in memory at once. KOLDOS keeps the active context well below the window on purpose.

KOLDOS 7B

Available

A 7B coding model run locally through llama.cpp. KOLDOS is built around it: deterministic checks, budgets and tools do the heavy lifting, so a 7B model can do reliable work on a mid-range GPU.

Parameter countNominal size from the model name.: About 7 billion
Architecture: Decoder-only transformer, about 7 billion parameters
Runtime: GGUF file, run with llama.cpp on CUDA. KOLDOS configures a 24,576-token window.
Quantization: Q5_K_M in use. IQ4_XS, Q4_K_M and Q6_K also measured
Context: Configured window: 24,576 tokens
Estimate settingsChosen on the left. They only change the memory estimates below, not the model's specification.: 4-bit · 4K tokens
VRAM estimateTo run entirely on a GPU.: 5.2 GBweights 4.0 + cache 0.1 + runtime 0.6 + kept free 0.5 GB
RAM estimateInstalled RAM to run on the CPU only.: 7.5 GBweights 4.0 + cache 0.1 + runtime 0.4 + system 3.0 GB
License: Apache 2.0
Status: Available. Works in the current build.
Release date: Pending. Not published yet
Through the API: Local API: served as the model "koldos". Hosted API: not started yet.
Designed for: Programming
Conversation
Tool use
Running locally on mid-range GPUs
Trade-offs: On a 6 GB GPU the 24K context may need an 8-bit KV cache, which can reduce performance.
Not fine-tuned yet. Specialization is on hold until enough real-use data has been collected.
Needs quantization to run on 6 to 8 GB GPUs, which can cost some quality compared with the unquantized model.

Memory figures are estimates from the parameter count and reference assumptions, not measurements.

Check it against your hardware

More parameters is not automatically better

A larger model can handle harder work, but it costs memory and speed on every machine. The right model is the one that fits your hardware and your task.

01It needs more memoryEvery parameter has to be stored. At the same quantization a model three times larger needs about three times the memory, so it fits on fewer machines.
02It generates more slowlyEach new token reads all the weights once. More weights means fewer tokens per second on the same hardware.
03Fitting in VRAM matters more than sizeA smaller model that runs entirely on the GPU is usually far faster than a larger one that has to be offloaded to system RAM.
04Heavy quantization erodes the gainSqueezing a large model into small memory with very few bits per weight costs quality, which can cancel out what the extra parameters add.
05Many tasks don't need itShort questions, rewriting and routine code are often handled well by the smaller model. The larger one pays off on long, multi-step work.

First model

Second model

Model size

Parameters, drawn to scale.

22B

Memory requirements

Estimates at 4-bit with a 4K context.

VRAM, all on GPU5.2 GB

RAM, CPU only7.5 GB

VRAM, all on GPU14.4 GB

RAM, CPU only17.1 GB

Expected hardware class

Smallest VRAM tier that holds the whole model. Official requirements are pending.

EntryUp to 6 GB of VRAM

High13 to 16 GB of VRAM

Use cases

Programming
Conversation
Tool use
Running locally on mid-range GPUs

Better reasoning
Better tool use and selection
Complex tasks through planning and verification
Native vision inside the model itself

Availability

Works in the current build.

API:Local API: served as the model "koldos". Hosted API: not started yet.

Long-term idea. Not committed.

API:Pending. A 22B model is not part of any plan.

Status

Available

Future

Same quantization, very different sizes

For every generated token the model reads its weights from memory. A larger model has more weights to read, so on the same GPU it is slower, and it needs a bigger card to stay in VRAM at all.

Size of the weights, drawn to scale

KOLDOS 7B

Available

4.0 GB

KOLDOS 22B

Future

12.4 GB

Weights only, at about 4.85 bits per weight (Q4_K_M-style). Context and runtime need more on top.

Performance

What performance depends on

There is no single speed number for a model. The same model can be fast on one machine and barely usable on another.

Quantization: How compressed the model's weights are. Fewer bits means a smaller file and less memory, with some loss of quality.; Going from 16-bit to about 4.85 bits per weight cuts weight memory by roughly 70%. Generation speed also improves, because fewer bytes have to be read per token.
VRAM: Memory on the graphics card. The most important number for local AI.; Token generation is limited mostly by memory bandwidth. GPU memory is several times faster than system RAM, so a model that fits entirely in VRAM runs much faster.
System RAM: Main memory. Holds whatever doesn't fit on the GPU, plus the operating system.; Layers that don't fit in VRAM can stay in RAM and run on the CPU. It works, but those layers become the bottleneck for every generated token.
CPU: Handles the parts of the model that aren't on the GPU, or all of it without one.; CPU inference speed depends on memory bandwidth and core count. More threads help up to a point, after which memory becomes the limit.
Backend: The software that actually runs the model on your hardware.; Different backends use the GPU differently (CUDA, ROCm, Vulkan, Metal or CPU paths) and support different quantization formats. Speed on the same hardware can differ noticeably between them.
Context length: How much text the model can consider at once: your conversation, files and instructions.; Every token of context stores keys and values for each layer (the KV cache). Memory grows linearly with context, and long prompts take longer to process before the first token appears.
Configuration: Settings like how many layers go on the GPU, batch size and threads.; Offloading as many layers as fit in VRAM is usually the biggest win. Which settings KOLDOS Local exposes will be documented with the first build.

Context costs memory too

Doubling the context doubles the KV cache. The larger model stores more per token, so long contexts push it out of VRAM sooner.

Estimated KV cache memory by context length
Context	KOLDOS 7B	KOLDOS 22B
4K	0.1 GB	0.9 GB
8K	0.2 GB	1.8 GB
16K	0.5 GB	3.5 GB
32K	0.9 GB	7.0 GB

Based on reference architectures for each size with a 16-bit cache. KOLDOS will publish exact figures with the model files.

Reference

Quantization levels used in estimates

These are common llama.cpp formats used as a reference. The files KOLDOS publishes may use different formats.

Level	Bits per weight	Reference format	7B weights	22B weights	Quality
3-bit	3.91	Q3_K_M	3.2 GB	10.0 GB	Smallest footprint. The quality loss is usually noticeable.
4-bit	4.85	Q4_K_M	4.0 GB	12.4 GB	The usual starting point for running models locally.
5-bit	5.69	Q5_K_M	4.6 GB	14.6 GB	Closer to the original weights for a little more memory.
6-bit	6.56	Q6_K	5.3 GB	16.8 GB	Very close to the original weights.
8-bit	8.5	Q8_0	6.9 GB	21.8 GB	Near-lossless, at almost twice the size of 4-bit.
16-bit	16	F16	13.0 GB	41.0 GB	Unquantized weights. Mostly useful as a reference point.

Want the full picture for your machine? Open the hardware checker.