FreeToken: A New System for Running Large MoE Models on Regular PCs

Wow, now even a gaming PC can run advanced models at interactive speeds using official weights without extreme quantization.

Researchers from UC Berkeley, MIT, and UT Austin have opened the source code of FreeToken—a system specifically designed to run ultra-large MoE models on regular computers.

In the paper, they show very impressive results:

• Qwen3.6 35B → laptop with RTX 4060 8GB → 39 tokens/s
• DeepSeek-V4-Flash 284B → RTX 5090 → 22–25 tokens/s
• GLM-5.2 753B → RTX PRO 6000 → 15 tokens/s

Compared to Ollama, the generation speed is 3–4 times higher, and the processing of input context is 6–30 times faster.

This means a gaming laptop with just 8GB of video memory can already run a model with 35 billion parameters at nearly 40 tokens per second.

And this is even faster than the median generation speed of Codex at 33 tokens/s, which the authors mention in the paper.

The main idea of FreeToken is to make the GPU, CPU, RAM, and PCIe work together.

Since MoE models activate only a small part of experts for each token, there is no need to keep hundreds of billions of parameters in video memory all the time.

FreeToken decides which parts of the model to keep in VRAM, which to temporarily move from RAM to GPU, and which to compute on the CPU.

During the input context processing stage, the system preloads the experts of the next layer. During generation, it dynamically distributes computations based on the current speed of the CPU, GPU, and PCIe.

There is even a separate Agent State Cache.

When an agent calls tools, changes context, or continues an old task, FreeToken can skip much of the reprocessing of context.

FreeToken comes with a full graphical interface.

There is no need to convert models to GGUF or build the project from source code.

Also, FreeToken Desktop already has agent bindings built-in—you select a model, choose an application, and run it.