KTransformers
A research framework that runs very large mixture-of-experts models by sharing the work between a GPU and CPU memory.
KTransformers is a research project on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. Its kt-kernel component supports scheduling mixture-of-experts work across the GPU and the CPU, with expert offload to system memory, which lets very large models run on a single consumer GPU together with plenty of RAM. CPU backends include AMX, AVX512 and AVX2-only options.
The README tracks day-0 support for new open models, including Kimi K2.5, GLM-5, GLM-5.2, GLM-5.3 Flash, MiniMax M3 and DeepSeek V4 Flash. It also has a fine-tuning part that trains mixture-of-experts models with LoRA together with LLaMA-Factory. For serving it integrates with SGLang through the project's own fork. Moonshot's Kimi K2.7 Code model card names KTransformers as a recommended runtime next to vLLM and SGLang.
The license is Apache 2.0. The PyPI packages require Linux on x86-64, and speed depends on memory bandwidth and CPU features, so it is slower than keeping every expert on GPUs. Treat it as a research tool that changes fast: the latest release is v0.7.2.
Good for
- Running giant mixture-of-experts models such as DeepSeek, GLM, Kimi or MiniMax on one GPU with a lot of system RAM
- LoRA fine-tuning of mixture-of-experts models on limited GPU memory
- Homelab and research setups
Watch out for
- The packages require Linux on x86-64
- Experts offloaded to the CPU run slower than experts on GPUs, and speed depends on memory bandwidth and CPU instruction support
- It is a research project with fast-changing releases, so expect rough edges