Microsoft has released BitNet, an official inference framework for 1-bit large language models (LLMs), including the BitNet b1.58 model. BitNet supports fast and lossless inference on CPUs and GPUs, with neural processing unit (NPU) support planned. The framework enables running large models efficiently on local devices, demonstrated by its ability to run a 100-billion-parameter BitNet b1.58 model on a single CPU at speeds comparable to human reading.
BitNet achieves significant performance improvements, delivering speedups ranging from 1.37x to 5.07x on ARM CPUs and 2.37x to 6.17x on x86 CPUs compared to full-precision models. It also reduces energy consumption by 55.4% to 70.0% on ARM and 71.9% to 82.2% on x86 processors. The BitNet-b1.58-2B-4T model, with 2.4 billion parameters trained on 4 trillion tokens, offers competitive performance while enabling faster and more energy-efficient inference. Microsoft provides an online demo and open-source code for users to test and deploy the framework.
The release of BitNet addresses growing demand for efficient AI model deployment on edge devices and local hardware, reducing reliance on cloud computing. Its ternary 1.58-bit precision models maintain accuracy comparable to full-precision counterparts but with lower computational cost. This positions BitNet alongside other frameworks aiming to optimize LLM inference, such as Open WebUI, which supports provider-agnostic AI interfaces. BitNet’s improvements in speed and energy efficiency could influence broader adoption of LLMs in resource-constrained environments.
The BitNet framework and its models are available on GitHub, with detailed technical documentation and an online demo accessible for testing. The project’s open-source status allows developers to build and run BitNet on various CPU and GPU platforms, with NPU support expected in future updates.