Tiny Language Model Runs on $10 Microcontroller
A developer has successfully implemented a 28.9-million-parameter language model on a $10 ESP32-S3 microcontroller, achieving 9.88 tokens per second offline. The model, trained on the TinyStories dataset, was scaled down to fit the hardware's 512KB of RAM and 8MB of PSRAM. This breakthrough has significant implications for the development of offline AI applications.
Key points
- A developer, slvDev, has implemented a 28.9-million-parameter language model on a $10 ESP32-S3 microcontroller.
- The model generates text at 9.88 tokens per second offline, using the TinyStories dataset trained by Microsoft Research.
- The model was scaled down to fit the hardware's limited RAM and PSRAM, using quantization to reduce memory requirements.
- The project, esp32-ai, is available on GitHub under an MIT license and has gained significant attention, with over 3,600 stars and 470 forks.
- This breakthrough has significant implications for the development of offline AI applications, particularly in resource-constrained environments.
Tiny Language Model Runs on $10 Microcontroller
A developer has successfully implemented a 28.9-million-parameter language model on a $10 ESP32-S3 microcontroller, achieving 9.88 tokens per second offline. This breakthrough has significant implications for the development of offline AI applications.
The model, trained on the TinyStories dataset by Microsoft Research, was scaled down to fit the hardware's 512KB of RAM and 8MB of PSRAM. This was achieved by using quantization to reduce memory requirements, trading a little accuracy for a 75% reduction in memory usage.
The project, esp32-ai, is available on GitHub under an MIT license and has gained significant attention, with over 3,600 stars and 470 forks. This development demonstrates the potential for offline AI applications in resource-constrained environments, such as sensor nodes and smart plugs.
Sources
The WireByte editorial team synthesises technology news from multiple primary sources, verifies the facts, and links every source. Articles are produced with AI assistance and reviewed under our editorial policy.