Making LLMs run on small and old devices
Archived — this story has rotated out of today’s deck. It is kept here in full.
The gist
Open source project smolmaxxing ran models up to 16B parameters on a 2013 iPad Air and a Xiaomi TV box.
It shows large language models can run on hardware with about 1 GB of usable memory.
Background
smolmaxxing is an open source GitHub project by developer hyperionaidev, posted to Hacker News by user mrashad74, who said he started it at his AI company. It aims to find the biggest language model each device can really run, with every claim measured on the hardware itself. The gains come from custom NEON (an ARM chip instruction set) and lookup-table kernels, a flash-based expert pager and weight streaming; above 2B parameters, flash bandwidth (278 MB/s) rather than the CPU becomes the bottleneck. The repo includes rebuildable model recipes, llama.cpp patches and 160 documented experiments.
How it unfolded
- Nov 12, 2026The project runs quantized LLMs from 260K to 16B parameters on a Xiaomi TV Box S (1.9 GB RAM, about 1 GB usable) and a 13-year-old jailbroken iPad Air; a 2B model hits 7.7 tok/s on the TV box and a 16B mixture-of-experts model reaches 5.1 tok/s.
- Nov 12, 2026On the first-generation iPad Air (2013, 1 GB RAM, CPU only), a 4B model runs at about 10 tok/s, an 8B mixture of experts at about 3.5 tok/s, and a 16B one at 2.6-4.5 tok/s with experts paged from flash.
- Nov 12, 2026A 1-bit lookup-table kernel ported to base NEON took a 1.7B model from 2.75 tok/s on stock llama.cpp to roughly 11 tok/s, and up to 23.7-25.8 tok/s after further fixes, an 8.6x speedup on the Apple A7 chip.
- Nov 12, 2026The project was posted to Hacker News by mrashad74, who said it started at his AI company and that modified models are on Hugging Face; the repository lists 33 commits and 16 stars.
Key terms
- LLMs
- A machine learning model trained on huge amounts of text to generate, summarize, translate, and analyze language.
- kernels
- Core program of an operating system that controls all hardware and system resources and mediates between processes.
- quantized
- A family of neural network architectures based on multi-head attention that processes tokens with positional information.
Source: Wikipedia
Who’s saying what
- Author
- mrashad74 said he is open sourcing the kernels and results for others to reproduce, with modified models on Hugging Face, and asked for a more challenging device.
- Community
- A daily.dev community take found the project broadly praised as a fun, technically impressive feat, with tension over practical usefulness versus novelty; 55% positive, 35% mixed, 10% skeptical.
Still unverified
The daily.dev community take notes some suspicion in the thread about AI-assisted replies.