For years, on-device machine learning meant shipping a compromise: models too big, too slow, or too dumb to be worth the battery. That calculus flipped in the last two hardware and tooling cycles, and the models that now run in your pocket are good enough that sending data to the cloud is often the wrong default.
What Actually Changed
Three things converged. Neural accelerators went mainstream: Apple's Neural Engine now does tens of trillions of operations per second, and Android's NNAPI and dedicated NPUs give the same story on the other side. The hardware to run real models is now in the mid-range phone, not just the flagship.
At the same time, the models got smaller without getting proportionally worse. Quantization from 32-bit floats to 8-bit or even 4-bit integers cuts model size roughly 4x and speeds up inference dramatically, often with under a percentage point of accuracy loss. Combined with distillation and architectures designed for mobile like MobileNet and EfficientNet, a model that needed a server three years ago now fits in an app bundle.
The Cloud Tradeoff, Honestly
On-device wins on latency, privacy, cost, and offline capability. There is no network round trip, so inference is bounded by silicon rather than by the user's flaky LTE. The data never leaves the phone, which turns a compliance headache into a non-issue. And you pay for the compute exactly once, at build time, instead of per inference for the life of the product.
The cloud still wins when the model is genuinely large, when you need to update it constantly, or when the task demands frontier capability that will not fit in a few hundred megabytes. A 70-billion-parameter language model is not running on a phone this year. The right architecture is often hybrid: run the common, latency-sensitive, privacy-sensitive path on-device and escalate the rare, hard cases to the cloud.
A Worked Example
Take a nutrition app that identifies food from a photo. The naive design uploads every image to a vision API, which means a spinner, a data cost, and a privacy prompt every time someone photographs their lunch. The on-device design ships a quantized image classifier converted to Core ML with coremltools, and the whole interaction, camera to prediction, completes in under two seconds with the phone in airplane mode.
Concretely: take an EfficientNet-Lite backbone fine-tuned on a food dataset, quantize it to 8-bit, and it lands around 5 to 10 megabytes. On a recent iPhone the Neural Engine runs a single inference in tens of milliseconds; the two-second budget is mostly camera and UI, not the model. The user gets an instant estimate offline, and the app only calls the cloud for the ambiguous cases where on-device confidence is low.
The Toolchain
On iOS the path is Core ML, with coremltools to convert from PyTorch or TensorFlow and the option to compile the model at build time. On Android and cross-platform, TensorFlow Lite (now LiteRT) and ONNX Runtime Mobile cover the same ground, and MediaPipe gives you ready-made pipelines for common vision and audio tasks. For on-device language models, llama.cpp, MLC, and Apple's own foundation models are making small LLMs practical.
The unglamorous work is in the conversion and validation, not the training. Converters drop or approximate unsupported ops, so you must verify that the on-device outputs match your reference within tolerance, and you must profile on real low-end hardware, not just your dev phone. A model that flies on an iPhone 15 Pro can thermally throttle on a three-year-old Android device, and that device is the one your median user is holding.
When To Reach For It
Reach for on-device first when the task is frequent, latency-sensitive, or privacy-sensitive: real-time camera effects, keyboard prediction, health data, offline translation, anything a user does dozens of times a day. These are exactly the cases where a network round trip is both a UX tax and a cost you pay forever.
Stay in the cloud when the model is large, changes weekly, or is your core defensible IP that you do not want extracted from an app bundle. Everything in between is a genuine engineering decision with real tradeoffs, which is the whole point: on-device AI is finally a serious option on the table rather than a science project.
The takeaway
The default is no longer “call the API”; for frequent, private, latency-sensitive tasks, the model belongs on the device.
