Edge AI or cloud AI: where should your model run?

The trade, side by side

What changes	On the device	In the cloud
When the network drops	Keeps working	Stops
Time to answer	No network round trip	Depends on the network
Size of model	Limited by the device	Limited by your budget
Your data	Stays on the device	Leaves the device
Cost	Paid up front, in hardware	Paid with every answer
Updates	Have to reach every device	Change it once

Decide by how it fails

Start from the failure, not the model. Ask what happens when the network drops, and what happens when the answer arrives late.

A hand that has to close on a cup cannot wait for a server. ReFlex chooses its grip on the device: a custom-trained MobileNetV2 runs through TensorFlow Lite on a Raspberry Pi 5, between the camera in the palm and the motors in the fingers. HARP runs its language model on the robot itself, so its conversation does not depend on the building’s Wi-Fi. A weekly report can wait for a server, and a server can run a model too large for any device.

Can a Raspberry Pi 5 run a language model?

Yes, if the model is small enough. A Raspberry Pi 5 has four Arm Cortex-A76 cores at 2.4 GHz and up to 16 GB of memory. llama.cpp, an open-source engine for running language models, supports integer quantisation, which shrinks a model so it uses less memory and runs faster.

For more speed, the Raspberry Pi AI HAT+ 2 adds a Hailo-10H accelerator with 8 GB of its own memory, made for running language models and vision-language models locally. The AI HAT+ boards, with Hailo-8 or Hailo-8L accelerators, are for vision work such as object detection, segmentation and pose estimation.

Those are capabilities, not speeds. How many words a second you get depends on the model, how far it is quantised and how well the board is cooled. Measure it on your hardware, with your prompts, before you promise anyone a response time.

Fitting a model to a device

Four things have to fit at once, and the model is only one of them:

• Memory. The model, its working data and everything else the device runs.
• Speed. Fast enough for the job, measured on the real device, not on a desktop.
• Power. A robot on a battery pays for every calculation in running time.
• Heat. A small board that overheats slows itself down, and the speed you measured on the bench is gone.

Tools exist for each step. TensorFlow Lite now continues as LiteRT, Google’s runtime for running models on devices. NVIDIA’s TensorRT compiles models for low latency on NVIDIA hardware, and SKYLOCK runs its detection model through it at its ground control station.

Where your data goes

We agree that before we build. A language model can run with a provider or on your own hardware, and the choice changes what you have to tell your customers and your regulators. If the data must not leave the building, that settles the question before anything else does.

If you are weighing this for a machine of your own, our AI practice does this work, and the HARP case study shows a model running on a robot.