# Why a vision model that works in the lab fails in the field

> A vision model learns the pictures it was trained on. New light, cameras and places shift the data. How to test for it, and what to ask for first.

Page: https://www.whitehatdesignstudio.com/guides/vision-model-lab-to-field/
Published by: WhiteHat Design Studio
Last updated: 7 November 2026

The short answer

A vision model learns from the pictures it was trained on, so its accuracy falls when the light, the camera, the angle or the place changes. Researchers call this distribution shift. The answer is to collect pictures from where the model will run, test on those, and give every accuracy figure with the conditions it was measured in.

By the engineers at WhiteHat Design Studio. Published 7 November 2026.

![HARP's head on a bench beside a laptop that shows the camera view used for face tracking.](https://www.whitehatdesignstudio.com/_astro/harp-face-tracking.DaZu4jSG_ZM3Nr3.jpg)

HARP's head and camera on the bench, during face tracking development.

## What changes between the lab and the floor

| What changes | What the model sees |
| --- | --- |
| Light | Sun through a window, a flickering tube, a dark corner |
| The camera | A different sensor, lens or resolution, a dirty lens |
| The angle | Objects seen from above instead of from the side |
| The background | A busy floor instead of a plain bench |
| The objects | New models, new packaging, wear and dirt |
| Movement | Blur from a moving robot or a moving part |

Each of these changes what the pictures look like. A person barely notices. A model can.

## What the research says

This is not a rare fault. The WILDS benchmark, from researchers at Stanford and elsewhere, gathers real cases, from camera traps for wildlife to scans from different hospitals, and finds that distribution shifts can substantially degrade the accuracy of machine learning systems deployed in the wild. Its authors report that standard training gives substantially lower accuracy on data from outside the training conditions than on data from inside them.

A second benchmark, by Hendrycks and Dietterich, measures how models cope with everyday corruptions of an image, such as noise, blur and changes in brightness, rather than deliberate attacks. These are the conditions a camera on a factory floor or a robot meets every day.

## How to test a vision model properly

1. **Collect from the real place.** Pictures from the cameras, the light and the angles the model will really have, including the worst hour of the worst day.
2. **Hold out by place and time, not at random.** Test on a site or a week the model never saw in training. A random split hides the shift you are trying to measure.
3. **Measure on the real hardware.** The speed and accuracy on the device that will run it, not on a desktop.
4. **Report by condition.** Accuracy in good light and in bad light, near and far, not one average that hides both.
5. **Keep the failures.** Every picture it got wrong goes into the next round of training and testing.

## How we work

We do not ship a model that has only seen its training set. It is tested on data from where it will run. And we do not quote a model’s accuracy without the data and conditions it was measured in.

Our own systems show why. HARP finds faces with the camera in its head and turns to follow the person in front of it. ReFlex chooses a pinch or tripod grip from what the camera in its palm sees, and we do not quote its grip accuracy. Auto Vendi runs YOLOv8 detection alongside its mapping as it drives a building, and SKYLOCK finds drones against a changing sky. Each one meets light and backgrounds no training set fully covers.

## What to ask before you buy a vision system

- Which data was it trained on, and which data was it tested on?
- Was the test data from a place or a time the model never saw in training?
- What is the accuracy in each condition, not on average?
- What does it get wrong, and how often?
- Do you receive the model, the training code, the data and the test results at the end?

The [handover checklist](https://www.whitehatdesignstudio.com/handover-checklist/) lists what to ask for when an AI supplier finishes. [Our AI practice](https://www.whitehatdesignstudio.com/services/ai/) builds vision that runs where the work is.

## Where the answer comes from

- [SKYLOCK](https://www.whitehatdesignstudio.com/work/skylock/) Finds drones in live camera feeds, predicts where each one is going and keeps a gimbal camera on it, from the operator's console or on its own.
- [HARP](https://www.whitehatdesignstudio.com/work/harp/) Navigates a mapped building, tracks faces and holds a conversation with an onboard language model, built around a Raspberry Pi 5.
- [Auto Vendi](https://www.whitehatdesignstudio.com/work/auto-vendi/) Maps its route, avoids obstacles and delivers in a locked compartment that the phone app releases.
- [ReFlex](https://www.whitehatdesignstudio.com/work/reflex/) Reads forearm muscle signals, chooses a pinch or tripod grip from what the palm camera sees, and stops each finger on encoder feedback.

Practices

- [AI](https://www.whitehatdesignstudio.com/services/ai/) Intelligence that runs where the work is.

C AI

## Questions people ask next

### What is distribution shift?

It is the difference between the data a model was trained on and the data it meets in use. New light, a new camera or a new place each shift the data, and accuracy usually falls with it.

### How much data do we need from our own site?

Enough to cover the conditions you will really have, including the bad ones: night, glare, dirt on the lens, the busiest hour. The number depends on the task, so we agree it with you before we build.

### Why will you not quote an accuracy figure up front?

Because a figure without its data and conditions tells you nothing. We give accuracy with the data and the conditions it was measured in, so your engineer can check it.

## Sources

- [Koh and others, WILDS, a benchmark of in-the-wild distribution shifts, 2020](https://arxiv.org/abs/2012.07421)
- [Hendrycks and Dietterich, Benchmarking neural network robustness to common corruptions and perturbations, ICLR 2019](https://arxiv.org/abs/1903.12261)

## More guides

- [How does drone detection work?](https://www.whitehatdesignstudio.com/guides/how-drone-detection-works/)
- [Edge AI or cloud AI: where should your model run?](https://www.whitehatdesignstudio.com/guides/edge-ai-vs-cloud-ai/)
- [Visual SLAM vs LiDAR SLAM for indoor robots](https://www.whitehatdesignstudio.com/guides/visual-slam-vs-lidar-slam/)

[All the guides](https://www.whitehatdesignstudio.com/guides/)

## Tell us what has to work.

One sentence is enough. An engineer will reply.

[Talk to an engineer](https://www.whitehatdesignstudio.com/contact/) [WhatsApp an engineer](https://wa.me/923112721007?text=Hello%20WhiteHat%20Design%20Studio%2C%20I%20would%20like%20to%20talk%20to%20an%20engineer%20about%20my%20project.)