Synthetic Data Generation & Computer Vision
R&D · Bench dataSynthetic Training Data for Railway Obstacle Detection
Real cab-view video · ego-path · world-locked placement · polygon labels · human review
Python · OpenCV · TEP-Net · SDXL · BiRefNet · GroundingDINO + SAM · YOLO-Seg
Last updated:
01Problem
Problem: Obstacle Imagery on the Track Can't Be Collected in the Field
Track obstacles are too rare and dangerous to film, so we generate labelled synthetic training data for railway obstacle detection on real cab-view video. A detector that has to recognise a rock, a branch or an animal on the rails needs many labelled frames of exactly that. These events are rare and hazardous, and nobody can put an obstacle in front of a train to collect data. The result is a shortage of positive examples and a skewed class distribution.
- 01Real obstacle imagery barely exists, so the model never sees the critical event during training
- 02Frame-by-frame manual labelling is expensive and error-prone across thousands of frames
- 03An obstacle pasted onto the screen moves with the train and never grows naturally; the model learns motion that does not exist
- 04A track region found with classical image processing can slide onto the adjacent track or a parallel road; obstacles land in the wrong place and the whole dataset is wrong
- 05If synthetic images leak into validation or test sets, measured model performance is inflated
02Pipeline Context and Key Terms
03Why the Simple Approaches Failed
- 01Screen-space anchoring: an obstacle placed relative to the screen (for example "at 60% of the visible rail") never approaches. It was the single root cause behind three recurring defects in earlier attempts: the obstacle drifts, does not grow naturally, and travels with the train.
- 02Texture-based track mask: it could not tell the rails from the adjacent track and a parallel gravel road, and landed in the wrong place in 62.6% of the test clip's frames.
- 03Box labels: a segmentation model needs polygons; box labels did not pass our training-set builder.
- 04A single visibility threshold: counting the part outside the frame as "not visible" left the obstacle unlabelled exactly when the train was closest to it.
- 05An open loop: the model reported which class it was weak on, but that report did not drive generation.
04How the Pipeline Works
Letterbox bars are cropped, the ego-path is extracted per frame with TEP-Net, and each clip is stored with a portable 720p proxy
→ The corridor where obstacles go is known for every frame; a clip folder opens on any machine
A label-free quality check: the ego track must leave the frame bottom-centre, so the corridor centre in the bottom band has to stay within 0.35–0.65; a clip with more than 5% of its frames outside that band is rejected
→ Masks sitting on the adjacent track never enter the data. On 1,672 frames of the same clip: classical CV 62.6% out-of-band frames, ego-path 0%
Trackable ground points are followed with KLT; distance travelled is measured through the inverse distance to the horizon line (1/q)
→ No focal length, camera height or train speed needed; works on uncalibrated video
An obstacle is defined by the frame at which the train reaches it; size and lateral offset are given in gauge units, and colour harmonisation is computed once at placement
→ The obstacle stays fixed in the scene and grows as the train approaches; 1/height vs. time R² of 0.985–0.996 across 8 obstacles
Labels are YOLO-Seg polygons taken from the obstacle's visible alpha; parts hidden by a nearer obstacle are excluded, and leaving the frame no longer drops the label
→ On the same validation scene, obstacle instances left unlabelled while leaving the frame fell from 96 to 11. The rest are the last one or two frames before an obstacle leaves the frame entirely, with a visible part under 10 px
When the model is weak on a class or distance, the policy proposes a scenario (on track, far and small at 10–40 px, trackside) and generation runs class-balanced
→ Synthetic data targets a measured weakness instead of being generated at random
Faulty images are rejected on review sheets; only approved ones move into training. Synthetic images cannot enter validation or test sets; above 25% synthetic the builder warns, above 33% the UI will not start training
→ Model scores are not inflated by synthetic data; every image's source and reviewer are on record
05Tried Approach → Chosen Approach
Synthetic data generation with automatic labelling
This video shows a second generation path for railway obstacle detection: the entire scene is generated in Blender. It is separate from the real cab-view video path described on this page. Bounding boxes for the objects placed in the scene are not drawn by hand, they are extracted automatically during rendering. The boxes in this video are that automatic output.
The footage is synthetic; it is not a recording from a field inspection. The purpose is to produce a labelled training set for scenarios where collecting real data is expensive and hazardous.
06What We Can Do With This Pipeline
For a railway project with forward-facing camera footage, we can generate the rare-scenario data an obstacle detector needs, label your real data, and set up a real test set that makes the contribution of synthetic data measurable. The capabilities below are built and running in this work; in a client project, scope, classes and footage are defined per project.
- 01Footage usability analysis: we extract the track region frame by frame from your cab-view video and report, with a label-free test, which clips are suitable for obstacle generation
- 02Class-balanced synthetic training data with YOLO-Seg polygon labels in the obstacle classes you need: on-track, far-and-small (10–40 px) and trackside scenarios
- 03A real-data labelling and active-learning loop: GroundingDINO + SAM pre-labels, human review, YOLO-Seg training and per-class iteration reports
- 04Sim-to-real evaluation design: real test clips held out from synthetic generation, synthetic-ratio limits and a traceable approval record
- 05A new obstacle class needs no field campaign: it is added with a new asset set generated on GPU, while compositing runs on CPU
- 06The same labelling and active-learning stack is used for airspace (bird / drone) datasets; moving a trained detector onto edge hardware is the subject of our Edge AI pipeline capability
To evaluate your Synthetic Data Generation & Computer Vision needs on your own platform, schedule an embedded architecture audit or scope your platform class with the system requirements calculator.
Frequently Asked Questions
07Frequently Asked Questions
Why does railway obstacle detection need synthetic data?
Because obstacles on the track are rare and dangerous, and you cannot collect enough labelled, class-balanced real examples in the field to train a model. Synthetic data fills that gap: it produces frames with obstacles of the class, distance and position on the track you need. It does not replace real data, and its contribution has to be measured on a real test set.
What is world-locked placement and why does it matter?
World-locked placement means the synthetic obstacle is tied to the scene, not to the screen. An obstacle placed at a fixed share of the visible rail neither stays put nor grows as the train approaches, so the model learns motion that does not exist. We track the obstacle's depth with a single variable: it stays fixed in the scene, grows as the train closes in and leaves the frame at the bottom. In validation, 1/height vs. time had an R² of 0.985–0.996 across 8 obstacles.
Do you need camera calibration?
No. Track gauge in the image changes linearly with the image row, so the horizon and scale come straight from the rails, and distance travelled comes from ground points tracked with KLT. Focal length, camera height and train speed cancel out of the computation, so it works on cab-view video without calibration data. The computation assumes a flat ground plane; accuracy drops on sharp gradient changes.
How are the labels produced? Is anything drawn by hand?
Nothing is drawn by hand: the label comes out of the geometry as a YOLO-Seg polygon the moment the obstacle is placed. The polygon follows the visible part of the obstacle, so parts hidden by a nearer obstacle are excluded and leaving the frame does not drop the label. Human effort goes into review, not drawing: faulty images are rejected on review sheets and the reviewer is recorded.
Can synthetic data inflate model scores?
It can if the setup is wrong, which is why three rules are enforced in code. Synthetic images go into the training set only, never into validation or test sets. Clips held out as the real test set are never used for generation. Above 25% synthetic data the builder warns, and above 33% the UI will not start training.
Which obstacle classes exist, and can new ones be added?
This work covers 8 classes: rock, fallen branch, bicycle/scooter, metal debris, wood plank, plastic waste, tool and animal, with 657 images in the library. A new class needs no field campaign: images are generated with SDXL, cut out with BiRefNet and added to the library once they pass the automatic quality checks. That step needs a GPU; compositing and labelling run on CPU.
How well does it work in the field?
We have not measured that yet, and we do not publish a number before we do. What we have measured so far is the correctness of the pipeline: the track region lands on the right track, the obstacle stays world-locked, and the label follows the visible part. The contribution of synthetic data to detection will be measured on real test clips that are held out from generation and labelled by hand.
Films on this topic
Methodology, Measurement Conditions and Limits
- This is an R&D / capability case, not a delivered client project. Every figure on this page was measured by Spikedge under the conditions below.
- Track-region comparison: 1,672 frames of a cab-view clip containing an adjacent track and a parallel gravel road (3600×2338, H.264). Metric: share of frames where the horizontal centre of the corridor's bottom 10% band falls outside 0.35–0.65. August 2026; repeated on 26 September 2026 with the same result.
- World locking: a 960-frame, 8-obstacle validation scene on a second clip. R² of a linear fit of 1/height against frame index, over frames where the obstacle is at least 10 px tall. For an object approached at constant speed this relation must be linear. Because the obstacle's size is computed from the measured ego-motion, this is an internal consistency check: it shows the obstacle grows the way a stationary object would; it is not a position comparison against a real object.
- Obstacles leaving the frame: on the same scene we counted obstacle instances (obstacle × frame) that were at least 10 px tall but carried no label: 96 under the old rule, 11 under the new one. Every remaining instance is one of the last one or two frames before an obstacle leaves the frame entirely; its visible part is under 10 px, so it is deliberately left unlabelled.
- Timing: 60 labelled images, 10 scenes; Apple M5 Pro (24 GB), CPU only, ego-motion pre-computed; wall-clock time. The 4K source and the 720p proxy were compared on the same request; ego-motion measured from the two differed by at most 0.9%.
- Asset count: a count of the library folder of assets that passed automatic quality checks; 657 images, 57–109 per class (26 September 2026).
- The contribution of synthetic data to detection performance in the field has not been measured yet. Real test clips are held out from generation and labelled for that purpose; no accuracy claim is published before this is done.
- Limits: the ego-path model cannot extract a corridor without visible rails (track fully under snow, inside tunnels); such clips are rejected by the quality check. The ego-motion model assumes a flat ground plane; sharp gradient changes break the linearity. Night and thermal cameras are outside the scope of this work.
- Open models used: TEP-Net for the ego-path (Railenium; code MIT, weights CC BY-NC-SA 4.0, commercial use requires retraining under a suitable licence), SDXL and BiRefNet for obstacle images. The fully synthetic Blender scene is a separate generation path.
Sources
- 1.Laurent, T. (2024). Train Ego-Path Detection on Railway Tracks Using End-to-End Deep Learning. arXiv:2403.13094
- 2.TEP-Net code (MIT) and ego-path weights (CC BY-NC-SA 4.0)
- 3.Zendel et al. (2019). RailSem19: A Dataset for Semantic Rail Scene Understanding. CVPR Workshops
- 4.Podell et al. (2023). SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. arXiv:2307.01952
- 5.Zheng et al. (2024). Bilateral Reference for High-Resolution Dichotomous Image Segmentation (BiRefNet). arXiv:2401.03407
Do you have cab-view footage, and is your obstacle detector short of data for rare scenarios?
In a free technical assessment, let's review your cab-view footage, your target obstacle classes and the scenarios where your model falls short.
* R&D / capability case. Figures were measured by Spikedge on the stated clips and hardware; they are not field-validated detection accuracy. The contribution of synthetic data on a real test set has not been measured yet.