Synthetic Data: Weather as a Parameter for Paired Testing

You cannot film an obstacle on the track in snow, at dusk or in haze: the event is rare, dangerous and cannot be scheduled. In a full CGI scene, weather, light and visibility are render parameters. For our film "We Didn't Wait for Winter" we rendered one Blender railway scene along one camera path in four conditions; the orange boxes are ground truth written by the renderer as it draws each frame, not detections. That enables a paired test: geometry and labels stay fixed and only the condition changes, so a detector's metric difference between two conditions is the effect of the rendered condition.

In short

• An obstacle on the track, let alone in snow or at dusk, cannot be collected in the field; in full CGI the condition is a parameter and you choose the distribution.
• Labels come from geometry at render time, but you write the rules for occlusion, truncation, minimum size and objects hidden by haze.
• Randomize conditions for training, change one at a time for testing: a paired test isolates the condition's effect.
• A synthetic test score does not prove field performance; decide on held-out real images.

Why is the data you cannot film always missing?

An obstacle on the track is the data a rail perception system needs most and has least, hence the shortage of positive samples and the heavy class imbalance. The rock, animal and box in our film Training Data You Cannot Film were rendered for this reason.

The condition multiplies the problem. The few obstacle recordings in an archive show whatever weather that day had; "an obstacle on the track, in snow, at dusk" is the intersection of two rare events and practically never appears in field data. With synthetic data you choose the class and condition mix yourself, which is also a claim: you have to show that it matches the field.

Our compositing pipeline, which places obstacles on real cab-view footage, is described in a separate write-up (in Turkish).

How are labels written at render time?

The renderer knows each object's 3D position, scale and class. The film's scene has 12 object classes, with boxes exported automatically at render time (3840×2160, 30 fps). A box has two possible sources that measure different things. Projection maps the object's vertices onto the image plane; the box covers the whole object, including parts hidden behind a pole (amodal). An instance mask from the Object Index pass in Cycles or from Cryptomatte gives a box around the visible part only (modal). Match your real test set's annotation convention, or the difference will look like a model error in IoU.

Either way, write down four rules: the visible fraction below which an object leaves the labels; whether a box that leaves the frame is clipped or dropped; the minimum size in pixels; and what happens to an obstacle that is geometrically there but invisible in dense haze. Label that one positive and you ask the detector to see the invisible.

The snippet below is not our production pipeline and ignores occlusion; it was run on a Blender 5.2 test scene and checked by drawing the boxes over the render.

import bpy
from bpy_extras.object_utils import world_to_camera_view

MIN_PX, MAX_TRUNC, TRAIN_SIDE = 12, 0.5, 640   # example values

def yolo_lines(scene, cam, objects, depsgraph):
    r = scene.render
    W = r.resolution_x * r.resolution_percentage / 100
    H = r.resolution_y * r.resolution_percentage / 100
    k = TRAIN_SIDE / max(W, H)                  # render px -> training px
    out = []
    for obj in objects:
        cls = obj.get("class_id")
        if cls is None:
            continue
        ev = obj.evaluated_get(depsgraph)        # animation, modifiers applied
        pts = [world_to_camera_view(scene, cam, ev.matrix_world @ v.co)
               for v in ev.to_mesh().vertices]
        ev.to_mesh_clear()
        if not pts or min(p.z for p in pts) <= 0:
            continue
        x0, x1 = min(p.x for p in pts), max(p.x for p in pts)
        y0, y1 = min(p.y for p in pts), max(p.y for p in pts)
        cx0, cx1, cy0, cy1 = max(x0, 0), min(x1, 1), max(y0, 0), min(y1, 1)
        if cx1 <= cx0 or cy1 <= cy0:
            continue
        trunc = 1 - (cx1 - cx0) * (cy1 - cy0) / max((x1 - x0) * (y1 - y0), 1e-12)
        if trunc > MAX_TRUNC or min((cx1 - cx0) * W, (cy1 - cy0) * H) * k < MIN_PX:
            continue
        # Blender: (0, 0) is bottom left; YOLO: y grows downwards
        out.append(f"{cls} {(cx0 + cx1) / 2:.6f} {1 - (cy0 + cy1) / 2:.6f} "
                   f"{cx1 - cx0:.6f} {cy1 - cy0:.6f}")
    return out

scene = bpy.context.scene
for f in range(scene.frame_start, scene.frame_end + 1):
    scene.frame_set(f)
    # Geometry Nodes instances: iterate depsgraph.object_instances
    lines = yolo_lines(scene, scene.camera, bpy.data.collections["Obstacles"].all_objects,
                       bpy.context.evaluated_depsgraph_get())
    with open(f"labels/{f:06d}.txt", "w") as fh:
        fh.write("\n".join(lines))

Worked example: Assume Blender's default camera (50 mm focal length, 36 mm sensor width), a 3840 px wide image, an ideal pinhole and a 40 cm tall object. The focal length in pixels is 50/36 × 3840 ≈ 5,333 px, so at 100 m the object is 5,333 × 0.40 / 100 ≈ 21 px tall. Scaled to a 640 px long side for training (the Ultralytics default imgsz), the scale is 1/6 and the object shrinks to about 3.6 px. This is geometry, not a measurement: set the minimum pixel threshold at the training input size.

How does a condition become a parameter?

Light, sun angle, clouds, haze, snow cover, precipitation and exposure are all scene settings: sun_elevation on the Sky Texture node, the Density of a Principled Volume on the World output, exposure in color management. Light, clouds and exposure leave the box unchanged; snow can partly bury a low object and precipitation can cover its front, changing the visible box.

Sample these parameters for training; sweep them for testing. The training idea is domain randomization: Tobin et al. (2017) randomize rendering so the real world looks to the model like one more variation, and report that an object localizer trained only in simulation with non-realistic random textures reached 1.5 cm accuracy in the real world. At the other end is photorealism: Hodaň et al. (2019) report that physically based rendering of full scenes raised mAP@.75IoU by 24% (Rutgers APC) and 11% (LineMod-Occluded) in absolute terms over objects rendered on random photographs. Prakash et al. (2019) sit in between, placing objects with distributions from the scene context; for a railway scene already structured by rails, ballast and catenary, that is the natural choice.

Haze can have a physical unit too. Sakaridis et al. (2018) model fog with the attenuation coefficient β: transmission t = exp(−β·ℓ) (ℓ: distance to the camera) and visibility (MOR) = 2.996/β, so β = 0.01 m⁻¹ means about 300 m. Do not assume what a Blender volume density corresponds to: place black and white targets at known distances and derive β from the contrast loss in linear (EXR) output. Worked example: with β = 0.01 m⁻¹, transmission is exp(−1.5) ≈ 0.22 at 150 m and exp(−3) ≈ 0.05 at 300 m; labeling an obstacle further away as positive may ask for something physics does not allow.

Paired testing: when only the condition changes

Compare a "haze" subset from the field with a "clear" one and the route, distance distribution, objects and camera have changed too; the causes cannot be separated. In We Didn't Wait for Winter the same scene, camera path and labels are rendered four times; only the condition changes.

The closest examples are corruption benchmarks. ImageNet-C applies 15 corruption types at five severity levels to the same validation images (Hendrycks and Dietterich, 2019); Michaelis et al. (2019) did the same for object detection and report that standard detectors drop to 30–60% of their original performance. These corruptions act in the image plane (ImageNet-C generates fog with the diamond-square algorithm); in a render, haze thickens with distance and snow lands on the geometry.

The measurement plan is below, with no numbers in the cells. Two of the film's four conditions are winter, so some pairs change more than one thing; the table therefore adds a reference condition that is not in the film.

For interactions, use a full factorial design: snow (yes/no) × light (day/dusk) × haze (yes/no) is eight renders of the same camera path. Rules that keep the test honest:

• Keep the label fixed; store what depends on the condition next to it. Distance, visible fraction and transmission go into a side file; objects beyond the MOR are marked "ignore". KITTI does the same: detections in "don't care" areas or below the minimum size do not count as false positives.
• Count object passes, not frames. A 24-second path at 30 fps gives 720 frames, not 720 independent observations. Per object, compare across conditions the distance at which detection becomes consistent.
• Do not train with a test condition's generator. Michaelis et al. keep test corruptions out of training; train and test with the same haze model and you measure fit to your renderer.

What does success on a synthetic test not prove?

A high synthetic test score shows that the model fits your renderer: training and test data from one renderer share texture and noise. The decision comes from a held-out set of real images: split by route, day and camera, kept out of tuning, with empty-track frames for false alarms. Report the few real positives with confidence intervals.

Raise the synthetic share step by step and test every model on the same real set, plus a variant with synthetic pre-training and real fine-tuning. Both beat real-only training in the literature: fine-tuning on real data in Tremblay et al. (2018), combining synthetic and real data in Prakash et al. If the curve falls as the synthetic share grows, the model is most likely learning the renderer. With even a few real dusk or haze frames, check that the synthetic test ranks the conditions as the real slices do; if not, your condition model has drifted from the field. The pixel threshold also depends on the target device's input resolution; we cover that chain on our edge AI pipeline page.

Full CGI, compositing or generative models: which one when?

Our film Auto-Labeled Synthetic Obstacles on Real Railway Footage is an example of compositing: world-locked objects scaled against the rail gauge, so a 40 cm object stays 40 cm at any distance; that run produced 960 frames, 8 classes and 3,145 annotations in YOLO format. Here the condition comes from the recording; adding haze to a sunny recording is a depth-based image operation, as in Foggy Cityscapes. The full pipeline is in the railway case study.

Where it goes wrong

1. Shortcut cues. Geirhos et al. (2020) define shortcuts as decision rules that perform well on standard benchmarks but fail to transfer to more challenging test conditions. In CGI the shortcut can be the render itself: denoiser traces, edges that are too clean, obstacles with more texture detail than the ground.
2. The same object model in every scene. With one 3D model per class, the detector memorizes that model instead of learning the class. Keep test assets that never appear in training.
3. Labels that ignore visibility. Counting a fully occluded object, or one lost in haze, as positive teaches hallucination and punishes the detector unfairly.
4. Test set leakage. If frames from one camera path are in both training and test, the test measures memory. Split by scene, path and asset.
5. A condition mix that does not match the field. A quarter per condition is not the field distribution, and one average mAP hides the mix. Report per condition.

What this article cannot tell you

• We have not yet measured how a detector trained on this data behaves when conditions change, or in the field; the film's description names this as the next step. The table is a plan.
• The scene is an internal Spikedge R&D scene, not customer work and not a deployed system.
• We have not compared in numbers how close the rendered snow and haze are to the real thing.
• The cited results come from other tasks, such as robotic grasping and car detection on KITTI, and do not carry over to railway obstacles as they are.

Frequently asked questions

Does synthetic data replace real data?

No. Synthetic data fills in the rare and dangerous cases the field will not give you; whether the model works is decided on held-out real images.

Why are the labels not drawn by hand?

The renderer knows every object's 3D position, scale and class, so the box is computed from geometry as the frame is drawn and the same object gets the same label in every condition. You still write the rules for occlusion, truncation and invisible objects.

What do you gain by rendering the same scene in four conditions?

A paired test: geometry, camera path and labels are fixed, so the metric difference between two conditions is the effect of the rendered condition. Field data from different days and places cannot separate that.

Full CGI or compositing on real footage?

Full CGI suits controlled condition tests and scenarios you cannot stage; compositing suits multiplying rare objects on real backgrounds. The two are often combined, and both are tested on real data.

Sources

• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World: Tobin et al., IROS 2017.
• Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization: Tremblay et al., 2018.
• Structured Domain Randomization: Prakash et al., ICRA 2019.
• Photorealistic Image Synthesis for Object Instance Detection: Hodaň et al., ICIP 2019.
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations: Hendrycks and Dietterich, ICLR 2019.
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming: Michaelis et al., 2019.
• Semantic Foggy Scene Understanding with Synthetic Data: Sakaridis et al., IJCV 2018.
• Shortcut Learning in Deep Neural Networks: Geirhos et al., 2020.
• KITTI Object Detection Evaluation: the "don't care" rule.
• Blender API: bpy_extras.object_utils: world_to_camera_view.
• Blender API: Camera: default focal length and sensor.
• Blender Manual: Render Passes: Object Index, Cryptomatte.
• Ultralytics: Object Detection Datasets: YOLO label format.
• Ultralytics: Model Training: default imgsz.

Accessed 26 September 2026.