Our First Public Datasets, and a Model Trained Only on Game Engines That Beats the Baselines on Real-World Depth

originlab/game-depth: 48,600 RGB frames at 1920 pixels wide with raw, uncompressed depth measured inside the engine, from 10 licensed titles. Split into train, test, and an extended set.

originlab/game-recordings-v3: full multimodal sessions, video with frame-synced telemetry, inputs, and events, growing toward 150 hours as we add samples.

originlab/frame-synced-multiplayer: 16 matches, 108 player-hours, up to eight players each on a separate machine, synchronized to one frame clock and verified. About 3.8 terabytes, with per-frame cross-player annotations and the verification script.

originlab/lotus-game-depth: one depth model on the Lotus recipe, shipped as two checkpoints. One trained from scratch on game-depth alone, with no real images. One fine-tuned from that on NYU Depth V2.

Playable previews of every modality, side by side and in sync, are at app.originlab.ai/data.

We built a capture system that pulls 10 modalities, including RGB before and after the HUD, depth, normals, camera telemetry, keyboard and mouse inputs, in-engine event labels, audio, and world state, out of commercial game engines. Everything is frame-synced to the video at 1080p and 60 fps. The question we most wanted answered before scaling into the millions of hours was simple: does a model pretrained on this data generalize to the real world?

There are two versions of the question. Can a model trained on engine data alone, with no real photographs in the mixture, produce useful predictions on real scenes? And if you then fine-tune on a small amount of real data, does the engine pretraining help, hurt, or wash out?

The intuition from language modeling is that volume and quality of pretraining data produce representations that transfer, and that fine-tuning on a narrow real distribution is cheaper when the pretraining was broad. We wanted to know whether that holds when the pretraining corpus is rendered rather than photographed, and when the labels are the engine's own computation rather than a sensor's sample or a model's estimate.

In August, researchers at the University of Hong Kong and the Kuaishou Kling team released PlayWorld, which puts agent players inside nine current world models across 171 long-horizon scenarios. Every model was unreliable on long-horizon interactive objectives, particularly spatial consistency and persistent state.[5] Two weeks later Tencent unified its internal AI stack at Gamescom around GIGA, treating games and virtual worlds as research environments for general-purpose agents.[6] Labs without that first-party access need the data from somewhere.

We used the Lotus recipe, a Stable Diffusion 2 based latent diffusion depth model, because it has strong published baselines and because Lotus and Marigold were themselves trained on synthetic data. That makes the comparison clean: same architecture, same recipe, different synthetic source.

Zero-shot to real outdoor scenes — KITTI AbsRel, lower is better: Marigold, pretrained on Hypersim and Virtual KITTI: 0.244. Lotus, pretrained on Hypersim and Virtual KITTI: 0.224. Ours, pretrained on game-depth only: 0.191.

All three models were pretrained on rendered data only. The one pretrained on game-engine capture produces better depth on real driving footage than the two pretrained on the standard synthetic sets, on the same architecture.

Fine-tuned on real indoor scenes — NYU AbsRel: fairly tuned Lotus baseline, with indoor pretraining and roughly four times our pretraining frames: 0.115. Ours, game-pretrained then NYU fine-tuned, with no indoor pretraining and roughly 18,000 pretraining frames: 0.116.

That second result is the one we find most interesting. The game-depth corpus has no living rooms, offices, or interiors at all. A representation learned entirely from outdoor game worlds transferred well enough that a small amount of real indoor data closed the gap completely. The pretraining was doing work that had nothing to do with matching the target domain.

We want to be careful about what that does and does not show. Engine data did not replace real data here. It made a small amount of real data go much further. That is the claim, and it is the one we think matters for anyone with a narrow real-world dataset and a large training budget.

Both checkpoints are on Hugging Face so anyone can reproduce these numbers or try to beat them.

On coverage. Without fine-tuning, the game-pretrained model is stronger outdoors than indoors, which is what a corpus with no interiors would predict. It is still a usable indoor model at that stage, and the gap is coverage rather than method. Coverage is something we can change by capturing different titles, and more indoor and more varied scenes are in the next version.

The multiplayer release is the one we have not seen done before, and it is worth explaining why it is hard.

Multi-view gameplay datasets to date are replay-rendered. One demo file gets re-rendered from several viewpoints, and the views are synchronous because there is only one clock. What that loses is the real thing: the actual client-side frames each player saw, their actual input timing, and per-view depth.

In frame-synced-multiplayer, six to eight players on separate consumer PCs over residential internet each record their own live view of one match, and frame k on every machine is the same server instant. Across all 318 player pairs in the release, the worst slides 0.44 frames over a full hour and most matches hold under a quarter frame. Of eighteen matches captured, sixteen certified within the half-frame gate and ship; the two that missed it were withheld.

For every ordered player pair at every instant, the data records whether one holds the other in view, at what angle, screen position, and distance, plus interval events including mutual sight, first sight, and blindside. All of it is computed deterministically from engine camera pose, with no annotators in the loop.

At each matched hit event, the attacker's camera ray from their own telemetry has to point at the victim's position from the victim's telemetry. Median aim error collapses to 1.4 degrees only at true alignment, across roughly 4,800 matched events per match. The consistent optimum sits 70 to 100 milliseconds before the hit, which is the game's lag-compensation window measured from the data.

Sixteen matches, 108 player-hours, 377,000 engine events, and 73,676 view-crossing intervals. Every player ships the full engine stack at 1080p and 60 fps, and the verification script ships alongside so the synchronization claims can be re-derived rather than trusted.

It opens work in multi-agent world models, opponent-conditioned imitation learning, geometry-derived theory-of-mind labels, collaborative SLAM and 3D reconstruction, and behavior modeling with verified-human aim against synchronized target visibility.

Every single-player session is a continuous recording of two to eight hours from a licensed interactive world; multiplayer matches run about an hour each. All modalities sit on a common frame clock at 60 fps constant frame rate.

Pre-HUD RGB: the rendered world without interface overlays, up to 4K. Post-HUD RGB: the same frame as the player saw it. Depth: raw, uncompressed per-pixel depth. Surface normals: per-pixel orientation. Camera telemetry: position, rotation, field of view, and velocity in world coordinates.

Keyboard and mouse: captured at the engine layer as the game processes them, already aligned. World state: object positions, velocities, collision events, and environment state as synchronized JSON. Audio: separated dialogue, environment, and effects tracks. Action and event labels: model-generated, human-reviewed timelines across more than 200 activity types.

Available on request: per-object motion vectors, full-scene optical flow computed from depth and camera motion, and lighting-independent albedo.

Our capture technology reads directly from the game engine and is patent-pending. The full catalog spans more than fifty licensed titles from twenty-plus publisher partners. Our turnaround from a research request to delivered data is about twenty-four hours.

Every hour is played by a professional human on capture-grade hardware, working from an individualized mission that a planning system builds from that title's mechanics. Sessions are constructed for coverage, including rare events, failure and recovery, and the ordinary middle of play that nobody uploads.

The GameFactory authors showed that undirected gameplay corpora carry severe action imbalance, with the forward key pressed more than a hundred times as often as the backward key in VPT.[3][4] We measure that directly. An analysis pass reports action dominance per title, flags underrepresented actions, and feeds the next round of mission planning.

Quality assurance runs in two cycles, with automated checks during capture covering frame-rate gating, input-sync verification, and redundancy, followed by a human supervisor reviewing and approving every session before it enters the catalog.

Eleven research tracks, each described in full at originlab.ai/research:

1. Depth and normal estimation. Single-step depth and surface-normal prediction on engine ground truth rather than pseudo-labels. The results above are the first output of this track.

2. Segmentation. Instance and semantic masks captured at the render pass, used to train dense segmentation models and condition world models.

3. 4D reconstruction. Dynamic scenes reconstructed from per-frame depth, intrinsics, and measured camera pose, with measured-versus-estimated signal comparisons.

4. World value scoring. Diversity, redundancy, and task-coverage scoring that ranks titles, explains pricing, and directs capture spend.

5. Scene intelligence indexing. Dense scene labels fused with telemetry, input, and audio under a semantic query layer.

6. Automated data quality and AI QA. Real-time and post-hoc signals for frame pacing, duplicates, sync drift, coverage, and fitness for purpose.

7. Synthetic players for capture. Human-like, task-conditioned agents whose capture policies adapt toward whatever the corpus is missing.

8. Generative world building. Large, high-fidelity environments evaluated by downstream training utility rather than visual appeal.

9. World model. Action-conditioned dynamics trained over synchronized RGB, depth, and inputs, scaling from one title to multi-game and multiplayer systems.

10. Open world model benchmark. Ground-truth probes for action fidelity, camera controllability, and physical consistency, with public leaderboards and baselines.

11. Open synthetic data benchmark. Fixed downstream tasks and real held-out evaluations that compare generated corpora against real capture of equal size.

If you train on game-depth and get a different number, tell us. If you run the multiplayer sync verification and it does not hold, tell us. If you find a failure mode we did not report, tell us. If there is a modality or genre that would make the corpus useful for your problem and it is not in the catalog, tell us that too, because we can capture to spec.

The point of building capture this fast was to let researchers iterate on what they need rather than what happens to exist, and a request that comes back as data in a day is the version of this company we are trying to run.

Discord for discussion and questions.

X for updates.

LinkedIn for jobs and longer posts.

For research groups that want more than the public sets, we provide data, compute credits, and engineering support. Reach out with your ideas and requests.

Each dataset carries its license on its Hugging Face card. The multiplayer release uses the Origin Lab Data License with two tracks: internal evaluation, which gives ninety days to train and assess with no obligation to publish, and non-commercial research, which permits papers, open weights, and benchmarks with attribution.

The model checkpoints are gated under CC-BY-NC-4.0 with one added condition: any model derived from them must be released publicly with open weights and a model card. Commercial use requires a separate agreement.

Every title in the corpus is licensed from its rights holder with a documented chain of title, every player recorded consented in writing and was compensated, and nothing was scraped. For a research group that means results can be published without an institutional review problem. For a lab it means a training corpus that is an asset rather than a contingent liability.

A second post from the research team on statistical evaluation of game worlds as training sources: which properties of a title predict downstream value, and how we score them before capture. After that, our first public benchmark and our first playable world model.

Origin Lab is hiring ML researchers and engineers.