Omniverse Remote Rendering
Remote rendering with Kognitwin data in Omniverse.
Lesson
Seeing is believing; but GPUs are expensive; and WebRTC is a pain.
The lesson in full: While the technical challenge is not trivial, the biggest hurdle is the cost of GPU cloud rendering and licensing. However, even with that in mind, to be able to see physically-accurate rendering of your asset with thousands of light sources really gives a different sense of ‘being there’, compared to a typical real-time 3D engine.
What it is
Omniverse for high-fidelity 3D remote-rendering and simulation. This requires deep integration with both third-party services and the internal AI orchestration engine. Our 3D Interaction Agent can control the Omniverse extension in the same way it controls our internal 3D engine, as I created a message bus that translates commands like ‘fly to equipment’, ‘highlight part’, ‘show annotation’, etc. via WebRTC to the Omniverse remote rendering machine. Now with coding agents like Claude and GitHub Copilot, setting things up and adding testing features is very simple. In April 2026 the collaboration with NVIDIA was announced at Hannover Messe. In 2026 the integration moved from one monolithic Omniverse app to the decomposed library stack: a render engine, a scene-state owner, a streaming layer and a physics library, each consumed as plain Python functions. On top of that I built two small apps: a streamer that serves frames to the browser over WebRTC, and an agent-facing reasoner, an MCP server over the render engine, so the same agents that drive the Surface client can set view modes, overlays and camera paths in a photorealistic scene. Validated on a GPU box against an open-data offshore facility: physically simulated lighting at night and at dawn, selection and flow overlays, and rigid-body experiments such as a crane-lift clearance check.
Tools and technologies
NVIDIA Omniverse · OpenUSD · MCP · Python · WebRTC
Links
Lineage
Built on lessons from UX Reboot and Design System, Self-Service Twin Portal, Video Calls in XR, Multi-Agent Reasoning Engine.
Lessons carried into Spatial Intelligence for Agents, Drawings to 3D Scene, Agent-Operable Engines.