AR Surgical Guidance

Role

Own the full stack of the AR headset, from computer vision and tracking to the Unity front end, a custom OpenXR driver, and the headset prototype itself.
Train the detection and landmark models behind the guidance, and build the tools that capture and annotate their training data.
Validate the system on sawbones and cadaver specimens.
Help build the company's new surgical planning software, which replaced its previous planning system.

An AI computer vision stack that shows the surgeon exactly what they need to see, depending on what is visible during surgery.
Built the whole tracking system around one idea: detect and track the 6-DoF pose of the patient's anatomy from the RGB and depth sensors, and use it for anatomy mapping, instrument tracking, and surgical measurement.
Trained custom detection and landmark models and fused them with the IMU and depth camera odometry to hold a stable pose on the shoulder during surgery.
Run the Python host as a multi-process system for real-time throughput, with IMU prediction covering render latency so the overlay does not lag a moving head.
Wrote a custom OpenXR driver for the unity front end and put the headset together myself: dual RealSense depth cameras, a custom binocular micro OLED display, and a fanless design for the sterile operating room.
Built a suite of Python data tools that capture annotated 3D sessions straight from the rig into versioned datasets, label them, assemble training sets for each condition, and test every trained model against its own data.
For the new planning software, helped write the save system and an AI model for automatic landmark placement, and built the packaging tool for the Windows and macOS releases.