FX Shoulder Solutions

AR Surgical Guidance

Own the full stack of the AR headset, from computer vision and tracking to the Unity front end, a custom OpenXR driver, and the headset prototype itself.
Train the detection and landmark models behind the guidance, and build the tools that capture and annotate their training data.
Validate the system on sawbones and cadaver specimens.
Help build the company's new surgical planning software, which replaced its previous planning system.

We discovered quickly that we needed an AR system with no surgeon input, ruling out buttons, hand tracking, and eye tracking.
Surgeons need guidance placed directly on the patient's anatomy, without intrusive markers.
The overlay has to stay locked to the shoulder while the surgeon's hands occlude the anatomy.
No off the shelf headset fits the job. Consumer devices lack the depth sensing and compute power to run the computer vision models with high accuracy.
The AI models need training data that does not exist.

An AI computer vision stack that shows the surgeon exactly what they need to see, depending on what is visible during surgery.
Built the whole tracking system around one idea: detect and track the 6-DoF pose of the patient's anatomy from the RGB and depth sensors, and use it for anatomy mapping, instrument tracking, and surgical measurement.
Trained custom detection and landmark models and fused them with the IMU and depth camera odometry to hold a stable pose on the shoulder during surgery.
Run the Python host as a multi-process system for real-time throughput, with IMU prediction covering render latency so the overlay does not lag a moving head.
Wrote a custom OpenXR driver for the unity front end and put the headset together myself: dual RealSense depth cameras, a custom binocular micro OLED display, and a fanless design for the sterile operating room.
Built a suite of Python data tools that capture annotated 3D sessions straight from the rig into versioned datasets, label them, assemble training sets for each condition, and test every trained model against its own data.
For the new planning software, helped write the save system and an AI model for automatic landmark placement, and built the packaging tool for the Windows and macOS releases.