GPT-6 + SAM 3 Auto Labeling: Build a Step-by-Step Process Guidance App

​When you're doing delicate work, like assembling a product, setting up an experiment in a lab, or repairing an item, it's important to get every step of the process right. To do that, you can build a computer vision application that watches the process and provides guidance on the next step. In this session, you'll see how one such process guidance app was built, from automated image labeling to running on a mobile device.

​Roboflow Engineer Lucas Fochesatto will demonstrate a computer vision app that guides you through a process: solving a Rubik's cube step by step. You'll see how Lucas used GPT-6 along with Segment Anything Model 3 (SAM 3) to automatically annotate images, and how he got the app running on an iPhone for real-time process guidance.

​In this live session, we will cover:

• ​Auto-labeling with GPT-6 and SAM 3: Generate instance segmentation masks for small, partly hidden objects, and review what the auto-labels got right and wrong.

• ​Model fine-tuning: Train a custom segmentation model on those masks, plus what Lucas learned about objects hidden behind fingers.

• ​Guiding the next step in real time: Export the model to Core ML so it runs on an iPhone's Neural Engine and turns each frame into the next instruction.