Our investigation revealed that PTC/Vuforia should intervene in this industry for several key reasons. PTC already had the capacity to provide end-to-end Create-Manage-Deliver service information on spare parts. This efficiency was due to its established CAD infrastructure, which spanned authoring, management, and delivery of all needed content types. PTC already possessed training content that could help the system establish a reasonable universe of objects and narrow down potential parts to 20-200. Additionally, AR technology was feasible for ubiquitous smartphone technology held by service technicians, and PTC had a customer base of 30,000 who regularly worked with CAD and 3D imaging, providing an immediately accessible market. Through our user research, we identified four specific personas with distinct jobs to be done. These insights directly shaped our product requirements and user experience design:

1. Jim Lee (Service Operator/Technician)
   • “Today, I’m using a part catalog (physical book or online) to find the part I’m looking for. Many times, I make mistakes, and the process takes time.”
   • Needs help to avoid making mistakes when ordering parts.
2. Hannah Jamser (AR Service Provider)
   • “Today, I provide AR apps to manufacturing units, but there is a limited set of 3D models users can automatically place in the physical world.”
   • Serves as the intermediary between manufacturers and software companies.
3. Alez Rodriguez (Factory Operator)
   • “I end up ordering obsolete parts that I don’t need and have increased operational costs.”
   • Responsible for optimizing the processes in the factory.
4. James Romero (CAD Author)
   • “It’s hard to organize my database so that CAD models can be used by AI algorithms directly.”
   • Organizes and maintains CAD database.

Our prototyping process developed a seamless user experience across multiple platforms while building a robust AI recognition system. We created a detailed user flow sequence that included both the annotation and training phases:

Annotation Flow (Steps 1-3):

1. Take a screenshot with a mobile app that has a class 1 object (e.g., screwdriver).
2. Annotate in the mobile app (draw a bounding box).
3. Send annotated images to the Azure web server.

Training Flow (Steps 4-8):

   4. Train in Azure Custom Vision server (advanced training may take       hours).
   5. Export trained model to iOS/Android,
   6. Load the trained model in app project.
   7. Predict class in an image taken from the mobile app in real-time.
   8. The user accepts, modifies, or rejects the bounding box prediction.

We developed an AI training model for the backend utilizing tools like Onshape, Blender, and ParaSOLID to generate synthetic images and point clouds, which fed into a vanilla AI network for classification. Instead of using traditional annotation methods, we developed the RTO2-3D network, which was trained on both images and point cloud data. For the front-end user experience, I designed high-fidelity interfaces for mobile devices, tablets, and RealWear AR headsets, adhering to platform-specific best practices. For mobile, we focused on 2D interactions, allowing users to act as a magic hand, making inputs into the semi-real world. For tablets, we incorporated placeable pins on CAD models to help users make annotations, ensuring clickable functions were within reach. For RealWear AR, we limited background functions to preserve battery life. Our implementation strategy was structured with clear execution parameters:

Time:

• 6 months for gradual adoption with Vuforia Studio widgets
• 1 year for RTOD services suite
• Across four sprint phases

Budget:

• Approximately $1.7M

Human Resources:

• 7-10 FTE

Working within these parameters, we successfully delivered a product with four unique characteristics:

1. Efficient: Instantaneous predictions with constant improvement.
2. Versatile: Adaptable to customer environments.
3. Persistent: Constant predictions around the 3D space.
4. Cross-platform: Integration between Unity Engine and Vuforia Studio.