About the 6D.ai SDK Texturing Sample App
We are currently publishing an example of real time 3D reconstruction with texture overlays. We are not providing sample code for this right now, but rather providing a prebuilt app you can install on iOS directly from TestFlight.
What is this app?
This is a fun preview from a real customer project that we thought we'd share with our developer community! In basic terms, texturing means we are using the live RGB camera frames from the phone camera and draping them over the triangles of the mesh that we are creating in real time. We found adding the texturing to the scanned mesh really helped spaces come to life and demonstrated some of 6D's core strengths. If you have a project that could benefit from some of this tech, reach out to email@example.com.
What this isn't
This example is explicitly not for asset creation, but rather a simple way to demonstrate 6D's ability to create very large scale, real-time scans for rough human readable models of large spaces. We know many of our developers would love to create photorealistic 3D models of scanned objects, but for that we'd recommend using an offline photogrammetry or depth camera based system (which are not real time).
Why is real time important?
Generally, the 6D Reality Platform is different from other spatial computing platforms in that we compute a dense 3D reconstruction (mesh) in real time using standard smartphones. This is important so that the mesh can be used immediately versus processed offline and used later. For example, a developer could enable a user to scan a space and play a game in it at the same time. Or an enterprise worker could broadcast a machine in 3D to a remote expert in a live session in order to collaborate more effectively.
How does it work, where does my data go?
For this app, we decided not to push any of your scanned data to our AR Cloud in order to ensure privacy and really prove that our tech does everything on the device. This means that whatever you scan never leaves the phone as all of our 3D reconstruction occurs directly on your phone using our proprietary neural networks. Note that you do need a network connection when you first use the app, just for a quick license check on our server, so we can roughly see how popular the app is.
Keep in mind this means that your data is only available during the time you are actively scanning with the app, and everything is deleted when you close the app. If you want to save a screenshot or video of what you capture, go for it!
Please reference #6d_ai in any social media if you feel so inclined and we'll rebroadcast anything that catches our eye.
How do I actually use the app?
First you need to install it, so make sure you've registered for free on the developer site, then login to your account. Our TestFlight page can be found under our Downloads menu, or simply by visiting the page from here: 6D TestFlight.
When you first launch the app, we have a brief instructions screen and then you will see a gray-scale image with a (-) and plus sign (+) in the bottom left of the screen.
As you move around, the app will start to texture the gray scale image. You will see the screen fill up with the textured mesh. Click the (-) button to zoom out. This will subsequently blacken out the parts where the mesh hasn’t been created yet.
The textured mesh will have a higher resolution closer to you, and have lower resolution textures farther away up to a 6 meters radius. Since storing these textures are memory intensive, after the 6 meter radius, we color the mesh instead of storing the textures.
Please be aware of your environment while scanning! Don't forget that the coffee table you see in the beautiful 3D model in front of you is about to say hello to your shins ;)
Have fun with it! After a bit of scanning, try zooming out with the (-) button; look around the space you've scanned while you are zoomed out to really get a feel for what you've created. Another idea is to try scanning your space while you've zoomed out from the map. Let us know what worked best for you - you'll find us on our 6D Developer Slack, Twitter and LinkedIn!
When will this be part of the generally available 6D Reality Platform?
We are still evaluating market demand versus other priorities, but our default is to share early and often. We look forward to feedback on the channels mentioned above, specifically around how developers would want this technology to solve new problems that weren’t previously solvable.