Meshing API Guide

Please review to the SDK Function Definitions for additional info on functions mentioned in this guide. This guide describes the meshing API of the beta SDK version 0.21.0 onwards.

Meshing is one of the unique features of the beta SDK. The mesh made available to developers is a combination of on-the-fly meshing generated in the current AR session through depth estimation, and mesh loaded from the AR Cloud in the case of a successful relocalization.

Mesh Geometry 101

Mesh geometry is made of vertices and faces. Vertices are 3D points; faces are triangles made of three vertices.

In our meshing API, vertices are made of 6 floating point numbers each: 3 for the position of the point (x, y, z); and 3 for its normal vector (nx, ny, nz). The normal vector describes the direction the point is facing, which is used for light rendering and collision physics.

The coordinate system used is right-handed, and by convention X points to the right of the initial position of the user, Y points up, and Z points towards the user. This convention is widely adopted by ARKit, OpenGL, Metal, SceneKit, etc. Unity uses a different convention, for which a sign inversion on the Z axis is necessary.

Faces are made of 3 integers each: they each represent an edge of the triangle with its index in the list of vertices. This way, vertices can be re-used and shared between faces. Each triangle faces only one direction, the one for which the three edges are in counter-clockwise order.

For a more detailed description of the concepts of mesh geometry, read Demystifying Meshes on the blog. Warning: the SDK was different at the time that article was written, so some API details are now inaccurate, but the general information is still current.

Mesh Buffers

In the beta SDK API, mesh data is exposed through three different buffers:

Blocks are subdivisions of the overall mesh, cut along cubes 1.4m (4½ ft) a side. Blocks keep nearby vertices and faces together, which simplifies application-side mesh operations such as collision physics, distance-based culling, navigation, etc.

The exact size of a mesh block can be obtained in meters with the API call float SixDegreesSDK_GetMeshBlockSize().

In our meshing API, blocks are made of 6 integers each: 3 for the position of the block (x, y, z), 1 for the number of vertices, 1 for the number of faces, and 1 for the version number, which can be used to prevent unnecessary updates.

The three buffers must be allocated on the app side. The API call int SixDegreesSDK_GetBlockMeshInfo(int* blockBufferSize, int* vertexBufferSize, int* faceBufferSize) will return the overall version number of the mesh, and populate the passed integer pointers with the number of elements in the buffer.

The sizes provided by SixDegreesSDK_GetBlockMeshInfo() are the number of elements in the buffers assuming they are allocated as int and float arrays. It is important to allocate those arrays on the heap, not the function stack, as they can easily become very large.

In other words, the number of blocks is blockBufferSize/6; the number of vertices is vertexBufferSize/6 and the number of faces is faceBufferSize/3.

Once the buffers are allocated, the API call int SixDegreesSDK_GetBlockMesh(int* blockBuffer, float* vertexBuffer, int* faceBuffer, int blockBufferSize, int vertexBufferSize, int faceBufferSize) will populate them, and return the number of blocks successfully populated.

Parsing the Buffers

The buffers are indexed by block. Every vertex and face belongs to one block only.

To illustrate the buffers, let's use a simple flat mesh spanning two blocks.

The first block in yellow contains a square made of 2 faces and 4 vertices; the second block in blue contains one triangle represented by 1 face and 3 vertices.

Overall, this mesh contains 2 blocks, which means the block buffer has a size of 12 ints.

Here is what the block buffer looks like: the two blocks show different coordinates, and vertex and face counts corresponding to their respective contents.

In reality, blocks will contain hundreds of vertices and faces each. Iterating through blocks to split the mesh into submeshes can be a winning strategy to avoid unnecessary updates and costly recalculations.

The overall mesh contains 7 vertices, which means the vertex buffer has a size of 42 floats. The vertex buffer is made of two halves, the first one containing all positions as (x, y, z) 3D coordinates, and the second one containing all normal vectors as (nx, ny, nz) vectors, in the same order.

In our case the first 4 vertices belong to the first yellow block, and the next 3 belong to the second blue one. All positions are listed back-to-back, then all normals.

In this example, all vertices share a same normal vector pointing up, consistent with the geometry of a flat horizontal surface.

Note that vertices 2 & 4, and 3 & 5, are identical but in different blocks.

Finally, this overall mesh contains 3 faces, which means the face buffer has a size of 9 ints. All faces are listed back-to-back in counter-clockwise vertex order, with the two faces of the first yellow block first, and the only face of the blue block last.

Each int value is the vertex index in the vertex buffer. Note that the index is not reset to 0 in new blocks.

The sample apps for Unity and SceneKit provide code examples for feeding mesh data to rendering and physics.