April 20, 2017 Powering the mobile AR revolution with human-aware AI
Augmented reality visuals are the new norm in social messaging as the camera is becoming the new keyboard. With the aid of face filters and stickers, users are decorating their photos and videos to express themselves creatively and to attract the attention of more viewers. On April 18, Facebook announced the Camera Effects Platform as its focal point at the F8 developer conference, while Snapchat launched World Lenses with 3D stickers. So where is this AR battle leading to next?
Two months ago, Body Labs released a filter app for bodies called Mosh, which stands for ‘motion and shape’ and is powered by our core technology (human-aware artificial intelligence that can see and predict 3D human motion and shape). We trained our core tech using a vast library of 3D body scans of thousands of people in different poses to understand human body shape, pose and motion. By integrating with Body Labs, Mosh can easily detect 2D body joints and enable users to render fun filters on their body.
Beyond images, we also detect 3D body joints and rotations in real time. Up until now, marker-based mocap has been the best way to capture human movements with high fidelity. However, Body Labs has markerless mocap capabilities to capture subtle surface nuances because it leverages our neural networks with SMPL (Skinned Multi-Person Linear Model), which is our patented parametric body model that includes our template mesh, blend weights, corrective blend shapes, and a rigged skeleton. By combining with 3D joints and landmark locations, we can predict the body projected movements that are pivotal for realistic animations, video games, and interactive AR experiences.
Why haven’t we seen body filters sooner?
Unlike heavily commoditized facial recognition, body tracking needs much more information to achieve high accuracy and precision. The human body has 680 muscles, 206 bones and 340 joints. It bulges, breathes, flexes, jiggles, and is covered with clothing. Furthermore, our body shape, density, and flexibility change with age and fitness; not to mention our pose is dynamic and unpredictable. In order to create the most lifelike 3D body effect, Mosh takes into account the depth, scale and volume of the body to identify the joints and anthropometric landmarks. Mosh then estimates the 3D shape and pose to align the filters to the whole body. This enables Mosh users to step up their dancing skills like this:
Body Labs will be releasing the a Mobile SDK for real-time body tracking soon. Mobile app companies can integrate this SDK to equip their users with body tracking capabilities and fun AR body filters. Body filters are even more fun to render than face filters because you can curate your own storyboard by acting in response to the filters to tell a more compelling and engaging story. Users will also be able to use motions like extending your arms to trigger a responsive filter. Our Mobile SDK could be built out to understand hand gestures, multiple people, and occluded bodies. Down the line, action recognition like Snapchat’s “barfing rainbow” could be replaced with clapping or jumping. The use cases are limitless.
What does this mean for businesses?
Social messaging will get another facelift with new augmented reality features for the entire body. In this information-rich world, content is king but curation is key. Users are looking for more ways to express themselves and stay connected with their network. AR body filters offer not only an unprecedented way for storytelling, but also enhance user engagement by encouraging users to collaborate and take photos of each other. Businesses can use our Mobile SDK to build apps that apply action and motion recognition into their UI to create better content and experiences for their users.
As of now, the Mosh app is a tech demo to showcase our core tech’s aptitudes and product offerings. Our Mobile SDK will soon be available for licensing. The outputted images and video clips can be saved or seamlessly shared to your social media platform with a one-click export function. You can try out our technology by downloading our new app on the app store (download Mosh).
For information about the Body Labs Mobile SDK email shin.woon@bodylabs.com.