Volumetric data streaming from smartphones

Abstract

Smartphones are increasingly being equipped with dedicated depth cameras that directly measure the distance to objects in a scene. These cameras enhance the 3D perception of the world and enable a wide range of immersive applications (such as 3D tele-presence and tele-surgery). However, such applications often require real-time processing of depth data (for instance, to carry out 3D reconstruction), which can be infeasible to run on a mobile device. On the other hand, the depth data may be processed on the cloud. However, this requires streaming large amounts of volumetric data. To this end, the emergence of edge computing plays a key role in providing the necessary communication and computational resources required to enable the next wave of immersive applications.

The goal of this thesis is to study the feasibility of a real-time 3D tele-presence application that captures depth data from one user's smartphone and presents their volumetric 3D model to the other as a 3D avatar. We leverage a server-class edge device to overcome the limited computational capabilities of smartphones. We implement a mobile application that offloads the captured depth data to the edge server. An application on the edge server performs the 3D reconstruction that can be visualized on another smartphone or a head-mounted display.

We evaluate the proposed solution in terms of the reconstruction quality and the overall performance. The experiments prove the feasibility of our solution, while achieving acceptable visual quality and reconstruction frame rate. In addition, we present several insights into designing such an application and highlight open questions for future work.