Subscribe to NewscastStudio for the latest news, project case studies and product announcements in broadcast technology, creative design and engineering delivered to your inbox.
Imagine a world where digital information isn't confined to screens but becomes an integral part of our surroundings. Our computer disappears, and our screen is blown up to encompass the entire universe. We can hang app windows anywhere we want to in the real world around us, or alternatively have a huge screen encompassing the space within our vision. Our interaction with these apps will no longer be governed by a mouse or a touchscreen. We will be able to manage apps with a simple hand gesture, eye movement, or facial expression.
This is the promise of spatial computing. But what exactly is spatial computing, and what opportunities and challenges does it present to the media and entertainment industry?
Spatial computing represents a paradigm shift in how we interact with the world around us. At its core, it involves the seamless integration of digital elements into the physical world, enabling seamless interaction between humans and machines. For instance, viewers can be immersed into the action through real-life virtual scenes where characters are portrayed in 3D stereoscopic video. It incorporates immersive audio coming from all directions and even bouncing off the objects in one's surroundings, for a true-to-life experience. It also includes 3D virtual graphics and textual metadata to help more easily navigate and interact with the scene.
Endless opportunities: from immersive experiences to monetizationSpatial computing can be applied in a multitude of ways in the media and entertainment industry. It can bring spectacular 3D movies to users on demand, wherever they are — even on a cramped airplane seat! Imagine the immersive experience of being at the stadium or concert venue, while still at home. You can then layer on interactive extras: dive deep into sports statistics, or shop for the guitar being played by the lead singer.
The implications of spatial computing extend far beyond the realm of entertainment, however. As technology matures, it has the potential to redefine various aspects of our daily lives. From enhanced productivity and immersive learning experiences to revolutionizing industries such as healthcare and manufacturing, spatial computing opens countless possibilities.
It can also provide new revenue and monetization opportunities. Apps can seamlessly integrate with the content by adding overlays enabling new interactions such as:
Key to spatial computing's success are the headsets that enable interaction with the immersive environment. Older generations suffered from poor video resolution, leading to unsatisfactory user experience. The latest generation headsets have much better resolution but can be heavy — making them uncomfortable for users watching a whole movie or an entire sports event or engaging in a long training session. They are also expensive. But the electronics and computer industries have a long track record of making things faster, smaller/lighter and cheaper over time, and spatial computing headsets should be no exception.
On the streaming services side, one of the main challenges when encoding 3D video is to ensure user comfort by delivering video in quality that is perceived as consistent between the two eyes for each area of the image. A traditional constant bitrate approach is not sufficient here, and it is critical to use a more advanced rate-control scheme to deliver constant visual quality based on a very accurate video quality model.
Advertisement
Another important challenge is the need for high bandwidth (one stream per eye, each of which will typically be 4K or higher resolution and potentially at twice the frame rate of regular movies). To avoid buffering and saturating the network it is therefore critical to use an efficient encoder.
The current state-of-the-art solution is to use the MultiView High Efficiency Video Coding standard (MV-HEVC) developed by the ITU and MPEG Joint Video Exploration Team (JVET). With MV-HEVC, one eye (typically left) is encoded as an independent stream while the other is differentially encoded with respect to the other (similar to how scalable video encoding is done). This leads to an average of 30% reduction in the required bandwidth, up to 50% depending on the content.
Another major hurdle is performing spatial computing in real-time. Currently, only VOD content, which is processed offline, is commercially available. Real-time processing is required, however, for live events such as sports or music concerts. With the latest generation of chipsets, combined with careful software optimizations, real-time spatial computing has been demonstrated in lab environments and should become commercially available in the near future.
With spatial computing challenges on the cusp of being solved, it's just a matter of time before the merger of our digital and physical worlds becomes an everyday reality. It's an exciting time for the media and entertainment industry, and beyond.
Subscribe to NewscastStudio for the latest news, project case studies and product announcements in broadcast technology, creative design and engineering delivered to your inbox.
No comments:
Post a Comment