Whether you want to work, study, or play from home, a smart screen is for you.
Simply put, a smart screen is a battery-powered touchscreen device that connects, shares, and interacts with other devices. Also known as a wireless monitor, it is used for activities that require little or no keyboard involvement. It’s a relatively inexpensive and easy way to connect web apps, video conferencing calls, and workspace sharing with built-in Wi-Fi connectivity.
Smart screens have emerged as an exciting innovation platform for consumer, industrial and enterprise client computing. It started gaining popularity during the COVID-19 pandemic when people started working from home. As detailed in a recent Synaptics article, by combining artificial intelligence (AI), wireless connectivity, touch, sound, and image recognition, scalable and secure processing, and advanced display drivers and algorithms, they are capable of fully immersive, Enable context-aware user experiences that turn them into a multifunctional control point from which to securely interact with a variety of devices. This article looks at the ins and outs of hardware and software development and collaboration to enable next-generation intelligent displays by explaining how Visidon software, running on Synaptics’ VS680 multimedia SoC, enables the Smart -Improved screen capabilities and supported a variety of use cases.
Although definitions may vary, the rise of smart displays as the central hub for personal, commercial, and industrial applications is well documented. For example, Research and Markets expects the market to reach $16.16 billion by 2027, at a CAGR of 22.56% from $4.79 billion in 2021. IoT-enabled smart home devices are expected positively influence market growth.
However, for smart displays to reach their full potential, the underlying hardware must meet critical performance, robustness, security, cost, and efficiency requirements, while the algorithms must be efficient, intuitive, seamless, and able to leverage the capabilities of the hardware fully exploit. This requires close collaboration and advanced, easy-to-use development tools.
Synaptics and Visidon are collaborating to provide a complete solution that improves user experience in smart displays through Synaptics VS680 Mutimedia SoC (hardware) and Visidon algorithms (software). This article dives deeper into two groups of Visidon algorithms, such as B. Face Analysis and Depth Calculation ported and optimized for Synaptics VS680 to address different use cases.
Visidon face analysis algorithms include face detection, tracking and recognition, age, gender and smile analysis, as well as glasses and face mask detection. These algorithms enable intelligent screen access control and content personalization. For example, face recognition is useful for organizing photos, securing devices like laptops and phones, and helping the blind and partially sighted.
For smart screen video conferencing and communication enhancements, Visidon algorithms perform depth calculations and portrait segmentation, and offer a variety of fun filters. One of the filters is Bokeh, which helps focus on the foreground subject by providing clear and sharp edges in the foreground and gradual blurring of the background. Also, video conferences can be further enriched with Visidon’s sketch, black and white, retro or cartoon effects for more entertaining communication.
In addition to video conferencing improvements, depth estimation algorithms are used in augmented reality applications such as assisted sports, virtual wardrobe, and interactive gaming.
Visidon’s facial analysis and depth calculation algorithms consist of various components implemented on the VS680’s NPU, CPU and GPU (Figure 1).
Visidon’s face analysis algorithms, face detection and tracking run on the CPU, while age detection, gender, smile and glasses detection run on the NPU. It’s important to note that while some NPUs are “attached” to the CPU and GPU in a SoC, the VS680’s NPU and AI pipeline are seamlessly embedded, allowing them to operate in a trusted execution environment, which is important for security and privacy is of crucial importance (Figure 2 ).
For enhanced video conferencing, Visidon offers depth estimation and portrait segmentation (Figure 3) to separate the main subject in the image and replace the background. These networks run on the NPU. In addition, face detection selects the focus point in the frame and depth map for the bokeh effect. Mask refinement improves the depth map and the accuracy of the segmentation mask. And finally, effect rendering creates background blurring, background substitution, and playful effects. These networks run on the CPU and GPU. Visidon offers depth estimation for both monocular and stereo cameras. Compared to segmentation-based counterparts, Visidon bokeh offers a more natural look with less object clipping and less visible flaws. It also features high stability with no body parts flickering.
In depth-sensing applications, depth estimation algorithms allow realistic depth and distance calculations of objects in images, while segmentation can be used to extract objects and entities from the image context. For example, if a person sits in a chair where the floor and the chair are the same distance from the camera as the person, the segmentation can semantically distinguish what is a person, what is a chair, and what is a floor.
Synaptics VS680 is powered by its SyNAP toolkit, which allows customers to optimize ML/AI models to fully leverage the capabilities of the VS680 (Figure 4). These models include those for video, image, and audio AI. Working with VS680 was easy and didn’t require the use of other special platform tools, which is often a disadvantage when working with special hardware.
Synaptics and Visidon will present their solution at the IBC Show 2022 (https://show.ibc.org/) September 9-12 in Amsterdam. (SF)
About the author
Marketing Manager Visidon
Sharmin Farah is Visidon’s Marketing Manager with a mission to grow the company’s video enhancement brand across multiple market segments. She has a passion and expertise in digital marketing and communications, but also experience in project management, branding, public relations and research and development.