Google announced a new feature in the Google Meet application that is an in-browser machine learning solution. This new feature enables the users to blur and replace the background in a Live video. Rather than the surrounding, this enables a better focus on the person.
With the new normal on the move, the world is adapting how to best manage, learn, and continue to stay connected. Thus, Google Meet deployed a new in-browser ML solution for blurring as well as replacing the background. This can run efficiently on the web with the new ML models and OpenGL shaders. The latest features achieve real-time performance by low power consumption and even on low-power devices.
The new feature works directly in the browser and does not need any extra steps like installing additional software. These features are built with MediaPipe and are powered by revolutionary web machine learning (ML) technologies. The main intention for developing these features was to provide real-time as well as in-browser performance on almost all contemporary devices. This was established using XNNPACK and TFLite, by combining efficient on-device ML models, WebGL-based rendering, and web-based ML inference.
MediaPipe, one of Google’s open-source frameworks was used to develop the new features of the Google Meet. In applied ML pipelines, this helps to build multimodal like video, audio, and any time series data as well as cross-platform that are Android, iOS, web, and edge devices. ML solutions like on-device real-time poses and body parts tracking are also possible. WebAssembly that is leveraged by MediaPipe’s web pipeline can improve speed for compute-heavy tasks. The browser can convert WebAssembly instructions while running into a native machine code that can execute faster than the conventional JavaScript (JS) code.
By segmenting the user from their background, this new solution develops each video frame and works out a low-resolution mask. This cover is then refined to align it with the image boundaries. Further, it is used to generate the output video through WebGL2, through either blurred or replaced background.
During the COVID-19 pandemic, the use of the Google Meet App grew by a factor of 30 during January and April of 2020. Google Meet reported having 100 million daily active users. Google Meet is a popular video-communication service provided by Google. Google Meet was introduced as the combined replacement of two other Google apps, namely Google hangouts and Google chat.
