Over the past several years, young trendsetters around the world have flocked to the OPPO Reno series to get their hands on top-level camera features capable of creating stunning portrait photos and videos filled with creativity and personality. Since the Reno series was first introduced, OPPO has continued to evolve its camera technologies to meet growing demand when it comes to smartphone imaging capabilities. Starting from the Reno4 series, this has meant focusing on making the Reno series the Portrait Expert and a one-stop solution for shooting fantastic visual content using portrait photos and videos.
Now, on the newly-launched Reno8 Pro, OPPO has taken a step further in building the Portrait Expert with the introduction of its first self-developed imaging neural processing unit (NPU), MariSilicon X. With its unprecedented, dedicated image processing power, MariSilicon X brings a next level portrait video experience to the new Reno8 Pro.
Extending Possibilities in Computational Videography
OPPO’s MariSilicon X arrives after years of investment in imaging technology aimed at finding the optimal combination of hardware and software-based image processing capabilities. However, despite OPPO and the rest of the smartphone industry making significant breakthroughs in computational videography, it became clear that bottlenecks in both hardware and software were preventing the boundaries from being pushed even further.
On the hardware side, for example, the limited physical size of smartphones places constraints on the size of the camera sensors that can be used. In the case of the Reno8 Pro, the phone is already equipped with best-in-class Dual Sony Flagship Sensors in the form of the IMX709 and the IMX766.
Co-developed by OPPO and Sony, the IMX709 has been completely redesigned with an all-new RGBW pixel array that contains additional white pixels not found in conventional RGGB sensors. With this new design, users can easily shoot stunning selfie photos and videos using Reno8 Pro’s front camera, even in the presence of strong backlight. The IMX766 on the other hand features a large 1/1.56-inch sensor size with built-in DOL-HDR technology and All Pixel Omni-Directional PDAF. Together, these properties ensure a better shooting experience on Reno8 Pro’s main camera with an extended dynamic range and strong preservation of original colors and details even in poor light conditions.
On the software side, OPPO’s continuous development of imaging algorithms has meant that the Reno series has always come packed with powerful and creative shooting features. As the latest model in the series, Reno8 Pro also comes with in-built AI algorithms for portrait retouching, image segmentation, bokeh effects, and more, allowing it to capture professional-looking photos and videos in virtually any scenario.
Unlike hardware, algorithms aren’t restricted by physical space, meaning they can essentially offer unlimited potential when it comes to boosting image processing performance. But one area in which they do face limitations is in computing power, especially algorithms which are dedicated to video processing.
Generally speaking, running a nighttime portrait optimization algorithm on a single image takes a few seconds using current algorithms and smartphone hardware. But if you try taking this process and completing it on the 30 images per second needed to shoot 4K nighttime portrait videos at 30fps, it’s easy to see that the math simply does not work out. The only way to achieve such processing speeds is to significantly increase computing power while also lowering power consumption if the solution is to be viable in a consumer device like a smartphone. While this sounds easy enough, current off-the-shelf SoCs are not yet able to handle this level of dedicated image processing power, which ultimately causes a bottle neck in algorithm performance.
This is where MariSilicon X comes in. Developed in-house by OPPO, the dedicated imaging NPU delivers both the immense computing power and the low power consumption that is needed for advanced image processing performance on mobile devices. What’s more, the NPU can be upgraded through continuous software-level tuning and optimizations, making it possible to further extend the upper limits of computational videography on smartphones in the near future. Such dedicated computational power is the key to taking the Portrait Expert experience on Reno8 Pro to an entirely new level, offering even more possibilities for Reno users to look forward to.
MariSilicon X: OPPO’s First Self-developed Imaging NPU
Arriving at MariSilicon X — OPPO’s first imaging NPU to be designed entirely in-house — involved a three-year R&D process supported by a team of over one thousand people.
MariSilicon X is built on the cutting-edge 6nm manufacturing process and features an industry-leading DSA architecture. Together, these form the foundations of its incredibly powerful AI computing capabilities. The NPU can deliver up to 18 trillion operations per second (18 TOPS), making it even more powerful in AI computation than some industry-leading flagship-level SoCs, such as the Apple A15 (15.8 TOPS), and giving it more than enough dedicated image processing power to support OPPO’s advanced AI algorithms. In terms of power efficiency, MariSilicon X delivers an incredible 11.6 TOPS per watt, offering top-end performance at low power consumption to enable 4K night video capture with crystal-clear frames.
Thanks to this outstanding computing performance, MariSilicon X can process images in the Bayer RAW domain. Typically, most image processing takes place at the very end of the image processing pipeline in what is known as the YUV domain. This means that much of the original image information can be lost during the various processes that are performed between the camera sensor and the YUV domain.
Unlike the YUV domain, the Bayer RAW domain lies at the front end of the image pipeline, where images contain richer lossless information, meaning that there is a lot more data for image processing algorithms to work on. One downside of this increase in available data is the additional pressure on hardware to perform more computations. Fortunately for Reno8 Pro, this is the exact use case for which MariSilicon X has been designed. Through its real-time, lossless processing in the RAW domain, MariSilicon X can help capture 4K video with up to 20bit ultra dynamic range and an 8dB improvement in signal-to-noise ratio.
Enabling 4K Video Shooting on Reno8 Pro
Following its original launch on OPPO’s flagship Find X5 series, MariSilicon X has been further optimized for improved AI processing performance on Reno8 Pro. OPPO has been working closely with MediaTek to get the most out of the partnership between MariSilicon X and Reno8 Pro’s powerful integrated 5G SoC, the MediaTek Dimensity 8100-MAX. Working in tandem with this SoC, MariSilicon X delivers enough dedicated image processing power to solve typical problems that occur during nighttime video shooting.
To start with, the IMX766 sensor on Reno8 Pro’s main camera first detects that the user is trying to capture video in low light conditions. Making use of its embedded DOL-HDR technology, the sensor collects as much image information as possible by capturing both long and short exposures of the same frame almost simultaneously.
After receiving this data from the image sensor, MariSilicon X will conduct real-time lossless processing in the Bayer RAW domain. The first of these processes is HDR Fusion, which combines the multiple frames of different exposures into one single image with four-times the original dynamic range. For night video, this extended dynamic range allows the details in dark areas of the image to be brought out without causing overexposure in the bright areas.
For the next step, MariSilicon X performs AI noise reduction (AINR) to deliver a significant increase in overall image clarity. The AINR algorithm has been trained on a million-point dataset and can intelligently recognize and reduce unwanted noise in nighttime portrait images. On Reno8 Pro, with the support of MariSilicon X, AI noise reduction can be performed with an increase in power efficiency of 1,116% compared with Reno7 Pro. On top of this, the combination of HDR Fusion and AINR algorithms improves the dynamic range of 4K Ultra HDR Video recorded on Reno8 Pro by 42.9%.
Following HDR Fusion and AINR, the image information arrives at the Dimensity 8100-MAX where OPPO’s 3A algorithms and AI Retouching algorithm get to work restoring the video to its most natural appearance by intelligently adjusting the exposure and color.
Through this multistep process, even portrait videos taken in the most challenging low light and backlit conditions will come out looking ultra-clear on Reno8 Pro, with colors and details in each frame that look true to life.
Thanks to MariSilicon X, this is the first time that 4K video shooting has been brought to a smartphone at this price point. Using 4K Ultra Night Video or 4K Ultra HDR Video on Reno8 Pro’s main camera, users can shoot videos at four times the resolution of 1080p videos. The sharpness and clarity of the ultra-clear night videos shot on Reno8 Pro are even comparable to those recorded on a high-end flagship device of other brands.
Empowered by MariSilicon X, Reno8 Pro becomes a next level Portrait Expert equipped with everything users need to enjoy the best possible portrait camera experience in even the most challenging situations. With even more creative possibilities to be discovered, MariSilicon X takes video to all new realms on the OPPO Reno8 Pro.
