Welcome to ‘Kagoo Explains’ - a series of short articles de-mystifying some of the confusing terminology used to describe tech. This week we’re concluding our in-depth look at upscaling - if you haven’t already, you can read part 1 here: Kagoo Explains: Upscaling Part 1. Today we’ll be looking at how the latest generation of televisions use Neural Networks and Machine Learning to make upscaling more powerful than ever!
Missing the forest for the trees
The technology and methodology for upscaling discussed in Part 1 has been around for years. It’s been improved and refined with time, but the basic premise is the same: using mathematical algorithms to decide the statistically-likely solution to missing visual data, taking each pixel individually.
The problem with this is the system has little concept of the context for each pixel - it doesn’t know if it’s part of a wall, or a poster, or a blood splatter. This limits how effective the upscaling can be - the algorithms can handle a lot of the standard scenarios, but unusual circumstances, edge cases and missing/corrupted visual information in the original image can cause strange results and artifacts that detract from the clean video you’re trying to watch.
Put your trowels and Alan Grant posters away - we aren’t talking archeology! ‘Artifacts’ is the name used for any anomalies, distortions or glitches in digital images and video. In upscaling, these are often the result of mistakes in the algorithms - wrong guesses for the missing visual data - or by corruption during data transfer. Many of these won’t be noticeable - a single blip only on screen for 1/30th of a second - but a large number of small artifacts at once can impact the quality of the video.
Trading Maths for Machines
However there is a new approach being used by the big companies to make upscaling more powerful than ever - AI-based upscaling technology. This new approach looks at the image as a whole, analyses it, and then improves each individual element (such as a face, a gun or an apple) in isolation, before piecing the image back together in fully-upscaled glory.
First the TV receives the data, and then has to understand what is actually being shown (i.e - a person talking, a town at night, a gun in someone’s hand). This is a process called ‘Image Recognition'. It’s a complicated process, but the gist is that a ‘neural network’ is created, and this system is fed an enormous amount of labelled images. This collection of data is used to ‘train’ the system to recognise certain images - so with time, it learns that a certain group of pixels is likely to be an eye, or a collection of pixels arranged in a set way is likely to be a dog. The larger the sample size the neural network is given, the more likely it will be to correctly identify an image correctly.
Neural Networks & Machine Learning
There are two important concepts to understand: first, Neural Networks. When we talk about a neural network, we are describing a massive collection of algorithms that are designed to analyse and understand patterns, in order to solve difficult problems. They are loosely based on the networks that make up our brains, and are designed to simulate something called ‘associative memory’ - the act of understanding and remembering connections between objects.
These systems ‘remember’ the correct answer for a problem (e.g - is this image showing a cat or a dog) and over time learns through experience, providing more accurate results. The act of a neural network improving itself through constant training is known as ‘Machine Learning’.
Once trained, the system can be used to take any image and analyse the pixel makeup, providing a list of things it believes is included. This is what the TV accesses when it attempts to work out what is being shown in each frame of the video it’s trying to upscale. Once the system has identified the components of the image, the really clever stuff happens!
The secret formula to upscaling
Companies such as Samsung have ‘trained’ a second AI to learn how to upscale objects. The process is similar to image recognition - a neural network is given an enormous database of videos and images, in both low and high-definition. It downgrades the high-def images to match the lower quality, and tracks what visual data was lost. In effect it’s learning what the blank pixels should be, rather than guessing what they could be.
From this exercise, it can learn how to do the reverse - putting the missing info back *into* the low-res image, effectively learning how upscale it. This is then turned into a formula to upscale the specific object - so over time, the machine ‘learns’ how to approach upscaling anything from a banana, to a cat’s face, to a rainy cityscape at night. This allows for the creation of a massive collection of formulae, each giving exact instructions for upscaling a single specific object.
Back to the TV itself - this bank of formulae can be accessed by the television as it’s trying to upscale the video. Therefore if it decides the image shows a man, a dog and a tree, it will use the formula for each element and upscale them individually - meaning each part of the image is being processed in context, and with an understanding of what it’s *meant* to end up looking like, rather than just another pixel in a group of millions.
Once the individual parts have been processed and upscaled, they are then brought back together as a whole, and the image is further processed to smooth over the joins between the different upscaled elements. The end result is an image that looks far higher-quality than any math-based upscaling algorithm could manage!
AI-based upscaling: An evolving technology
I’ve barely scratched the surface of the science behind neural networks and AI-based upscaling - if you want to read more there are a multitude of detailed and fascinating articles on the subject. However hopefully from this introduction you can see that AI-based upscaling relies on more than just maths to work out what should fill the blanks - allowing for *far* more reliable results, with fewer artifacts and mistakes to break the illusion.
Because it is still an infant technology, it isn’t found on many televisions on the market, and you will pay a premium for the capability. Indeed Samsung is one of the only companies pushing AI-based upscaling on TVs. However the nature of machine learning is that it will continue to grow and evolve - the systems will learn more, generating better results, and with time the technology will become more prevalent, more powerful and more affordable! Right now you can find AI upscaling in several 8K TV sets - Kagoo has got you covered for the best prices and deals on these cutting-edge sets!