Deep learning super sampling

Deep learning super sampling (DLSS) is a family of real-time deep learning image enhancement and upscaling technologies developed by Nvidia that are exclusive to its RTX line of graphics processors^[1], and available in select video games. The goal of these technologies is to allow the majority of the graphics pipeline to run at a lower resolution for increased performance, and then infer a higher resolution image from this that contains the same level of detail as if the image had been rendered at this higher resolution. This allows for higher graphical settings and/or frame rates for a given output resolution, depending on user preference.^[2]

As of June 2021, this technology is available exclusively on GeForce RTX 20 and GeForce RTX 30 series GPUs.

History[]

Nvidia advertised DLSS as a key feature of the GeForce RTX 20 series GPUs when they launched in September 2018.^[3] At that time, the results were limited to a few video games (namely Battlefield V^[4] and Metro Exodus) because the algorithm had to be trained specifically on each game on which it was applied and the results were usually not as good as simple resolution upscaling.^[5][6]

In 2019, the video game Control shipped with ray tracing and an improved version of DLSS, which did not use the Tensor Cores.^[7][8]

In April 2020, Nvidia advertised and shipped an improved version of DLSS named DLSS 2.0 with driver version 445.75. DLSS 2.0 was available for a few existing games including Control and Wolfenstein: Youngblood, and would later be added to many newly released games and game engines such as Unreal Engine^[9] and Unity^[10]. This time Nvidia said that it used the Tensor Cores again, and that the AI did not need to be trained specifically on each game. ^[3][11] Despite sharing the DLSS branding, the two iterations of DLSS differ significantly and are not backwards-compatible.^[12][13]

Release history[]

Quality presets[]

Algorithm[]

DLSS 1.0[]

The first iteration of DLSS is a predomenantly spatial image upscaler with two stages, both relying on convolutional auto-encoder neural networks^[17]. The first step is an image enhancement network which uses the current frame and motion vectors to perform edge ehancement, and spatial anti-aliasing. The second stage an image upscaling step which uses just the current frame to upscale the image to the desired output resolution. The neural networks are trained on a per-game basis by generating a "perfect frame" using traditional supersampling to 64 samples per pixel, as well as the motion vectors for each frame. The data collected must be as comprehensive as possible, including as many levels, times of day, graphical settings, resolutions etc as possible. This data is also augmented using common augmentations such as rotations, colour changes, and random noise to help generalise the test data. Training is performed on Nvidia's Saturn V supercomputer. ^[13][18]

This first iteration received a mixed response, with many critising the often soft appearance and artifacting in certain situations^[19][5][4]; likely a side effect of the heavy reliance on the neural networks to produce the output image which could not be trained to perform optimally in all scenarios and edge-cases.^[13]

Nvidia also demonstrated the ability for the auto-encoder networks to learn the ability to recreate depth-of-field and motion blur^[13], although this functionality has never been included in a publicly released product.^{[citation needed]}

DLSS 2.0[]

DLSS 2.0 works as follows:^[20]

• The neural network is trained by Nvidia using "ideal" images of video games of ultra-high resolution on supercomputers and low resolution images of the same games. The result is stored on the video card driver. It is said that Nvidia uses DGX-1 servers to perform the training of the network.^[21]
• The neural network stored on the driver compares the actual low resolution image with the reference and produces a full high resolution result. The inputs used by the trained neural network are the low resolution aliased images rendered by the game engine, and the low resolution motion vectors from the same images, also generated by the game engine. The motion vectors tell the network which direction objects in the scene are moving from frame to frame, in order to estimate what the next frame will look like.^[22]

Architecture[]

DLSS is only available on GeForce RTX 20 and GeForce RTX 30 series GPUs, using dedicated AI accelerators called Tensor Cores.^[22][23]

Tensor Cores are available since the Nvidia Volta GPU microarchitecture, which was first used on the Tesla V100 line of products.^[24] Their specificity is that each Tensor Core operates on 16 bits floating point 4 x 4 matrices, and seem to be designed to be used at the CUDA C++ level, even at the compiler level.^[25]

The Tensor Cores use CUDA Warp-Level Primitives on 32 parallel threads to take advantage of their parallel architecture.^[26] A Warp is a set of 32 threads which are configured to execute the same instruction.

Anti-Aliasing[]

DLSS requires and applies its own anti-aliasing method.

It operates on similar principles to TAA. Like TAA, it uses information from past frames to produce the current frame. Unlike TAA, DLSS does not sample every pixel in every frame. Instead, it samples different pixels in different frames and uses pixels sampled in past frames to fill in the unsampled pixels in the current frame. DLSS uses machine learning to combine samples in the current frame and past frames, and it can be thought of as an advanced and superior TAA implementation made possible by the available tensor cores.^[12]

See also[]

• Image scaling
• Deep learning
• Supersampling
• GeForce
• Nvidia DGX
• FidelityFX Super Resolution, AMD FSR is the equivalent upsampling technology for Radeon
• , Intel XeSS is an AI-augmented upscaling technology
• Tensor Processing Unit, an AI accelerator application-specific integrated circuit (ASIC) developed by Google
• List of games with DLSS support

References[]