AI and GPU Evolution: Tensor Cores Redefine Computing and 5G

The discussion began with a series of quick questions. The first computer was an Apple II, preceded by a teletype connected to a mainframe. The favorite keyboard shortcut is "W, A, S, D," indicating a gaming background. When programming, tabs are preferred over spaces. The favorite programming language is O, with Fortran and Pascal also mentioned as languages extensively used. Python is used for tasks where O isn't sufficient. The least favorite programming language is C++. The first computer game was Asteroids. Regarding beverages, tea is increasingly preferred over coffee.

Reading Archived Papers and AI for Research

A hobby involves reading archived papers. While many are too complex, scanning them always yields new knowledge. A recent paper of interest was DeepSeek R1, which demonstrated reinforcement learning without supervised fine-tuning to achieve excellent results. The discussion then shifted to using AI, specifically tools like ChatGPT, to summarize and analyze research papers. This approach allows users to effectively "talk to the researcher" by asking the AI questions about the paper's content. This transforms the AI into an expert on the topic, capable of answering questions that might not have been explicitly considered during initial research. For instance, when researching silicon photonics, the AI can provide insights into its history, future, and key players.

Evolution of GPU Architectures

Historically, different GPUs were designed for specific tasks, such as Quadro cards for video editing and GeForce cards for gaming. While all GPUs share CUDA, their internal resource mixes varied. Some might have more texturing units, others more ROPs, or different memory types like HBM or graphics memory. At the computational architecture level, some GPUs excelled in double-precision floating-point operations, while others focused on FP32 for graphics.

However, there's a growing convergence due to the increasing use of tensor cores. Tensor cores are now a significant component in graphics, AI, and even physics applications. In computer graphics, AI allows rendering one pixel and inferring the other 15, leading to higher resolutions, greater complexity, and improved image quality that surpasses purely rendered approaches. AI is not just an approximation; it expands the reach of physics and many other fields.

The first major architectural fork occurred when computing required high double precision, while computer graphics needed very low double precision. Compatibility across GPUs was a priority, so while all programs would run, FP64 might be slow on graphics-focused GPUs and vice-versa. The second fork emerged with the realization of the importance of tensor cores for AI in data centers. This led to increased tensor processing capabilities, sometimes at the expense of FP64 performance. The current strategy involves leaning more into tensor cores, using emulation for double precision, and moving towards hybrid approaches that combine principal solvers with partial emulation and AI.

For computer graphics, the focus initially shifted to FP32 and programmable shaders. However, AI's potential led to the reintroduction of tensor cores from data centers back into graphics cards. This created a full circle: GeForce brought CUDA to the world, CUDA enabled AI by providing researchers with supercomputers on their PCs, and now AI has revolutionized GeForce, making computer graphics heavily AI-driven.

The Relentless Pace of AI and Hardware Scaling

The pace of change in AI models and hardware is relentless. AI models are becoming twice as fast every seven months, and the amount of data is also accelerating. This results in a tenfold increase in computational requirements annually.

To keep up with this demand, the approach to computing has evolved:

1. Pre-packaged Software Era: Initially, software was pre-packaged and compiled, limiting computer scaling to Moore's Law, driven by semiconductor physics and CPU architecture advancements (microcoded to pipelining, superscalar, out-of-order execution, vectors). This approach was constrained by how the software was packaged.

2. Accelerated Computing (CUDA): The breakthrough came with accelerated computing. Software packages could remain the same, but CUDA allowed for the introduction of new algorithms and underlying GPU architectures simultaneously. This "co-design" or "full-stack optimization" enabled much faster software acceleration than Moore's Law predicted.

3. AI-Driven Optimization: With AI, further dimensions of optimization became possible:

   • Precision: AI's statistical nature allows for reduced precision (e.g., FP32 to FP16 to FP8), effectively quadrupling computation or reducing energy by a factor of four with each step.
   • Computation Structure: The structure of computation can be changed from vector algorithms to more efficient tensor core structures, matching the algorithm's nature and allowing for simultaneous execution of many instructions.
   • Data Center Scale Parallelization: Algorithms can be parallelized across multiple chips, nodes, and racks, moving beyond single-chip limitations. This has led to a million-fold increase in computation scale over the last decade, compared to Moore's Law's hundred-fold prediction.

Simultaneously, new neural network architectures are constantly being invented. Neural networks are essentially software and are highly adaptable. Concepts like transformers have evolved significantly, incorporating innovations like speculation techniques, multi-token prediction, and compression techniques to further accelerate their performance.

Scaling Up vs. Scaling Out

• Scaling Up: This involves increasing the capability of a single computer to make software run much faster with minimal changes. It's limited by semiconductor physics, memory bandwidth, and other physical constraints. NVIDIA's NVLink allows connecting multiple GPUs to act as one giant GPU, overcoming these physical limits.
• Scaling Out: This involves breaking an algorithm into smaller, distributed parts to spread the workload across many machines. Hadoop, based on Google's MapReduce, is an example of a scale-out approach.

The most efficient parallel processing combines both. First, scale up as much as possible to create the most powerful individual units (like a giant GPU via NVLink), then scale out by connecting multiple racks. This is analogous to having the smartest, most productive people in the smallest possible team, as scaling out too much introduces overheads in communication and coordination (the "mythical man-hour"). Scale-up stays within this "mythical man-hour" by leveraging the inherent parallelism within a GPU, a core invention of CUDA.

The Role of CPUs in a Parallel World

CPUs are still necessary due to Amdahl's Law, which states that any program will have a sequential component. Even if 90% of a task can be parallelized and sped up infinitely, the remaining 10% sequential part will limit the overall speedup. Therefore, it's crucial to make the single-threaded, sequential part as fast as possible. This is why NVIDIA builds its own CPUs, ensuring excellent single-threaded performance, while offloading multi-threaded tasks to CUDA.

Unconventional Uses of NVIDIA Technology

One unexpected and exciting application of NVIDIA's technology is in 5G radio. Instead of using custom chips for radio baseband processing, CUDA is used. This makes 5G radio software-defined, allowing for the integration of AI. AI can replace pipeline layers, enable fully deep learning-based radio networks, be used at the massive MIMO layer, or orchestrate traffic across multiple radios. This leads to "AI RAN" (Radio Access Network), transforming the radio network into a robotics network that can leverage reinforcement learning for adaptive and autonomous operations. This could significantly save energy and expand spectrum efficiency.

Applying AI to communication networks can also dramatically increase effective bandwidth. For example, in video conferencing, after the initial few frames, subsequent frames might not need to be fully encoded. AI can predict facial movements based on sound, reanimating faces using neural networks. This could reduce bandwidth usage on 5G or 6G by a thousandfold. By incorporating human prior knowledge into transmissions, generative processes at the receiving end can replace network bandwidth with neural network computation, opening up many interesting possibilities.