Alternative Compute: Brain‑Inspired, Optical and Neuromorphic Insights

The Alternative Compute Club began with a thought-provoking question: if one could ask super-intelligent aliens a single question, what would it be? While some might inquire about power solutions like Dyson spheres or nuclear energy, the speaker found the question of how aliens compute their floating-point operations (flops) more intriguing. This led to a discussion on the co-adaptation of hardware and software, particularly hardware architecture and optimizers, in the context of deep learning over the past 14 years.

The Evolution of Deep Learning Hardware

Around 2012, with the advent of AlexNet, Convolutional Neural Networks (CNNs) dominated the deep learning landscape until approximately 2019-2020. During this period, NVIDIA, under Jensen Huang, was primarily focused on maximizing compute per joule (flops per joule), especially for CNNs, which were not memory or bandwidth-bound. The speaker noted that Jensen's interest in AI seemed to significantly increase around 2020, likely spurred by the demands of GPT-2 and GPT-3. These transformer models required unprecedented memory capacity and bandwidth, shifting the focus from raw compute efficiency to memory capabilities. This led to the development of architectures like Ampere, which prioritized memory capacity and bandwidth, even at the expense of compute efficiency. This shift is largely attributed to the N-squared complexity of attention mechanisms in transformers.

The speaker highlighted a plateau in the improvement of G-flop per joule over the last two years, indicating a need for alternative approaches. Drawing from their PhD research, which focused on understanding the brain and exploring alternative computational methods, the speaker suggested that backpropagation, a cornerstone of deep learning, might be replaced within the next decade.

The Brain as an Alternative Compute Model

The human brain, operating on a mere 20 watts (comparable to a light bulb), can perform complex tasks with incredible efficiency. This stark contrast to current AI systems, which require massive energy, underscores the potential of brain-inspired computing. The speaker pointed out that while there's a growing bifurcation of chips into training and inference categories, most still adhere to the transformer architecture, often incorporating large amounts of SRAM, which may not always be necessary.

A key argument against the brain using backpropagation is its feed-forward nature. The idea of neurons firing backward to update synapses, or the "weight transfer problem" where weights would need to be exactly equal in forward and backward passes, seems biologically implausible. Instead, the brain likely utilizes cyclic graphs, chemistry, and precise timings. Another crucial aspect of brain function is massive inhibition, which allows cortical columns (assemblies of neurons) to learn independently.

Promising Alternative Computing Paradigms

The speaker introduced several promising alternative computing paradigms:

• Brain-inspired computing: Leveraging the 4.5 billion years of evolutionary optimization of the brain.
• Optical computing: Utilizing light for computation. The speaker argued that since data centers already communicate in light, performing the entire computation in light would be instantaneous and consume minimal power, potentially what aliens are already doing.
• Neural computing: This is a broad category, and the speaker's own work falls into this.
• Neuromorphics: Mimicking the brain's analog circuitry.

Challenges and Opportunities in Optical Computing

Ilker, a PhD from EPFL, elaborated on optical computing. He highlighted that photons offer significantly lower loss and higher bandwidth (10,000 times) compared to electrons, making them ideal for data transmission. The convergence of AI's tolerance for lower bit precision and optical communication's ability to support higher bit depths creates an interface for optical computing.

Optical computing offers massive parallelization because photons do not interact with each other. In terms of energy consumption, electronic matrix-vector multiplications scale with N-squared, while optical methods, by modulating light passively, can achieve N-scaling.

Despite these advantages, optical computing faces challenges:

• Digital-to-analog conversion: Converting digital data from electronic memories to analog optical signals and back incurs significant energy costs.
• Calibration and programming: Maintaining precision and programming optical devices requires active energy expenditure.
• Nonlinear activation functions: Passive optics excel at linear operations, but nonlinearities, crucial for deep learning, are difficult to implement optically and often require complex mechanisms.

Ilker presented a study where they programmed light propagation for diffusion-based image generation. Their approach involved using an application-specific, passive optical device to perform denoising steps in a diffusion model. By iterating this optical system, they could generate clean images from random pixel distributions. They achieved this by modulating the phase of light through diffractive layers. To address the challenge of changing weights in optics, they divided the diffusion process into fixed-parameter subunits, allowing for repeated use of the same optical device. While currently demonstrated on small datasets like MNIST, the system showed similar power-law scaling behavior to digital neural networks, suggesting potential for scaling. Their small optical model demonstrated a significant energy advantage over similarly performing digital models on GPUs, assuming passive weights.

A key takeaway from Ilker's presentation was the importance of co-designing hardware and algorithms for optical computing to maximize its advantages. The next milestone is to demonstrate an end-to-end system at a billion-parameter scale to validate energy and latency benefits.

Neuromorphic Computing: Borrowing from the Brain

Alok, a venture capitalist and former semiconductor researcher, discussed neuromorphic computing. He emphasized the brain's miraculous capabilities: language, reasoning, creativity, multimodality, real-time processing, continuous learning, and efficient learning from few examples, all while consuming only 20 watts.

Carver Mead coined the term "neuromorphics" in the late 1980s, aiming to design circuits inspired by brain function. Key principles of neuromorphic computing include:

• Intertwined memory and computation: Connections (synapses) in the brain have memory (weights) and directly participate in computation.
• Event-driven communication: Information is transmitted via "spikes," which are quasi-digital and quasi-analog signals.
• Adaptive structure and function: The brain continuously rewires itself based on experience.

Alok described a toy model of a spiking neuron as a leaky capacitor that fires a digital spike when charge accumulates above a threshold. This mixed-signal behavior is unique to the brain.

While modern neural networks are "brain-inspired" in their use of interconnected neurons and adaptive connection strengths, many core concepts like backpropagation, transformer attention, and large-scale training on digital hardware do not directly map to brain function. Alok argued that the brain is the ultimate hardware-software co-design, and any attempt to apply its principles to other substrates must be careful to leverage the native strengths of that substrate.

Current neuromorphic research focuses on:

• Energy efficiency: Inspired by the brain's low power consumption, researchers are exploring co-mingling memory and computation (e.g., SRAM and logic) and analog computing for training and inference.
• Edge computing: For real-time sensing and adaptation, spiking neural networks and on-device learning are being investigated. Intel's Loihi chip is an example of this.
• Exploiting new physics: Utilizing exotic electronic devices like resistive memories, memristors, and coupled oscillators, which exhibit analog and nonlinear properties, to build new computational models.

Alok concluded that neuromorphic computing is still largely in the R&D phase, seeking to find the right match between biological inspiration and physical implementation. He noted that like early neural networks, neuromorphics might require a long gestation period before becoming widely useful.

Brain-Inspired Computing: Human Brain Cells Playing Doom

Sean, CEO of Parasma, presented his work on using human brain cells to play Doom, in collaboration with Cortical Labs. His approach treats the brain as a dynamical system, focusing on input-output relationships rather than forcing it to perform innate calculations like matrix multiplications.

The process involves:

1. Encoding: Game state (ammo, health, screen image) is encoded into electrical stimuli (channel, amplitude, frequency) for a 2D multi-electrode array.
2. Spiking: The neural culture processes these stimuli and generates spikes.
3. Decoding: These spikes are decoded into specific actions in the game (strafing, turning, attacking).

Previous work involved brain cells playing Pong, where encoding and decoding could be hardcoded due to the game's simplicity. For Doom, with its 3D environment and larger action space, end-to-end learning of encoding and decoding is necessary.

Sean's team used a closed-loop Proximal Policy Optimization (PPO) architecture. The encoder network, comprising a CNN and MLP, outputs stimulation frequencies and amplitudes for each channel. Since gradients cannot be propagated through the biological neural culture, stochastic stimulation (beta sampling) is used to find optimal parameters. This also contributes to policy learning for the cells, as the culture changes with stimulation.

A linear readout decoder was used, intentionally undersized and with zero bias, to prevent it from overfitting and ensure the brain cells themselves were learning. Overfitting the decoder would allow it to play the game without the cells actually learning.

Feedback to the cells is crucial. Good and bad feedback are provided through synchronous and asynchronous stimulation, respectively, based on the free energy principle. Asynchronous stimulation is believed to be something the brain avoids. To address the issue of constant negative feedback at the start of learning, they scaled feedback based on "surprise" (TD error) from a silicon critic. This allowed for modulated feedback, promoting connection building and minimizing surprise.

Sean acknowledged the scalability challenge, noting that while intelligence can be achieved on cells, distributing this intelligence to billions of people remains an unsolved problem. He emphasized that human intelligence is optimized for survival, not necessarily compute, and that a different form of intelligence might emerge from this biological substrate.