How To Advance AI Without New Architectures or Algorithms

When we think about the next big leap in AI, we often think in terms of architecture. A shift away from transformers perhaps, or maybe a way to update model weights on the fly. As exciting as those possibilities are, what we generally consider "AI" is actually a five layer cake (As Jensen Huang puts it: energy, chips, cloud infrastructure, models, and applications). And there are some very pragmatic things we can do today at different levels of this cake to dramatically improve model capabilities without needing new architectures or algorithms.

Take resource utilization for example. Most GPUs today are only 40-60% utilized due to factors like scheduling gaps, fragmentation or mismatched workloads. If we can somehow figure out how to get them to 80-90% utilization, our effective supply of compute nearly doubles without needing any new hardware.

Here are some more bottlenecks and promising avenues on the physical side of building AI that are often overlooked:

(A quick note on the numbers. Some of the figures here are broad strokes. Utilization rates, energy splits, thermal ceilings; these vary enormously with workload, cluster design, and what you are even choosing to measure. I have kept them simple because the argument doesn't hinge on any single figure, it hinges on the direction they all point in.)

It all starts with,

The von Neumann Problem

In 1945, John von Neumann established that compute and memory should be physically separate: data and instructions stay in memory while the actual processing happens somewhere else. This decision, which came to be known as the von Neumann architecture, is what gave rise to general purpose computing and is fundamental to every processor. However, while we were able to scale processing power exponentially over the decades, memory growth has been much, much slower (because scaling compute was way easier). If data has to constantly travel between two places (processor and memory), the movement itself eventually becomes a new bottleneck. This disparity is not new. It has been a central chokepoint across computing science from the beginning (from sorting datasets larger than your RAM to walking large graphs).

AI models today are subject to the same limitations. Autoregressive decoding is fundamentally constrained by memory bandwidth. Each time you want to generate a token, you will need to load the model's entire active weight matrices from memory, which can run to tens or hundreds of gigabytes depending on model size. You can buy more GPUs and add FLOPs/s to your inference system but that will do little to no help. If your arithmetic intensity is below your hardware's ridge point, you will be memory-bound. PERIOD. This is why some companies see diminishing returns on their GPU spending.

This problem is exemplified during training, where petabyte-scale datasets have to be streamed batch after batch through the storage, host and device hierarchy, with the CPU sitting right in the middle of that path. Fetching a piece of data from off-chip DRAM can cost orders of magnitude more energy than the floating point operation that eventually consumes it. To put this in perspective, data movement (shuttling tensors between memory, GPUs, and racks) alone eats up 60-80% of the energy, not the actual math operations training the model; more on this later. There are technologies like Nvidia's GPUDirect Storage that let data flow directly from NVMe drives to GPU memory over PCIe, skipping the CPU hop. But it doesn't remove the memory-bandwidth wall inside the chip.

So what if, instead of taking the data to the processor, we take the processor to the memory? This is called Processing-In-Memory (PIM). Although some companies like Samsung have produced PIM systems inside High Bandwidth Memory stacks, it still remains a niche category because you can only embed so much compute inside the HBM. Fitting a GPU-class ALU inside a memory die challenges density and thermal properties. But it's not too far-fetched for LLM inference, so I'm excited to see how this evolves.

The Networking Layer

Remember we talked about how GPUs are only 40-60% utilized? A large part of that is because distributed training between GPUs demands microsecond-scale latencies to prevent compute stalls. Let me explain.

Within a single DGX server consisting of Hopper GPUs, NVLink 4 gives you about 900 GB/s of bidirectional bandwidth between them. However, the moment you cross the server boundary over InfiniBand, that speed can drop 25-35x. It means that although intra-node communication can be blazing, inter-node is glacially slow by comparison. And this cliff is the single biggest hidden constraint on how large a model you can parallelize efficiently. It determines the topology you need. A minor misconfiguration in the layout can waste millions in GPU-hours, and no software update will fix it.

It's not the only factor to consider though. You see, even with fast memory, each accelerator can only hold so many parameters and activations locally. When a model spills across devices, you pay in communication and synchronization. The bigger your scale, the worse that sync overhead is going to be. Large training runs rely on frequent all-reduce operations to keep every device in lockstep, so as clusters grow, the cost of that lockstep explodes.

This is why Mixture-of-Experts architectures are not just clever mathematically, they're also an adaptation to the hardware since sparse activations mean fewer activations need to cross the inter-node cliff.

So what can we do about it? Well, one solution is to simply make the node bigger, which is what Nvidia has been doing. With the GB200 NVL72 rack, you can tie 72 GPUs together into a single non-blocking NVLink domain, effectively making your entire rack the fast zone. Or, another interesting direction of engineering is replacing copper communications with light. Lightmatter's Passage uses co-packaged optics and photonic interconnects to do just that, aiming to link thousands of GPUs inside a single high-bandwidth domain rather than hundreds. Last month, they joined Nvidia's NVLink Fusion ecosystem, and it's up to our imagination what possibilities this is going to unlock.

Which brings me to the next topic I want to discuss. What if instead of only using light to transport data between dies and packages, we use it to do the actual computing?

Optical Computing

A transformer model is, at its mathematical core, just billions of multiply-accumulate operations. That's what GPUs do all day. Since light, passing through optical media, naturally performs weighted sums via interference; we can leverage its properties to do our matrix-vector products. A digital GPU needs O(n²) operations for a matrix-vector product, but an optical one only costs us the time it takes for light to propagate through the diffractive optical element, effectively O(1) per layer regardless of the matrix size.

Around this time last year, researchers from China and Singapore produced the world's first highly parallel optical architecture called Meteor-1, reaching a theoretical peak of 2,560 TOPS at 50 GHz optical frequency for heavy AI workloads. Only a few months later, a different group from Tsinghua University introduced OFE2, a photonic accelerator that only takes 250.5 picoseconds per matrix-vector op at 12.5 GHz. While Meteor-1 is great for throughput, OFE2 is the champion of latency, suitable for real-time applications.

It's not a magical solution to our dream of reaching singularity though. Neural networks almost always have non-linear activations, yet light is fundamentally linear. Meteor-1 for example, is a specialized accelerator and not a general-purpose processor. It operates alongside conventional electronics and currently, the optical-to-electric conversion costs eat the gains. You also still have to encode your numbers into light and read them back out again, and those steps scale with the size of the problem even when the propagation does not. What we have today are just proofs-of-concept but it's reasonable to bet on them going mainstream in the next 5-10 years. The market for optical computers is tiny at the moment with enormous potential. However, there's also some other kinds of brilliant technology already out there today, hiding in plain sight. The first of them is

Compute Express Link (CXL)

Have you ever wondered why Apple produces extremely powerful yet efficient chips for consumer and pro laptops that give their competitors a run for their money? A large part of that is thanks to the Unified Memory Architecture which pools memory from across their SoCs to be shared between the CPU, GPU and Neural Engine. Without the need to constantly copy data back and forth between separate memory banks, you get reduced latency and enormous power savings. Combine that with ultra-wide memory buses (allowing bandwidth to scale past 500 GB/s to 800 GB/s) and you're basically unbeatable.

CXL is the same idea spiritually. Instead of every GPU being boxed in by whatever HBM happens to sit on its own package, CXL lets you attach a much larger pool of memory that can be shared and tiered across the rack. The HBM stays exactly where it is, fast and close; the CXL pool sits behind it, slower but far bigger and shareable between GPUs and CPUs. If you haven't heard of it already, it might be the most consequential memory standard since DDR4. You see, Apple Silicon is great for on-device computation. Enterprise data-centers need modularity. If you want to add new GPUs or expanders over external slots or cables, CXL allows you to do that.

Released in November 2025, CXL 4.0 doubled its bandwidth to 128 GT/s via PCIe 7.0, allowing massive data throughputs of up to 1.5 TB/s. Even on earlier generations, the results have been striking. One demonstration showed a 6.5x speedup over 100G RDMA for KV-cache offloading and more than 5x over SSD-based KV cache. In a separate demo, Astera Labs showed 3x concurrent LLM instances at higher throughput with 3x lower latency using CXL.

It's not all perfect though. The protocol handling is no cake-walk and power consumption is definitely more than a monolithic on-chip integration. But given the pace of innovation going on around it, there's no doubt that it gets much, much better very soon. Another very interesting type of technology we already have today is the Neural Processing Unit (NPU).

Neural Processing Units (NPU)

Whenever someone talks about scaling AI, it's hard to not talk about GPUs. First introduced by Nvidia in the late 1990s, a Graphics Processing Unit was purpose built for what its name suggests: rendering graphics. The genius of this invention is that instead of executing one piece of information at a time, you could apply the same instruction to thousands of pieces of data (all independent of each other) simultaneously. And since AlexNet leveraged this massive parallelism potential to prove that GPUs can be used to scale neural networks, we never looked back.

However, GPUs might be more general-purpose than we might need. That's where a Neural Processing Unit (NPU) comes in. While GPUs have thousands of programmable cores built to handle everything from matrix-multiplications to ray tracing and texture mapping, NPUs are designed to prioritize data flow and memory hierarchy specifically for the access patterns that neural networks exhibit. And as a result, we have systems that can consume dramatically less power than a GPU for the same inference workload. It's not insignificant given that for a heavily deployed model, inference rather than training ends up dominating lifetime energy; we'll take a deeper look at this in the next section. Google's TPUs are a popular example of NPUs, while Apple's Neural Engine is probably the most widely used among consumers.

Basically, a lot of this advantage comes from minimizing data movement rather than raw FLOP throughput. It's trying to solve the same memory-bandwidth problem we saw earlier, just from a different perspective. Then why doesn't everyone just abandon their GPUs and go after NPUs? For one, programmability. NPUs are great at doing a fixed set of tensor operations extremely efficiently. But if we know anything about deep learning, every few years, we see architectures churn. The kind of architecture we needed for CNNs is completely different from LSTMs, then Transformers, followed by MoEs and diffusion, and now hybrid state-space models.

On an iPhone, it's important for the neural nets to be fast and energy-efficient. You have limited battery, and it's not like the Face ID algorithm is going to change drastically very often. But for other types of inference, when a new attention variant or a new activation function or a new sparsity pattern shows up, frameworks like CUDA are what allow the hardware to adapt. Once an ASIC is burned onto a die, there is no going back. That's how Nvidia grew from a fledgling startup in California to the behemoth it is today, by being general enough to change and resilient enough to continue making amazing chips. As co-founder Chris Malachowsky said at our commencement back in May, "I feel like the over night success, it only took 30 years in the making."

Which makes Quadric a very, very interesting player in this space. For almost any kind of edge AI system today, you need a CPU plus a DSP (Digital Signal Processor) plus a GPU/NPU. Their Chimera GPNPU (General Purpose NPU) is a fully C++ programmable processor that runs all three workloads on a single core, instead of partitioning them three different ways. Each processing element has its own Local Register Memory (LRM) and the programming model's central design goal is to keep data in the LRM for as long as possible. In a way, it tackles the memory-wall problem we keep running into; while using a hybrid von Neumann with 2D SIMD matrix architecture. I don't know about you, but if you like to geek out on engineering, this is one of the coolest ideas out there.

That's all great for inference, but what about training? Tesla's Dojo was somewhat adjacent to the idea of an NPU. It was built as a high-throughput, general purpose processor using a RISC-V-inspired instruction set with custom vector instructions for ML. They deliberately stripped out virtual memory support (no TLBs or page walks) giving them low latency and a small die area; allowing Tesla to optimize for training throughput at massive scale with a custom interconnect. It all comes down to where you're willing to sacrifice flexibility for efficiency.

After all this, there is still one more mega (maybe even the biggest) factor that limits how well we can build AI.

Energy, Heating and Cooling

We all know that training AI models needs a lot of energy. What people generally forget is that training is a one-time cost. Sure, you go back and forth iteratively. But once you build the model, it's built. For any model that actually gets deployed at scale, inference is what really consumes 80-90% of its lifetime energy. Want something more interesting? It's not the actual math operations that need the most energy. Data movement (shuttling tensors between memory, GPUs, racks) alone eats up to 80% of it. They are two very different problems structurally. Training prioritizes throughput (tokens/sec), inference prioritizes latency. Optimizing for one can make the other one bad.

But let's say you have all the compute you need to build the most powerful model you possibly can. You will still need power on the order of what a small nuclear plant produces in order to train it. And it's not just how much power you have that matters, it's also about how efficiently you can deliver it. Your current goes from grid-level AC to facility-level, then to rack-level, then to the Voltage Regulator Modules (VRMs) on your boards, and finally to the sub-1V rails your GPU dies actually run on. At every stage, you shed energy as heat, which needs to be cooled, which itself consumes more power. Some new data centers are rearchitecting around DC power distribution end-to-end. Reducing the number of AC-DC-AC-DC hops cuts conversion losses and reduces cooling load simultaneously.

We know from Landauer's principle that erasing one bit of information requires a minimum of kT ln 2 (about 2.85 × 10⁻²¹) joules at room temperature. Modern logic gates operate roughly 1,000x above this theoretical minimum. In principle there's headroom to reach that floor, but conventional CMOS approaches it slowly and unevenly. The theoretical floor isn't what binds us in practice, though. What binds us is the heat we generate while sitting a thousandfold above it. For air-cooled parts the practical ceiling tends to sit around 300W, and for advanced liquid cooling closer to 1kW, before thermal throttling or reliability failures set in. Three-dimensional chip-stacking (already used in HBM) makes cooling harder because inner layers can't radiate heat efficiently.

Our big problem now isn't whether we can power something, rather it's how fast we can remove heat from it. Traditional air cooling is designed for racks at 5-10 kW. In a standard 42U rack, six AI servers would be consuming around 37 kW. This is why direct liquid cooling and immersion cooling is crucial. And that is also a huge reason why many are turning to two-phase immersion cooling, which uses refrigerants that boil and re-condense directly on chip surfaces. Pretty cool, eh? The infrastructure needed is incompatible with most data centers today, but we can surely give it a try in the new ones.

Finally, there's one last physical property that gets overlooked often. It's that DRAM cells are capacitors, and capacitors leak charge. Every 64 ms, every single cell has to be refreshed, bank by bank, in a process called row refresh. At AI scale (using petabytes of DRAM), this is a constant background tax consuming bandwidth and power even when no data is being accessed. As cells shrink, they leak faster, so refresh frequency must increase, and our penalty worsens.

Alternative memory technologies like MRAM (magnetic), PCM (phase-change based) and RRAM (resistive) do not require refresh because they're non-volatile by design, meaning they can retain their state without needing power. If any of these achieves production-scale cost-parity with DRAM for AI workloads, it would simultaneously reduce power consumption, increase effective bandwidth, and potentially enable in-memory persistence that changes how model states are managed. The gap between promise and production here is sobering though. All three have been a few years away for a couple of decades now.

Some closing thoughts

In case you didn't notice, memory, networking and energy are essentially the same problem. Moving bits dominates all three. Solving one eases the others. Utilization is as important as raw supply, and inference is the long-term cost center. Often, physical constraints are what dictate algorithm design. Many algorithmic innovations are actually responses to hardware limits (e.g. sparsity, MoE, quantization).

Not so long ago, I had the privilege to tour the University of Florida's HiPerGator Data Center. It was really cool to see up close how all these parts come together to make the fastest and most powerful university-owned AI supercomputer in the US. Today, I hope you got a deeper appreciation of everything that goes into building Artificial Intelligence as well. Next time you send a message to ChatGPT or Claude, think about how all these tiny little things come together. Thank you for having the patience to read till the end. Cheers.