NVIDIA’s chief Jensen Huang in a recent interview said that the era of traditional scaling might be coming to an end. He sees the emergence of what he calls “Hyper Moore’s Law” as the future.
“We’re going to be on some kind of a hyper Moore’s law curve, and I fully hope that we continue to do that,” Huang said, indicating a shift towards new paradigms in computing that extend beyond the conventional model of chip performance improvement.
Historically, Moore’s Law relied on Dennard scaling and Carver Mead’s VLSI (Very Large-Scale Integration) techniques to boost chip performance. However, Huang acknowledges that these methods have reached their limits. “The two fundamental technical pillars were Dennard scaling and Carver Mead’s VLSI scaling. Both of these techniques were rigorous. However, these techniques have really run out of steam,” he explained.
The future, according to Huang, lies in “codesign,” a method where both hardware and software are developed together to ensure optimal performance. “Unless you can modify or change the algorithm to reflect the architecture of the system or change and then change the system to reflect the architecture of the new software and go back and forth—unless you can control both sides of it, you have no hope,” he said.
This codesign approach, which integrates hardware and software, allows for innovations in precision, such as moving from floating point operations like FP64 to FP32, and even as low as FP4, according to Huang.
“The second part of it is the data centre. This is a big part of the equation. Full-stack approaches and innovative scaling strategies will reshape our data centre infrastructure,” he added.
Earlier this year, when NVIDIA released Blackwell, it was said that Moore’s Law is dead. Reflecting on the rapid advancement in computing power, Huang said that in just eight years, NVIDIA has increased computational capacity by a thousandfold, a progress that far exceeds the benchmarks set during the heyday of Moore’s Law.
However, he rued that even after this remarkable growth, the industry’s accelerating demands are far from being met. “In the past eight years, we’ve increased computation by 1000 times, and we have two more years to go. So that puts it into perspective [the fact that] the rate at which we’re advancing computing is insane. And it’s still not fast enough,” said Huang.
