Power Optimization Techniques and a PI Perspective

Designers optimize chip power consumption during the floorplanning stage by strategic placement of large 'macro' blocks, changing aspect ratios to minimize interconnect wire lengths, and by creating isolated power and thermal zones. With these techniques, they minimize the energy spent in driving signals across the floorplan and curtail energy consumed in static leakage. These techniques are primarily physical; other electrical techniques such as DVFS (dynamic voltage and frequency scaling) and clock and power gating address active/leakage power consumption separately. The physical and electrical optimization domains remain disconnected: without early information on how a floorplan arrangement impacts cumulative voltage droop, or how electrical activity and power-gating disturbs the power grid / disconnects block capacitance and changes droop and noise spread within the floorplan, there’s little cooperation between them to determine the true low-power optimum or avoid detrimental PI consequences of either.

Power and Optimization efforts in the NVIDIA® H100 Chip

Numerous floorplan optimization techniques were employed in the H100 (considered here as an example of an AI chip) that are briefly summarized ahead. The NVIDIA^® H100^[5] draws over 700 watts TDP (thermal design power) with ~80 billion switching transistors forming 144 Streaming Multiprocessors (SMs), memory, and I/O units. This power consumption in the H100 came about despite advanced architectural enhancements and floorplanning that included large (50MB) L2 Caches structured as localized 'grids' physically separating the SMs in order to minimize energy loss in the interconnect to external high-bandwidth memory (HBM3). Distributed 'power islands' have also been implemented along with asynchronous execution and 'data locality' ensuring data flow within localized clusters of SMs. Dedicated functional units called Tensor Memory Accelerators (TMAs) handle data transfers asynchronously, permitting switching off large computational units (Tensor Cores) when their work is done. Floorplanning in the H100 also considered thermal and IR Drop hotspots, rearranging power-dense units and widening power rails over high-power zones.

Information in the public domain on DVFS design within the SM Cores is minimal, while it's known that they operate at a base clock frequency that is boosted depending upon the platform the H100 is integrated into. Presumably, the clock frequency boost is associated with an operating supply voltage increase. But there is scant information on the determination of exact operating voltages other than that they may be based upon early (fast?) IR Drop estimates. Nevertheless, it's a safe assumption that isolated power islands form part of the H100 SM Cores' power consumption optimization. Overall, the following techniques show up in the H100:

• Interconnect length minimization Done with 'analytical placers' that employ force-directed equations and nonlinear optimization to find an equilibrium where the overall Spring Tension (wire length) is minimized, or with Deep Reinforcement Learning agents that optimize placement based on a cost function that includes wire lengths, congestion, and power density. Minimizing wire lengths directly reduces power consumption in the capacitance and resistance of these wires that has been estimated to have exceeded the power consumed in switching device capacitance.
• Isolated Power Domains and gating Primarily to eliminate leakage-related static power consumption, permitting switching off functional units once their work is done.
• Thermal and IR Drop awareness Leakage and static power consumption increase dramatically with temperature and thermally-aware placement (near "heat sinks" or at "cooler" edges of the chip) helps minimize this component. Thermal hotspots are correlated with high IR Drop regions.
• Dynamic voltage and frequency scaling (DVFS) Power consumption is regulated by the performance (speed) desired; voltage and clock frequency are reduced when high performance is unnecessary.

The Power Integrity (PI) Perspective

A measure of power integrity degradation is included in present-day floorplanning methodology in the form of IR Drop awareness. For instance, EDA software that estimates IR Drop and locates a 'hotspot' often shifts high-switching macros in that region away from each other and/or widens the power delivery network (PDN) rails feeding that region. While these are anticipatable actions, there are a few challenges with such steps — not the least of which is the gross approximation of PI degradation as i · r or some variant of such calculation. Anasim refers to PI degradation^[3] as Cumulative Voltage Droop^[2] that includes the resistive, inductive, and capacitive (RLC) effects of a load current demand upon physical components such as PDN wires and distributed device capacitance. The use of a simplistic equation by current EDA tools not only breaks the laws of nature (c, the speed of light) but also does not capture the effects of local PDN inductance, constructive or destructive wave intereference, and resonance. The measure of PI degradation employed in their optimization, to put it bluntly, is simply inadequate.

Relaxing the Springs (interconnect wire lengths)

Absent a means to rapidly estimate true spatiotemporal^[7] noise in a developing floorplan, current EDA tools assume 'shorter is always better' and change macro placements and aspect ratios to obtain the shortest possible interconnect lengths. This is motivated by interconnect power (CV²f + I²R ) having exceeded estimated switching device power in modern chips. This assumption, they claim, gives them the lowest unit power consumption.

A simple thought experiment employing reverse scaling helps highlight the importance of true PI awareness to floorplan optimization. Consider the old Roots-of-Two Scaling scenario applied to a functional block with switched gate capacitance C_g, switched interconnect capacitance C_int, and power consumption P. Assume that the functional block is as compact as it can be (C = C_g , no additional decoupling gate capacitance) and that 50% of power P is interconnect power. Now let the width and length of the functional block scale by √2, doubling its area, and let the additional area be padded with quiet gate-oxide decoupling capacitance thus effectively doubling charge storage capacitance in the block. As the previously compact floorplan is expanded, power consumed increases because interconnect lengths scale by √2, and rises to P_new = √2×(P/2) + (P/2), where the second term is device power that remains unchanged. Power increases in the reverse-scaled floorplan by a factor of ((√2+1)/2) which increases the average current by the same factor. Now a measure of true PI degradation, the transient droop metric^[4] of ΔI · √(L/C), is changed by a factor of (√2+1)/2)×√(1/(√2 · 2)), since the total charge storage capacitance C_new=2×C (ignoring increased parasitic capacitance) and inductance L scales by (1/√2) due to a greater number of parallel power rails in either direction in the expanded floorplan. The droop metric scales by ~0.72, which is ~(1/√2) — the RECIPROCAL of the reverse scaling factor of √2 ! This estimation follows exactly as our prior simpler derivation of PI degradation with forward scaling discussed in our books [6] and other published articles. Since droop reduces in the expanded floorplan, we are now justified in reducing supply voltage by a factor of 1/√√2, which changes power again to P · (1+√2)/2√2, or by a factor of ~0.85 which, coincidentally, is about the same as the supply voltage scaling factor of 1/√√2 in our experiment.

In studies conducted a quarter century ago [4], it was clear that the droop metric of ΔI · √(L/C) accurately captures voltage excursions in systems with multiple resonant loops that modern integrated circuits tend to be. As we see in the thought experiment above that employs the droop metric, rearranging a floorplan, or "relaxing the springs," can, at some area cost (not double, but no free lunch!), not only improve power integrity, but also reduce power consumption. What we gain by doing so is a more robust implementation that is simultaneously lower in power consumption.

Wider is Always Better (Or is it?)

Employing 'Predictive Thermal and IR Drop mapping,' EDA tools often widen automated power delivery network rails over hotspots and calculate that the correspondingly reduced effective resistance mitigates rail heating and electromigration concerns. While this is valid from i · r and i² · r estimates (though wider rails may carry more current, not just to the hotspot region), it's an incomplete picture from a true-physical, maxwell-accurate CVD perspective. Our work with PDNLab™ has shown in repeated instances that reducing resistance to a very low value reduces the damping factor in power grid oscillations and actually increases CVD when extended too far. This is of particular concern with a technology under adoption, Backside Power Delivery, that decimates PDN resistance and greatly increases the Q factor of the near-die power delivery network.

Power Switches and Gating

EDA tools excel in placing "thousands" of power-gating switches that "smoothly" power down functional units not in use and help eliminate static leakage power. What they may not do is evaluate the true transient droop impact of the inrush currents caused by powering up such "dark" silicon, or estimate the local droop impact of a large chunk of silicon capacitance dynamically detached from the local PDN! We hypothesized on detrimental effects of power gating early in 2008^[1] from a PI perspective. These are experiments easily built and run in PDNLab.

Floorplanning with true PI awareness

We built PDNLab™ for this very purpose — to inform designers, early, of holistic and accurate PI degradation during floorplan development and optimization. Simulations in our previous article (link below) modify the activity in a fixed floorplan through software, and this dynamic reconfiguration, including activity at edges and in the middle of the chip, shows significant variation in cumulative voltage droop. These simulations are completed in minutes on ordinary PCs, not hours or days on powerful machines, and facilitate rapid convergence to an optimal floorplan that has the least PI degradation and therefore the least power consumption as well. We believe PDNLab™ through its true-physical models and Maxwell-accurate simulation bridges the physical and electrical optimization domains for modern digital chips.

← PI Analysis experiments on the H100's SM Core