As experts lay out the roadmap for AI capabilities, many are suggesting that the automation and efficiencies AI will provide in many industries are not just “step changes,” but “leap changes.” The missing component of this conversation is how important the underlying data center structure is to enable AI. To me, it’s analogous to the hydrogen economy. For years, many experts talked about the great benefits of green hydrogen, but very few people spoke about how to get there, which inspired me to write: Pieces that Need to Fall into Place to Make Green Hydrogen Viable at scale.
For AI to meet the promise, commensurate data center capacity will need to be constructed and come on-line.
A new type of compute is needed to Support AI
Training LLM AI data centers is different than legacy data centers. The computers or servers are larger and heavier, have new components like GPU accelerators and DPUs, use more energy, and give off more heat. The ideal arrangement is in a high-density cluster. All the GPUs, servers, and clusters operate in parallel or as one. This approach is much different than legacy servers all acting independently (serial vs parallel). Adding legacy compute capacity will not meet the AI capacity challenge because it must be AI specific compute, which requires more power and cooling to deploy. Overall, demand for AI compute to train LLMs is far outpacing the AI data center capacity that the industry can build today.
More utility power capacity is needed
Utility companies are highly regulated. They have a long history of large capacity, centralized power plants with turbines fueled by fossil fuels mixed with some hydro and nuclear plants. The processes to apply for a permit and connect to the grid were designed for a small number of large power plants, only a few per decade per country. This inefficient process is resulting in lengthy grid connection queues as the market has moved to adding smaller, more renewable projects to the grid. These queues sometimes get very complicated as companies have applied for too many permits and put projects in the queue at the same time interest rates were spiking. The grid connection pricing escalated very quickly, resulting in the projects becoming nonviable while blocking viable projects from ever getting started.
Improvements needed to manage the data center design and build process
Unfortunately, you can’t just build and drop a data center anywhere you want, and you can’t simply repurpose an unused, vacant facility as a data center. Data centers require permitting, which is sometimes a lengthy process. First, the location must attain the appropriate zoning, then a proposal and subsequent study must be done to determine what sort of demand the data center is going to place on municipal services, particularly power and water. The design also must be approved to comply with air quality and noise ordinances, meet efficiency and sustainability standards, and, lately, esthetic approvals may be required. There are also requirements for network bandwidth and fiber to get the massive data in and out. If that isn’t enough, let’s add in personnel skills and capacity shortages in the design and construction phases, and in the operation phase. Work is being done in cooperation with universities to booster appropriate skill sets, but this will take time to materialize.
Migrate from training LLMs to deploying models to do actual work
Edge computing will see a boost from the need for AI models close to the user and data called edge AI. This will be a year of workflow automation where AI models at the edge will work to make existing systems and processes more efficient for many industries, such as transport, manufacturing, and medical. Different factors determine the generative AI model that needs to be deployed at the edge including the scope, how fast a decision needs to be made, the accuracy of the output, and the output type. For example, if your AI model is operating an emergency vehicle routing, it will need to analyze high-definition videos and traffic patterns to perform real-time operations to clear the traffic. This would probably require an IT network of high speed, networked, edge AI data centers that could process a large amount of data and deliver real-time decisions and predictions. The more cost-effective option is to run compressed or downsized models at the edge.
As AI matures and scales, significant transformation of systems and process in all industries will occur in this latest and potentially great technology wave. This will need to be accompanied by the scaling of data center to capacity to enable AI. To read about how data centers will need to transform, check out our White Paper, “The AI Disruption: Challenges and Guidance for Data Center Design."
