Summer of AI: Hyperscale, Colocation Data Center Infrastructure Focus Tilts Slightly Away From Cloud

Assembled data center industry analysis emerging over the course of the summer reveals a certain trend in data center infrastructure capital expenditures, discernible over both short- and long-term estimates, where AI capex is shown encroaching just a bit on the growth of the traditional cloud, particularly among the large hyperscalers.

Half trillion dollar AI investment has hidden cost

Notable research from Dell'Oro Group released in July predicts that AI infrastructure investments will raise data center capital expenditures to over $500 billion by 2027. But in the wake of this half trillion dollar burst of investment fueled by AI, the researcher cautioned that near-term cloud and enterprise capex growth is expected to decelerate "as the market undergoes digestion," in the analyst's phrasing.

Dell'Oro Senior Research Director Baron Fung conceded that, in the firm's estimation, despite near-term data center capex growth experiencing some resistance as major cloud service providers and enterprises optimize their infrastructure, forthcoming technology transitions will stimulate long-term growth.

Fung explained, “Most notably, the hyperscale cloud service providers will prioritize their investments toward accelerated systems for AI applications for both their public cloud platform and SaaS offerings. We will see continuous optimization across the entire data center stack, with the deployment of next-generation servers featuring high-core counts and deeper memory that are attached to next-generation networks. Meanwhile, the rest of the market will invest in accelerated systems more selectively, with most enterprises adopting a hybrid cloud strategy.”

Dell'Oro's July 2023 Data Center IT Capex 5-Year Forecast Report ultimately projects worldwide data center capex to grow 15% by 2027. The report states that over 20% of global server deployments in 2027 "may be accelerated."

The full report forecasts data center and server capex and server unit shipments for categories and customer segments including Top 4 US Cloud, Top 4 China Cloud, Top 3 Tier 2 Cloud, Rest-of-Cloud, Telco, and Enterprise. The report also penetrates breakout markets for data center infrastructure equipment, along with sub-segments for servers, storage systems, and other data center equipment revenues.

Interestingly, Dell'Oro noted it has trimmed its edge computing forecast, explaining that "the ecosystem and compelling use cases have been slow to materialize."

Counterpoint confirms AI driving immediate data center capex, amid otherwise cautious spend

Viewed from an immediate global perspective, the industry researchers at Counterpoint, in data also released in July, describe how AI is driving cloud service providers' capex this year, amid overall spending conditions the analyst characterized as "cautious."

Counterpoint states that cloud service providers’ capex is expected to grow by about 8% YoY in 2023 due to investments in AI and networking equipment.

"Microsoft and Amazon are among the highest spenders as they invest in data center development," notes the analyst, who assessed that Microsoft will spend over 13% of its capex on AI infrastructure.

The analyst expects that global cloud service providers will grow capex by an estimated 7.8% YoY in 2023.

According to the latest research from Counterpoint’s Cloud Service, higher debt costs, enterprise spending cuts and muted cloud revenue growth are all factors impacting infrastructure spend in data centers, as compared to 2022.

Counterpoint estimates that Microsoft will spend proportionally the most on AI-related infrastructure, with 13.3% of its capex directed towards AI, followed by Google at approximately 6.8% of its capex spend. The analyst recalled that Microsoft has already announced its intention to integrate AI within its existing suite of products.

Commenting on the large cloud service providers’ 2023 plans, Counterpoint Senior Research Analyst Akshara Bassi explained, “Hyperscalers are increasingly focusing on ramping up their AI infrastructure in data centers to cater to the demand for training proprietary AI models, launching native B2C generative AI user applications, and expanding AIaaS (Artificial Intelligence-as-a-Service) product offerings."

According to Counterpoint’s estimates, around 35% of total cloud capex for 2023 is earmarked for IT infrastructure, including servers and networking equipment, compared to 32% in 2022.

The analyst notes that AI infrastructure can be 10 to 30 times more expensive than traditional. general-purpose data center IT infrastructure.

Counterpoint assessed that in 2023, Microsoft and Amazon (AWS) will account for 45% of total data center development capex, with US-based hyperscalers expected to contribute to 91.9% of overall global capex in 2023.

Interestingly, the firm's global data reveals that Chinese hyperscalers’ capex is decreasing due to these companies' decreasing cloud revenues and inability to access Nvidia’s GPU chips.

The analyst stated that, "Chinese hyperscalers are spending less due to slower growth in cloud revenues amid a weak economy and difficulties in acquiring the latest Nvidia GPU chips for AI due to US bans."

Counterpoint further speculates that "the scaled-down version – A800 of the flagship A100/H100 chips – that Nvidia has been supplying to Chinese players may also come under the purview of the ban, further reducing access to AI silicon for Chinese hyperscalers."

The firm's current outlook also observes that though Chinese players are investing a larger portion of their spends towards AI, the amount is significantly less than that of the US counterparts due to a lower overall capex.

Is Nvidia tilting the playing field for specialized cloud infrastructure?

If Nvidia can tilt the table away from cloud hyperscalers in China, it may be trying to tilt it toward certain hyperscalers within America. On this front, 'Exhibit A' might be CoreWeave.

If there's any current data center operation that knows about large-scale investment in AI data center infrastructure, it's the former crypto mining specialist, who has grown up in a big way since its relatively humble beginnings with GPUs assembled on a pool table in lower Manhattan. We can't talk about the Summer of AI without talking about CoreWeave's many coups this year, and also its relationship with Nvidia.

Earlier in August, CoreWeave announced its securing of a $2.3 billion debt financing facility, led by investors Magnetar Capital and Blackstone, with strategic participation from Coatue, DigitalBridge Credit, and funds and accounts managed by BlackRock, PIMCO, and Carlyle, all to meet surging demand and ongoing expansion of specialized cloud infrastructure to power AI computing.

That's a lot of investment, coming on the heels of CoreWeave's announcement in April that it had raised $221 million in Series B funding, led by Magnetar Capital, with contributions from Nvidia and others, along with its securing, one month later, of $200 million in Series B extension funding, also led by Magnetar Capital.

In June came an eye-widening CNBC report that Microsoft had signed a deal to leverage some AI computing power from the Nvidia-backed CoreWeave - its competitor - which sources in the story estimated could be worth billions of dollars over multiple years. This news was nearly contiguous with reporting from The Information that Nvidia appears to closing in on a deal (which sources estimated could total $300 million in new capital) to take an equity stake in chip designer Lambda Labs, a startup competitive with AWS and other major cloud providers in renting servers with Nvidia chips.

The Information report notes that Nvidia has also become one of the biggest venture capital investors cloud and AI software startups, a key category of customers who need its chips. In additional reporting this summer, The Information described how CoreWeave is being given preferential treatment by Nvidia over CoreWeave's cloud rivals.

In a recent email to this editor, DCF Editor at Large Rich Miller opined on this topic, "This is smart because it provides ready infrastructure for Nvidia’s in-house cloud services that should be a cheaper alternative to AWS and other cloud platforms. This...makes it pretty clear that Nvidia has ambitions to control chunks of cloud infrastructure."

"Nvidia’s role in the AI chip supply chain is another interesting wrinkle," added Miller. "There’s no sign yet that they are wielding their market position to tilt the playing field, but their ability to invest in infrastructure could become more interesting over time."

In July, CoreWeave unveiled what it contends is "the world's fastest AI supercomputer," built in partnership with Nvidia, measured by an industry standard benchmark test called the MLPerf. CoreWeave in a press release noted that its publicly available supercomputing infrastructure trained the new MLPerf GPT-3 175B large language model (LLM) in under 11 minutes, which the company stated was "more than 29x faster than the next best competitor and 4x larger than the next best competitor."

But the set piece CoreWeave announcement for any article about AI demand fueling the spend on hyperscale and cloud data center infrastructure must be the company's July news that it will pump $1.6 billion in property improvements into its new data center in Plano, Texas.

As a key expansion to CoreWeave's nationwide footprint of campuses - a fleet to be comprised of 14 data centers in place by the end of 2023, the company anticipates - the 454,421 SF Plano facility located at 1000 Coit Road "will help meet the unprecedented demand for high-performance cloud solutions for artificial intelligence, machine learning, pixel streaming and other emerging technologies that CoreWeave is uniquely positioned to deliver," said CoreWeave's CEO, Michael Intrator.

As reported by DCD, CoreWeave's Plano site is publicly listed as having been owned and operated by Lincoln Property Co.’s Lincoln Rackhouse, and was formerly the North American campus for hoary French telecom giant Alcatel. Reporting in the Plano Star Courier further described how the Plano City Council voted unanimously to pass a tax rebate agreement for at least $1.6 billion used to make building improvements at the site over the next two years; whereby, each year, CoreWeave must show the city that it designated at least $800 million to property improvements to its building to receive the rebate.

Michael Intrator, CoreWeave's CEO, this summer joined CNBC's 'Closing Bell Overtime' to talk about the AI boom and how CoreWeave has raised $2.3 billion in debt collateralized by Nvidia chips.

DCF reported previously this summer on the limitations placed on leasing and building by data center power availability contraints, confirming that the demand for data center development is being limited by the availability of power. The question of at what point the availability of power might ever impact the trajectory of data centers' capex for cloud infrastructure remains open.

Shifts in the clouds

Seemingly echoing the forecasts by Dell'Oro and Counterpoint, for its part, in data announed this month, Synergy Research Group reports that Q2 enterprise spending on cloud infrastructure services was close to $65 billion globally, up $10 billion from the second quarter of last year.

The networking and telecoms market intelligence and analytics firm noted that this is the third consecutive quarter in which the cloud market grew by $10 billion from the same time last year. However, the analyst demonstrated that the consecutive quarters of spending growth are somewhat deceiving, belied by the market's YoY growth rate being assessed at 18% in Q2, down 19% from the previous quarter, and down 20% from the fourth quarter of 2022.

Also indicative of a leveling trend amid apparent growth, Synergy notes that Q2 enterprise spending on cloud infrastructure services was seen rising by only a moderate 3% from Q1, similar to the quarter-on-quarter growth rate seen in Q2 of last year. Paradoxically, while growth rates are coming down, "in absolute terms the quarterly market keeps on growing by $10 billion from last year," summarized the researcher.

Highlighting the market's overall resiliency, Synergy's analysis admits that "the current economic climate has crimped some growth in cloud spending, but the market continues to expand at a healthy rate despite those short-term challenges." Notwithstanding, the analyst concludes that the ongoing nudge-down in the growth rate of global cloud spending is driven by "macroeconomic pressures, belt-tightening by enterprises, local market issues in China, "and, above all else, the law of large numbers," as reckoned by a Synergy press release.

Among the largest cloud providers, Synergy assessed that Google and Microsoft had the strongest year-on-year growth numbers, resulting in both increasing their worldwide market share by a percentage point from the second quarter of last year. Synergy noted that Q2 worldwide market shares for both companies were 22% and 11% respectively. Meanwhile, the analyst stated that market leader Amazon "stayed within its long-standing market share band of 32-34%, though towards the bottom end of that range."

In aggregate, Synergy discerned the three cloud giants as accounting for 65% of the worldwide cloud services market. Among tier two cloud providers, Synergy stated those with the highest YoY growth rates included Oracle, Snowflake, MongoDB, VMware, Huawei and China Telecom.

With most of the major cloud providers having now released their earnings data for Q2, current Synergy estimates place quarterly cloud infrastructure service revenues,including IaaS, PaaS and hosted private cloud services, at $64.8 billion, with trailing twelve-month revenues reaching $247 billion.

The analyst said that public IaaS and PaaS services account for the bulk of the market, and that those grew by 19% in Q2. Synergy said that the dominance of the major cloud providers "is even more pronounced in public cloud, where the top three control 72% of the market," as stated in a Synergy press release.

Echoing the other analysts' findings, Synergy concludes that geographically, the cloud market continues to grow strongly in all regions of the world.

"When measured in local currencies., the APAC region had the strongest growth, with India, China, Australia and South Korea all growing by well over 20% year over year," stated the analyst, who further added that "with the Chinese market potentially returning to somewhat more normal circumstances, many economic pressures easing, and enterprises having rationalized historic cloud usage and spending," it expects future cloud growth rates "to remain buoyant."

Server shipments slip

Reminiscent of Dell'Oro's forecast of a period of middling near-term cloud and enterprise capex growth as deployed infrastructure gets filled out, The Register's Dan Robinson, in an article which also highlights the conundrum described by the Synergy data, describes how Omdia data from the analyst's latest Cloud and Data Center Market Update report shows that global annual server shipments have declined 11% YoY.

In kind, the analyst reduced its annual server shipments forecast by 1 million units to 12.5 million. Robinson speculates that "this seemingly paradoxical situation of increasing demand yet falling server shipments can be explained by a shift in priorities," noting how "average unit prices (AUPs) for servers are rising due to a richer mix of servers configured for AI processing."

Robinson's reporting adds:

Server AUPs are on track for an almost 20 percent year-on-year increase as a result, Omdia claims.

The company said this data backs up the hypothesis made in its previous report that investment in infrastructure for AI model training is now the top priority for data center operators, and the expense associated with procuring the costly high performance server kit for AI is being offset by delaying the refresh of existing server fleets and investment in other new projects.

Having more servers configured with AI co-processors should mean greater demand for physical infrastructure that can support higher power densities within each rack, Omdia said – and it claims there is an uptick in infrastructure enabling higher rack densities to support this."

DCPI is booming

Unsurprisingly in wake of the rate of recent AI infrastructure investments lifting data center capex, in more new data announced this this month, Dell'Oro sees a concurrent burst in AI workloads accelerating the growth trajectory for the Data Center Physical Infrastructure (DCPI) market.

Dell’Oro said its forecast for DCPI market growth has been revised upward, with the segment now forecast to grow from 2022 to 2027 at a 10% CAGR. The analyst stated that DCPI vendor backlogs and softening supply chain constraints are propelling the market's double-digit market growth in 2023.

"As these market dynamics wane throughout the rest of year, AI workloads, including Generative AI, are emerging as new, significant long-term drivers for DCPI market growth," added the analyst.

Lucas Beran, Research Director at Dell’Oro Group, commented, “I have been tracking the DCPI market for nearly a decade, observing annual revenue growth rates in the mid-to-low single digits. There is a change unfolding. The excitement Generative AI is creating at industry events and in my discussions with ecosystem vendors is palpable.”

According to Dell'Oro's Data Center Physical Infrastructure 5-Year July 2023 Forecast Report, the stakes for data center cabinet power distribution and busway and thermal management equiment are expected to grow at the fastest rates during the forecast period.

Beran added, “I am already seeing AI workloads leading to a broad proliferation of accelerated computing infrastructure. This will require investments in next-generation data center physical infrastructure to support new architectures with higher power and thermal management requirements."

Beran said that planning and designing for AI deployments at scale is already underway, while noting that "this won’t materialize in market growth overnight."

By way of example, Beran recalled how deployments of Meta’s recently announced data center architecture changes will occur through 2027.

Notably, Beran added, “One of the key enablers to deploying AI infrastructure is liquid cooling. As we’ve seen in the High Performance Computing industry, air cooling isn’t able to support the thermal management requirements of accelerated servers, with cost, performance, and sustainability in mind. This has led to an upward revision of our liquid cooling forecast, now approaching $2 billion by the outer years of our forecast. The mix has also shifted towards more Direct Liquid Cooling (DLC) based on early momentum in ecosystem support,” he concluded.

Digital Realty and AI have entered the building

As the summer's marquee (not to say "living, breathing") example and confirmation of the AI capex trend in data center infrastructure, comes Digital Realty's announcement this month of its launch of a new, global high-density colocation offering to address data and AI growth challenges.

The firm announced the availability of the new high-density colocation services across its global PlatformDIGITAL data center platform.

With high-performance computing (HPC)-ready infrastructure configurations billed as "extensible and sustainable" and able to "scale up and out with flexible and sustainable consumption models," Digital Realty said the new high-density colocation services will "enable businesses to overcome the processing and proximity challenges associated with the unstructured and exponential growth of data, as well as the use of artificial intelligence (AI)."

Supporting high-density workloads of up to 70 kW per rack, the new service offering utilizes innovative Air-Assisted Liquid Cooling (AALC) technologies. Digital Realty said the new offering is supported in 28 markets across three global regions, including North America, EMEA and Asia-Pacific.

Notably, a press release stated that in partnership with Digital Realty, Avnet Integrated will be acting as a point of cooling engagement in the new launch, offering end-to-end services and virtual platforms for the evaluation, design, deployment, and maintenance of liquid-cooled technologies across Digital Realty's PlatformDIGITAL

"Undoubtedly AI, HPC and video streaming workloads are driving increased demand for capital-intensive, dense computing solutions. To effectively implement these solutions at scale, customers require facilities and services that not only support seamless deployment but also maximize capital utilization," observed Nicole Enright, President of Avnet Integrated Solutions.

Enright added, "By leveraging PlatformDIGITAL, Digital Realty's global data center platform where companies, technologies and data come together to innovate and grow, customers can capitalize on a comprehensive ecosystem of connected data communities that facilitate secure, low-latency connections and high-speed data transfer to the businesses that matter most, enabling them to get the most out of their high-density colocation deployments."

In the following clip, Digital Realty's President and CEO Andy Power kicks off the firm's MarketplaceLIVE 2023 event in July. A preceding video illustrates Digital Realty’s key role in what the images depict as "the dynamic, unpredictable, and infinitely promising world of digital interconnectivity," as stated by the company.

In the video, Digital Realty's Power expands on the firm's MarketplaceLIVE 2023 theme for this year, which was 'The Crossroads of the Digital World.'

Scott Mills, Senior Vice President Engineering and Customer Solutions at Digital Realty, added, "We recognize the data management challenges that companies are facing, where the exponential growth in data can hinder performance and innovation. As a trusted infrastructure partner, Digital Realty is committed to addressing these challenges by leveraging our longstanding experience in delivering high density infrastructure solutions."

Mills continued, "While we already support hundreds of high-performance computing deployments across our portfolio, we've listened to our customers and taken it a step further, providing standardized configurations and ultra-high-power densities. This offering will enable businesses to unlock new possibilities, quickly deploy high performance infrastructure on a global platform, and achieve optimal performance in the age of AI."

As visually depicted in the trade show video cited above, the wording of Digital Realty's announcement ringing in the AI-oriented colocation services launch nicely sums up the AI imperative for data center infrastructure capex, describing how:

"In today's digital landscape, businesses face the ever-increasing challenge of managing and harnessing the power of vast amounts of data. This rapid data expansion presents numerous challenges including performance bottlenecks, compliance complexities, and scalability limitations. Businesses are increasingly turning to emerging technologies like AI to analyze and process this data more effectively. These new technologies consume large amounts of power, which require specialized infrastructure to support them, providing high-power densities, while safely and efficiently regulating the temperature to ensure performance is not impaired."

In testimonial to the new Digital Realty AI offering, Dave Driggers, Chief Technology Officer, Cirrascale Cloud Services, explained, "As Cirrascale continues to deploy the latest AI accelerators for its AI Innovation Cloud, we're relying on global data center providers like Digital Realty to supply high-density colocation services that can support the massive per rack kilowatt demands that the next generation of accelerators require. By supporting up to 70 kilowatts per rack, Digital Realty is enabling us to be able to quickly deploy our infrastructure in a dense environment without fear of overheating, throttling, or damage to our high-density servers."

Courtney Munroe, Research Vice President of IDC, said the analyst is forecasting data consumption to grow at a rate of over 20% annually in coming years. IDC is forecasting that enterprise applications stimulated by the increasing adoption of AI capabilities will drive a significant portion of that data consumption.

"Colocation providers such as Digital Realty are central to the digital ecosystem for managing data workflows," concluded Monroe. "As a global leader in the colocation segment, Digital Realty is now ideally positioned to cater to these burgeoning high-density colocation requirements of enterprises as they implement and leverage AI capabilities."