Regional Data Center Clusters Power Amazon’s Cloud

A map of Amazon Web Service’s global infrastructure (Image: AWS)

When Internet platforms go hyperscale, they typically pursue a “bi-coastal” strategy, building one data hub on the East Coast and another on the West Coast. This data center site selection strategy allows a provider to deliver content and services to the continental U.S. with minimal lag, as well as supporting a global audience. The East Coast hub serves content to Europe, while the West Coast hub delivers data to Asia. This configuration also supports business continuity, allowing data to be be backed up to the opposite coast for disaster recovery purposes.

This pattern has been seen among most of the major cloud players, including Google and Facebook, which each built server farms in Oregon and North Carolina to start their global expansion. For Amazon, the initial infrastructure focus was northern Virginia.

Northern Virginia (US East)

While Amazon’s cloud network spans the globe, a huge chunk of its Internet infrastructure is concentrated across Loudoun and Prince William counties in northern Virginia, where the company has 23 data centers.

Why is so much of Amazon’s cloud infrastructure focused on northern Virginia? It’s the busiest Internet intersection, the place where all the networks meet, anchored by the Equinix interconnection hub in Ashburn. Amazon’s initial company-built data centers sprang up in Ashburn and Sterling (the town next door).

An analysis of Amazon’s data centers by Greenpeace identified 23 facilities in northern Virginia, including nine in Sterling and five in Ashburn, as well six in Manassas and two in Chantilly. This clustering of data centers supports Amazon’s focus on low-latency connections between its Availability Zones, allowing customers to spread application instances and data across several isolated data centers to implement failover strategies.

As a result, the US East region is far and away the largest region within Amazon Web Services. This is reflected in the Greenpeace analysis (PDF), which bases its power capacity estimates on state permits for backup generators at Amazon locations.

A research note: Although Greenpeace has done a thorough job locating Amazon’s facilities and permits, it has a mixed track record in estimating data center capacity. In 2012, it “conservatively” estimated that Apple’s North Carolina data center required 100 megawatts of power, while Apple said the correct number was closer to 20 megawatts. Greenpeace’s methodology has improved in recent years (it now reports the Apple NC facility at 23 megawatts) and its 2015 estimates represent design capacity rather than current power consumption – and thus include permits to support future expansion.

What’s not in dispute is that a very large chunk of the AWS cloud is housed in US East. Several of the Availability Zones in the region appear to be full, with two of the five AZs off-limits to new customers. One notable Greenpeace finding is that Amazon is accelerating its expansion in US East.

“Amazon significantly expanded its data center infrastructure in Northern Virginia in 2014,” the group wrote. “Based on our analysis … AWS expanded the capacity of US East by over 200 MW, as much as the total amount of capacity in the rest of the US Availability Zones combined, bringing the total energy demand capacity of AWS facilities in Northern Virginia to 500 MW.”[clickToTweet tweet=”AWS expanded the capacity of its US East region by as much as 200 MW in 2014.” quote=”AWS expanded the capacity of its US East region by as much as 200 MW in 2014.”]

Amazon hopes to add more capacity in Haymarket in Prince William County, but requires an expansion of power infrastructure by Dominion Power, which is facing resistance from community activists over the proposed route of a new power line.

Oregon & California (US-West)

In 2009, Amazon began adding data centers in northern California. Some of this capacity is in Santa Clara, which is the leading data center hub in Silicon Valley, as local utility Silicon Valley Power has slightly lower electricity rates than offered by PSE&G in surrounding towns. As a result, just about all major data center providers have facilities in Santa Clara, offering a density of network providers similar to that seen in Ashburn.

But power and real estate are expensive in Silicon Valley, which is one reason why Amazon is now focusing its West Coast operations in northern Oregon, where it is building data centers in several small towns lining the Columbia River.

Oregon has become an attractive data center destination in recent years, offering a trifecta of advantages:

• A cool climate that, allowing data centers to cool their servers using fresh air instead of power-hungry chillers and air conditioners.
• A favorable tax climate.
• Access to green hydro-electric power from dams on the Columbia River

That hydro power allows Amazon to offer carbon-neutral cloud hosting from its Oregon infrastructure, which is available in both US West and its GovCloud region, which runs atop AWS’ Oregon infrastructure. (It also offer carbon-neutral hosting from Frankfurt, Germany for European customers).

Amazon has built three data center buildings in the Port of Morrow in Boardman, Oregon and has another campus about 20 miles down the road in Umatilla. Two of these facilities feature hybrid designs that combine a traditional building with modular data centers. These pre-fabricated enclosures allowed AWS to accelerate its initial deployment of cloud capacity in Oregon, housing servers while the adjacent, larger data center building was under construction.

Rows of servers inside an Amazon data center (Image: James Hamilton, Amazon Web Services)

Amazon has big expansion plans in the Northwest. AWS hopes to build as many as 15 data centers in Oregon, according to Amazon state public policy manager Eileen Sullivan, who shared the company’s plans with the Senate Committee on Finance and Revenue earlier this year to encourage it to exempt data centers from an expansion of a state tax (they were exempted).

Ohio (Under Construction)

With major beachheads on each coast, Amazon is now building big in the center of the country. This also follows a familiar template: Google, Facebook and Microsoft have all built huge server farms in Iowa, adding capacity to meet demand in the center of the U.S.

Instead of Iowa, AWS has chosen to focus on Ohio, where it will invest as much as $1.1 billion to construct data center campuses in three locations: Dublin, Hilliard and New Albany. Plans filed with these municipalities indicate that each campus will include up to five data centers, each backed by 18 backup generators (2.5 megawatt units).

That suggests power capacities of about 32 MW per facility, near the upper end of the 25 to 30 megawatt design standard for data centers outlined by Amazon’s James Hamilton. By that math, AWS has a potential long-term buildout of between 450 and 480 megawatts of data center capacity – or more than 750,000 physical servers.

In theory, those three campuses should provide lots of runway for growth over time, with Amazon adding data centers for years to come.

Global Cloud Footprint

Amazon began its global expansion in 2007 in Dublin, Ireland, which like Oregon offers an ideal environment for using fresh air to cool data centers, as well as a favorable tax climate and solid pool of IT workers. By 2010, Amazon accounted for a third of all physical servers in Ireland, according to Netcraft. Earlier this year Netcraft found 52,000 web-facing servers in Ireland, more than 75 percent of the country’s total. That number may be low, since it only counts web-facing servers.

A chart from Netcraft showing Amazon’s steady growth in web-facing servers in EMEA.

Amazon’s Dublin operation has expanded to include three Availability Zones (and hence at least three data centers). In 2014, AWS added two data centers in Frankfurt, a leading center for financial computing and network interconnections.

From 2010 to 2012, Amazon has added cloud regions in Sao Paulo, Brazil (3 AZs) to serve South America, and Asia-Pacific regions in Singapore (2 AZs), Tokyo, Japan (3 AZs), Beijing, China (2 AZs) and Sydney, Australia (2 AZs).

One of Amazon’s large new data centers in China is being built in the Ningxia West Cloud Valley development in  Zhongwei, as documented in a blog post by Dave Ohara.

AWS also maintains 40 edge nodes around the world to support its CloudFront content delivery network (CDN), which allows customers to move their content closer to end users. Here’s an overview of those locations (via the AWS Global Infrastructure directory):

• North American Edge Locations: Ashburn, VA (3), Atlanta, GA, Dallas/Fort Worth, TX (2), Hayward, CA, Jacksonville, FL, Los Angeles, CA (2), Miami, FL, New York, NY (3), Newark, NJ, Palo Alto, CA, San Jose, CA, Seattle, WA, South Bend, IN, St. Louis, MO
• South American Edge Locations: Rio de Janeiro, Brazil, São Paulo, Brazil
• Europe/Middle East Edge Locations: Amsterdam, The Netherlands (2), Dublin, Ireland, Frankfurt, Germany (3), London, England (3), Madrid, Spain, Marseille, France, Milan, Italy, Paris, France (2), Stockholm, Sweden, and Warsaw, Poland
• Asia/Pacific Edge Locations: Chennai, India, Hong Kong, China (2), Manila, the Philippines, Melbourne, Australia, Mumbai, India, Osaka, Japan, Seoul, Korea (2), Singapore (2), Sydney, Australia, Taipei, Taiwan, Tokyo, Japan (2)

For more details, see Part One of our series, Inside Amazon’s Cloud Computing Infrastructure.