Server Farms Writ Large: Super-Sizing the Cloud Campus

FOREST CITY, N.C. – When it comes to data center operations and maintenance, walking is old school. The hallway outside the office at Facebook’s data center is lined with Xooter kick scooters, which allow staffers to zoom across data halls in buildings which are a quarter-mile long – the equivalent of three Wal-Marts lined end-to-end.

The scooters are no longer enough. Facebook also operates a small fleet of electric golf carts, which staffers use to cruise across the company’s 160-acre campus here in western North Carolina, which houses two major data centers and three smaller data storage facilities.

The golf carts are a symbol of the super-sizing of the cloud campus. Data centers have been growing in scale for years, with single facilities spanning up to 600,000 square feet. The cloud builders are now focused on multi-building campuses that concentrate massive amounts of computing power in regional hubs, often in rural areas offering cheap land and power.

These huge campuses are where the cloud lives – the physical manifestation of the Internet. Data center hubs offer economies of scale, enabling companies to rapidly add server capacity and electric power as more workloads shift from in-house IT rooms into these massive server farms.

The cloud campuses are leaving a large mark not only in the IT landscape, but also in the communities where they set up shop.  Politicians welcome them as shining symbols of the new economy, rising in the rusted remnants of factories and industrial zones that were once the linchpin of the local business scene.

As cloud computing gains momentum – driven by social networks, Big Data and the Internet of Things – the scale of these facilities will grow ever larger. It has become routine for companies like Apple, Google, Facebook and Microsoft to invest more than $1 billion in a single location where they place a cloud campus. Just last week, Facebook said it will build a huge new server farm near Dallas, Google filed plans for a mammoth 800,000 square foot data center in Atlanta, and Equinix announced 1 million square feet of new data centers on its campus in Silicon Valley.

This much is certain: As the Internet grows, we will see more cloud campuses in more places, and they’ll be bigger than the ones we see today. At Facebook’s North Carolina operation, we got a look inside one of these operations.

Site Selection

As he pilots the golf cart around the grounds in Forest City, Facebook Datacenter Manager Keven McCammon points out the sites on the campus, a brownfield which previously housed a textile factory. Once it acquired the property, Facebook planted more than 6,800 trees on the campus, and restored a stream that had been interrupted by previous tenants of the site.

One of those tenants has to stay. McCammon pulls over beside a large wooded area, which is preserved as the habitat for the endangered “dwarf leaf” flower, found only in parts of North Carolina and China.

Geography played a large role in Facebook’s site selection process. Forest City is located in an isothermal belt along the eastern edge of the Blue Ridge mountains, which provides a constant temperature. “It creates calm, so we don’t get a lot of dramatic swings in weather, and we get a prevailing wind,” said McCammon.

That wind influenced the placement of the data center buildings, which are lined with louvers that take in fresh air to cool the armada of servers inside. This strategy, known as air-side economization or “free cooling,” can slash energy costs by eliminating the need for chillers, the energy-hungry refrigeration units used in most data center cooling infrastructure. Free cooling is easiest in cool climates such as Oregon, where Facebook built its first data center.

After monitoring its performance in Oregon, Facebook determined it could use fresh-air cooling in warmer climates, including North Carolina. Since commencing construction in 2010, Facebook has loaded two huge data centers with servers and storage. Each 350,000 square foot data center has four data halls, which were filled in phases as the social network added capacity.

Adding capacity isn’t “rack and stack” at this scale. Vendors delivering Facebook’s equipment – which is open sourced through the Open Compute Project – will pre-populate the racks with servers,  which are on casters so they can be rolled into place inside the data hall and hooked up to power and network. All connections are on the front of the server to make it easy for techs to repair and swap out equipment, with no need to enter the enclosed “hot aisle” where exhaust heat from the servers creates sweltering conditions.

Facebook is not done building. It has space on the Forest City campus to add a third huge server farm in between the two existing data centers, although that’s a decision for the future. “Our site selection team is always looking at our capacity needs,” said McCammon.

The Heat Map

Data centers are expensive to build. And the Internet can’t grow without them. That’s why data center design and construction has become a crucial skillset for companies whose core business is selling phones, software, gaming systems and search engine ads.

Most of the new Interent titans started out by running servers in leased space in colocation facilities, or perhaps a wholesale data center space. But as they grow to hyperscale status, they inevitably build their own infrastructure, customizing the designs for everything from servers to power distribution to cooling systems and data halls, seeking to optimize their entire IT operations to be as efficient as possible.

Facebook began to develop and own its real estate after determining it could build a custom data center that was 38 percent more energy-efficient than leased space, while saving 24 percent in operating costs.  The company estimates it has saved more than $2 billion in the first three years of building its own infrastructure.

A big chunk of that savings is realized before the first brick is laid. The data center site selection process can identify ideal locations to save money on power, cooling and land, as well as areas where economic incentives can reduce the taxes a company will pay on the hundreds of millions of dollars it may spend on servers, storage and power equipment.

“Power and connectivity and climate are all factors,” said McCammon. “There’s also the economic development issues. In a campus environment, you get economies of scale and you end up with the ability to attract good talent.”

For service providers, site location decisions are guided by proximity to customers, which often means big cities where real estate, power and connectivity are expensive. When infrastructure grows to Internet scale, site selection priorities shift from local to global, with a different set of criteria. This gave rise to the rural server farm, a concept pioneered by Google and adopted in short order by Yahoo, Microsoft, Apple and Facebook.

Capacity Planning

Huge regional campuses allow cloud builders to address their toughest challenge: capacity planning. With demand for cloud services growing rapidly, it’s difficult to accurately predict how much data center space will be needed, and how soon.

“We have teams that are dedicated to the modeling and capacity planning,” said Joe Kava, Vice President, Data Centers at Google, who called it a “very rigorous” process that looks at the historic growth of Google applications and then runs computer simulations to guide their projections.

Computer models are great, but real estate is another matter. With even small cloud campuses costing up to $200 million, running out of capacity would be an expensive mistake.

The solution: Build bigger. Find the very best places to build a data center, and acquire enough land to accommodate years of growth.

Large campuses also offer benefits for the workforce. Data centers are highly automated operations, and don’t typically hire large amounts of on-site staff. Facebook says that a single data center operations staffer can manage up to 20,000 servers.

McCammon says that having multiple data centers in a single location provides an ideal work environment for data center teams.

“It really is about the social side of it,” he says. “When you’re on a campus, you get a feeling of togetherness and being connected.”

McCammon shares this observation over lunch at the cafeteria at Building 3 on the Forest City campus, where about 50 staffers are dining. The cafeteria offers catered breakfast and lunch every weekday, with a buffet, cold drinks and ice cream. Just outside, visible through the ceiling-high windows, a ping pong tournament is underway, even in the 95-degree heat of a June afternoon.

Heat is the primary weather challenge Facebook faces in western North Carolina, a seismically quiet area far enough inland to reduce the impact of major hurricanes. Rutherford County is also in the lowest tier of tornado risk, averaging just one tornado a year since 1950. These factor into site selection, which is often guided by “heat maps” incorporating data on risk from earthquakes, hurricanes, tornadoes, wildfires, sandstorms and even proximity to airports and flight paths.

Facebook and other hyperscale companies serve large geographies from these regional campuses. Most started out with a bi-coastal strategy, with one hub on the West Coast and another on the East Coast.

The need for East Coast sites has provided a huge opportunity to North Carolina, which has used aggressive tax incentives to land cloud campuses for three of the largest Internet titans – Google (Lenoir), Apple (Maiden) and Facebook. The tax breaks prompted discussion about the merit of offering huge financial incentives to some of the world’s largest and wealthiest corporations, especially since data centers bring lots of investment, but a modest number of permanent jobs.

Local Impact

Although residents in some parts of North Carolina may debate tax breaks for server farms, at the local level the direct benefits in jobs and local investment have won over most skeptics. The impact of massive data center campuses on small towns was not immediately clear when they began appearing on the landscape in 2007.

In North Carolina, the largest server farms have all landed in small communities. Lenoir, where Google has a large data center campus, is the largest with about 18,000 residents. A few miles south in Maiden, where Apple has built a massive server farm and solar fields, there are 3,300 residents.

Forest City is a town of about 7,500 people located just off Route 74, which makes it the largest municipality in Rutherford County. It’s known for its museums (including the Rutherford County Farm Museum and the Bennett Classics Antique Auto Museum) and its Christmas celebration, which features more than 1 million lights. Forest City boasts a drug store with a soda fountain. A stroll down its charming Main Street, with a tree-lined median and rocking chairs available on the sidewalks, recalls an earlier era in American small town life.

Less than 2 miles from Main Street sits one of the largest concentrations of cutting-edge Internet technology found anywhere on earth. Blinking blue lights fill the rows of racks in the data halls of Facebook’s data center, delivering status updates and photos to more than 1.4 billion users.

What happens when servers outnumber residents? The data center industry’s fondness for secrecy and codenames caused some early tensions in The Dalles, Oregon, a town of 13,000 where Google built its first company-designed data center. Google and its cloud-building brethren have learned from those early mistakes, working hard to get to know their new homes.

“We’ve really integrated ourselves into the community,” said McCammon, a long-time resident of Rutherford County.

The company has made more than $500,000 in grants to community groups, including providing laptops for schools. The hallways of the data center are lined with art from local students, many of whom have had the opportunity to tour the facility and see what goes on behind the scenes in creating their Facebook newsfeed.

[clickToTweet tweet=”Facebook invested $526 million to build & equip the Forest City cloud campus from 2011-13. ” quote=”Facebook invested $526 million to build & equip the Forest City cloud campus from 2011-13.”]

It soon became clear that many Forest City students lacked decent Internet access at home, so Facebook partnered with the town, of Forest City, Rutherford County Schools, and PANGAEA Internet on a pilot program to provide free Wi-Fi access to 100 students in the local school district.

The company has also worked to make training available so that area residents can develop the job skills to work in the data center industry. Facebook worked with nearby Isothermal Community College to create the Data Center Institute. Isothermal instructors worked with Facebook to develop a 12-day certification workshop providing the technical knowledge required of an entry-level data center employee. The course is followed by a two-day simulation and a certification exam. The program has graduated 68 students, nearly all of whom are now working in information technology.

“Quite simply, it gave Isothermal students the amazing opportunity to work closely with cutting-edge technology,” said Walter Dalton, president of Isothermal Community College.

Between 2011 and 2013, Facebook  invested $526 million on capital expenditures for data center infrastructure and construction costs at the Forest City cloud campus. In 2013, the company had $22 million in expenses to operate the data center, which includes the cost of power, employee wages and contractors. During that year, Facebook also paid $198,000 in local property taxes to Forest City and Rutherford County, and paid $530,000 in taxes to the state of North Carolina through a gross receipts tax on electricity ($336,000) and franchise taxes ($194,000).

The construction of the two data centers supported more than 300 full-time construction workers, and an economic impact study suggests the impact of Facebook’s spending helped support nearly 5,000 jobs across the state.

Retrofitting for the Future

Some of the construction workers are back on the job, as Facebook has begun to retrofit Building 1 in Forest City to support the newest generation of technology from the Open Compute Project.

“We’re still evolving our server design to make it even easier to repair,” said McCammon. “Part of the retrofit is our electrical distribution, which has changed since our first generation.” There’s also a new network configuration, with fiber reaching every server.

The upgrades and server refreshes are done in phases, with several rows of the data hall being overhauled at a time, which allows Facebook’s operations to proceed with no visible impact to users. As servers are taken out of duty, capacity is shifted to elsewhere in the data center. Electrical and network upgrades are installed as the servers in the surrounding rows continue their work. When each phase of work is completed, the row is repopulated with next-generation OCP servers. Full racks on casters are slid into place.

“You just roll stuff in,” says McCammon.

The retrofit of Building 1 will take between six months and nine monts, allowing a data center begun in 2010 to operate with the 2015 edition of Facebook’s technology.

Some of the coming changes in cloud campuses will involve renewable energy. Apple has built a huge solar array to support its data center in Maiden, N.C. and will build a similar facility at its Reno, Nevada server farm. Apple also is planning huge new facilities in Ireland and Denmark that will tap into utility-scale wind energy. Meanwhile, Google says access to renewable energy via the grid connections was a key factor in choosing a former coal plant in Alabama for its newest data center.

On-site solar panels and wind farms would require even larger campuses and more land. In Maiden, Apple owns 100 acres for its data centers, and acquired an additional 200 acres of land for the massive solar arrays to generate 20 megawatts of renewable power.

Pretty soon, traversing the cloud campuses will require more than scooter and golf carts.

“The campus is a living and breathing environment,” said McCammon. “It’s ever-changing.”