• About Us
  • Partnership Opportunities
  • Privacy Policy

Data Center Frontier

Charting the future of data centers and cloud computing.

  • Cloud
    • Hyperscale
  • Colo
    • Site Selection
    • Interconnection
  • Energy
    • Sustainability
  • Cooling
  • Technology
    • Internet of Things
    • AI & Machine Learning
    • Edge Computing
    • Virtual Reality
    • Autonomous Cars
    • 5G Wireless
    • Satellites
  • Design
    • Servers
    • Storage
    • Network
  • Voices
  • Podcast
  • White Papers
  • Resources
    • COVID-19
    • Events
    • Newsletter
    • Companies
    • Data Center 101
  • Jobs
You are here: Home / Cloud / Server Farms Writ Large: Super-Sizing the Cloud Campus

Server Farms Writ Large: Super-Sizing the Cloud Campus

By Rich Miller - July 13, 2015 Leave a Comment

Server Farms Writ Large: Super-Sizing the Cloud Campus

Facebook Datacenter Manager Keven McCammon inside one of the massive data halls in the company's campus in North Carolina. (Photo: Rich Miller)

LinkedinTwitterFacebookSubscribe
Mail

FOREST CITY, N.C. – When it comes to data center operations and maintenance, walking is old school. The hallway outside the office at Facebook’s data center is lined with Xooter kick scooters, which allow staffers to zoom across data halls in buildings which are a quarter-mile long – the equivalent of three Wal-Marts lined end-to-end.

The scooters are no longer enough. Facebook also operates a small fleet of electric golf carts, which staffers use to cruise across the company’s 160-acre campus here in western North Carolina, which houses two major data centers and three smaller data storage facilities.

The golf carts are a symbol of the super-sizing of the cloud campus. Data centers have been growing in scale for years, with single facilities spanning up to 600,000 square feet. The cloud builders are now focused on multi-building campuses that concentrate massive amounts of computing power in regional hubs, often in rural areas offering cheap land and power.

These huge campuses are where the cloud lives – the physical manifestation of the Internet. Data center hubs offer economies of scale, enabling companies to rapidly add server capacity and electric power as more workloads shift from in-house IT rooms into these massive server farms.

An aerial view during construction shows the scale of Facebook’s quarter-mile long data center building in Forest City, N.C. (Photo: AerialPhotoPros for Facebook)

An aerial view during construction shows the scale of one of Facebook’s quarter-mile long data center buildings in Forest City, N.C. (Photo: AerialPhotoPros for Facebook)

The cloud campuses are leaving a large mark not only in the IT landscape, but also in the communities where they set up shop.  Politicians welcome them as shining symbols of the new economy, rising in the rusted remnants of factories and industrial zones that were once the linchpin of the local business scene.

As cloud computing gains momentum – driven by social networks, Big Data and the Internet of Things – the scale of these facilities will grow ever larger. It has become routine for companies like Apple, Google, Facebook and Microsoft to invest more than $1 billion in a single location where they place a cloud campus. Just last week, Facebook said it will build a huge new server farm near Dallas, Google filed plans for a mammoth 800,000 square foot data center in Atlanta, and Equinix announced 1 million square feet of new data centers on its campus in Silicon Valley.

This much is certain: As the Internet grows, we will see more cloud campuses in more places, and they’ll be bigger than the ones we see today. At Facebook’s North Carolina operation, we got a look inside one of these operations.

Site Selection

As he pilots the golf cart around the grounds in Forest City, Facebook Datacenter Manager Keven McCammon points out the sites on the campus, a brownfield which previously housed a textile factory. Once it acquired the property, Facebook planted more than 6,800 trees on the campus, and restored a stream that had been interrupted by previous tenants of the site.

One of those tenants has to stay. McCammon pulls over beside a large wooded area, which is preserved as the habitat for the endangered “dwarf leaf” flower, found only in parts of North Carolina and China.

Geography played a large role in Facebook’s site selection process. Forest City is located in an isothermal belt along the eastern edge of the Blue Ridge mountains, which provides a constant temperature. “It creates calm, so we don’t get a lot of dramatic swings in weather, and we get a prevailing wind,” said McCammon.

Free Resource from Data Center Frontier White Paper Library

Edge Cloud
The Benefits of Edge Cloud On-Ramps for CSPs & End Users
Download the new report from EdgeConneX to explore the public cloud's performance, security and cost issues and examine what can be done to resolve them. The white paper also explains the cost, performance, security and other benefits of leveraging edge cloud on-ramps access to access the hyper scale public cloud providers, and examines how CSPs can use this local infrastructure to promote their own connectivity infrastructure. 
We always respect your privacy and we never sell or rent our list to third parties. By downloading this White Paper you are agreeing to our terms of service. You can opt out at any time.

Get this PDF emailed to you.

That wind influenced the placement of the data center buildings, which are lined with louvers that take in fresh air to cool the armada of servers inside. This strategy, known as air-side economization or “free cooling,” can slash energy costs by eliminating the need for chillers, the energy-hungry refrigeration units used in most data center cooling infrastructure. Free cooling is easiest in cool climates such as Oregon, where Facebook built its first data center.

After monitoring its performance in Oregon, Facebook determined it could use fresh-air cooling in warmer climates, including North Carolina. Since commencing construction in 2010, Facebook has loaded two huge data centers with servers and storage. Each 350,000 square foot data center has four data halls, which were filled in phases as the social network added capacity.

facebook-campus

The view from the roof of Building 2 of the Facebook data center campus in North Carolina. Int he distance is one of the “cold storage” facilities, with the initial data center looming behind it. The large open field in the middle of the campus may eventually house a third huge data center. (Photo: Rich Miller)

Adding capacity isn’t “rack and stack” at this scale. Vendors delivering Facebook’s equipment – which is open sourced through the Open Compute Project – will pre-populate the racks with servers,  which are on casters so they can be rolled into place inside the data hall and hooked up to power and network. All connections are on the front of the server to make it easy for techs to repair and swap out equipment, with no need to enter the enclosed “hot aisle” where exhaust heat from the servers creates sweltering conditions.

Facebook is not done building. It has space on the Forest City campus to add a third huge server farm in between the two existing data centers, although that’s a decision for the future. “Our site selection team is always looking at our capacity needs,” said McCammon.

The Heat Map

Data centers are expensive to build. And the Internet can’t grow without them. That’s why data center design and construction has become a crucial skillset for companies whose core business is selling phones, software, gaming systems and search engine ads.

Most of the new Interent titans started out by running servers in leased space in colocation facilities, or perhaps a wholesale data center space. But as they grow to hyperscale status, they inevitably build their own infrastructure, customizing the designs for everything from servers to power distribution to cooling systems and data halls, seeking to optimize their entire IT operations to be as efficient as possible.

Facebook began to develop and own its real estate after determining it could build a custom data center that was 38 percent more energy-efficient than leased space, while saving 24 percent in operating costs.  The company estimates it has saved more than $2 billion in the first three years of building its own infrastructure.

facebook-datacenter-cooling

How Facebook saves money on cooling: fresh air moves through a wall of filters (seen at right) and is guided through louvers and cooling media (at left) before entering the data hall. No chillers! (Photo: Rich Miller)

A big chunk of that savings is realized before the first brick is laid. The data center site selection process can identify ideal locations to save money on power, cooling and land, as well as areas where economic incentives can reduce the taxes a company will pay on the hundreds of millions of dollars it may spend on servers, storage and power equipment.

“Power and connectivity and climate are all factors,” said McCammon. “There’s also the economic development issues. In a campus environment, you get economies of scale and you end up with the ability to attract good talent.”

For service providers, site location decisions are guided by proximity to customers, which often means big cities where real estate, power and connectivity are expensive. When infrastructure grows to Internet scale, site selection priorities shift from local to global, with a different set of criteria. This gave rise to the rural server farm, a concept pioneered by Google and adopted in short order by Yahoo, Microsoft, Apple and Facebook.

Capacity Planning

Huge regional campuses allow cloud builders to address their toughest challenge: capacity planning. With demand for cloud services growing rapidly, it’s difficult to accurately predict how much data center space will be needed, and how soon.

“We have teams that are dedicated to the modeling and capacity planning,” said Joe Kava, Vice President, Data Centers at Google, who called it a “very rigorous” process that looks at the historic growth of Google applications and then runs computer simulations to guide their projections.

Computer models are great, but real estate is another matter. With even small cloud campuses costing up to $200 million, running out of capacity would be an expensive mistake.

The solution: Build bigger. Find the very best places to build a data center, and acquire enough land to accommodate years of growth.

Large campuses also offer benefits for the workforce. Data centers are highly automated operations, and don’t typically hire large amounts of on-site staff. Facebook says that a single data center operations staffer can manage up to 20,000 servers.

facebook-servers

Racks packed with servers inside the Facebook data center in North Carolina. (Photo: Rich Miller)

McCammon says that having multiple data centers in a single location provides an ideal work environment for data center teams.

“It really is about the social side of it,” he says. “When you’re on a campus, you get a feeling of togetherness and being connected.”

McCammon shares this observation over lunch at the cafeteria at Building 3 on the Forest City campus, where about 50 staffers are dining. The cafeteria offers catered breakfast and lunch every weekday, with a buffet, cold drinks and ice cream. Just outside, visible through the ceiling-high windows, a ping pong tournament is underway, even in the 95-degree heat of a June afternoon.

Heat is the primary weather challenge Facebook faces in western North Carolina, a seismically quiet area far enough inland to reduce the impact of major hurricanes. Rutherford County is also in the lowest tier of tornado risk, averaging just one tornado a year since 1950. These factor into site selection, which is often guided by “heat maps” incorporating data on risk from earthquakes, hurricanes, tornadoes, wildfires, sandstorms and even proximity to airports and flight paths.

Facebook and other hyperscale companies serve large geographies from these regional campuses. Most started out with a bi-coastal strategy, with one hub on the West Coast and another on the East Coast.

The need for East Coast sites has provided a huge opportunity to North Carolina, which has used aggressive tax incentives to land cloud campuses for three of the largest Internet titans – Google (Lenoir), Apple (Maiden) and Facebook. The tax breaks prompted discussion about the merit of offering huge financial incentives to some of the world’s largest and wealthiest corporations, especially since data centers bring lots of investment, but a modest number of permanent jobs.

Local Impact

Although residents in some parts of North Carolina may debate tax breaks for server farms, at the local level the direct benefits in jobs and local investment have won over most skeptics. The impact of massive data center campuses on small towns was not immediately clear when they began appearing on the landscape in 2007.

forest-city

The sidewalk of Main Street in Forest City, N.C. is lined with mowers and small tractors, across the street from the hardware store. (Photo: Rich Miller)

In North Carolina, the largest server farms have all landed in small communities. Lenoir, where Google has a large data center campus, is the largest with about 18,000 residents. A few miles south in Maiden, where Apple has built a massive server farm and solar fields, there are 3,300 residents.

Forest City is a town of about 7,500 people located just off Route 74, which makes it the largest municipality in Rutherford County. It’s known for its museums (including the Rutherford County Farm Museum and the Bennett Classics Antique Auto Museum) and its Christmas celebration, which features more than 1 million lights. Forest City boasts a drug store with a soda fountain. A stroll down its charming Main Street, with a tree-lined median and rocking chairs available on the sidewalks, recalls an earlier era in American small town life.

Less than 2 miles from Main Street sits one of the largest concentrations of cutting-edge Internet technology found anywhere on earth. Blinking blue lights fill the rows of racks in the data halls of Facebook’s data center, delivering status updates and photos to more than 1.4 billion users.

blue-light-2

The blue lights from servers illuminate the rows inside the Facebook data center in Forest City, N.C. (Photo: Rich Miller)

What happens when servers outnumber residents? The data center industry’s fondness for secrecy and codenames caused some early tensions in The Dalles, Oregon, a town of 13,000 where Google built its first company-designed data center. Google and its cloud-building brethren have learned from those early mistakes, working hard to get to know their new homes.

“We’ve really integrated ourselves into the community,” said McCammon, a long-time resident of Rutherford County.

The company has made more than $500,000 in grants to community groups, including providing laptops for schools. The hallways of the data center are lined with art from local students, many of whom have had the opportunity to tour the facility and see what goes on behind the scenes in creating their Facebook newsfeed.

[clickToTweet tweet=”Facebook invested $526 million to build & equip the Forest City cloud campus from 2011-13. ” quote=”Facebook invested $526 million to build & equip the Forest City cloud campus from 2011-13.”]

It soon became clear that many Forest City students lacked decent Internet access at home, so Facebook partnered with the town, of Forest City, Rutherford County Schools, and PANGAEA Internet on a pilot program to provide free Wi-Fi access to 100 students in the local school district.

The company has also worked to make training available so that area residents can develop the job skills to work in the data center industry. Facebook worked with nearby Isothermal Community College to create the Data Center Institute. Isothermal instructors worked with Facebook to develop a 12-day certification workshop providing the technical knowledge required of an entry-level data center employee. The course is followed by a two-day simulation and a certification exam. The program has graduated 68 students, nearly all of whom are now working in information technology.

“Quite simply, it gave Isothermal students the amazing opportunity to work closely with cutting-edge technology,” said Walter Dalton, president of Isothermal Community College.

Between 2011 and 2013, Facebook  invested $526 million on capital expenditures for data center infrastructure and construction costs at the Forest City cloud campus. In 2013, the company had $22 million in expenses to operate the data center, which includes the cost of power, employee wages and contractors. During that year, Facebook also paid $198,000 in local property taxes to Forest City and Rutherford County, and paid $530,000 in taxes to the state of North Carolina through a gross receipts tax on electricity ($336,000) and franchise taxes ($194,000).

The construction of the two data centers supported more than 300 full-time construction workers, and an economic impact study suggests the impact of Facebook’s spending helped support nearly 5,000 jobs across the state.

Retrofitting for the Future

Some of the construction workers are back on the job, as Facebook has begun to retrofit Building 1 in Forest City to support the newest generation of technology from the Open Compute Project.

“We’re still evolving our server design to make it even easier to repair,” said McCammon. “Part of the retrofit is our electrical distribution, which has changed since our first generation.” There’s also a new network configuration, with fiber reaching every server.

facebook-nc-retrofit

An area of the Facebook data center in North Carolina is retrofitted with new power and network configurations. Once the retooling is complete, racks with the latest Open Compute server designs will be rolled into place. (Photo: Rich Miller)

The upgrades and server refreshes are done in phases, with several rows of the data hall being overhauled at a time, which allows Facebook’s operations to proceed with no visible impact to users. As servers are taken out of duty, capacity is shifted to elsewhere in the data center. Electrical and network upgrades are installed as the servers in the surrounding rows continue their work. When each phase of work is completed, the row is repopulated with next-generation OCP servers. Full racks on casters are slid into place.

“You just roll stuff in,” says McCammon.

The retrofit of Building 1 will take between six months and nine monts, allowing a data center begun in 2010 to operate with the 2015 edition of Facebook’s technology.

Some of the coming changes in cloud campuses will involve renewable energy. Apple has built a huge solar array to support its data center in Maiden, N.C. and will build a similar facility at its Reno, Nevada server farm. Apple also is planning huge new facilities in Ireland and Denmark that will tap into utility-scale wind energy. Meanwhile, Google says access to renewable energy via the grid connections was a key factor in choosing a former coal plant in Alabama for its newest data center.

On-site solar panels and wind farms would require even larger campuses and more land. In Maiden, Apple owns 100 acres for its data centers, and acquired an additional 200 acres of land for the massive solar arrays to generate 20 megawatts of renewable power.

Pretty soon, traversing the cloud campuses will require more than scooter and golf carts.

“The campus is a living and breathing environment,” said McCammon. “It’s ever-changing.”

LinkedinTwitterFacebookSubscribe
Mail

Tagged With: Facebook

Newsletters

Stay informed: Get our weekly updates!

Are you a new reader? Follow Data Center Frontier on Twitter or Facebook.

About Rich Miller

I write about the places where the Internet lives, telling the story of data centers and the people who build them. I founded Data Center Knowledge, the data center industry's leading news site. Now I'm exploring the future of cloud computing at Data Center Frontier.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

  • Facebook
  • Instagram
  • LinkedIn
  • Pinterest
  • Twitter

Voices of the Industry

How to Simplify Distributed IT Management? Focus on the Tools to Improve Flexibility and Efficiency

How to Simplify Distributed IT Management? Focus on the Tools to Improve Flexibility and Efficiency Russell Senesac, Director of Data Center Strategy at Schneider Electric, explores how to address distributed IT management and the tools available today.

DCF Spotlight

The COVID-19 Crisis and the Data Center Industry

The COVID-19 pandemic presents strategic challenges for the data center and cloud computing sectors. Data Center Frontier provides a one-stop resource for the latest news and analysis for decision-makers navigating this complex new landscape.

An aerial view of major facilities in Data Center Alley in Ashburn, Virginia. (Image: Loudoun County)

Northern Virginia Data Center Market: The Focal Point for Cloud Growth

The Northern Virginia data center market is seeing a surge in supply and an even bigger surge in demand. Data Center Frontier explores trends, stats and future expectations for the No. 1 data center market in the country.

See More Spotlight Features

White Papers

digital transformation

Transforming Your Business Through Data — Creating a Virtuous Cycle for Your High-Value Data

Change is always hard, because it demands that people move out of their comfort zones. Even with the commitment to see it through, the precise nature and requirements can be elusive. This has never been more true than with digital transformation. Get the new paper from Iron Mountain that explores digital transformation in full, including today’s data landscape, current challenges, opportunities and more.

Get this PDF emailed to you.

We always respect your privacy and we never sell or rent our list to third parties. By downloading this White Paper you are agreeing to our terms of service. You can opt out at any time.

Newsletters

Get the Latest News from Data Center Frontier

Job Listings

RSS Job Openings | Peter Kazella and Associates, Inc

  • MEP Commissioning Manager - Salt Lake City, UT
  • Data Center Facility Manager - Montreal, Quebec
  • Mechanical Construction Sales Rep - Ashburn, VA
  • Data Center Facility Manager - Richmond, VA
  • Data Center Facility Manager - San Jose, CA

See More Jobs

Data Center 101

Data Center 101: Mastering the Basics of the Data Center Industry

Data Center 101: Mastering the Basics of the Data Center Industry

Data Center Frontier, in partnership with Open Spectrum, brings our readers a series that provides an introductory guidebook to the ins and outs of the data center and colocation industry. Think power systems, cooling, solutions, data center contracts and more. The Data Center 101 Special Report series is directed to those new to the industry, or those of our readers who need to brush up on the basics.

  • Data Center Power
  • Data Center Cooling
  • Strategies for Data Center Location
  • Data Center Pricing Negotiating
  • Cloud Computing

See More Data center 101 Topics

About Us

Charting the future of data centers and cloud computing. We write about what’s next for the Internet, and the innovations that will take us there. We tell the story of the digital economy through the data center facilities that power cloud computing and the people who build them. Read more ...
  • Facebook
  • LinkedIn
  • Pinterest
  • Twitter

About Our Founder

Data Center Frontier is edited by Rich Miller, the data center industry’s most experienced journalist. For more than 15 years, Rich has profiled the key role played by data centers in the Internet revolution. Meet the DCF team.

TOPICS

  • 5G Wireless
  • Cloud
  • Colo
  • Connected Cars
  • Cooling
  • Cornerstone
  • Coronavirus
  • Design
  • Edge Computing
  • Energy
  • Executive Roundtable
  • Featured
  • Finance
  • Hyperscale
  • Interconnection
  • Internet of Things
  • Machine Learning
  • Network
  • Podcast
  • Servers
  • Site Selection
  • Social Business
  • Special Reports
  • Storage
  • Sustainability
  • Videos
  • Virtual Reality
  • Voices of the Industry
  • White Paper

Copyright Data Center Frontier LLC © 2021