Building the Phone Rack for Facebook’s Mobile Lab

July 14, 2016
To scale up its Mobile Device Lab, Facebook built custom racks to house 2,000 smartphones for application testing in its Prineville data center. Here’s a look at the design process for this offbeat challenge.

PRINEVILLE, Ore. – The call may have gone something like this:

HQ: “Hey, we need you guys to build a custom cabinet for us.”

Data Center: “OK. What kind of hardware will run inside this cabinet?”

HQ: “It’s a few dozen mobile phones.”

Data Center: “Wait. What?”

That conversation led to the creation of Facebook’s Mobile Device Lab at its huge data center complex in central Oregon. Facebook uses the lab to test drive revisions to its applications across up to 2,000 different types of smartphones, representing the diverse universe of technology found among Facebook’s 1.6 billion users.

This testing is crucial to maintaining a good user experience. But why does a rack full of phones need to live in one of the world’s most advanced data centers?

Facebook tried doing this testing at its headquarters campus in Menlo Park, Calif., but the task proved unwieldy, as engineers couldn’t create a hardware device testbed that could give it the scale and consistency it was seeking. (See Facebook’s blog post for a description of their initial attempts to create a lab in their Silicon Valley offices).

A Home in the Cold Storage Aisles

Still, it didn’t make sense to house cabinets full of phones in Facebook’s production data centers in Prineville, which run the latest Open Compute in racks that can support up to 12kW of power capacity.

One of the custom racks for Facebook’s Mobile Device Lab, which houses up tom32 smartphones. At right you can see the insulation and other adaptations to the door of the rack. (Photo: Rich Miller)

Fortunately, there was a better option: the Facebook cold storage facility in Prineville, which houses exabytes of rarely accessed photos uploaded by Facebook users. The cold storage facilities are simpler data centers with no generators or UPS (uninterruptible power supply). Emergency backup power isn’t needed because the facility doesn’t serve live production data.

The primary challenge in building racks for the Mobile Device Lab was managing wifi access to simulate how apps will run for end users – who typically don’t hold their phone next to hundreds of other phones when using Facebook.

“If we were to use the same racks for mobile device testing that we use for our servers in our data center, our tests would be unsuccessful because the Wi-Fi signal from one rack would interfere with the Wi-Fi signal from the next rack,” said Antoine Reversat, who demonstrated the lab for media during a tour of the Prineville facility is week. “We needed racks to be well-isolated from their neighbors — ensuring a consistent environment — in order for our tests to be reliable and to have reproducible results.”

To Each His Own Wireless

To solve this problem, the Facebook data center team built custom racks that function as an electromagnetic isolation (EMI) chamber. Each rack contains a wireless access point and an overhead camera. The phones lie on a slightly tilted pegboard so the cameras can record their screens.

There were several considerations in determining how many phones could be housesdin each cabinet. A key factor was the network infrastructure and hardware to monitor the tests.

Each rack holds eight Mac Minis (or four OCP Leopard servers for Android testing) that drive the phones to install, test, and uninstall the applications for Facebook, Instagram and WhatsApp. Each Mac Mini is connected to four iPhones, and each OCP Leopard server is connected to eight Android devices, for a total of 32 phones per rack.

Insulation, and More Insulation

The early prototypes needed some work.

“When we tested our EMI enclosure, we found that it was about as efficient at blocking radio signal as a sieve is at blocking water,” said Reversat.

To turn that sieve into a bucket, Facebook took the following steps:

  • Grounded the EMI enclosure by connecting it to one of the metallic surfaces of the rack itself.
  • Covered the inside seams with copper tape.
  • Added insulation foam inside the rack.
  • Added a 100 dBm power filter (i.e., a filter for power, USB, and ethernet that’s rated to filter out electromagnetic radiation and to ensure that it doesn’t interfere with the Wi-Fi signal inside the enclosure).
  • Added even more insulation foam.

Facebook now has around 60 racks for mobile device testing in one of its cold storage facilities in Prineville. The data center team continues to refine the design, and hopes to densify the current rack design to hold up to 64 phones per rack (versus the current 32).

“There are multiple challenges we’re working through here,” Reversat wrote in his blog post. “For all devices, we have to maintain a 4-foot air gap between the phones and the antenna for the signal to be sufficiently attenuated once it reaches the phones. We don’t have a lot of space to stack the phones if we want to have two shelves instead of one. Additionally, we intend to get larger form-factor phones in the rack next year, which we will need to consider in the redesign.”

Some of the 60 custom racks for Facebook’s Mobile Device Lab inside the cold storage facility at the Prineville, Oregon data center campus. (Photo: Facebook)

About the Author

Rich Miller

I write about the places where the Internet lives, telling the story of data centers and the people who build them. I founded Data Center Knowledge, the data center industry's leading news site. Now I'm exploring the future of cloud computing at Data Center Frontier.

Sponsored Recommendations

Guide to Environmental Sustainability Metrics for Data Centers

Unlock the power of Environmental, Social, and Governance (ESG) reporting in the data center industry with our comprehensive guide, proposing 28 key metrics across five categories...

The AI Disruption: Challenges and Guidance for Data Center Design

From large training clusters to small edge inference servers, AI is becoming a larger percentage of data center workloads. Learn more.

A better approach to boost data center capacity – Supply capacity agreements

Explore a transformative approach to data center capacity planning with insights on supply capacity agreements, addressing the impact of COVID-19, the AI race, and the evolving...

How Modernizing Aging Data Center Infrastructure Improves Sustainability

Explore the path to improved sustainability in data centers by modernizing aging infrastructure, uncovering challenges, three effective approaches, and specific examples outlined...