Building the Phone Rack for Facebook’s Mobile Lab

PRINEVILLE, Ore. – The call may have gone something like this:

HQ: “Hey, we need you guys to build a custom cabinet for us.”

Data Center: “OK. What kind of hardware will run inside this cabinet?”

HQ: “It’s a few dozen mobile phones.”

Data Center: “Wait. What?”

That conversation led to the creation of Facebook’s Mobile Device Lab at its huge data center complex in central Oregon. Facebook uses the lab to test drive revisions to its applications across up to 2,000 different types of smartphones, representing the diverse universe of technology found among Facebook’s 1.6 billion users.

This testing is crucial to maintaining a good user experience. But why does a rack full of phones need to live in one of the world’s most advanced data centers?

Facebook tried doing this testing at its headquarters campus in Menlo Park, Calif., but the task proved unwieldy, as engineers couldn’t create a hardware device testbed that could give it the scale and consistency it was seeking. (See Facebook’s blog post for a description of their initial attempts to create a lab in their Silicon Valley offices).

A Home in the Cold Storage Aisles

Still, it didn’t make sense to house cabinets full of phones in Facebook’s production data centers in Prineville, which run the latest Open Compute in racks that can support up to 12kW of power capacity.

One of the custom racks for Facebook’s Mobile Device Lab, which houses up tom32 smartphones. At right you can see the insulation and other adaptations to the door of the rack. (Photo: Rich Miller)

Fortunately, there was a better option: the Facebook cold storage facility in Prineville, which houses exabytes of rarely accessed photos uploaded by Facebook users. The cold storage facilities are simpler data centers with no generators or UPS (uninterruptible power supply). Emergency backup power isn’t needed because the facility doesn’t serve live production data.

The primary challenge in building racks for the Mobile Device Lab was managing wifi access to simulate how apps will run for end users – who typically don’t hold their phone next to hundreds of other phones when using Facebook.

“If we were to use the same racks for mobile device testing that we use for our servers in our data center, our tests would be unsuccessful because the Wi-Fi signal from one rack would interfere with the Wi-Fi signal from the next rack,” said Antoine Reversat, who demonstrated the lab for media during a tour of the Prineville facility is week. “We needed racks to be well-isolated from their neighbors — ensuring a consistent environment — in order for our tests to be reliable and to have reproducible results.”

To Each His Own Wireless

To solve this problem, the Facebook data center team built custom racks that function as an electromagnetic isolation (EMI) chamber. Each rack contains a wireless access point and an overhead camera. The phones lie on a slightly tilted pegboard so the cameras can record their screens.

There were several considerations in determining how many phones could be housesdin each cabinet. A key factor was the network infrastructure and hardware to monitor the tests.

Each rack holds eight Mac Minis (or four OCP Leopard servers for Android testing) that drive the phones to install, test, and uninstall the applications for Facebook, Instagram and WhatsApp. Each Mac Mini is connected to four iPhones, and each OCP Leopard server is connected to eight Android devices, for a total of 32 phones per rack.

Insulation, and More Insulation

The early prototypes needed some work.

“When we tested our EMI enclosure, we found that it was about as efficient at blocking radio signal as a sieve is at blocking water,” said Reversat.

To turn that sieve into a bucket, Facebook took the following steps:

Grounded the EMI enclosure by connecting it to one of the metallic surfaces of the rack itself.
Covered the inside seams with copper tape.
Added insulation foam inside the rack.
Added a 100 dBm power filter (i.e., a filter for power, USB, and ethernet that’s rated to filter out electromagnetic radiation and to ensure that it doesn’t interfere with the Wi-Fi signal inside the enclosure).
Added even more insulation foam.

Facebook now has around 60 racks for mobile device testing in one of its cold storage facilities in Prineville. The data center team continues to refine the design, and hopes to densify the current rack design to hold up to 64 phones per rack (versus the current 32).

“There are multiple challenges we’re working through here,” Reversat wrote in his blog post. “For all devices, we have to maintain a 4-foot air gap between the phones and the antenna for the signal to be sufficiently attenuated once it reaches the phones. We don’t have a lot of space to stack the phones if we want to have two shelves instead of one. Additionally, we intend to get larger form-factor phones in the rack next year, which we will need to consider in the redesign.”

Some of the 60 custom racks for Facebook’s Mobile Device Lab inside the cold storage facility at the Prineville, Oregon data center campus. (Photo: Facebook)