Mapping Netflix: Content Delivery Network Spans 233 Sites

Netflix is renowned for its use of the Amazon Web Services cloud to support its bandwidth-intensive video service. But the company also has plenty of hardware, operating a content delivery network (CDN) of Netflix-designed servers running in data centers around the world.

British researchers have now mapped the Netflix CDN, which houses Open Connect servers in at least 233 locations across six continents, with a heavy concentration in North America and Europe. Scientists at Queen Mary University of London’s School of Electronic Engineering and Computer Science believe they are the first to map Netflix’s physical server distribution around the world.

The findings suggest how Internet infrastructure will evolve to address future capacity, as more content providers adapt to video delivery at scale. With online video use surging to new highs, and more rich content applications in the pipeline (including virtual reality) the network of the future may look a lot like Netflix.

The data on the Netflix CDN reinforces predictions that this surge in content will drive two trends in infrastructure: massive data traffic at the network core, and a growing role for edge data centers to cache content closer to end users. Although there has been lots of buzz about edge computing, the Netflix CDN focuses much of its hardware in Internet connection points (IXPs), the peering and interconnection data centers in major cities around the globe.

The Importance of IXPs

“This study highlights the importance of the still vastly underappreciated IXP ecosystem,” said Professor Steve Uhlig, the senior supervisor of the Queen Mary University team. “Indeed, IXPs are vital to supporting high-speed connectivity in the Internet, especially for large content delivery players such as Netflix.”

The study shows that the USA accounts for a vast majority of Netflix traffic, followed by Mexico, UK, Canada, and Brazil. Their findings confirm the importance of various regions as major Netflix markets, judging by the sheer server deployment in them.

Netflix houses its core infrastructure in the AWS cloud. The company’s video delivery accounts for more than 36 percent of all downstream Internet traffic. To prevent all that data from clogging Internet backbones, Netflix caches copies of popular content in data centers operated by IXPs and Internet Service Providers (ISPs). When a user requests a video file, it can be delivered from a local data center, rather than traversing the entire Internet.

The geography of the Netflix Open Connect presence in major IXPs is not a mystery, as that information is public available through PeeringDB, which outlines the company’s use of both public Internet exchanges (like LINX, AMS-IX and DE-CIX) as well as private peering facilities operated by Equinix, Interxion, Telehouse, EdgeConneX and Telx, among others.

The map of Netflix servers in Internet connection points (IXPs) shows larger concentrations of hardware in major markets. (Image: Queen Mary University of London)

What’s new is the view that Queen Mary University provides of Netflix’ presence in ISP data centers and PoPs (points of presence) around the world. Netflix distributes content through its Open Connect network, partnering with local providers to store cached copies of videos on Open Connect hardware.

The server appliances are designed by Netflix and built by systems integrators Sanmina and Intequus, and use deployment architectures that are tailored for each ISP, according to Netflix.

The UK researchers said they were somewhat surprised by the Netflix network in North America, its largest market. “An overwhelming majority of the servers in this region is located at IXPs,” says the report (PDF). “The number of servers deployed at each IXP location is much larger than the number of servers deployed at ISP locations. We find the limited deployment at ISP locations surprising, given that the USA is the main market for Netflix.

The Politics of Peering

“The absence of deployment inside specific US ISPs (AT&T, Comcast, Time Warner Cable and Verizon) can be explained by the fact that they publicly refused to deploy Netflix servers and instead insisted on signing paid peering contracts with Netflix,” the researchers concluded.

An overview of Netflix Open Connect deployments within the networks of Internet service providers (ISPs) around the world. (Source: Queen Mary University)

By contrast, almost all of the Netflix servers in Canada and Mexico are deployed through ISP networks rather than central exchange points, which are less established in these countries.

The deployment situation in Europe is different. For most countries in Europe, Netflix servers are deployed at only a few locations per country. The notable exception to this scheme is the United Kingdom, where the researchers observed widespread deployment across the whole country, particularly within ISPs.

It’s also worth noting the historic divergence in interconnection practices. A non-profit model is widely used in Europe, and typically spreads exchange operations across multiple data centers in a market. In the U.S. these exchanges are typically hosted by commercial providers, with interconnections focused in a single facility or campus operated by that provider.

A Virtual Expansion In Some Areas

A Netflix Open Connect server appliance. (Photo: Netflix)

While Netflix is expanding globally, that’s not yet true of its infrastructure, the researchers said. “The latest expansion in 130 countries announced by Netflix in January 2016 was only a virtual expansion,” the report concludes. “Obviously, obtaining the necessary licensing agreements with each local movie industry had to be done for the content to be available in these countries. Interestingly, from the server deployment footprint we do not observe this huge expansion. We observe some additional servers deployed in Internet exchanges in Asia, but many countries where Netflix is officially available still remain without any Netflix infrastructure.”

Server deployment in Africa is particularly limited, with the study finding just found eight servers in an IXP in Johannesburg.

The paper, titled “Open Connect Everywhere: A Glimpse at the Internet Ecosystem through the Lens of the Netflix CDN,” is based on research conducted in April and May 2016. Five researchers requested videos from university computers, localizing the requests using a browser extension. They studied the traffic delivered by the servers in each region, highlighting the relative reliance on servers in IXPs and ISPs.

“The different deployment strategies observed are caused by inherent regional differences, forcing Netflix to adapt its strategy to ensure low movie start-up times and to avoid video stalling during playback,” said Timm Boettger, lead author of the paper. “These differences are not only caused by how well an ISP connects its end-users, but also by how well different intermediary networks and ISPs interconnect and exchange traffic. The study is important as it provides an insight into how today’s Internet works.”