If it rains enough in Thailand, cloud storage could run dry. That’s why Amazon Web Services now looks differently at the supply chain for its massive data center operation, according to VP of Infrastructure Jerry Hunter.
In 2011, an unusually heavy monsoon season led to massive flooding in Thailand, which at the time manufactured nearly half of the world’s supply of hard disk drives (HDDs). The human toll was steep, with more than 800 deaths. The business impact was felt globally, especially in the market for hard drives. Prices soared, shortages developed, and Amazon’s usual vendors were unable to deliver the volume the company sought to support its fast-growing cloud computing platform.
Amazon’s response offers insights into the importance of the global supply chain for data center hardware. Hunter’s presentation at re:Invent Wednesday will be of interest if you’ve been following our recent series on AWS’ Internet infrastructure,
Hunter refers to the Thailand flooding as a “wake-up call.” He and other Amazon executives hopped on a plane and flew to Thailand, hoping to find providers who could supply them with the hard drives they needed. They were startled by what they saw as they toured the landscape of factories that supply the world’s Internet infrastructure.
“When one flood hits half the manufacturing supply, and you don’t have a direct relationship with suppliers, it turns out to be hard to get what you need,” said Hunter. “We want to understand our supply chain.”
When he says “understand,” Hunter means controlling as much of the supply chain as possible. The company has built the relationships to do that, and since 2012 has been building its own servers, storage and networking equipment. In his talk at AWS re:Invent in Las Vegas, Hunter spoke about how the experience in Thailand motivated Amazon to examine its supply chain.
“Just about everyone relied upon parts from Thailand,” said Hunter. “These facilities were under water. I’m not talking about 6 inches of water, either. I’m talking about several feet of water.
Hunter and his team spent time working directly with manufacturers. “We were able to get what we needed through these direct relationships,” he said. “The real lesson we learned is that disasters happen.”
Amazon Web Services VP Jerry Hunter speaks about “Innovation at Scale” at the AWS ReInvent conference Wednesday. (Image: Amazon Web Services)
Amazon’s S3 cloud storage service is almost certainly one of the largest users of storage hardware. At the time of the Thailand crisis, S3 was hosting about 750 billion objects on its service (that number has since moved past a trillion).Other storage providers faced major challenges as well, including Backblaze, which deployed family and friends on “drive farming” missions to buy up all available inventory at local retail stores.
Prices tripled nearly overnight, and it took nearly two years for prices to return to previous levels.
But Amazon came away stronger, says Hunter. The relationships it established with component suppliers and original design manufacturers (ODMs) have brought more control and predictability to Amazon’s infrastructure. This is critical, Hunter explained, because AWS must provide seamless capacity expansion for its customers.
“We’ve turned our supply chain from a potential liability to a strategic advantage,” said Hunter.
Snowball: Big Data Rolling Down the Highway
Amazon added to its hardware offerings yesterday with AWS Snowball, a portable storage appliance that can be used to deliver up to 100 terabytes of data to Amazon’s data centers. In the age of cloud computing, why send hardware back and forth? AWS estimates that if a company committed 100 megabits per second of their total bandwidth to moving data to AWS, it would take about 100 days to transfer 100 TB of data over that connection.
It’s a reminder of the old saying: “Never underestimate the bandwidth of a truck filled with backup tapes driving down the highway.”
AWS ships the Snowball appliance directly to the customer, who plugs it into their local network and uses a software tool to encrypt and transfer data to the appliance, and can use a pre-filled shipping label to ship the drive to AWS and track its progress.
An overview of the Amazon Snowball storage appliance. (Image: Amazon Web Services)
Early adopters include BuzzFeed, which serves news and video more than 200 million unique monthly visitors. “Buzzfeed generates a massive amount of content that we need to preserve and maintain,” said Eugene Ventimiglia, Director of Technical Operations at Buzzfeed. “We’ve been wanting to implement an archive solution based on Amazon Glacier, but our 250TB of archive data was too much data to migrate on individual disk drives, and using high-speed Internet would simply take too long. The release of AWS Snowball completely changes our thinking here as we now have a simple, secure, and cost-effective way to migrate the entire 250TB library to AWS in less than a week.”[clickToTweet tweet=”Never underestimate the bandwidth of a truck filled with backup tapes driving down the highway.” quote=”Never underestimate the bandwidth of a truck filled with backup tapes driving down the highway.”]
AWS also introduced Amazon Kinesis Firehose, is a fully managed service for loading streaming data into AWS via a network connection.
“As customers have realized that their data contains key insights that can lead to competitive advantage, they’re looking to get as much data into AWS as quickly as possible,” said Bill Vass, Vice President, AWS Storage Services. AWS Snowball and Amazon Kinesis Firehose give customers two more important tools to get their data into AWS.”
