Spotify Heads To The Google Cloud

Is this the future of company infrastructure?
On the 23 February 2016, Spotify announced their infrastructure move to the Google Cloud Platform. This is a major shift for Spotify who until now have been doing things the traditional way: “buying or leasing data-center space, server hardware and networking gear as close to our customers as possible. This approach has allowed us to give you music instantly, wherever you are in the world.”
In essence, Spotify’s growth rate has become too much to keep up with infrastructure demands, following their expansion into features such as videos. Until now Spotify didn’t feel that cloud services were up to scratch when compared to their own data centers, but this migration to Google’s cloud platform highlights their newfound confidence in the ability of the cloud to support their global service. Their announcement elaborated on how “The storage, compute and network services available from cloud providers are as high quality, high performance and low cost as what the traditional approach provides. This makes the move to the cloud a no-brainer for us. Google, in our experience, has an edge here, but it’s a competitive space and we expect the big players to be battling it out for the foreseeable future.”
Nicholas Harteau, VP of engineering and infrastructure at Spotify, commented in a Wired interview: “Historically we’ve run our own data centres. That’s using space, its buying hardware and equipment but now we’re doing a big push to partner with Google on running the Spotify backend — the guts that power the Spotify service — in Google’s cloud platform.”
Spotify continue by explaining why they chose Google’s cloud services. Their experience in the past with Google’s data platforms and tools has been positive, including the focus on efficiency for all of the teams involved. Google’s ability to deal with big data is core to the decision. They boast the following features which Spotify say will provide them with an advantage to “where it matters most”:
Bigquery-Large-scale data analytics
Dataflow-real-time data processing service for batch and stream data processing
Dataproc-a managed Hadoop MapReduce, Spark, Pig, and Hive service, to easily process big datasets at low cost
Datalab-an interactive tool for large scale data exploration
Pub/Sub-scalable messaging middleware
The actual move to Google’s cloud is expected to take around two years, and it will be a complex undertaking. The aim is to move without causing any service disruptions.
Google comments that its platform will enable complex database queries with fast return answers. Guillaume Leygues from the Google Cloud Platform says: “This lets Spotify perform more frequent in-depth, interactive analysis, guiding product development, feature testing and more intelligent user-facing features.”
Being powered by Google’s Cloud will enable Spotify to roll out new features relatively instantaneously as the cloud will support these changes. Otherwise, using the previous own infrastructure, Spotify would have had to physically made alterations to enable the new features. Moving to the Cloud would allow Spotify to be even more cutting edge.
Meanwhile, Google’s competitor Amazon with their cloud hosting AWS, is targeting big US banks to move their business to AWS. Amazon’s offering “in theory, could help banks shift tech spending to newer areas and away from maintaining so many data centers. The hurdle is that Amazon’s is a public cloud business, open to anybody. Banks, which have relied on their own private infrastructure to this point, would need to feel comfortable about security and ease any concerns from regulators before moving ahead.”
The advantages of using a cloud are a reduction in cost of storage space and an increase in flexibility. This allows extra computing power on days where massive transactions periodically peak, such as Black Friday sales. JP Morgan is one of the many banks considering the move.
For now Amazon’s AWS is leading Google Cloud. AWS have been building data centers all over the world. But from history we can be sure that Google is hot on their heels with their cloud which without any doubt will grow exponentially. Let’s see who joins them next.
Is your business planning to make the move to a Cloud service? If so we would love to hear your story.