GOTO Amsterdam (June 17-19, 2015) is a vendor independent international software development conference with more that 50 top speakers and 500 attendees. The conference covers topics such as AngularJS, Disruption, Docker, Drones, Elasticsearch, Hadoop, Microservices & Scrum.

<< previous speaker next speaker >>

Rafal Wojdyla, Engineer at Spotify

Rafal Wojdyla

Biography: Rafal Wojdyla

Rafal is an engineer at Spotify, a member of Hadoop squad responsible for operating, maintaing and growing one of the biggest Hadoop cluster in Europe. He is a core committer to snakebite – super fast pure python HDFS client.

Twitter: @ravwojdyla

Presentation: The Evolution of Hadoop at Spotify - Through Failures and Pain

Track: Hadoop / Time: Friday 11:30 - 12:20 / Location: Veilingzaal

The quickest way to learn and evolve infrastructure is by encountering obstacles and being forced to overcome limitations that keep you inches away from project goals. At Spotify, we've encountered many of these obstacles and frustrations as we grew our Hadoop cluster from a few machines in an office closet aggregating played song events for financial reports, to our current 900 node cluster that plays a large role in many features that you see in our application today.

Two members of Spotify's Hadoop 'squad' will weave in war stories, failures, frustrations and lessons learned to describe the Hadoop/Big Data architecture at Spotify and talk about how that architecture has evolved.

We'll talk about how and why we use a number of tools, including Apache Falcon and Apache Bigtop to test changes; Apache Crunch, Scalding and Hive w/ Tez to build features and provide analytics; and Snakebite and Luigi, two in-house tools created to overcome common frustrations.