Loading…
#MesosCon 2015 has ended
Friday, August 21 • 2:20pm - 3:00pm
Scaling a Highly-Available Scheduler Using the Mesos Replicated Log: Pitfalls and Lessons Learned - Kevin Sweeney, Twitter

Sign up or log in to save this to your schedule and see who's attending!

Mesos frameworks can go from prototype to production at scale very quickly. As your users realize how easy it is to spin up tasks, they will start doing so gratefully and rapidly. This talk will arm you with techniques for writing a framework scheduler for a large-scale Mesos cluster using Apache Aurora as a case study. Aurora routinely manages 100s of thousands of tasks per Mesos cluster. To do this it maintains the state of each task and quickly reacts to changes in the cluster. To talk to Mesos Aurora relies on the Mesos JNI library. For high-availability support it relies on the Mesos replicated log for durable storage. As its clusters grew, Aurora pushed against the limits of these libraries. This talk will explore the tools the Aurora scheduler has used to meet these challenges, including Apache Thrift for schema management, and the importance of normalization.

Speakers
avatar for Kevin Sweeney

Kevin Sweeney

Software Engineer - Cloud Infrastructure, Twitter
"Kevin Sweeney is a software engineer at Twitter and committer on Apache Aurora, working on it since 2012. You can find him on Twitter as @kts.



Friday August 21, 2015 2:20pm - 3:00pm
Grand Ballroom C

Attendees (0)