Loading…
This event has ended. View the official site or create your own event → Check it out
This event has ended. Create your own
View analytic
Friday, August 21 • 2:20pm - 3:00pm
Scaling a Highly-Available Scheduler Using the Mesos Replicated Log: Pitfalls and Lessons Learned - Kevin Sweeney, Twitter

Sign up or log in to save this to your schedule and see who's attending!

Mesos frameworks can go from prototype to production at scale very quickly. As your users realize how easy it is to spin up tasks, they will start doing so gratefully and rapidly. This talk will arm you with techniques for writing a framework scheduler for a large-scale Mesos cluster using Apache Aurora as a case study. Aurora routinely manages 100s of thousands of tasks per Mesos cluster. To do this it maintains the state of each task and quickly reacts to changes in the cluster. To talk to Mesos Aurora relies on the Mesos JNI library. For high-availability support it relies on the Mesos replicated log for durable storage. As its clusters grew, Aurora pushed against the limits of these libraries. This talk will explore the tools the Aurora scheduler has used to meet these challenges, including Apache Thrift for schema management, and the importance of normalization.

Speakers
avatar for Kevin Sweeney

Kevin Sweeney

Software Engineer - Cloud Infrastructure, Twitter
"Kevin Sweeney is a software engineer at Twitter and committer on Apache Aurora, working on it since 2012. You can find him on Twitter as @kts.



Friday August 21, 2015 2:20pm - 3:00pm
Grand Ballroom C