Loading…
#MesosCon 2015 has ended

Sign up or log in to bookmark your favorites and sync them to your phone or calendar.

Internals / Extensions [clear filter]
Thursday, August 20
 

11:00am

Challenges in Optimizing Job Scheduling On Mesos - Alex Gaudio, Sailthru
As a distributed systems kernel, Mesos solves complex scheduling problems. These problems include: Bin Packing of tasks given resource constraints; Dominant Resource Allocation to prioritize Framework offers; user-defined optimistic or pessimistic Scheduler algorithms. When job allocation stops working properly, we see Framework starvation, dead locks, underutilization and other issues. In this talk, Alex Gaudio will present a unique interpretation of task scheduling on Mesos. By drawing on his experiences building Mesos Frameworks and infrastructure for Sailthru's Data Science team, he will provide attendees with ways to fundamentally question how they use Mesos, and offer practical advice to improve sub-optimal usage patterns.

Speakers
avatar for Alex Gaudio

Alex Gaudio

Senior Data Scientist and Engineer, Sailthru
Alex Gaudio is a Senior Data Scientist and Engineer at Sailthru, where he builds machine learning algorithms and distributed data pipelines on Mesos. He is the primary developer of Relay.Mesos, an intelligent auto-scaler for Mesos that functions like a thermostat. He also develops... Read More →


Thursday August 20, 2015 11:00am - 11:40am
Willow

11:50am

The New Mesos HTTP API - Vinod Kone, Twitter, Isabel Jimenez, Mesosphere
The previous Mesos API provided language bindings (a.k.a. drivers) for frameworks to interact with Mesos. Framework developers don’t like dealing with dependencies in native libraries, and with Mesos’ new HTTP API, they no longer have to! In this session, (Vinod Kone, Technical Lead for Mesos at Twitter and Isabel Jimenez, Software Engineer at Mesosphere) will provide a comprehensive walk-through of recent advancements with the Mesos API, explaining the design rationale and highlighting specific improvements that simplify writing frameworks to Mesos. 

Speakers
avatar for Isabel Jimenez

Isabel Jimenez

Distributed systems engineer, Mesosphere
Isabel Jimenez is a Software Engineer at Mesosphere.
VK

Vinod Kone

Vinod Kone is the Technical Lead for Mesos at Twitter. Vinod leads the Mesos team. Mesos is a fundamental component of Twitter's data center infrastructure powering a lot of its key services.


Thursday August 20, 2015 11:50am - 12:30pm
Willow

11:50am

Preemptive Task Scheduling in Mesos Framework - Li Jin, Two Sigma
Two Sigma uses Apache Mesos as the core of its computing infrastructure. To solve the scheduling problem of lots of resource-hungry computation tasks, we extended Dominant Resource Fairness (DRF), the core algorithm in Mesos allocator, to support preemption so the resource allocation remains fair with continuously changing demands. We will explain our extension of DRF and how it allows us to achieve very short per-user wait times in a constantly saturated Mesos cluster. We will also discuss how the core Mesos allocator itself could be enhanced to use this technique.

Speakers
LJ

Li Jin

Two Sigma LLC, Software Engineer
Li Jin is a distributed systems developer in Two Sigma. Li works on a Mesos framework that runs thousands of computing tasks every day. In his spare time, Li likes to read distributed system papers, hiking and winter sports.


Thursday August 20, 2015 11:50am - 12:30pm
Grand Ballroom C

2:50pm

Supporting Stateful Services on Mesos using Persistence Primitives - Jie Yu, Twitter & Michael Park, Mesosphere
This talk will discuss the persistence primitives recently built into Mesos. Persistent primitives provide native support for running stateful services like Cassandra and MySQL in Mesos, which makes it possible for us to further consolidate workloads in the data center and improve overall cluster utilization. The goal of persistent primitives is to allow a framework to have guaranteed access to its lost state even after task failover or slave restart. To achieve that, we first discuss a primitive that allows tasks to write persistent data which won’t be garbage collected after tasks finish. Then, we discuss the primitive that allows frameworks to dynamically reserve resources for their stateful tasks upon launching so that the stateful tasks can always be re-launched on the same slave that contains their persisted state.

Speakers
avatar for Michael Park

Michael Park

Software Engineer, Facebook
I'm a software engineer at Facebook, working on the C++ libraries and standards team. My focus for C++ is to introduce pattern matching to facilitate better code.
avatar for Jie Yu

Jie Yu

Tech Lead, Mesosphere
Jie Yu is a Tech Lead at Mesosphere, Inc, focused on containerization, storage and networking. Before joining Mesosphere, he was a software engineer at Twitter. Jie obtained his PhD in Computer Science and Engineering from the University of Michigan where he conducted research for... Read More →


Thursday August 20, 2015 2:50pm - 3:30pm
Willow

4:00pm

Mesos Gets Pluggable: Introducing Mesos Modules - Kapil Arya, Mesosphere & Niklas Nielsen, Mesosphere
One of the most exciting new innovations is a simple, pluggable approach to building customizations for Mesos. This makes it as easy to extend or customize Mesos as it is to install plug-ins into your web browser. In this talk, (Niklas Nielsen and Kapil Arya, Distributed Systems Engineers at Mesosphere) will provide a technical introduction to Mesos Modules - the new way to customize Mesos without having to recompile / relink for each specific use case. They will demonstrate how to build and use Mesos Modules, including an example of how to write an allocator module for custom scheduling policies across a cluster. The presenters will also explain how Mesos Modules will help grow the ecosystem of developers building new features for Mesos. Customization will be one of strongest and most flexible attributes of the Mesos distributed systems kernel!

Speakers
avatar for Kapil Arya

Kapil Arya

Kapil Arya is an Engineer at Mesosphere focussing on the core team. He recently finished his PhD at Northeastern University where he was the lead developer of the open source distributed checkpointing project DMTCP, and contributed to the reversible debugger FReD. Interning at VMware... Read More →
NN

Niklas Nielsen

Niklas Nielsen is an engineer at Mesosphere. He is a C++ hacker and is works on the Mesos core. Before joining Mesosphere he worked at Adobe on the Flash VM and Google V8, as well as a secret next-generation VM. He did his Master's thesis at Lawrence Livermore National Lab, working... Read More →


Thursday August 20, 2015 4:00pm - 4:40pm
Willow

4:50pm

Per Container Network Monitoring and Isolation in Mesos - Jie Yu, Twitter
This talk will discuss the per container network monitoring and isolation feature introduced in Mesos 0.21.0 and our experiences in using it at Twitter. With this feature enabled, each container is assigned a separate network stack which isolates the network traffic from other containers and allows operators and users to easily get its network statistics. This greatly simplifies the network monitoring, health checking and troubleshooting for jobs, and provides a more fairly shared and predictable network to users. This feature is completely transparent to tasks, and does not require changes to the underlying network infrastructure. To achieve that, the key idea is to let containers share the same host IP and use ports to multiplex network packets. We will show you the implications of this approach and the lessons we learned during the deployment and ongoing use of this feature.

Speakers
avatar for Jie Yu

Jie Yu

Tech Lead, Mesosphere
Jie Yu is a Tech Lead at Mesosphere, Inc, focused on containerization, storage and networking. Before joining Mesosphere, he was a software engineer at Twitter. Jie obtained his PhD in Computer Science and Engineering from the University of Michigan where he conducted research for... Read More →


Thursday August 20, 2015 4:50pm - 5:30pm
Willow
 
Friday, August 21
 

10:40am

Securing your Mesos Cluster - Adam Bordelon, Mesosphere & Joris Van Remoortere, Mesosphere
Security features are in high demand as organizations increasingly trust their Mesos clusters with critical production workloads and sensitive data, while at the same time providing cluster access to naive/untrusted users. The same questions keep coming up:
  • How can Mesos prevent unauthorized users from intercepting sensitive data or forging communications?
  • How can Mesos prevent unauthorized frameworks from launching malicious jobs and prevent unauthorized slaves from stealing tasks and data?
  • How can Mesos ensure users and frameworks can only access their own tasks/data?
In this first ever MesosCon security deep-dive, Adam and Joris will walk through the best practices and latest advances in securing Mesos clusters, including encryption, authentication, and authorization features.

Speakers
AB

Adam Bordelon

Mesosphere, Distributed System Engineer
Adam Bordelon is a distributed systems architect at Mesosphere and an Apache Mesos committer. Before joining Mesosphere, Adam lead development on Hadoop core at MapR, built distributed systems for recommendations at Amazon, and re-architected the LabVIEW compiler at National Instruments... Read More →
JV

Joris Van Remoortere

Joris Van Remoortere is a Senior Software Engineer at Mesosphere. He focuses on large-scale, low-latency, high-throughput systems. He alleviates scaling issues and makes performance and reliability improvements on many projects, including Orly, messaging and payment systems, web tiers... Read More →


Friday August 21, 2015 10:40am - 11:30am
Willow

11:30am

Rocket and the Need for the App Container Specification - Brandon Philips, CoreOS
Rocket is a simple daemon-free tool that enables users to run containerized apps on their systems free of host dependencies. Containers running under rocket execute like regular processes and can be managed using existing process management tools like upstart, systemd, runit and etc.

Rocket is also an implementation of the “App Container spec,” which defines how to define and build containerized applications using tooling like tar and PGP, and then host these files easily using standard protocols like HTTP. The goal of the spec is to enable independent and creative implementations of container runtimes and build tools.

This presentation will give an overview of the App Container spec and why the spec is important to the Mesos community. This session invites attendees to learn how they work and how to get involved.


Speakers
BP

Brandon Philips

Brandon Philips is the CTO of CoreOS where he is helping to build modern Linux server infrastructure. Prior to CoreOS, he worked at Rackspace hacking on cloud monitoring and was a Linux kernel developer at SUSE. As a graduate of Oregon State's Open Source Lab he is passionate about... Read More →


Friday August 21, 2015 11:30am - 12:10pm
Willow

1:30pm

Satellite: A Monitoring Application for Mesos - Sunil Abraham, Two Sigma Investments
Satellite is a monitoring and alerting solution for Mesos. Satellite’s powerful event streams processing engine allows /ops teams to build custom monitoring and alerting solutions that will improve uptime and reliability of the underlying compute resources in the cluster. In this talk, you'll learn how Two Sigma is using Satellite to monitor and take action on the hosts in our Mesos cluster. After a quick introduction to the architecture of Satellite, we'll walk through the process of defining a rule, adding a Pagerduty alert, and programmatically taking action on the state of the cluster by enabling and disabling hosts in the cluster.

Speakers
SA

Sunil Abraham

Sunil Abraham is a developer at Two Sigma Investments. Previously, Sunil was a Fulbright Scholar at the University of Waterloo working on public health and studied math and statistics at Oxford and Cambridge.


Friday August 21, 2015 1:30pm - 2:10pm
Willow

2:20pm

The Scheduler Meets the Network - Arunabha Ghosh, Moz
A vital part of datacenters is the network which enables communications. As the size
and complexity of applications deployed on Mesos grows, close integration with the underlying network will become crucial. Traditionally, networks have been statically configured, but the recent rise of network virtualization combined with techniques like overlay networks have enabled unprecedented network programability and control.

Mesos provides an ideal platform to leverage the dynamic capabilities of modern computer networks due to the scheduler’s complete knowledge and control over jobs in the system. The talk will showcase several features enabled by such integration such as IP per job, programmable network isolation, easy service discovery etc. The talk will conclude with a look towards the future evolution of networks and tighter integration with Mesos.

Speakers
AG

Arunabha Ghosh

Moz
Arunabha Ghosh is the Director of Engineering at Moz where among other things he leads the effort to move Moz onto Mesos. Prior to Moz, Arunabha worked at Yahoo and Google focusing on building large scale infrastructure. Arunabha leads the Systems Research group at the HackerDojo... Read More →


Friday August 21, 2015 2:20pm - 3:00pm
Willow