Loading…
#MesosCon 2015 has ended

Sign up or log in to bookmark your favorites and sync them to your phone or calendar.

Experience: Intermediate [clear filter]
Thursday, August 20
 

11:00am

Mesos at Bloomberg - Skand Gupta, Bloomberg
BVault is a massive scalable, archiving and e-discovery solution for communications and has been adopted by more than 800 enterprises, processing more than 220 million daily messages and archived more than 90 billion communication objects. The service is optimized for fast deployment of data-centric and processing-intensive applications using elastic cloud computing strategies namely, Container-based resource management (Mesos) to abstract heterogeneous data center assets as a homogeneous set of resources, prefabricated hardware in secure and geographically distributed data centers to provide on-demand capacity management, and Continuous development and integration using containers as an emerging standard in cloud infrastructure.

We share the lessons learned along the way, and tools we needed to build that enable us to validate, monitor, and operate the platform in production.

Speakers
SG

Skand Gupta

Skand Gupta is technical lead and engineering manager at Bloomberg LP. He has experience building large scale distributed systems in shared infrastructure and search space. He currently leads the Mesos infrastructure team for Bloomberg Vault.


Thursday August 20, 2015 11:00am - 11:40am
Grand Ballroom C

11:00am

Fully Fault tolerant Streaming workflows at Scale using Apache Mesos & Spark Streaming - AkhilDas, Sigmoid
Reliability, maintainability and scalability are the key concerns while designing any data intensive application . With the advent of realtime streaming platforms like Apache Spark, Storm etc, it is important that these computation frameworks adhere to all of them. One has to face a lot of challenges while implementing a realtime spark streaming pipeline in production. In this presentation we will showcase how to overcome these challenges and develop a fully, scalable fault tolerant streaming system with the help of Apache Mesos which not only makes it easier to deploy and manage the resources but also helps in handling varying data loads through dynamic resource management and allocation .

Speakers
A

AkhilDas

AkhilDas is a Software Developer at Sigmoid with focus on distributed computing, big data analytics, scaling and optimising performance. Sigmoid has worked with over 25 customers in the Big data space to get them real time insights on Tbs of data using Apache Spark and Spark Streaming.Previously... Read More →


Thursday August 20, 2015 11:00am - 11:40am
Grand Ballroom B

11:50am

Preemptive Task Scheduling in Mesos Framework - Li Jin, Two Sigma
Two Sigma uses Apache Mesos as the core of its computing infrastructure. To solve the scheduling problem of lots of resource-hungry computation tasks, we extended Dominant Resource Fairness (DRF), the core algorithm in Mesos allocator, to support preemption so the resource allocation remains fair with continuously changing demands. We will explain our extension of DRF and how it allows us to achieve very short per-user wait times in a constantly saturated Mesos cluster. We will also discuss how the core Mesos allocator itself could be enhanced to use this technique.

Speakers
LJ

Li Jin

Two Sigma LLC, Software Engineer
Li Jin is a distributed systems developer in Two Sigma. Li works on a Mesos framework that runs thousands of computing tasks every day. In his spare time, Li likes to read distributed system papers, hiking and winter sports.


Thursday August 20, 2015 11:50am - 12:30pm
Grand Ballroom C

11:50am

How to Monitor Mesos - Alexis Le-Quoc, Datadog
By providing a robust abstraction over core computing resources, Mesos does away with what has been until now the foundation of most monitoring systems: the individual host. With that gone, what should monitoring of applications running on Mesos revolve around? In this talk, Alexis Le-Quoc argues that imperative monitoring of hosts must give way to declarative monitoring, built on tags, tasks and queries. With concrete examples and live monitoring data, he will present a better way to monitor Mesos.

Speakers

Thursday August 20, 2015 11:50am - 12:30pm
Grand Ballroom B

2:00pm

Resource Optimization with Charmander - Marcel Neuhausler, AT&T Foundry
Charmander is a closed loop data analysis and optimization environment for a container based infrastructure. Charmander is open source (https://github.com/att-innovate/charmander) and is based on Mesos, Docker, cAdvisor, InfluxDB, Apache Spark, and Redis. In this presentation, Marcel Neuhausler will showcase different use-cases for Charmander, provide a technical overview of the components, and deep dive into its own Mesos Framework. 

Speakers
MN

Marcel Neuhausler

Marcel Neuhausler is a Principal Software Engineer at the AT&T Foundry in Palo Alto focusing on resource-conscious distributed systems.


Thursday August 20, 2015 2:00pm - 2:40pm
Grand Ballroom C

2:50pm

Supporting Stateful Services on Mesos using Persistence Primitives - Jie Yu, Twitter & Michael Park, Mesosphere
This talk will discuss the persistence primitives recently built into Mesos. Persistent primitives provide native support for running stateful services like Cassandra and MySQL in Mesos, which makes it possible for us to further consolidate workloads in the data center and improve overall cluster utilization. The goal of persistent primitives is to allow a framework to have guaranteed access to its lost state even after task failover or slave restart. To achieve that, we first discuss a primitive that allows tasks to write persistent data which won’t be garbage collected after tasks finish. Then, we discuss the primitive that allows frameworks to dynamically reserve resources for their stateful tasks upon launching so that the stateful tasks can always be re-launched on the same slave that contains their persisted state.

Speakers
avatar for Michael Park

Michael Park

Software Engineer, Facebook
I'm a software engineer at Facebook, working on the C++ libraries and standards team. My focus for C++ is to introduce pattern matching to facilitate better code.
avatar for Jie Yu

Jie Yu

Tech Lead, Mesosphere
Jie Yu is a Tech Lead at Mesosphere, Inc, focused on containerization, storage and networking. Before joining Mesosphere, he was a software engineer at Twitter. Jie obtained his PhD in Computer Science and Engineering from the University of Michigan where he conducted research for... Read More →


Thursday August 20, 2015 2:50pm - 3:30pm
Willow

2:50pm

Mesos at OpenTable - Pablo Delgado
Opentable has been using Apache Mesos for production workloads and for running critical parts of their production services for more than a year. Not only did Mesos help deploying resilient / elastic standalone applications and services , but also the distributed / fault-tolerant frameworks like Apache Spark for Data processing and machine learning. Mesos enabled Opentable to run multiple distributed applications across the same infrastructure at scale. Pablo will tell the story of how Opentable started with Mesos, the pain points of dealing with an hybrid Mesos + non-Mesos environment and how to survive in the transition.

Speakers
PD

Pablo Delgado

Senior Data Engineer, OpenTable, Inc.


Thursday August 20, 2015 2:50pm - 3:30pm
Grand Ballroom D

2:50pm

Scaling at eBay with Mesos - Soma Shekar Oruganti, eBay
eBay Inc. has embraced Jenkins for it’s continuous integration and delivery processes. It was challenging to efficiently use the infrastructure resources to accommodate the continuous integration process at the scale of eBay. Apache Mess and Marathon have made it possible for eBay to build a highly scalable, available and self healing processes, with efficient utilization of resources. Along the journey, eBay had challenges and overcome them effectively, and contributed back the solutions to the open source community with contributions to Jenkins framework for Mesos.

Speakers
SS

Soma Shekar Oruganti

MTS 1, Paypal Inc
Soma Shekar works as principal software engineer with Global Platform and Infrastructure Team at eBay Inc. The team is responsible for the tools used by eBay Inc.(both eBay and Paypal) developer and QA community for their Continuous Integration and Delivery processes. eBay have embraced... Read More →



Thursday August 20, 2015 2:50pm - 3:30pm
Grand Ballroom B

4:00pm

Apache Cotton: MySQL on Mesos - Yan Xu, Twitter
Apache Cotton (previously named Mysos) is an Apache Mesos framework for running MySQL instances. It dramatically simplifies the management of a MySQL cluster and is designed to offer:
- Efficient hardware utilization through multi-tenancy (in performance-isolated containers)- High reliability through preserving the MySQL state during failure and automatic backing up to/restoring from HDFS- An automated self-service option for bringing up new MySQL clusters- High availability through automatic MySQL master failover- An elastic solution that allows users to easily scale up and down a MySQL cluster by changing the number of slave instances
We’ll share our experience developing and using this framework.

Speakers
avatar for Yan Xu

Yan Xu

Software Engineer, Apple


Thursday August 20, 2015 4:00pm - 4:40pm
Grand Ballroom C

4:00pm

Running Stateful Services with Mesos - Arunabha Ghosh, Moz & Ankan Mukherjee, Moz
The Mesos system allows services to be decoupled from machines, however this decoupling creates problems for legacy apps that rely on persistent state. Traditional SQL databases are a prime example of such apps. It is still possible, however to run such apps under Mesos and gain the operational advantages it provides. We will cover techniques used at Moz to successfully run SQL databases and other legacy persistent state services under Mesos. The presentation will also cover challenges, best practices and a look at how to leverage Mesos primitives for stateful services. We will demonstrate several such services running on Mesos.

Speakers
AG

Arunabha Ghosh

Moz
Arunabha Ghosh is the Director of Engineering at Moz where among other things he leads the effort to move Moz onto Mesos. Prior to Moz, Arunabha worked at Yahoo and Google focusing on building large scale infrastructure. Arunabha leads the Systems Research group at the HackerDojo... Read More →
AM

Ankan Mukherjee

Ankan Mukherjee is a senior engineer at Moz and is currently focussed on building the next generation cluster operating system for Moz's datacenters. Prior to working at Moz he donned many different roles in the enterprise software world - software engineer, technical architect, technical/project... Read More →


Thursday August 20, 2015 4:00pm - 4:40pm
Grand Ballroom B

4:50pm

Per Container Network Monitoring and Isolation in Mesos - Jie Yu, Twitter
This talk will discuss the per container network monitoring and isolation feature introduced in Mesos 0.21.0 and our experiences in using it at Twitter. With this feature enabled, each container is assigned a separate network stack which isolates the network traffic from other containers and allows operators and users to easily get its network statistics. This greatly simplifies the network monitoring, health checking and troubleshooting for jobs, and provides a more fairly shared and predictable network to users. This feature is completely transparent to tasks, and does not require changes to the underlying network infrastructure. To achieve that, the key idea is to let containers share the same host IP and use ports to multiplex network packets. We will show you the implications of this approach and the lessons we learned during the deployment and ongoing use of this feature.

Speakers
avatar for Jie Yu

Jie Yu

Tech Lead, Mesosphere
Jie Yu is a Tech Lead at Mesosphere, Inc, focused on containerization, storage and networking. Before joining Mesosphere, he was a software engineer at Twitter. Jie obtained his PhD in Computer Science and Engineering from the University of Michigan where he conducted research for... Read More →


Thursday August 20, 2015 4:50pm - 5:30pm
Willow

4:50pm

Distributed CI: Scaling Jenkins on Mesos and Marathon - Roger Ignazio, Puppet Labs
As engineering teams and product offerings expand, monolithic Jenkins instances grow and become single points of failure. By running Jenkins on Mesos and Marathon, masters can be scaled horizontally and executors provisioned dynamically as demand increases. This presentation will demonstrate that running Jenkins at-scale is more feasible than ever. The architecture provided by Mesos moves Continuous Integration (CI) from being synonymous with Jenkins to Jenkins being a component of the overall CI system.

This presentation will cover:
- Conventional scaling techniques and their shortcomings
- Scaling Jenkins masters and slaves horizontally on Mesos and Marathon
- Dynamically creating and executing jobs
- Using Elasticsearch, Logstash, and Kibana (ELK) to store, query, and visualize run data

Thursday August 20, 2015 4:50pm - 5:30pm
Grand Ballroom C
 
Friday, August 21
 

10:40am

Global Control of Decentralized Mesos Clusters - Daniel Giribet, Schibsted
Apache Mesos and the frameworks running on top of them are a great resource management solution but they lack some features to make it suitable for automatic scaling and communication of microservices that run across different locations and vendors. In this presentation Schibsted will discuss these limitations and present their plan for a Global Scheduler capable of controlling independent Mesos clusters across multiple distant datacenters and service providers.

Speakers
avatar for Daniel Giribet

Daniel Giribet

Daniel Giribet is the Infrastructure Platform Development leader at Schibsted Products and Technology. He holds a Computer Science degree and has been focusing on video processing, web engineering, systems architecture and content management. Has worked in indie projects and also... Read More →


Friday August 21, 2015 10:40am - 11:20am
Grand Ballroom B

10:40am

Securing your Mesos Cluster - Adam Bordelon, Mesosphere & Joris Van Remoortere, Mesosphere
Security features are in high demand as organizations increasingly trust their Mesos clusters with critical production workloads and sensitive data, while at the same time providing cluster access to naive/untrusted users. The same questions keep coming up:
  • How can Mesos prevent unauthorized users from intercepting sensitive data or forging communications?
  • How can Mesos prevent unauthorized frameworks from launching malicious jobs and prevent unauthorized slaves from stealing tasks and data?
  • How can Mesos ensure users and frameworks can only access their own tasks/data?
In this first ever MesosCon security deep-dive, Adam and Joris will walk through the best practices and latest advances in securing Mesos clusters, including encryption, authentication, and authorization features.

Speakers
AB

Adam Bordelon

Mesosphere, Distributed System Engineer
Adam Bordelon is a distributed systems architect at Mesosphere and an Apache Mesos committer. Before joining Mesosphere, Adam lead development on Hadoop core at MapR, built distributed systems for recommendations at Amazon, and re-architected the LabVIEW compiler at National Instruments... Read More →
JV

Joris Van Remoortere

Joris Van Remoortere is a Senior Software Engineer at Mesosphere. He focuses on large-scale, low-latency, high-throughput systems. He alleviates scaling issues and makes performance and reliability improvements on many projects, including Orly, messaging and payment systems, web tiers... Read More →


Friday August 21, 2015 10:40am - 11:30am
Willow

11:30am

Rocket and the Need for the App Container Specification - Brandon Philips, CoreOS
Rocket is a simple daemon-free tool that enables users to run containerized apps on their systems free of host dependencies. Containers running under rocket execute like regular processes and can be managed using existing process management tools like upstart, systemd, runit and etc.

Rocket is also an implementation of the “App Container spec,” which defines how to define and build containerized applications using tooling like tar and PGP, and then host these files easily using standard protocols like HTTP. The goal of the spec is to enable independent and creative implementations of container runtimes and build tools.

This presentation will give an overview of the App Container spec and why the spec is important to the Mesos community. This session invites attendees to learn how they work and how to get involved.


Speakers
BP

Brandon Philips

Brandon Philips is the CTO of CoreOS where he is helping to build modern Linux server infrastructure. Prior to CoreOS, he worked at Rackspace hacking on cloud monitoring and was a Linux kernel developer at SUSE. As a graduate of Oregon State's Open Source Lab he is passionate about... Read More →


Friday August 21, 2015 11:30am - 12:10pm
Willow

1:30pm

Docker Swarm on Mesos - Tim Chen, Mesosphere & Victor Vieux, Docker
Docker has become a popular way for developers to package their applications for deployment, but as applications get more complex there is a growing need to deploy them at scale, orchestrating them across dozens or thousands of machines. With Docker Swarm, you can talk to Mesos using the Docker command line client and start Docker containers on your Mesos cluster as easily as you would do it on your development laptop. In this presentation, Victor Vieux (Docker Engineer, Docker) and Timothy Chen (Mesosphere Engineer, Mesosphere) will explain why Mesos is the “gold standard” for running Docker containers at scale and will demo how to setup Docker Swarm on Mesos (using DCOS) and how to do a `docker run` against your Mesos cluster easily.

Speakers
TC

Tim Chen

Tim Chen is on the Mesosphere core team and focuses on containerization and big data frameworks. He is also a PMC/committer on Apache Drill and Apache Mesos, and contributed to other open source projects such as Spark, Kafka, Hive. Before joining Mesosphere, Tim past experiences includes... Read More →
avatar for Victor Vieux

Victor Vieux

Docker Core Engineer, Docker, Inc
Victor Vieux works as a software engineer at Docker, Inc. He was one of the few guys working on the Docker internals, he's now working on Docker Swarm, a Docker-native clustering system. Originally from Paris, France, where he was famous for his work on robots (programming them in... Read More →


Friday August 21, 2015 1:30pm - 2:10pm
Grand Ballroom C

2:20pm

Scaling a Highly-Available Scheduler Using the Mesos Replicated Log: Pitfalls and Lessons Learned - Kevin Sweeney, Twitter
Mesos frameworks can go from prototype to production at scale very quickly. As your users realize how easy it is to spin up tasks, they will start doing so gratefully and rapidly. This talk will arm you with techniques for writing a framework scheduler for a large-scale Mesos cluster using Apache Aurora as a case study. Aurora routinely manages 100s of thousands of tasks per Mesos cluster. To do this it maintains the state of each task and quickly reacts to changes in the cluster. To talk to Mesos Aurora relies on the Mesos JNI library. For high-availability support it relies on the Mesos replicated log for durable storage. As its clusters grew, Aurora pushed against the limits of these libraries. This talk will explore the tools the Aurora scheduler has used to meet these challenges, including Apache Thrift for schema management, and the importance of normalization.

Speakers
avatar for Kevin Sweeney

Kevin Sweeney

Software Engineer - Cloud Infrastructure, Twitter
"Kevin Sweeney is a software engineer at Twitter and committer on Apache Aurora, working on it since 2012. You can find him on Twitter as @kts.



Friday August 21, 2015 2:20pm - 3:00pm
Grand Ballroom C

2:20pm

The Scheduler Meets the Network - Arunabha Ghosh, Moz
A vital part of datacenters is the network which enables communications. As the size
and complexity of applications deployed on Mesos grows, close integration with the underlying network will become crucial. Traditionally, networks have been statically configured, but the recent rise of network virtualization combined with techniques like overlay networks have enabled unprecedented network programability and control.

Mesos provides an ideal platform to leverage the dynamic capabilities of modern computer networks due to the scheduler’s complete knowledge and control over jobs in the system. The talk will showcase several features enabled by such integration such as IP per job, programmable network isolation, easy service discovery etc. The talk will conclude with a look towards the future evolution of networks and tighter integration with Mesos.

Speakers
AG

Arunabha Ghosh

Moz
Arunabha Ghosh is the Director of Engineering at Moz where among other things he leads the effort to move Moz onto Mesos. Prior to Moz, Arunabha worked at Yahoo and Google focusing on building large scale infrastructure. Arunabha leads the Systems Research group at the HackerDojo... Read More →


Friday August 21, 2015 2:20pm - 3:00pm
Willow