August 25, 2014Web · Data · Infra · Mobile · Culture · Open Source · Open Compute · @Scale

@Scale 2014 Agenda is Live!

@Scale 2014 agenda now live!

We're gearing up for @Scale 2014, the latest event in our conference series for engineers who build or maintain systems that are designed for scale.

Building applications and services that scale to millions or even billions of people presents a complex set of engineering challenges, many of them unprecedented. The @Scale community is focused on bringing people together to openly discuss these challenges and collaborate on the development of new solutions.

@Scale 2014 will 21 talks across data, mobile, and web tracks from companies like Box, Dropbox, Facebook, Google, Instagram, Khan Academy, LinkedIn, Netflix, Pinterest, Reddit, Twitter, Wikimedia, and YouTube.

Check out the agenda below and stay tuned for updates as we add more talks and speakers. And if you haven't already, register for @Scale 2014 at

Data Track

Recent advances in large-scale online data management systems and data analytics infrastructure. Covering real-time stream processing and analytics, cache infrastructure, scalable storage systems, graph databases, and massive warehousing techniques.

Data Platform Architecture, Evolution, and Philosophy

Kurt Brown, Data Platform at Netflix and Justin Becker, Senior Software Engineer at Netflix

The data platform at Netflix has evolved from a traditional BI stack to a modern, cloud-based architecture. Some current production components include Hadoop 2, Pig (on Tez), Hive, Presto, Teradata (Cloud), and S3 as our central data hub, along with lots of custom tooling (including Netflix open sourced techs like Inviso, Lipstick, and Aegisthus). We'll run through our data platform evolution, what our current architecture looks like and why, and the philosophical principles which drive how we get things done.

We'll also do a deep dive into Mantis, a reactive stream processing system under development at Netflix. With over two billion hours of content streamed per month, accounting for over a third of North American downstream internet traffic at peak, our operational systems are large, complex, and distributed. Mantis will further enhance our ability to analyze, detect, and resolve issues at scale. In this talk, we'll discuss why we're building it, how it differs from other stream processing systems, and explore the technical details regarding system design and implementation. Highlights include: designing for the cloud, progressive processing, push vs. pull, support for high order functions, stream locality, and more.

Growing Facebook on mobile, a realtime analytics story

Weizhe Shi,Engineer at Facebook; Anshul Jasiwal, Engineering Manager at Facebook; and Will Wirth, Product Analyst at Facebook

In the recent years Facebook has grown tremendously on mobile. In this talk we’ll discuss how realtime mobile analytics helped accelerate the growth by enabling faster feedback loops during product iteration cycles. We’ll discuss the challenges involved in logging data on mobile, in processing it and analyzing it in real-time. We’ll present the systems we developed to solve these problems. We’ll end the talk with specific examples of how we used these systems to carry out improvements in our mobile app performance, stability and engagement.

Zen: Pinterest's graph storage service

Raghavendra Prabhu, Engineering Manager at Pinterest and Xun Liu, Engineer at Pinterest

Zen is a storage service built at Pinterest that offers a graph data model on top of HBase and potentially other storage backends. Zen was originally conceived and built in summer 2013 and since then, has grown to be one of the our most widely used storage solutions, powering the home feed, interest graph, notifications, messages and other upcoming features. In this talk, we'll go over the design motivation for Zen and briefly describe the data model, API, and internals. We will also talk about our learnings from running the system in production for the last year, and features added and performance improvements made to accommodate the fast adoption we have seen since launch.

Building Scalable Caching Systems via McRouter

Rajesh Nishtala, Engineer at Facebook and Ricky Ramirez, Engineer at Reddit

Modern large scale web infrastructures rely heavily on distributed caching (e.g memcached) to process user requests. These caches serve as a temporary holding spot for the most commonly accessed data. However, this makes these services very sensitive to cache performance and reliability. McRouter is the lynchpin of Facebook’s caching infrastructure. It handles the basics of routing requests to the appropriate hosts and managing the responses in a highly performant way. In addition, there are a lot of features in McRouter that have been designed to dramatically improve the reliability of the caching infrastructure. The problems that McRouter addresses are not specific to Facebook, but distributed caching systems in general. As a result Instagram and Reddit have also adopted McRouter as the primary communication layer to their cache tiers. This talk gives a very brief overview of McRouter and the basics of integrating it into different pieces of infrastructure. We are excited to announce that we will be open sourcing McRouter today so that others can leverage this work.

Structured Data at Box: How we're Building for Scale

Tamar Bercovici, Senior Engineering Manager at Box

Serving hundreds of thousands of businesses, tens of millions of users, and storing metadata for billions of files, our database and caching tier are at the very core of our stack. We’ll start with an overview of our current MySQL-based sharded database architecture, including a few of the unique design choices we made and how they’ve panned out for us. We'll then talk about our new clustered memcache service, which helps us guarantee both the uptime and consistency of our cache. Finally, we'll discuss our current challenges, and some future directions we're considering. We'll take a sneak peek into Credence, our new structured data store, and discuss why we chose to take on the daunting task of rewriting our PHP data access layer in Scala, where we are in its development, and what open questions remain.

Scaling Youtube's backend: the Vitess trade-offs

Sugu Sougoumarane, Software Engineer at YouTube

If your website or mobile service becomes hugely successful, you will quickly realize that one of the hardest parts to scale is the storage. You will also find yourself making various trade-offs related to transactions, consistency, availability and durability. We faced these challenges at YouTube, which gave rise to the development of Vitess, an open source project. This talk will cover Vitess's architecture and discuss how we found our sweet spot by balancing all these requirements.

Mobile Track

Best practices and innovation in mobile development. Focusing on performance, fluid user interfaces, data modeling and persistence, large-scale development and build cycles, and the tooling required to support native development (Android and iOS) at scale.

Facebook's iOS architecture

Nick Schrock, Engineering Director at Facebook; Adam Ernst, Engineer at Facebook; Jonathan Dann, Engineer at Facebook; Ari Grant, Software Engineer at Facebook

Details to come.

Practical Cross-Platform Mobile C++ Development


We'll discuss the motivation of why xplatform C++ makes sense and then focus on the "how" of making it work - tying languages together, dealing with platform differences, and the tools we've developed to do so. That'll focus on a language generator we wrote that generates objective-C++ and jni bindings automatically, as well as other aspects of tooling and dealing with cross-platform mobile C++.

Unbundling and Its Impact on Mobile and Web Architectures

Kiran Prasad, Senior Director of Engineering at LinkedIn

As mobile becomes a larger part of user traffic, there is a natural question of should we combine or separate the client/server architecture for feature development. In addition to system architecture, the same question arrises around testing, build tooling, deployment systems and cadence of releases. Come learn where we started, where we are and where we are going with LinkedIn Front End Architecture.

Speeding up Android Development with Exopackage

David Reiss, Engineer at Facebook

Buck is a build system for Android that encourages the creation of small, reusable modules consisting of code and resources. Because Android applications are predominantly written in Java, Buck also functions as a Java build system. Buck has an advanced feature to speed up iterative Android development called exopackage. An exopackage is a small shell of an Android app that contains the minimal code and resources needed to bootstrap loading the code for a full-fledged Android application. Loading the application code at runtime avoids a full reinstall of the app when testing a Java change, which dramatically reduces the length of edit/refresh cycles.

Android Apps for Emerging Markets

Alex Sourov, Engineering Manager at Facebook; Tyler Kieft, Engineer at Instagram; Chris Marra, Product Manager at Facebook

In the past few years Android's growth has been staggering, and now more than half a billion people connect to Facebook each month from more than 40,000 different device models. In order to develop great products for these people, getting smart about performance, connectivity, and device characteristics has become crucial for Facebook and Instagram. We'll discuss the qualitative and quantitative approaches we've taken to better understand the Android device market, and present the way we've scaled this in the organization. We'll cover performance optimizations from a recent redesign of the Instagram application. We'll also cover the ways we've made the Facebook for Android application smarter about its device and network, and the optimizations we're exploring for the future.

Web Track

Best practices and innovation in mobile development. Focusing on performance, fluid user interfaces, data modeling and persistence, large-scale development and build cycles, and the tooling required to support native development (Android and iOS) at scale.

Migrating Wikipedia from PHP5 to HHVM

Ori Livneh, Senior Performance Engineer at the Wikimedia Foundation

As a top 10 website, Wikipedia serves more than 500 million readers ever month. Performance is a serious challenge for the site, particularly when serving users via a legacy PHP application more than a decade old. Any action which bypasses caching, such as editing Wikipedia, takes a serious hit from the limitations of the PHP run time. To solve this problem, we decided to take on the ambitious task of migrating to a new stack centered on HHVM, a virtual machine for PHP with an associated just-in-time compiler. In this talk, we'll walk you through the successes and failures we encountered during our migration to HHVM, and share what lessons we've derived for anyone seeking to improve performance of PHP applications.

Creating interactive learning interfaces at Khan Academy

Ben Alpert, Software Engineer at Khan Academy and Pamela Fox, Software Engineer at Khan Academy

In this talk, hear from two Khan Academy engineers about how they build two very different learning interfaces: the interactive math exercises, which are built with React and work on both web and mobile, and the live coding editor, which depends on the latest HTML5 features like web workers and content security policy.

JavaScript: Testing at Scale and Static Type Systems

Jeff Morrison and Avik Chaudhuri, Engineers at Facebook

At Facebook, we're writing more and more JavaScript every day. The code we write not only exploits the full flexibility and expressiveness offered by the language, but it also demands strong assurances of robustness, performance, and hackability. In such an environment, the lack of static typing becomes extremely costly, because all the tooling that is needed to support development at this level of scale and complexity requires code intelligence, and the foundations of code intelligence are static types. Flow is a static type system for JavaScript that we've built at Facebook, with the overall goal of making JavaScript development an awesome experience. Flow supports several flexible and expressive JavaScript idioms that require advanced type system features that go beyond traditional languages.

In this talk we'll also discuss how we test the tens of thousands of JavaScript modules at Facebook that interact with both each other and complex server-side APIs. Although these modules always start out owned and understood by a small number of engineers, over time the set of people who own/have context on them becomes broader. At times it even becomes unclear who has *any* context. To help address this, we spent some time identifying various important traits around automated testing that can help restore faith in engineers who change our codebase. The goal was to impose as little burden on developers as possible, while still encouraging them to write tests that help them (and others) succeed. As the backbone of this effort we built a tool (called Jest) to make it as simple and straightforward as possible to write tests according to these ideas. This resulted in a stronger culture around testing at Facbeook and we now have well over 2000 automated JavaScript tests in our codebase.

Advanced debugging techniques with Chrome

Paul Irish, Developer Advocate for Google Chrome

Advanced debugging techniques with Chrome - The bigger your app, the harder to trace down bugs to their sources. Luckily, a slew of functionality is in the Chrome DevTools that's dedicated to helping you avoid the bug hunt and keeping you productive. In this session, Paul will cover some advanced JavaScript debugging techniques and show how to be effective. From framework blackboxing to asynchronous callstacks and promises, there's plenty to incorporate into your development.

WebP at YouTube: Faster images on Desktop and Mobile

Mikhail Sychev, Software Engineer at Google

Despite being a video-sharing website YouTube is one of the largest image hosting platforms on the internet serving over a hundred billions images a day. For the last year we have been working on a site-wide WebP adoption project and as a result now serve it to Mobile, Desktop and TV. We want to share both highlights and lowlights of our effort to bring faster video thumbnails to the web.

Reactive Extensions

Jafar Husain, Technical Lead at Netflix

What's does a mouse drag event have in common with an Array of numbers? The answer to this question may surprise you: they are both collections. This key insight holds the key to dramatically simplifying asynchronous programming in JavaScript. In this talk you will learn how you can use the familiar JavaScript Array methods to create surprisingly expressive asynchronous programs. Using just a few functions, you will learn how to do the following:

    • Declaratively build complex events out of simple events (ex. drag n' drop)
    • Coordinate and sequence multiple Ajax requests
    • Reactively update UI's in response to data changes
    • Eliminate memory leaks caused by neglecting to unsubscribe from events
    • Gracefully propagate and handle asynchronous exceptions
  • In this talk we'll be exploring the Reactive Extensions (Rx) library which allows us to treat events as collections. You'll learn about how Netflix uses Rx on the client and the server, allowing us to build end-to-end reactive systems. We'll also contrast Rx with Promises, another popular approach to building asynchronous programs in JavaScript.

    Adaptable Systems and UI Components

    Nicolas Gallagher, Engineer at Twitter

    Details to come.

    @Scale 2014 will take place on September 15 at the InterContinental Hotel in San Francisco. @Scale conferences are open by invitation only to engineers who build or maintain systems that are designed for scale. If you’d like to apply to attend, please visit

    Previous @Scale conferences have featured talks from Box, Dropbox, Facebook, Google, Instagram, LinkedIn, Pinterest, Square, and Twitter. Here are some videos from previous meetings of the @Scale community:

    Facebook: Building Mobile Apps with Open Source Tools - Mobile @Scale

    Dropbox: Building for Cross Platform Apps at Dropbox - Mobile @Scale
    Twitter: Scaling Mobile Development at Twitter - Mobile @Scale
    Facebook: RocksDB: A High Performance Embedded Key-Value Store for Flash Storage – Data @Scale
    Instagram: Storage Systems at a Rapidly Scaling Startup with a Small Team – Data @Scale
    Pinterest: Hbase and its Associated Services – Data @Scale
    Facebook: Introducing Presto – Analytics @Scale
    Google: Security Team 2.1 - Security @Scale
    Square: Encrypted Data at Rest in Hadoop - Security @Scale
    Facebook: 2FAC: Facebook's Internal Multi-factor Auth Platform - Security @Scale

    We will announce speakers, agenda, and details in the coming weeks. Looking forward to seeing you in September!

    Want to work with us?

    Join the team, we're hiring! Here are some of our current open positions: More Engineering Positions

    Keep Updated

    Stay up-to-date via RSS with the latest open source project releases from Facebook, news from our Engineering teams, and upcoming events.

    Facebook © 2015