Engineering Blog

Blog posts tagged 'Production Engineering'

Saving capacity with HDFS RAID

Posted about 4 years ago
blog post · Data · Infra · Production Engineering

As we continue to evolve our data infrastructure, we’re constantly looking for ways to maximize the utility and efficiency of our systems. One technology we’ve deployed is HDFS RAID, an implementation of Erasure Codes in HDFS to reduce the replication factor of data in HDFS. We finished putting this into production last year and wanted to share the lessons we learned along the way and how we increased capacity by tens of petabytes. Read more...

Scaling the Facebook data warehouse to 300 PB

Posted about 4 years ago
blog post · Data · Infra · Production Engineering

At Facebook, we have unique storage scalability challenges when it comes to our data warehouse. Our warehouse stores upwards of 300 PB of Hive data, with an incoming daily rate of about 600 TB. In the last year, the warehouse has seen a 3x growth in the amount of data stored. Given this growth trajectory, storage efficiency is and will continue to be a focus for our warehouse infrastructure. Read more...

Steaphan GreeneEngineering

WebScaleSQL: A collaboration to build upon the MySQL upstream

Posted about 4 years ago
blog post · Data · MySQL · Production Engineering

To help the more than 1.23 billion people who use Facebook to share and connect with each other, we’ve had to build an expansive and incredibly advanced infrastructure -- including one of the largest deployments of MySQL in the world. Along the way, we’ve learned and benefited from code changes made by the MySQL community. Today we’re announcing WebScaleSQL, a collaboration among engineers from several companies that face similar challenges in running MySQL at scale and seek greater performance from a database technology tailored to their needs. Read more...

Emil HesslowSoftware engineer at Facebook

HHVM - Implementing MySQLi

Posted about 4 years ago

The MySQLi extension is very popular amongst PHP developers. We knew we had to support it in our HHVM runtime, and thought it would be interesting to share the process we went through to implement it. Read more...

Dhruba BorthakurEngineering

Under the Hood: Building and open-sourcing RocksDB

Posted about 4 years ago
blog post · Data · Infra · Backend · Production Engineering · Open Source · Storage

Every time one of the 1.2 billion people who use Facebook visits the site, they see a completely unique, dynamically generated home page. There are several different applications powering this experience--and others across the site--that require global, real-time data fetching. Read more...

Ashoat TevosyanSoftware engineer at Facebook

Under the Hood: Building posts search

Posted about 4 years ago
blog post · Infra · Data · Production Engineering · Graph Search

Last week we added the ability to search posts using Graph Search, a feature that has been two years in the making. With one billion new posts added every day, the posts index contains more than one trillion total posts, comprising hundreds of terabytes of data. Indexing these posts and building a system to return real-time results has been a significant engineering challenge — and this is just beginning. Read more...

Lachlan MulcahyEngineering

Windex: Automation for database provisioning

Posted about 5 years ago
blog post · Data · Infra · Production Engineering · Storage

Windex was originally developed to wipe data from hosts coming out of production, reinstall everything from the OS through to MySQL, and then configure them so they could be placed back into the spares pool all shiny and new. Now, Windex has expanded its role to cover all provisioning of MySQL DB hosts, whether they are freshly racked and set up by our site operations team or taken out of production for an offline repair like RAM replacement. Read more...

Mark MarchukovEngineering

TAO: The power of the graph

Posted about 5 years ago
blog post · Data · Infra · Caching · Production Engineering

Facebook puts an extremely demanding workload on its data backend. Every time any one of over a billion active users visits Facebook through a desktop browser or on a mobile device, they are presented with hundreds of pieces of information from the social graph. Users see News Feed stories; comments, likes, and shares for those stories; photos and check-ins from their friends -- the list goes on. The high degree of output customization, combined with a high update rate of a typical user’s News Feed, makes it impossible to generate the views presented to users ahead of time. Thus, the data set must be retrieved and rendered on the fly in a few hundred milliseconds. Read more...

Laurent DemaillySoftware engineer at Facebook

Wormhole pub/sub system: Moving data through space and time

Posted about 5 years ago
blog post · Data · Infra · Production Engineering · Caching

Over the last couple of years, we have built and deployed a reliable publish-subscribe system called Wormhole. Wormhole has become a critical part of Facebook's software infrastructure. At a high level, Wormhole propagates changes issued in one system to all systems that need to reflect those changes – within and across data centers. Read more...

Scaling memcache at Facebook

Posted about 5 years ago
blog post · Data · Infra · Caching · Production Engineering · Storage

Facebook started using memcached in August 2005 when Mark Zuckerberg downloaded it from the Internet and installed it on our Apache web servers. At that time, Facebook was starting to make increasingly sizable database queries on every page load, and page load times were significantly increasing. Providing a fast, snappy user experience has always been a high priority for Facebook, and memcached came to the rescue. Read more...

Eric BarrettData Plumber at Facebook

Under the Hood: Automated backups

Posted about 5 years ago
blog post · Web · Data · Infra · Production Engineering

Facebook has one of the largest MySQL installations in the world, with thousands of database servers in multiple regions, so it's no surprise that we have unique challenges when we take backups. Our job is to keep every piece of information you add safe, while ensuring that anything that's been deleted is purged in a timely manner. Read more...

Sean LynchEngineering

Monitoring cache with Claspin

Posted about 5 years ago

When I started at Facebook, I joined the newly formed cache performance team in production engineering. Our goal was to get a handle on the health of our various cache systems and to facilitate quick troubleshooting, starting with answering the question, "Is this problem being caused by the cache?". Read more...

Keep Updated

Stay up-to-date via RSS with the latest open source project releases from Facebook, news from our Engineering teams, and upcoming events.

Facebook © 2018