Engineering Blog

Blog posts tagged 'Storage'

Dhruba BorthakurEngineering

Under the Hood: Building and open-sourcing RocksDB

Posted about 4 years ago
blog post · Data · Infra · Backend · Production Engineering · Open Source · Storage

Every time one of the 1.2 billion people who use Facebook visits the site, they see a completely unique, dynamically generated home page. There are several different applications powering this experience--and others across the site--that require global, real-time data fetching. Read more...

Domas MituzasInfrastructure Engineer at Facebook

Flashcache at Facebook: From 2010 to 2013 and beyond

Posted about 4 years ago
blog post · Infra · Data · Storage · Caching · Performance · Optimization

We recently released a new version of Flashcache, kicking off the flashcache-3.x series. We’ve spent the last few months working on this new version, and our work has resulted in some performance improvements, including increasing the average hit rate from 60% to 80% and cutting the disk operation rate nearly in half. Read more...

Lachlan MulcahyEngineering

Windex: Automation for database provisioning

Posted about 4 years ago
blog post · Data · Infra · Production Engineering · Storage

Windex was originally developed to wipe data from hosts coming out of production, reinstall everything from the OS through to MySQL, and then configure them so they could be placed back into the spares pool all shiny and new. Now, Windex has expanded its role to cover all provisioning of MySQL DB hosts, whether they are freshly racked and set up by our site operations team or taken out of production for an offline repair like RAM replacement. Read more...

Scaling memcache at Facebook

Posted about 5 years ago
blog post · Data · Infra · Caching · Production Engineering · Storage

Facebook started using memcached in August 2005 when Mark Zuckerberg downloaded it from the Internet and installed it on our Apache web servers. At that time, Facebook was starting to make increasingly sizable database queries on every page load, and page load times were significantly increasing. Providing a fast, snappy user experience has always been a high priority for Facebook, and memcached came to the rescue. Read more...

Tim ArmstrongEngineering

LinkBench: A database benchmark for the social graph

Posted about 5 years ago
blog post · Data · Infra · Graph · MySQL · Performance · Optimization · Open Source · Testing · Storage

MySQL offers a good mix of flexibility, performance, and administrative ease, but the database engineering team continues to explore alternatives to MySQL for storing social graph data. There are several generic open-source benchmarks that could provide a starting point for comparing database systems. However, the gold standard for database benchmarking is to test the performance of a system on the real production workload, since synthetic benchmarks often don't exercise systems in the same way. When making decisions about a significant component of Facebook's infrastructure, we need to understand how a database system will really perform in Facebook's production workload. Read more...

Alex GartrellEngineering

McDipper: A key-value cache for Flash storage

Posted about 5 years ago
blog post · Infra · Data · Web · Storage · Caching · Performance · Server Infrastructure · Data Centers · Photos

Memcache has been used at Facebook for everything from a look-aside cache for MySQL to a semi-reliable data-store for ads impression data. Of course RAM is relatively expensive, and for working sets that had very large footprints but moderate to low request rates, we believed we could make Memcached much more efficient. Compared with memory, flash provides up to 20 times the capacity per server and still supports tens of thousands of operations per second, so it was the obvious answer to this problem. Read more...

Under the Hood: Scheduling MapReduce jobs more efficiently with Corona

Posted about 5 years ago
blog post · Data · Infra · Open Source · Storage

Nearly every team at Facebook depends on our custom-built data infrastructure for warehousing and analytics, with roughly 1,000 people across the company – technical and non-technical – using these technologies every day. Over half a petabyte of new data arrives in the warehouse every 24 hours, and ad-hoc queries, data pipelines, and custom MapReduce jobs process this raw data around the clock to generate more meaningful features and aggregations. Read more...

Sean LynchEngineering

Monitoring cache with Claspin

Posted about 5 years ago

When I started at Facebook, I joined the newly formed cache performance team in production engineering. Our goal was to get a handle on the health of our various cache systems and to facilitate quick troubleshooting, starting with answering the question, "Is this problem being caused by the cache?". Read more...

Andrew RyanEngineering

Under the Hood: Hadoop Distributed Filesystem reliability with Namenode and Avatarnode

Posted about 6 years ago
blog post · Infra · Data · Storage · Open Source

The Hadoop Distributed Filesystem (HDFS) forms the basis of many large-scale storage systems at Facebook and throughout the world. Our Hadoop clusters include the largest single HDFS cluster that we know of, with more than 100 PB physical disk space in a single HDFS filesystem. Optimizing HDFS is crucial to ensuring that our systems stay efficient and reliable for users and applications on Facebook. Read more...

MySQL and Database Engineering: Mark Callaghan

Posted about 6 years ago

Mark Callaghan joined Facebook in 2009. Now, working from home in Oregon on the database infrastructure team, he does whatever it takes to keep the database running and users like Trekkie George Takei, a favorite person to get updates from amongst Mark's team, happy. Read on for Mark's insights into fast-storage challenges, Facebook's MySQL roadmap, and more. Read more...

Nagavamsi PonnekantiEngineering at Facebook

Hybrid Incremental MySQL Backups

Posted about 7 years ago
blog post · Infra · Web · Data · MySQL · PHP · Storage · Performance

This post discusses enhancements to our database backups. As we deploy these enhancements to production servers, we may write additional posts about other improvements made along the way. Read more...

Jason EvansEngineering

Scalable memory allocation using jemalloc

Posted about 7 years ago
blog post · Infra · Data · Storage

The Facebook website comprises a diverse set of server applications, most of which run on dedicated machines with 8+ CPU cores and 8+ GiB of RAM. These applications typically use POSIX threads for concurrent computations, with the goal of maximizing throughput by fully utilizing the CPUs and RAM. This environment poses some serious challenges for memory allocation, in particular:. Read more...

Dhruba BorthakurEngineering

Looking at the code behind our three uses of Apache Hadoop

Posted about 7 years ago
blog post · Data · Infra · Open Source · MySQL · Storage · Messages

The size of the data warehouse cluster at Facebook has been increasing tremendously over the past few years. We use several pieces of open source software in our data warehouse including Apache Hadoop, Apache Hive, Apache HBase, Apache Thrift and Facebook Scribe. Together they keep this data processing engine humming. Read more...

Carlos BuenoFixer at Facebook

The Full Stack, Part I

Posted about 7 years ago
blog post · Infra · Data · Storage · Networking and Traffic · Compute · Hardware

One of my most vivid memories from school was the day our chemistry teacher let us in on the Big Secret: every chemical reaction is a joining or separating of links between atoms. Which links form or break is completely governed by the energy involved and the number of electrons each atom has. The principle stuck with me long after I'd forgotten the details. There existed a simple reason for all of the strange rules of chemistry, and that reason lived at a lower level of reality. Maybe other things in the world were like that too. Read more...

Chris PiroEngineering

Chat reaches 1 billion messages sent per day

Posted about 9 years ago
blog post · Web · Infra · Data · Compute · Storage · Chat · Messages

Facebook Chat usage has increased steadily since its launch last year, and this week we reached 1 billion messages sent per day. As a team we've been looking forward to this milestone; we track lots of statistics in the course of maintaining and improving Chat, but this number measures Chat's progress toward its ultimate goal: increasing communication between our users. We've invested a lot in making Chat stable and scalable in the past, and we continue making improvements even now. Read more...

Ashish ThusooEngineering at Facebook

Hive - A Petabyte Scale Data Warehouse using Hadoop

Posted about 9 years ago

A number of engineers from Facebook are speaking at the Yahoo! Hadoop Summit today about the ways we are using Hadoop and Hive for analytics. Hive is an open source, peta-byte scale date warehousing framework based on Hadoop that was developed by the Data Infrastructure Team at Facebook. In this blogpost we'll talk more about Hive, how it has been used at Facebook and its unique architecture and capabilities. Read more...

Peter VajgelEngineering

Needle in a haystack: efficient storage of billions of photos

Posted about 9 years ago
blog post · Infra · Data · Storage · Photos

The Photos application is one of Facebook’s most popular features. Up to date, users have uploaded over 15 billion photos which makes Facebook the biggest photo sharing website. For each uploaded photo, Facebook generates and stores four images of different sizes, which translates to a total of 60 billion images and 1.5PB of storage. Read more...

Paul SaabEngineering at Facebook

Scaling memcached at Facebook

Posted about 9 years ago
blog post · Data · Infra · Caching · Storage

If you've read anything about scaling large websites, you've probably heard about memcached. memcached is a high-performance, distributed memory object caching system. Here at Facebook, we're likely the world's largest user of memcached. Read more...

Avinash LakshmanEngineering

Cassandra – A structured storage system on a P2P Network

Posted about 9 years ago
blog post · Data · Compute · Open Source · Storage

When I joined Facebook I was eagerly looking forward to a new challenge. Fortunately, Facebook cannot be accused of a lack of challenging assignments. Prashant Malik,a colleague in Facebook from the Search team, was thinking about how to solve the Inbox Search problem. This challenge is about storing reverse indices of Facebook messages that Facebook users send and receive while communicating with their friends on the Facebook network. The amount of data to be stored, the rate of growth of the data and the requirement to serve it within strict SLAs made it very apparent that a new storage solution was absolutely essential. The solution needed to scale incrementally and in a cost effective fashion. Traditional data storage solutions just wouldn’t fit the bill. The aim was to design a solution that not only solved the Inbox Search problem but also provided a system as a storage infrastructure for many problems of the same nature. Hence was born Cassandra. To keep up with Facebook tradition, Prashant and I started the implementation of Cassandra about a year ago in one of our Hackthons. Cassandra is a distributed storage system for managing structured data that is designed to scale to a very large size across many commodity servers, with no single point of failure. Reliability at massive scale is a very big challenge. Outages in the service can have significant negative impact. Hence Cassandra aims to run on top of an infrastructure of hundreds of nodes (possibly spread across different datacenters). At this scale, small and large components fail continuously; the way Cassandra manages the persistent state in the face of these failures drives the reliability and scalability of the software systems relying on this service. Cassandra has achieved several goals – scalability, high performance, high availability and applicability. In many ways Cassandra resembles a database and shares many design and implementation strategies with databases. Cassandra does not support a full relational data model; instead, it provides clients with a simple data model that supports dynamic control over data layout and format. The rest of the material talks about the data model and the distributed properties, provided by the system. Data Model. Read more...

Keep Updated

Stay up-to-date via RSS with the latest open source project releases from Facebook, news from our Engineering teams, and upcoming events.

Subscribe
Facebook © 2017