Engineering Blog

Blog posts tagged 'Infra'

Robert JohnsonDirector, Software Engineering at Facebook

Scaling Facebook to 500 Million Users and Beyond

Posted about 7 years ago
blog post · Infra · Data · Culture

Today we hit an important milestone for Facebook - half a billion users. It's particularly exciting to those of us in engineering and operations who build the systems to handle this massive growth. I started at Facebook four years ago when we had seven million users (which seemed like a really big number at the time) and the technical challenges along the way have been just as crazy as you might imagine. A few of the big numbers we deal with: * 500 million active users * 100 billion hits per day * 50 billion photos * 2 trillion objects cached, with hundreds of millions of requests per second * 130TB of logs every day Over the years we've written on this page about a number of the technical solutions we've used to handle these numbers. Today, I'd like to step back and talk about some of the general ways we think about scaling, and some of the principles we use to tackle scaling problems. Like Facebook itself, these principles involve both technology and people. In fact, only a couple of the principles below are entirely technical. At the end of the day it's people who build these systems and run them, and our best tools for scaling them are engineering and operations teams that can handle anything. The scaling statistic I'm most proud of is that we have over 1 million users per engineer, and this number has been steadily increasing. Read more...

Carlos BuenoFixer at Facebook

Internet Cartography

Posted about 7 years ago

A telegram from San Francisco to Hong Kong in 1901 must have taken many hops through British Empire cables to Europe, through the Middle East, and so on. London to New York was fast and direct. The vestiges of the Spanish and Portuguese Empires show up in the many links between South America, the Caribbean archipelago, and the Iberian peninsula. A cool thing is that you can measure these relative latencies yourself, using the present-day internet. If you run a website with a decent amount of worldwide traffic, you can use that traffic to map out how the internet responds with regards to you, and see how that matches with the gross structure of the 'net. I wrote about a cheap and cheerful way to generate this data last year, and the code has since been open-sourced as part of Yahoo's Boomerang measurement framework. Read more...

Keith AdamsEngineer at Facebook

The Life of a Typeahead Query

Posted about 8 years ago
blog post · Web · Infra · Data · Performance · User Experience · Backend · Graph · Testing

In fall of 2009, some of us at Facebook imagined a more interactive search experience, where high quality results would appear as the user typed. We summarized the vision in a blog post introducing our new search box. Making this system real for 400 million users has been challenging at every level of the software stack. On the front-end, code running in users' browsers must consume and render results quickly enough to not distract the user as they type. We set a strict goal for ourselves that the new typeahead can't be slower than the existing typeahead for finding your friends, so every millisecond delay matters. Despite these performance constraints, the UI cannot be too minimalist; each result needs enough contextual clues to convey its meaning and relevance. Because the new typeahead auto-selects the first result when you hit Enter in the search box, we need near-perfect relevance so that the first result is always the one you're looking for. To satisfy all these constraints, we designed an architecture composed of several different backend services that were optimized for specific types of results. Let's follow a session as the request leaves a user's browser and becomes a set of navigable results. Read more...

Akhil WableEngineering

Intro to Facebook Search

Posted about 8 years ago
blog post · Infra · Data · Web · Caching · User Experience

Connecting and sharing with others is Facebook’s primary value. That value necessitates having the ability to easily and efficiently find the people and information we care about. The search team at Facebook is focused on building a search product to enable our more than 400 million users to quickly find what they're looking for. In July 2007 we explained the complexities of serving one of the largest user bases in the world and the reasons for building our own in-house search service. Serving more than 150 million queries a day, and supporting a user base that has grown by more than 10x since then reinforces that decision. Read more...

Marcel LaverdetSoftware Engineer at Facebook

XHP: A New Way to Write PHP

Posted about 8 years ago
blog post · Infra · PHP · Languages

Today, I'd like to take to this opportunity to share with you a project that's quickly becoming a cornerstone of front-end PHP development at Facebook. XHP is a PHP extension which augments the syntax of the language to both make your front-end code easier to understand and help you avoid cross-site scripting attacks. XHP does this by making PHP understand XML document fragments, similar to what E4X does for ECMAScript (JavaScript). While PHP is typically used to write front-end code, by itself it isn't a very good language for generating HTML (as evidenced by the popularity of templating engines like Smarty). XHP is something between a programmatic UI library and a full templating system. PHP has some tricks up its sleeve like the ability to switch between HTML and PHP syntax, but this often encourages poor security practices and hard to follow control flows. Consider this simple PHP application:. Read more...

Tao SteinEngineering

Facebook becomes a USENIX Patron

Posted about 8 years ago
blog post · Infra · Data · Web · Mobile · Compute · Open Source · Research · Graph · Languages · PHP · HipHop · Platformmore

From its beginnings, Facebook has had to solve novel systems challenges to help us scale and grow. The idea of the social graph, and its implementation as a web and mobile platform have repeatedly pushed our computer systems into uncharted territory. The workloads are non-traditional, graph-oriented and write-heavy, and the system has grown rapidly to a base of 350M users around the world. We have survived and thrived due to healthy innovation and creativity, but we haven't done it alone. We have benefited from innovation in both the open source and computer systems communities. The USENIX Association is an essential hub in the systems community and today we are pleased to announce that we are becoming a Patron of the USENIX Association. Read more...

Haiping ZhaoSenior Server Engineer at Facebook

Real-World Web Application Benchmarking

Posted about 8 years ago
blog post · Data · Infra

In order to adequately forecast compute capacity and financial expense, application developers must benchmark application performance. Three factors are important to consider when analyzing hardware performance: maximum throughput, acquisition cost and operating cost. Industry-standard benchmarks, such as those published by The Standard Performance Evaluation Corporation (SPEC) can be reasonable indicators of maximum throughput for certain workloads. At Facebook, we recognized these benchmarks wouldn’t necessarily represent our application behavior under real-world conditions and developed a proprietary analysis methodology. In this paper, we contrast the conventional wisdom of using industry-standard synthetic benchmarks as guidelines for assessing web application performance with organic traffic measurements performed in a controlled environment. Read more...

Names in translation and social language modeling

Posted about 8 years ago
blog post · Infra · Web · Culture · Languages · Framework · User Experience

Our Translations app allows users (translators) to click on a phrase as they browse the site, and see the original native string, vote on translations suggested by their peers or contribute their own. Here at Facebook, we offer an innovative approach to web site internationalization that leverages a unique infrastructure and a dedicated user community to keep our interface up-to-date in translation. Read more...

Daniel SchaferSoftware Engineer at Facebook

Reducing markup size

Posted about 8 years ago
blog post · Infra

Facebook's News Feed has become increasingly feature rich over time, allowing users to interact with their friends' stories in new and interesting ways. The introduction of the Like feature, launched in February, has been a huge success with an average of 0.3 Likes per user on the average day. The comment and Like box appearing below most feed stories is known as the "Universal Feedback Interface," or UFI to Facebook engineers, and is an important aspect of communication among Facebook users. These new methods of communication do come at a cost, though. In July, Lior Abraham, an engineer on the Feed team, presented the results of a study he had done on the markup size of the home page. He discovered that 34% of the markup on the home page was due to the UFI. Since the UFI appears on nearly every page on the site, reducing the markup on this feature would lead to page size wins across the board. In particular, the Like feature was selected as an aspect of the UFI that could be slimmed down. Read more...

Zizhuang YangEngineering

Every Millisecond Counts

Posted about 8 years ago

Site speed has always been an important factor in the development of Facebook, even as the site evolves over time to become more feature-rich and complex. As we grow beyond the 250 million user mark, every small change to the site causes a huge ripple, affecting throngs of web surfers and their experience on Facebook. My project this summer as an engineering intern on the Infrastructure team involved tackling this imposing fact by exploring data and finding out how various changes to fundamental parts of the user experience impacted and changed user behavior. Read more...

Tom CookEngineering

Hammering Usernames

Posted about 8 years ago
blog post · Web · Infra · Compute · Performance · User Experience

Site Reliability Engineering (SRE) at Facebook is always under pressure to keep the site and all the moving pieces behind the scenes running while still delivering an excellent user experience. The recent launch of usernames to our 200 million active users on a single night at the exact same time was unique in its preparation and potential for trouble. Our product teams had evaluated the various options for enabling people to register for their name and decided upon a single first come, first serve registration window for every user. Although this was the most fair, it was difficult operationally because predicting the number of users that would show up at that time to claim a name was impossible. The Memcached infrastructure that runs behind every page on the site was partitioned and expanded to cope with users checking the availability of names. Read more...

Chris PiroEngineering

Chat reaches 1 billion messages sent per day

Posted about 9 years ago
blog post · Web · Infra · Data · Compute · Storage · Chat · Messages

Facebook Chat usage has increased steadily since its launch last year, and this week we reached 1 billion messages sent per day. As a team we've been looking forward to this milestone; we track lots of statistics in the course of maintaining and improving Chat, but this number measures Chat's progress toward its ultimate goal: increasing communication between our users. We've invested a lot in making Chat stable and scalable in the past, and we continue making improvements even now. Read more...

Peter VajgelEngineering

Needle in a haystack: efficient storage of billions of photos

Posted about 9 years ago
blog post · Infra · Data · Storage · Photos

The Photos application is one of Facebook’s most popular features. Up to date, users have uploaded over 15 billion photos which makes Facebook the biggest photo sharing website. For each uploaded photo, Facebook generates and stores four images of different sizes, which translates to a total of 60 billion images and 1.5PB of storage. Read more...

Kannan MuthukkaruppanTechnical Lead at Facebook

XHProfLive: Continuous function-level performance stats from production

Posted about 9 years ago
blog post · Infra · PHP · Languages · Performance

The Facebook web tier serves billions of PHP requests every day across thousands of servers. Since new code or other changes to the site get pushed out frequently, it is critical to have near real-time performance data that is both representative of production traffic and rich in its ability to help pinpoint regressions down to specific functional areas. This blog post talks about our solution to this problem - a lightweight but powerful tool called XHProfLive. Read more...

Chuck RossiDirector of Engineering at Facebook

The way Facebook is made

Posted about 9 years ago
blog post · Infra · Culture

Release Engineering is the part of the software engineering process that is most akin to herding cats. As a release engineer you act as gatekeeper between hundreds of code changes that other engineers make and the people that use the systems we build, package, and release. I've been doing release engineering work for about 20 years on projects big and small. I've shipped everything from entire operating systems to small stand-alone apps to large web-based applications responsible for generating millions of dollars in revenue per day. Each of these environments handled release engineering in their own way, but were surprisingly similar in how the engineers developing the code interacted with the process: in general, those engineers were a bit removed from the nuts and bolts of how their code made it out the door. That's not the case at Facebook. I teach a class here on the release process at Facebook, and the first thing I explain is our unique development and deployment environment. Read more...

Paul SaabEngineering at Facebook

Scaling memcached at Facebook

Posted about 9 years ago
blog post · Data · Infra · Caching · Storage

If you've read anything about scaling large websites, you've probably heard about memcached. memcached is a high-performance, distributed memory object caching system. Here at Facebook, we're likely the world's largest user of memcached. Read more...

Robert JohnsonDirector, Software Engineering at Facebook

Facebook's Scribe technology now open source

Posted about 9 years ago
blog post · Web · Data · Infra · Open Source · Performance · Compute · Development Tools

Here at Facebook, we're constantly facing scaling challanges because of our enormous growth. One particular problem we encountered a couple of years ago was collection of data from our servers. We were collecting a few billion messages a day (which seemed like a lot at the time) for everything from access logs to performance statistics to actions that went to News Feed. We used a variety of different technologies for the different use cases, and all of them were bursting at the seams. We decided to build a unified system (called Scribe) to handle all of these cases, and do it in a way that would scale with Facebook's growth. Read more...

Jason SobelEngineer at Facebook

Scaling Out

Posted about 9 years ago
blog post · Data · Infra · Caching · Performance

I joined Facebook in April 2007 and, after getting settled over the course of a few weeks, my manager Robert Johnson approached me. We talked for a while but the conversation boiled down to: Bobby: "So, Jason, we're going to open a new datacenter in Virginia by 2008. Do you think you can help?" Me: "Uh.... yes?" Bobby: "Great!" My first project at Facebook was a tad more involved then I was expecting, but I think that is one reason why we have such a great engineering organization; we have a lot of hard problems to solve and everyone here is excited to jump in and tackle them. I set out to really understand why we were building a new datacenter and what problems we had to overcome to make it work. Read more...

David ReissEngineering

Thrift: (slightly more than) one year later

Posted about 10 years ago
blog post · Infra · Open Source · Languages

A little over a year ago, Facebook released Thrift as open source software. (See the original announcement.) Thrift is a lightweight software framework for enabling communication between programs written in different programming languages, running on different computers, or both. We decided to release it because we thought it could help other groups solve some of the same technical problems we have faced, and because we hoped that developers who found it useful would contribute improvements back to the project. A lot has happened in the year since we released Thrift. First and foremost, Thrift has gained a lot of cool features:. Read more...

Joydeep Sen SarmaTenured Engineer at Facebook

Hadoop

Posted about 10 years ago
blog post · Infra · Data · Open Source · Framework

With tens of millions of users and more than a billion page views every day, Facebook ends up accumulating massive amounts of data. One of the challenges that we have faced since the early days is developing a scalable way of storing and processing all these bytes since using this historical data is a very big part of how we can improve the user experience on Facebook. This can only be done by empowering our engineers and analysts with easy to use tools to mine and manipulate large data sets. About a year back we began playing around with an open source project called Hadoop. Hadoop provides a framework for large scale parallel processing using a distributed file system and the map-reduce programming paradigm. Read more...

Eugene LetuchyEngineering

Facebook Chat

Posted about 10 years ago
blog post · Web · Infra · Chat · Messages · Performance · User Experience · Development Tools

One of the things I like most about working at Facebook is the ability to launch products that are (almost) immediately used by millions of people. Unlike a three-guys-in-a-garage startup, we don't have the luxury of scaling out infrastructure to keep pace with user growth; when your feature's userbase will go from 0 to 70 million practically overnight, scalability has to be baked in from the start. The project I'm currently working on, Facebook Chat, offered a nice set of software engineering challenges:. Read more...

Aditya AgarwalDirector of Engineering at Facebook

Welcome to the Facebook Engineering Blog!

Posted about 10 years ago
blog post · Web · Mobile · Infra · Data · Culture

We are going to use this space to tell you a little about the code and systems that power Facebook. We thought it would be fun to share what goes on behind the scenes to ensure that the site scales smoothly and that we continue to provide the best overall user experience. Read more...

Keep Updated

Stay up-to-date via RSS with the latest open source project releases from Facebook, news from our Engineering teams, and upcoming events.

Subscribe
Facebook © 2017