Engineering Blog

Blog posts tagged 'Infra'

Alexey AndreyevEngineering

Introducing data center fabric, the next-generation Facebook data center network

Posted about 3 years ago

The more than 1.35 billion people who use Facebook on an ongoing basis rely on a seamless, “always on” site performance. On the back end, we have many advanced sub-systems and infrastructures in place that make such a real-time experience possible, and our scalable, high-performance network is one of them. Read more...

Announcing the Hack Transpiler

Posted about 3 years ago
blog post · Infra · Hack · Languages · PHP · Open Source

Today, we're proud to announce a first, experimental release of h2tp, or the “HH (Hack) Transpiler,” a tool which allows projects that have converted from PHP to Hack to still make releases that target the PHP language. Read more...

Audience Insights query engine: In-memory integer store for social analytics

Posted about 3 years ago
blog post · Web · Data · Infra · Production Engineering · Analytics · Data Science

A query engine with a hybrid integer store that organizes data in memory and on flash disks so that a query can process terabytes of data in real time. Read more...

Fernanda WeidenEngineering at Facebook

Security @Scale 2014 Recap

Posted about 3 years ago
blog post · Infra · Security

Making online services safe and secure for more than a billion people means that security solutions have to scale well. Recent internet-wide incidents involving SSL technology, such as POODLE and Heartbleed, only reinforce the importance of getting this stuff right, as well as the extent to which security technology impacts more than any single company. Read more...

Introducing Proxygen, Facebook's C++ HTTP framework

Posted about 3 years ago

We are excited to announce the release of Proxygen, a collection of C++ HTTP libraries, including an easy-to-use HTTP server. In addition to HTTP/1.1, Proxygen (rhymes with “oxygen”) supports SPDY/3 and SPDY/3.1. We are also iterating and developing support for HTTP/2. Read more...

Mike ArpaiaEngineering

Introducing osquery

Posted about 3 years ago

Maintaining real-time insight into the current state of your infrastructure is important. At Facebook, we've been working on a framework called osquery which attempts to approach the concept of low-level operating system monitoring a little differently. Read more...

@Scale 2014: Recap of Data Track

Posted about 3 years ago
blog post · Data · Infra · Mobile · Caching · Testing · @Scale

Facebook started the @Scale conference series as a forum for engineers to share how they solve problems in the real world. It's since evolved and grown to become a community that collaborates in the open in order to innovate further, faster, and more effectively. We saw this firsthand during the web track sessions of @Scale, where companies at all stages of growth — Wikimedia, Twitter, Khan Academy, Google, and Netflix — are pushing the boundaries of what we thought was possible in web development. For many of these teams, they're also contributing those technologies back to the open source community, ensuring that others can learn and benefit from these lessons. Read more...

Building Mobile-First Infrastructure for Messenger

Posted about 3 years ago
blog post · Mobile · Infra · Messages · Production Engineering · Backend · Storage

Messages have been part of Facebook for many years, beginning as direct messaging similar to email (available in your inbox the next time you visited the site) and then eventually evolving into a real-time messaging platform that provides access to your messages on a number of mobile apps or in a browser. But until recently the back-end systems hadn't evolved much from early iterations, and Messenger's performance and data usage started to lag behind — especially on networks with costly data plans and limited bandwidth. To fix this, we needed to completely re-imagine how data is synchronized to the device and change how data is processed in the back end to support our new synchronization protocol. Read more...

Tom OcchinoSoftware engineer at Facebook

@Scale 2014: Recap of Web Track

Posted about 3 years ago
blog post · Infra · Web · @Scale · Open Source · Front End

Facebook started the @Scale conference series as a forum for engineers to share how they solve problems in the real world. It's since evolved and grown to become a community that collaborates in the open in order to innovate further, faster, and more effectively. We saw this firsthand during the web track sessions of @Scale, where companies at all stages of growth — Wikimedia, Twitter, Khan Academy, Google, and Netflix — are pushing the boundaries of what we thought was possible in web development. For many of these teams, they're also contributing those technologies back to the open source community, ensuring that others can learn and benefit from these lessons. Read more...

Nick SchrockEngineering at Facebook

@Scale 2014: Recap of Mobile Track

Posted about 3 years ago
blog post · Mobile · Infra · @Scale · Android · iOS

It's never been a more exciting time to be a software developer. Two billion people — over a quarter of the entire human race — can access, with a single tap of a screen, software on smartphones, devices that would have been considered magic just a few decades ago. Read more...

Phil DibowitzProduction Engineer at Facebook

Facebook, configuration management, community, and open source

Posted about 3 years ago
blog post · Infra · Data · Open Source · Production Engineering

Last year we began speaking at conferences around the world about our approach to managing hundreds of thousands of servers. We had outgrown our existing system and needed something new. We wanted a system that would let any engineer make any change they needed to any systems they owned via simple data-driven APIs while also scaling to Facebook's huge infrastructure, and while also minimizing the size of the team that would have to own the system. We designed a new paradigm and built a framework to bring it to life. At the core of that framework is Chef — but the way we ended up using Chef is pretty unique. We wanted to share how and why we made those choices and the benefits they brought us. Read more...

James PearceEngineering at Facebook

Introducing TODO: Working together to make open source easier

Posted about 3 years ago
blog post · Web · Data · Mobile · Infra · @Scale · Open Source

Today at @Scale 2014 we joined a number of other companies in launching a new open source collaboration called TODO. The group — whose name is a backronym for “talk openly, develop openly” — was formed to address the challenges that companies like ours have encountered in consuming open source software and running open source programs. Read more...

Introducing mcrouter: A memcached protocol router for scaling memcached deployments

Posted about 3 years ago

Most web-based services begin as a collection of front-end application servers paired with databases used to manage data storage. As they grow, the databases are augmented with caches to store frequently-read pieces of data and improve site performance. Often, the ability to quickly access data moves from being an optimization to a requirement for a site. This evolution of cache from neat optimization to necessity is a common path that has been followed by many large web scale companies, including Facebook, Twitter[1], Instagram, Reddit, and many others. Read more...

@Scale 2014 Agenda is Live!

Posted about 3 years ago
blog post · Web · Data · Infra · Mobile · Culture · Open Source · Open Compute · @Scale

@Scale conferences are regular industry events for engineers who build or maintain systems that are designed for scale. Read more...

Bill WeihlEngineering

Sustainability @ Scale Recap

Posted about 3 years ago
blog post · Infra

Late last month, more than 200 people convened at Facebook’s offices in Menlo Park for Sustainability @ Scale, a day-long conference focused on corporate sustainability. Our hope, in bringing this group together, was that we could foster some open discussion of the sustainability challenges we all face and identify some opportunities to work together to address those challenges. Read more...

Debugging file corruption on iOS

Posted about 3 years ago
blog post · Mobile · Infra · iOS · Platform · Performance · Optimization

Recently, we solved a long-term mobile debugging problem and reduced the crash rate for people using the Facebook for iOS app by more than 50 percent. Read more...

Qiang WuInfrastructure Software Engineer at Facebook

Making Facebook’s software infrastructure more energy efficient with Autoscale

Posted about 3 years ago

Improving energy efficiency and reducing environmental impact as we scale is a top priority for our data center teams. We’ve talked a lot about our progress on energy-efficient hardware and data center design through the Open Compute Project, but we’ve also started looking at how we could improve the energy efficiency of our software. We explored multiple avenues, including power modeling and profiling, peak power management, and energy-proportional computing. One particularly exciting piece of technology that we developed is a system for power-efficient load balancing called Autoscale. Autoscale has been rolled out to production clusters and has already demonstrated significant energy savings. Read more...

Larry SchrofProduction Engineer at Facebook

#bewhoyoucansee: Larry Schrof

Posted about 3 years ago
blog post · Culture · Infra

As more and more people from underrepresented groups lift their voices and launch their careers in tech fields, it is more important than ever that we share their stories widely. We all need visible role models who can challenge, inspire, and motivate us. With this in mind, we’re starting the #bewhoyoucansee series. Every week, we'll profile someone from an underrepresented group working in tech to learn how they got started, what they're passionate about, and what advice they have for other people pursuing a technical career. If you're a role model, or want to share who and what inspires you, follow along with us on Facebook and Instagram by tagging your posts #bewhoyoucansee. Read more...

James PearceEngineering at Facebook

9.9 million lines of code and still moving fast - Facebook open source in 2014

Posted about 3 years ago
blog post · Mobile · Culture · Data · Infra · @Scale · Open Source

The first six months of 2014 have been very busy for our open source program. In the spirit of the World Cup, we thought it was time for a half-time review of some of the highlights so far. Read more...

Alex SourovSoftware Engineer / Engineering Manager / Product Manager at Facebook

Improving Facebook on Android

Posted about 4 years ago

In an effort to connect the next five billion, Facebook began to shift to a mobile-first company about two years ago. We trained hundreds of employees on mobile development, restructured internal teams to build for all platforms, and moved to a fast-paced release cycle. Read more...

Introducing “Wedge” and “FBOSS,” the next steps toward a disaggregated network

Posted about 4 years ago

We’re big believers in the value of disaggregation – of breaking down traditional data center technologies into their core components so we can build new systems that are more flexible, more scalable, and more efficient. This approach has guided Facebook from the beginning, as we’ve grown and expanded our infrastructure to connect more than 1.28 billion people around the world. Read more...

Nick PetroEngineering

F8 Developer Conference - Hacker Way Recap

Posted about 4 years ago
blog post · Data · Infra · Mobile · Web · Performance · Optimization · Open Source · Hack · Messages · Design Tools · Android · iOSmore

Over 1,700 developers traveled to the Concourse Exhibition Center in San Francisco for Facebook’s F8 Developer Conference last week. Read more...

Open-sourcing Haxl, a library for Haskell

Posted about 4 years ago
blog post · Infra · Data · Web · Backend · Open Source · Caching · Languages · Security · Data Science · Analyticsmore

Today we're open-sourcing Haxl, a Haskell library that simplifies access to remote data, such as databases or web-based services. Read more...

HydraBase – The evolution of HBase@Facebook

Posted about 4 years ago
blog post · Data · Infra · Messages · Analytics · Storage · Platform · Open Source

When we revamped Messages in 2010 to integrate SMS, chat, email and Facebook Messages into one inbox, we built the product on open-source Apache HBase, a distributed key value data store running on top of HDFS, and extended it to meet our requirements. At the time, HBase was chosen as the underlying durable data store because it provided the high write throughput and low latency random read performance necessary for our Messages platform. In addition, it provided other important features, including horizontal scalability, strong consistency, and high availability via automatic failover. Since then, we’ve expanded the HBase footprint across Facebook, using it not only for point-read, online transaction processing workloads like Messages, but also for online analytics processing workloads where large data scans are prevalent. Today, in addition to Messages, HBase is used in production by other Facebook services, including our internal monitoring system, the recently launched Nearby Friends feature, search indexing, streaming data analysis, and data scraping for our internal data warehouses. Read more...

Saving capacity with HDFS RAID

Posted about 4 years ago
blog post · Data · Infra · Production Engineering

As we continue to evolve our data infrastructure, we’re constantly looking for ways to maximize the utility and efficiency of our systems. One technology we’ve deployed is HDFS RAID, an implementation of Erasure Codes in HDFS to reduce the replication factor of data in HDFS. We finished putting this into production last year and wanted to share the lessons we learned along the way and how we increased capacity by tens of petabytes. Read more...

Nick CorralEngineering

Technical Program Managers at Facebook

Posted about 4 years ago
blog post · Culture · Infra · Web · Data Centers · Networking and Traffic · Backend

Technical Program Managers, or TPMs, are part of a program we implement when taking on large-scale engineering efforts. TPMs are responsible for seeing programs through from beginning to end, ensuring a better workflow and more effective communication. They are a diverse group with a wide range of backgrounds, but they share common characteristics: a love for execution and a knack for doing whatever is necessary to see a program to completion. Read more...

Scaling the Facebook data warehouse to 300 PB

Posted about 4 years ago
blog post · Data · Infra · Production Engineering

At Facebook, we have unique storage scalability challenges when it comes to our data warehouse. Our warehouse stores upwards of 300 PB of Hive data, with an incoming daily rate of about 600 TB. In the last year, the warehouse has seen a 3x growth in the amount of data stored. Given this growth trajectory, storage efficiency is and will continue to be a focus for our warehouse infrastructure. Read more...

Keep Updated

Stay up-to-date via RSS with the latest open source project releases from Facebook, news from our Engineering teams, and upcoming events.

Subscribe
Facebook © 2017