Engineering Blog

Blog posts tagged 'Open Source'

Celebrating a year of fighting email phishing with DMARC

Posted about 5 years ago
blog post · Infra · Security · Testing · Open Source · Messages · User Experience

A couple of years ago, Facebook joined a burgeoning alliance of major online services to support the development of DMARC, a protocol that seeks to augment new message authentication technologies with a strong policy layer focused on thwarting phishing attacks. Today, we’re celebrating the DMARC standard’s official one-year anniversary by announcing that DMARC now protects almost two-thirds of the world’s 3.3 billion consumer mailboxes, including 85% of Facebook’s user base. Read more...

Drew ParoskiSoftware Engineer at Facebook

Speeding up PHP-based development with HipHop VM

Posted about 5 years ago
blog post · Infra · HipHop · Open Source · PHP · Performance

Here at Facebook, the HipHop team constantly strives to improve the efficiency of PHP execution and increase productivity for our PHP developers. In late 2011 we announced that we were pursuing a just-in-time (JIT) compilation approach called HipHop VM (HHVM) as a successor to the HipHop PHP-to-C++ compiler (HPHPc). The goals of the HHVM project are two-fold: build a production-ready virtual machine that delivers superior performance, and unify our production and development environments. Read more...

Under the Hood: Scheduling MapReduce jobs more efficiently with Corona

Posted about 5 years ago
blog post · Data · Infra · Open Source · Storage

Nearly every team at Facebook depends on our custom-built data infrastructure for warehousing and analytics, with roughly 1,000 people across the company – technical and non-technical – using these technologies every day. Over half a petabyte of new data arrives in the warehouse every 24 hours, and ad-hoc queries, data pipelines, and custom MapReduce jobs process this raw data around the clock to generate more meaningful features and aggregations. Read more...

Paul TarjanWeb hacker at Facebook

Under the Hood: Timeline apps behind Facebook engineering

Posted about 5 years ago
blog post · Web · Infra · Data · Culture · Open Source · Timeline

A big part of working at Facebook is dogfooding our own products. Before new features roll out to the public, our team uses them frequently and thoroughly to figure out what we can do better. So when we started building Open Graph, we were all eager to give Open Graph apps a test run by creating our own and making the internal tools we were already using more social. Here's a look at some of our favorite in-house Facebook Open Graph apps for work and play, and what we've learned along the way that can help you, too. Read more...

Andrew RyanEngineering

Under the Hood: Hadoop Distributed Filesystem reliability with Namenode and Avatarnode

Posted about 6 years ago
blog post · Infra · Data · Storage · Open Source

The Hadoop Distributed Filesystem (HDFS) forms the basis of many large-scale storage systems at Facebook and throughout the world. Our Hadoop clusters include the largest single HDFS cluster that we know of, with more than 100 PB physical disk space in a single HDFS filesystem. Optimizing HDFS is crucial to ensuring that our systems stay efficient and reliable for users and applications on Facebook. Read more...

Jordan DeLongEngineering

Folly: The Facebook Open Source Library

Posted about 6 years ago
blog post · Infra · Backend · Open Source · Compute · Development Tools · Performance

Facebook is built on open source from top to bottom, and could not exist without it. As engineers here, we use, contribute to, and release a lot of open source software, including pieces of our core infrastructure such as HipHop and Thrift. Read more...

Michael AdkinsSoftware engineer at Facebook

DMARC: Building Open-Source Email Authentication Technologies

Posted about 6 years ago
blog post · Infra · Security · Open Source · Optimization · User Experience

As one of the largest senders of email on the Internet, Facebook is also one of the largest phishing targets. After working at an ISP on email security and abuse issues for several years, I came to Facebook because I wanted to have a greater impact on the email ecosystem. Supported by Facebook’s commitment to open technologies, in 2010 I started working with a handful of other email security experts on a major anti-phishing effort called DMARC. Read more...

Jason EvansEngineering

The HipHop Virtual Machine

Posted about 6 years ago
blog post · Infra · Open Source · HipHop · PHP · Performance · Languages

Read more...

Minghui YangSoftware engineer at Facebook

Making HPHPi Faster

Posted about 6 years ago
blog post · Infra · Web · PHP · HipHop · Open Source · Performance · Development Tools · Languages · C++ · Optimization · Caching · Testingmore

Read more...

Yael MaguireEngineering
John SichiSoftware engineer at Facebook

Working with Students to Improve Indexing in Apache Hive

Posted about 7 years ago
blog post · Culture · Compute · Open Source

In Facebook Engineering, we're always looking for ways to expand our academic involvement beyond internships and research projects, so for the 2010-2011 academic year, we sponsored a team of four undergraduates as part of the Harvey Mudd Computer Science Clinic. Clinic teams work with their faculty advisor plus an industry partner to explore an evolving area of computer science in a way that is valuable to the company. To help make that happen, Facebook engineers serve as liaisons and mentors, providing requirements and technical input. Along the way, the students learn a lot about real world software development and project management. Read more...

Jay ParkEngineering

Designing a Very Efficient Data Center

Posted about 7 years ago
blog post · Data · Compute · Hardware · Data Centers · Prineville · Open Source · Open Compute

When we started designing our Prineville, Ore., data center as part of the Open Compute Project, we did so with a "less is more" philosophy. We wanted a highly energy efficient, less costly, simpler and more reliable facility that could serve as a model for other data centers. Read more...

Amir MichaelEngineering

Inside the Open Compute Project Server

Posted about 7 years ago
blog post · Infra · Data · Data Centers · Hardware · Open Compute · Open Source · Optimization

We launched the Open Compute Project, an effort to nurture industry collaboration on the best practices and implementation of power- and cost-efficient compute infrastructure, yesterday. At the heart of the project lies the Open Compute server, a highly optimized server layout developed by Facebook engineers and industry partners. Read more...

Jonathan HeiligerVP Tech Operations at Facebook

Building Efficient Data Centers with the Open Compute Project

Posted about 7 years ago

A small team of Facebook engineers spent the past two years tackling a big challenge: how to scale our computing infrastructure in the most efficient and economical way possible. Read more...

Xin QiResearch Scientist at Facebook

HipHop for PHP: More Optimizations for Efficient Servers

Posted about 7 years ago

Facebook switched all its production servers to HipHop in early 2010, also releasing the project’s source code at that time. At the time of the switch, HipHop reduced our average CPU usage by 50%, the six months after its release saw an additional 1.8x performance improvement, and in the past six months the team in conjunction with the open source community has made an additional 1.7x improvement. Read more...

Cory OndrejkaAlleged Interim CTO at Facebook

HTML5 Games 0.1: Speedy Sprites

Posted about 7 years ago

Speed, it seems to me, provides the one genuinely modern pleasure. Read more...

Scott MacVicarEngineering

Supercell: test infrastructure for any open source project

Posted about 7 years ago
blog post · Infra · Open Source

Testing is an important part of any project; it allows the engineer to discover errors that could have occurred while adding a new feature or a regression introduced while fixing a bug. These are all a normal part of the software development process. Read more...

Liyin TangSoftware engineer at Facebook

Join Optimization in Apache Hive

Posted about 7 years ago
blog post · Infra · Data · Open Source · Optimization · Performance

With more than 500 million users sharing a billion pieces of content daily, Facebook stores a vast amount of data, and needs a solid infrastructure to store and retrieve that data. This is why we use Apache Hive and Apache Hadoop so widely at Facebook. Hive is a data warehouse infrastructure built on top of Hadoop that can compile SQL queries as MapReduce jobs and run the jobs in the cluster. Read more...

Dhruba BorthakurEngineering

Looking at the code behind our three uses of Apache Hadoop

Posted about 7 years ago
blog post · Data · Infra · Open Source · MySQL · Storage · Messages

The size of the data warehouse cluster at Facebook has been increasing tremendously over the past few years. We use several pieces of open source software in our data warehouse including Apache Hadoop, Apache Hive, Apache HBase, Apache Thrift and Facebook Scribe. Together they keep this data processing engine humming. Read more...

Scott MacVicarEngineering

Google Summer of Code 2010

Posted about 7 years ago
blog post · Culture · Mobile · Web · Compute · Open Source · Academics · PHP · HipHop · HTML5 · JavaScript · iOSmore

This summer we participated in Google Summer of Code (GSoC) for the first time and wanted to share an update on the progress our students made. GSoC is in its sixth year and exists to encourage university students to spend their summer coding for an open source / free software project. The students in turn are awarded a stipend which definitely makes this a cool summer job. Read more...

Scott MacVicarEngineering

HipHop for PHP: six months later

Posted about 7 years ago

It's been six months since we released HipHop and I wanted to share an update on its progress. In February we released 693,613 lines of source code which on average reduced our CPU usage here at Facebook by about 50%. Since February, the team has made HipHop another 1.8 times faster and all of that code is open source. We’ve also seen improvements to PHP itself with the additions to PHP's trunk in April being about 10% faster than 5.3. Read more...

Carlos BuenoFixer at Facebook

Internet Cartography

Posted about 7 years ago

A telegram from San Francisco to Hong Kong in 1901 must have taken many hops through British Empire cables to Europe, through the Middle East, and so on. London to New York was fast and direct. The vestiges of the Spanish and Portuguese Empires show up in the many links between South America, the Caribbean archipelago, and the Iberian peninsula. A cool thing is that you can measure these relative latencies yourself, using the present-day internet. If you run a website with a decent amount of worldwide traffic, you can use that traffic to map out how the internet responds with regards to you, and see how that matches with the gross structure of the 'net. I wrote about a cheap and cheerful way to generate this data last year, and the code has since been open-sourced as part of Yahoo's Boomerang measurement framework. Read more...

Tao SteinEngineering

Facebook becomes a USENIX Patron

Posted about 8 years ago
blog post · Infra · Data · Web · Mobile · Compute · Open Source · Research · Graph · Languages · PHP · HipHop · Platformmore

From its beginnings, Facebook has had to solve novel systems challenges to help us scale and grow. The idea of the social graph, and its implementation as a web and mobile platform have repeatedly pushed our computer systems into uncharted territory. The workloads are non-traditional, graph-oriented and write-heavy, and the system has grown rapidly to a base of 350M users around the world. We have survived and thrived due to healthy innovation and creativity, but we haven't done it alone. We have benefited from innovation in both the open source and computer systems communities. The USENIX Association is an essential hub in the systems community and today we are pleased to announce that we are becoming a Patron of the USENIX Association. Read more...

Haiping ZhaoSenior Server Engineer at Facebook
Ashish ThusooEngineering at Facebook

Hive - A Petabyte Scale Data Warehouse using Hadoop

Posted about 9 years ago

A number of engineers from Facebook are speaking at the Yahoo! Hadoop Summit today about the ways we are using Hadoop and Hive for analytics. Hive is an open source, peta-byte scale date warehousing framework based on Hadoop that was developed by the Data Infrastructure Team at Facebook. In this blogpost we'll talk more about Hive, how it has been used at Facebook and its unique architecture and capabilities. Read more...

Aditya AgarwalDirector of Engineering at Facebook

Adapting Open Source Software

Posted about 9 years ago
blog post · Web · Compute · Open Source · Languages · News Feed

Facebook is a fan of open source software. We are heavy users of packages like PHP, MySQL, Memcache and also believe in contributing technology like Thrift and Scribe back to the community. It's interesting not just to hear that companies are using open source software (everyone is!) but rather about how they adapt the software to better suit their needs. Read more...

Keep Updated

Stay up-to-date via RSS with the latest open source project releases from Facebook, news from our Engineering teams, and upcoming events.

Subscribe
Facebook © 2017