Skip to main content

Facebook outage was its biggest ever

Doug Gross
Facebook's outage Thursday was the site's worst in more than four years, lasting 2˝ hours for most users.
Facebook's outage Thursday was the site's worst in more than four years, lasting 2˝ hours for most users.
STORY HIGHLIGHTS
  • Facebook outage was worst in 4½ years, engineer says
  • Facebook was down for 2½ hours Thursday, the second day in a row
  • Outage was caused by a problem with how the Facebook network fixes errors
  • The problem required engineers to shut down the site, letting people back in slowly
RELATED TOPICS

(CNN) -- Facebook's outage that shut down the site for many users Thursday was its worst since it became a worldwide social-networking powerhouse, according to an engineer for the site.

In a blog post, Robert Johnson said an error in Facebook's automated system for fixing software problems made a problem worse instead of better.

"This is the worst outage we've had in over four years, and we wanted to first of all apologize for it," Johnson wrote.

He said the outage lasted for about 2½ hours for millions of Facebook's half-billion users. It began about 2:30 p.m. ET.

A Facebook spokeswoman said the issue was not related to the one from Wednesday, when problems with a third-party network provider shut the site down, or slowed it badly, for several hours. She didn't elaborate on the nature of Thursday's outage.

On Facebook's blog for developers, a post Thursday had said the site was "experiencing latency issues," or a time delay, with its platform.

In layman's terms, what Johnson described was a situation in which a problem popped up and lots of different parts of Facebook's system tried to fix it at the same time.

That led to hundreds of thousands of requests every second to the databases that handle error reports.

"We had entered a feedback loop that didn't allow the databases to recover," Johnson wrote. "The way to stop the feedback cycle was quite painful -- we had to stop all traffic to this database cluster, which meant turning off the site.

"Once the databases had recovered and the root cause had been fixed, we slowly allowed more people back onto the site."

If the last outage comparable to Thursday's was more than 4½ years ago, as Johnson said, then this one was the site's biggest by a massive margin. In mid-2006, Facebook had about 7 million registered users -- about 1.4 percent of the number it has now.

"We want you to know that we take the performance and reliability of Facebook very seriously," Johnson wrote Thursday.

Some users of Facebook Connect, which allows people to access other sites via Facebook, complained the outage affected their ability to surf the Web.

"Today I had to register for a different CNN.com account because the Facebook connect was broken," Joshua Patterson, 30, of Portland, Oregon, user told CNN's iReport.

Facebook's outage also led thousands of social networkers to Twitter to, well ... talk about Facebook.

"Somebody needs to fix Facebook ASAP," wrote Gizmodo blogger Rosa Golijan on her Twitter feed. "I don't like having to tweet to show that I 'like' all the stories about it being down."

Several sarcastic Twitter users bemoaned the fate of the undernourished virtual plants and animals of FarmVille, while others (presumably with a degree of self-awareness) remarked on the ramifications of the outage.

"OPB BREAKING NEWS: Facebook is down," read a message on Oregon Public Broadcasting's feed. "Worker productivity rises. U.S. climbs out of recession."

[TECH: NEWSPULSE]

Most popular Tech stories right now