GitLab down after it deletes wrong directory and backups stumble

GitLab.com, a similar site to GitHub, providing source code version control repository used by software developers, has come a cropper after an employee accidentally deleted a directory on the wrong server.

GitLab.com is currently offline due to issues with our production database.

Sign up to our free newsletter.
Security news, advice, and tips.

We’re working hard to resolve the problem quickly. Follow GitLabStatus for the latest updates.

The Register describes what went wrong:

Behind the scenes, a tired sysadmin, working late at night in the Netherlands, had accidentally deleted a directory on the wrong server during a frustrating database replication process: he wiped a folder containing 300GB of live production data that was due to be replicated.

Just 4.5GB remained by the time he canceled the rm −rf command. The last potentially viable backup was taken six hours beforehand.

To be honest, that’s the kind of mistake that anyone could make all too easily if there aren’t the right safeguards in place. But it’s bad news for anyone who was relying on GitLab to look after their code and merge requests properly, and weren’t keeping their own backup.

For, as the site explains, GitLab’s own backups have just added to the headaches:

Out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.

Although it would be easy to bash GitLab, it is trying to make the best out of a bad situation by showing remarkable transparency with its dev-head users.

We accidentally deleted production data and might have to restore from backup. Google Doc with live notes https://t.co/EVRbHzYlk8

— GitLab.com Status (@gitlabstatus) February 1, 2017

The incident affected the database (including issues and merge requests) but not the git repo's (repositories and wikis).

— GitLab.com Status (@gitlabstatus) February 1, 2017

Not only has the site set up a regularly updated Google Doc, containing details of the steps it is taking to try to recover data and resume regular site access, but it is even currently live streaming on YouTube so you can watch its employees slowly restore data where possible.

Remember this – don’t feel too smug if you’re remembering making backups. Yes, it’s great that you make backups, but a backup can’t be relied upon unless you have *tested* that it restores properly.

More details about the incident can be found on GitLab’s blog.

Found this article interesting? Follow Graham Cluley on Twitter or Mastodon to read more of the exclusive content we post.

Graham Cluley •   @gcluley

Graham Cluley is a veteran of the anti-virus industry having worked for a number of security companies since the early 1990s when he wrote the first ever version of Dr Solomon's Anti-Virus Toolkit for Windows. Now an independent security analyst, he regularly makes media appearances and is an international public speaker on the topic of computer security, hackers, and online privacy. Follow him on Twitter at @gcluley, on Mastodon at @[email protected], or drop him an email.