GitLab.com melts down after wrong directory deleted, backups fail

  • 1 February 2017
  • 3 replies
  • 21 views

Userlevel 7
Badge +54

Upstart said it had outgrown the cloud – now five out of five restore tools have failed

 
                               


 
1 Feb 2017 at 02:02, Simon Sharwood Source-code hub GitLab.com is in meltdown after experiencing data loss as a result of what it has suddenly discovered are ineffectual backups.
 
On Tuesday evening, Pacific Time, the startup issued a sobering series of tweets we've listed below. Behind the scenes, a tired sysadmin, working late at night in the Netherlands, had accidentally deleted a directory on the wrong server during a frustrating database replication process: he wiped a folder containing 300GB of live production data that was due to be replicated.
 
Just 4.5GB remained by the time he canceled the rm -rf command. The last potentially viable backup was taken six hours beforehand.
 
Full Article

3 replies

Ouch! Wouldn't want to be that sysadmin.
Userlevel 7
Badge +54
@ wrote:
Ouch! Wouldn't want to be that sysadmin.
This short clip says a lot:
 
"So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place."
Userlevel 7
Badge +56
@ wrote:
@ wrote:
Ouch! Wouldn't want to be that sysadmin.
This short clip says a lot:
 
"So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place."
The biggest thing i see many times from PC's I repair! "or set up in the first place" :(

Reply