Blog

Data Disasters - Fool Me Once (and hopefully never again)

Data Disasters - Fool Me Once (and hopefully never again)

Show first post

63 replies

Userlevel 3

Daily backup to cloud.

Monthly backup to offsite USB-drive

Userlevel 5
Badge +4

Reading this and the comments I am reminded of a friend who does voice-over work that I set up with an external drive as backup to his notebook computer that he produced from. He called me up one day and told me that the external drive had died, and when I replied that that was okay as it was only a backup and we can just start with a new drive, there was a long silence. That's when he decided to tell me that he had been making space on his primary machine by "archiving" projects to his backup drive and deleting them from the notebook. That long talk about single point of failure had not taken hold.

Userlevel 6
Badge +5

We always make sure we have local and cloud backups. We had a client who had been hit with ransomware and the only saving grace was their cloud backup. They didn’t have insurance, so their cloud backup was their only chance of restoring their data.

We require all of our managed clients to have on-prem and cloud backups. 

Userlevel 4

Daily backup to azure cloud and internal server.

Userlevel 4

Had an issue several years ago where I thought I had everything backed up completely. Suffered a drive failure and realized that a bulk of my music collection was toast, and not actually backed up. It wasn’t a critical business loss but it was still consequential to me. Lesson learned: Never take your backups for granted and have disaster test scenarios. 

Userlevel 3

Backups are a must, there are so many risks these days that you really need to be prepared.

Userlevel 7
Badge +25

I am big big big on backup, and push everyone I know to consider two backups. One local for fast recovery of deleted files or damaged drive, and one remote for disaster recovery so that the source and backup do not live in the same place. And I recommend that any cloud backup service include a “rewind” function to allow one to go back in time to a previous date , and recovery older versions of files. Why?

 

My PC was hit with ransomware. Sadly, it hit while backing up to a locally connected drive, and that drive was hit as well, Bad timing. But at the time, I used Dropbox as an offsite backup for important documents , files, and media. Sadly (I thought) the auto backup synced all my files with the encrypted ones. But a call to Dropbox and they rewound me to the first version of each file that was not encrypted.  I did a low level format on the PC, installed a fresh OS, reinstalled the apps I needed,  and turned on Dropbox. Within a half day, all my files were back to before the strike.  So yea, backup is your best protection against ransomware for sure. 

Userlevel 2

I always try to be prepared for sudden events, in fact every week a full system backup is scheduled... luckily I never needed to restore, but you never know!

Userlevel 7
Badge +54

I always try to be prepared for sudden events, in fact every week a full system backup is scheduled... luckily I never needed to restore, but you never know!

 

That is the just the point, we never do know what is around the corner.

Userlevel 2
Badge +1

I’ll keep it short, but my story is about one of my customers for whom I configured a backup solution and schedule for their server. The only thing they had to do is manually switch external drives on a daily basis. The reason for this is because we agreed on having a full backup outside of the premises. In addition the off-site backup was encrypted, so should they loose the drive the data would not be open to the public.

This setup worked for years and once in a while we needed the backup te restore some files a user accidentally deleted. So no restore required due to a disaster. Until one day they were hit by ransomeware. On one pc but a pc with access to some SMB shares on the server. A user clicked on a link in a email and downloaded some payload. Webroot was in place, but unable to detect the malware. Probably a zero-day attack. And files were being encrypted, and encrypted……

‘But we have a backup, right” they said. ‘Sure we do’ I responded and so I rushed in to start the restore the files (after first isolating and removing the  ransomware on the user’ pc). Only to discover that they didn’t follow my backup procedure and schedule for the last 3 weeks, because “it’s such a hassle to do once a day”!!.

Now they were confronted with a 3 week old backup as their only route back. 3 weeks of data lost of which a DB with all the scanned documents from their customers. It took them a while to digest this info, but they blamed themselves and started recovering the data in another way. They succeeded, but with a lot of extra time spend. A lot of extra time spend I can tell you. And since then? They want to backup twice a day and whenever they receive an email they don’t trust? They delete it and wait to see if it was important or not. If so, the customer will resent the email.

The morale? Whatever you put in place, whatever technology you have, as long as you don’t see the value of it you might end up like this customer of mine. Only once. You probably don’t make such a mistake twice….

Userlevel 7
Badge +8

We had a customer who refused to switch from a cheaper Backup software. We wernt firm enough and when the server died guess what let them down. 

Userlevel 5

A long time ago in a different lifetime, I thought just using a USB drive for my own personal backup was great.  And it was. Until I lost the drive. In theory no big deal but my PC was starting to have issues so did the full refresh back to factory and then remembered I had lost the backup. I have learned my lesson.

Userlevel 7
Badge +4

We had a client that declined to invest in a decent backup despite numerous recommendations. Suffice to say that after being Cryptolocker attacked (3 times!!) and many bills for recovery man hours, they are now a big advocate of DR!

Userlevel 4

I’ve had some form of file and folder backups even before joining a MSP. We try to get everyone to adopt onsite and cloud backups, at least for important machines.

So far the most important lesson I’ve learned was actually after taking a backup to redeploy a fresh OS to a machine, only to find out after wiping the HDD that the backup was corrupt…
Now I always verify my backups before trusting them, including audits of our clients backups, just in case.

Userlevel 4

I backup to OneDrive and also have continuous backups to a local nas drive.  

Userlevel 7
Badge +25

For large clients/companies, I have used Retrospect, which I have used for decades. It is sadly not cheap, thus big companies, but I like it because the users do not have to do anything at all, it connects through an encrypted client for access, and backup media is never connected to any user’s computer. It is also a server pull solution rather than a client push, so (nearly) impossible for a user machine to infect the backup server, especially since files are transferred encrypted and always as data to the backup drive, and never executed.)  It does network backup, and runs on a connected server with only LAN access. (And an occasionally remote user if needed, adding a specific firewall rule.) That server has all inbound traffic blocked, so it can only pull in data.  I run a personal copy at home to protect my home and wife’s business. I scripted a system that un-mounts and uses a power line controller to turn on and off a set of arrays, so I get a rotational stack of drives with only one array connected at a time. Serious overkill, but after being attacked by ransomware, I am paranoid now. This is in addition to Carbonite and Dropbox cloud backup. 

Maybe too much?  🤔

Userlevel 7
Badge +25

My Mac makes backups every hour (Time Machine). Once a month I’ll use SuperDuper to backup and clone the Mac. I also backup on a flash drive, plus I have a backup on iCloud.

~ Backup ~ Backup ~ Backup ~ 😀

Oh, I forgot, I also have TimeMachine on our Macs. Yea, that too!  ;-)

Userlevel 5
Badge +1

Have you ever experienced a severe data disaster?

I would call my personal 2TB failure a data disaster because it’s happened at least 3 times in the timespan of 6 years and I never had a backup solution because I was too lazy.. but realised I lost a lot old gems

 

What was the catalyst that led to you adopting a backup solution?

Realizing that I really need to stop being lazy because it only leads to carelessness and myself to blame for losing precious files!

 

Do you use cloud or local backups?

Local backups to a NAS Synology device

Userlevel 7
Badge +25

Do you use cloud or local backups?

Local backups to a NAS Synology device

One reason to also have a remote backup is disaster recovery. Earthquake, fire, flood, or burglary (for examples) can take out both your machine and it’s backup if stored in the sane location.
 

For people who do not want to use the cloud as the alternate (I’ve heard a lot of reasons), I suggest a rotating local device that gets swapped with another every ‘n’ days with your work location or a friends house that is a distance away. ‘n’ depends on how much of a loss you are willing to accept vs the hassle of swapping the backup device all the time. You may never need it. But if you do, it’s a life saver. 

Userlevel 7
Badge +4

Datto has got to have one of the best DR system available it got tons of service for Enterprise to SMB solution wouldn't consider using anything else when it come to DR and Backups 

Yes and their SaaS protection is decent as well.

Userlevel 3

At home, I have a 2nd encrypted HDD and perform a daily backup on the important staff (images and docs).

 

At work we have daily incremental backups to a backup server and one offsite backup to another vendor. 

 

Thankfully I never faced a data disaster. A customer of ours at work had a ransomware once and we needed to restore his vm the previous state.

Userlevel 4

So far our work has been lucky and our backups have always worked OK, my wife personally doesn’t trust every online backup option and saves all of our pictures of my son on various memory sticks twice, keeping them in a box.  I think she’d print them so she had a hardcopy if I let her!

Userlevel 6
Badge +1

A few years ago a customer experienced a severe ransomware attack in which the backup to disk was also infected and encrypted. Thanks to (what some believe was old-fashioned) tape backup we were able to restore the environment in a few hours for the most important parts, with the rest in the week after that. I use this real-life example to explain the importance of offline backup.

Userlevel 3

Cloud backup and DPM solution
 

 

Userlevel 7
Badge +6

A few years ago we had a customer’s office burn down. Since we had backups, we were able to spin up the servers in the cloud and get everyone up and running in a few hours.

Reply