I lost years' worth of work due to a hard drive crash

Yes I know, I should've had backups. And I have no one to blame but myself. There isn't any hope left of recovering any of the work, so this is kind of a eulogy (and a way to get myself to accept reality) before I start everything again from scratch.

A few days back I took my external hard disk drive to my sister's place to watch a movie. Stupidly enough I plugged that in to the TV directly instead of transferring it onto a pen drive as I always do. In fact, when I had the thought of transferring it onto a pen drive, a voice in my mind said, for some reason - "nah, this time we'll try and play it directly." And so I did. Not just that, I also plugged it into her new sound bar to play it off that when the TV didn't recognize it.

Of course, then I tried it on a laptop and it wouldn't recognize the drive. The usb plugged in sound would play and it would show up in Device Manager, but not in My Computer. Later at home, Disk Management saw the drive but labelled it "Unknown, not initialized". I tried it on multiple laptops with different cables but it wouldn't show. Partition and data recovery softwares wouldn't detect it either. There were no spinning sounds.

There was years worth of random data on it, but the important stuff I lost was my freelance work - artwork (digital paintings), photography and an indie game I had just started working on. Fortunately I hadn't built too many levels on it yet, and with a bit of time and patience I'll be able to rebuild that. But what I miss most is the script. It is a story driven game, so the script was full of random bits of dialogues, characters' personalities and research on psychology. If there was one file I could recover, it would be the script. I remember the plot, but there's no way all the details are still in my head. And there were so many tiny things, so many bursts of inspirations that I had scribbled down! There's no way to recollect those.

The worst bit is that only two weeks back, I left my job to focus completely on my art side business and on making this game. This doesn't change everything, but it just messed up my smooth transition from job to self employment. Unnecessary stress and hard work.

I had given the disk to a professional to check, who said that the data can't be recovered because the platters are damaged/ scratched, probably because of a sudden disconnection (or a power surge, my guess) I just came back from meeting him, getting my now useless HDD back with me. I guess professional services could still save the data, but I can't afford their upwards of 500-1000$ fee.

While the disk was with that guy, I spent two full days doing absolutely nothing. Playing video games, skipping meals, watching videos and generally being depressed while waiting for news from him. That feels like shit. These were the days that I was to spend immersed in my indie game work, while doing art on the side.

So this is the point where I accept reality, and start creating again from scratch. Now that I've written all this down, it's time to forget the past and work for the future. Maybe it's a blessing in disguise - that the script was getting bloated and was full of crappy stuff, and now I have a chance to forget that and create something brilliant! Or maybe not. Regardless, I don't have a choice. It's time to get back to work. And to backup that work.

Anyone have any similar hard disk heart break stories?

Yeah, I lost so much of my filmmaking work to an external drive as well. I learned to never use an external drive for anything I'd cry over losing, but then a HDD turned bad on me, too, so I don't know who to trust anymore.

This is obviously suggesting you shut the barn door after the horse has escaped, but the lesson going forward from these kinds of things is to always back up anything important to you. Always. Drives can break, be stolen, be destroyed in a fire or other calamity etc...

Backup to other media, at least, and preferably to the cloud. Storage solutions are cheap these days. For something like a script or art files you can just gmail them to yourself (or upload to Google Docs) to create a very secure backup. For something big like film files that's harder, but multi-terrabyte drives are cheap these days and you can also always buy a Blu Ray burner and create disc backups.

Backing up is a necessary aspect of good data management. Anything you can't afford to lose must be backed up.

I have had data loss myself when a hard drive went bad, erasing a lot of work, so I definitely know your pain, and I'm certainly not blaming anyone for not backing up (it is somewhat expensive, though getting cheaper, and can be a pain in the butt) but it HAS to be part of your workflow.

Cloud services are cheaper than ever, and once you go through the initial pain of uploading there are programs that can keep the backup relatively current.

@daiphyer: I know right? I guess keeping multiple copies of the most important work is what I'll do. Also stuff like dropbox. I did keep this script of mine in dropbox, but stopped updating it in around February cause it's inconvenient to copy paste it into the dropbox folder everytime. I wish it didn't need a dedicated folder from which to sync.

If you use Amazon Prime, you get unlimited photo/image storage - even RAW. It's pretty great.

That's what I had been doing, until around February when I stopped bothering to do it. Lesson learned! Fortunately, this happened towards the beginning of this project and will therefore not happen again.

Edit: GB's quoting can get weird

Yeah.

At my job we have an IT department and work on encrypted hard drives. At one point the OS on my hard drive became corrupt somehow (the hard drive itself didn't seem to be failing) and I couldn't do anything with it myself because of the encryption so I brought the laptop to IT and they gave me a "loaner" laptop said they would retrieve my data. A week later I asked what happened and they told me they forgot I wanted the data retrieved and they just wiped the laptop hard drive and re-issued it to someone else.

Fortunately my most important work was saved on the company servers so most of what I lost was just drafts and material for completed and already implemented projects, but it was a pretty good reminder of the importance of backing up (which I had done through the company servers, for the most part.) I wanted to yell at the IT guy but I knew that I would need his help at some point in the future so I just bit my tongue.

Data loss sucks.

@armaan8014: I'm so sorry to hear about what happened! It's a terrible thing and a scary reminder to us with a lot of data that backing up is a vital thing to do.

I'm sold on using a cloud backup service, so I don't have to drop money on a large physical HD that could break down the line, so what's a good service to subscribe to?

I know you said you know you ought to have backups but I will still post this info in hopes it comes in handy for you or another duder:

The saying that someone said or wrote to me years and years ago after losing some critical archive data and has always stayed in my head is the one that goes along the lines of "any data you do not have two backups of is not data worth saving".

On top of having at least one local backup solution, I cannot recommend an online backup system (I use Crashplan these days) enough. Having an off-site backup is critical when you might come up to things like fire, virus infection, malware, etc.

With that backup spiel out of the way, I hope you come out of this in an even better position, and are able to rebuild your script and game leaner, smarter, and tighter. All the best.

Obviously I haven't tested these, but pcmag has a reasonably good reputation (to my knowledge) so this is probably not a bad place to start.

Been there-ish. Double backup (preferably one off-site). I do a weekly Windows backup and I'll run a manual batch to backup my current working projects after I've done medium-major work on any of them. These days you could just crypt your data and sync it to cloud, I haven't done that though.

If the data is that important to you $500+ might not be too bad of a deal. How much is you time worth to you?

Maybe you've forgotten some of the old details, but you can come up with new, possibly even better stuff, just because of your past experiences. You still got the skills, actually you're even better than ever, getting better every day you create (but remember to relax and meditate every now and then ;)

Just keep on it, and remember to back that shit up.

@bonorbitz: Thanks :( I'm not sure either, I'll have to do my research now. But it seems there are some good answers here in this thread.

Thank you so much! The new design document/ script I just started work on now starts with the words "leaner, smarter and tighter" :)

Yep. It's a weird modern day thing that sucks more than we'd have expected some decades back.

Always keep a back-up of your back-up. A back-back-up (Or a dash-backwards, neutral jump as some may know it.) I, too, learned this the hard way. Not to the extreme degree that you did. My condolences. Gotta' say, I'm impressed by your attitude towards the entire thing. Definitely the right outlook. Wish you the best of luck with your game!

@willyod: That's exactly how I'm hoping it is! Optimism is vital, and a fresh start means I start again as a more experienced person, without any initial ideas holding me back. Meditation is on the agenda too. Thanks, your reply has got me pumped up :)

Great to hear, but I'm afraid I'm going to have to insist you add in the oxford comma to the phrase "leaner, smarter and tighter". Heh, best of luck! I've found in my experience that being forced to redo work sometimes results in you trimming the fat and distilling the process and the result into something even better.

Check out the Wirecutter's guide: http://thewirecutter.com/reviews/best-online-backup-service/

It's very detailed and will give you a good overview of the services. They recommend Crashplan, which I've used in the past and my wife still uses. The only complaint I have with Crashplan is that the client uses a ton of RAM if you have a lot to backup. They estimate 1GB of RAM for every 1TB of data. I've found it to be less than that, but still pretty hefty on a ~6TB backup. I liked the Backblaze client more, at least from a system resource perspective, but the feature set is a lot more limited than Crashplan.

@cameron said:

Thanks! I was planning on signing up for Crashplan based on @bigsocrates PC Mag link. Forgive my ignorance regarding the client using a ton of RAM, but can I just backup to the cloud at a time I designate so its system resource usage won't interfere with me gaming or using Photoshop or whatnot? I hope to just set it to backup and walk away when I want, so it doesn't just run in the background.

@bonorbitz: Crashplan has worked great for me, but know that by default it does auto backups throughout the day. The interface is clunky and a bit slow to open up so I find it easiest to set and forget, and allow to work in the background. I have had 2-3 instances (over the course of a couple of years; so pretty isolated) where it's tried to backup the large Illustrator file I am currently working on and I had to stop the backup process in order to be able to save the file.

edit: some settings you get to select for the auto back up include how long you need to be 'inactive' before a backup starts and CPU usage limits when backing up. If you want me to check on anything else let me know, hope this helps.

I did not like the very limited feature set/options in Backblaze; I bounced off that one immediately.

@bonorbitz: You can set up specific times for it to back up, or turn it off and on manually. As @whitegreyblack mentioned, you can also limit how many CPU resources it uses. Just be aware that, depending on your internet speed, any of those things are going to make the initial backup take a lot longer.

Paying $500 to get back years of work sounds like a pretty good option, especially considering it's work you're actively working on.

Finals are coming up in a week and I really should back everything up to avoid any ME KILLING MYSELF adventures. Thanks for the reminder!

I know it's a bit different, but when I lost my external HDD I lost about 500gb of anime that I had backlogged to watch.

Nowadays all the big torrent sites where you could get them from have been shut down, so I've lost pretty much the only way to get some more obscure shows. I also lost all my university work. I'd already finished uni for a few years, but it was nice to have something that I could look back on and see how good at writing I was

If anyone wants a recommended backup service, I use Backblaze. It pretty great, back up multiple drive and if you need lots of data recovered, they will send you a Hard Drive in the mail, I think that service is free too, as long as you return the drive to BackBlaze. Just make sure important file types are not excluded. It doesn't backup .exe by default (probably a good idea for security reasons) so make sure you have other ways to get your programs.

External drives are indeed a pain. They are portable but they can't handle being transported without breaking.

It takes guts to quit your job like that. I hope you somehow manage to make something good come out of it. Maybe make an even better game.

@whitegreyblack: Haha I did add the comma :P Hoping it works out, thanks again.

Thank you. Fortunately, I live in a country with low cost of living, and with a bit of planning and some freelance work (plus the fact that I don't have any debts or loans to pay off) I was able to take this risk, especially since I'm in my early 20s. It may be cliched, but you do only live once :)

I use crashplan on my two computers and I got my family to use crashplan as I was using a family account anyways. It has saved us a few times. It also keeps file revisions in case you need to go back for whatever reason. I just leave it on automatic and that works for me for the most part. If you want to disable it for a few hours while you game that works too.

For programming you should be using a source code manager. That too will save your butt.

Being paranoid kinda pays off when it concerns the stuff that you deem important. Back when i did my thesis I always saved my work in 3 different places just in case my harddrive would get corrupted for some reason. Doing those backups each day or each week might feel like extra bit of work that you dont really want to do but when you lose everything that extra effort pays itself off.

So in a nutshell my advice is when it comes to backing up stuff be paranoid.

A few things you can try if you're comfortable with it:

Since your drive is in an enclosure, it's possible that the controller on the enclosure has failed and the drive itself is ok. First thing I do with failed external drives is remove the drive from the enclosure and use a known, working external dock. Should just be a standard SATA/power connector. I can count on one hand the number of times where the drive wasn't standard and I was able to get an adapter from Fry's for under $10. If you're going to write off your drive as a loss, it wouldn't hurt to open the enclosure and just try the drive on its own.

Two big things I see with failed drives are stiction and failed main boards. If you're willing to write off your drive and data as gone and are going to toss the hard drive out, give these a try. It may work. It probably won't.

Easiest thing to do is if you have access to an identical drive, swap the PCB boards. Clean the contacts when you swap the board.

You could try freezing the drive for a few hours in a water-proof bag with some silica gel packets (condensation). Then plug the drive in and if it's recognized, pull off what you can in the 10 or so minutes you have to work with it.

As far as the game goes, you should look into version control for the next time. GitHub or Bit Bucket will keep copies of your scripts in case of HDD failure, with the added benefit of being able to revert to previous versions of your game in case something you do breaks a lot of shit.

@armaan8014: Yev from Backblaze here -> it's not a matter of if but when hard drives will fail. Sorry to hear about your loss, hopefully you can find a service that will avoid it in the future! Ping @Backblaze on twitter or msg me here if you have any questions about our service if you do look into it.

@zelyre: I am thinking of trying the hard disk from a different enclosure and connecting to the desktop. It's pretty much trash now anyway. Thanks!

Thanks for the support and suggestions everyone :) I've got some backup systems in place now and have started working on the script and game again. Also, I never knew how insanely easy it is to automatically save your word files to the Onedrive cloud. After you set it up, every time you save normally it saves to the cloud by default. Seamless and discreet. Oh well, lesson learned.