The Wonderful World of Video Data Management

Giant Bomb's Game of the Year awards are fast approaching, and you know what that means! Asset compilation! This is the time of year when we descend into the dusty catacombs of Giant Bomb's video archives and pull out all the relevant footage of games we're talking about. Since a few of you have requested insight into such topics, I thought now would be a good time to shed some light on how we manage our video data.

 Capturing some exciting gameplay footage.
The life of a Giant Bomb gameplay video, such as a Quick Look, begins at the capture stage. We plug our consoles into our capture gear, turn on the microphones, and hit "record" (it's a little more complicated than that, but that's for another post). The resulting video file is saved onto our capture machine, which resides in the control room. From there, the producer copies this "master" file over the network to their machine so they can work on it.

At the same time, the master is copied from the capture machine's local hard drive to an attached external hard drive. This is so we can delete the master from the local drive to save space (the local drive is only 500GB, which lasts us a week at most). When the external hard drive gets full, we unplug it, label it "Capture Archive ##," and set it on the shelf. A new hard drive goes in its place, ready to accept hours upon hours of game footage.

 The compressor machine's output log.
Meanwhile, the producer is toiling away on a video, using the master file brought over from the capture machine. Once the video is edited and complete (at this point, we call it an "export"), it gets copied, over the network, to our compressor machine, where magical things happen. Because we use an off-site compression solution (i.e., we send our videos across the Internet for someone else to compress), we first have to get our videos down to an Internet-friendly size. To do this, we do a preliminary compression using the compressor machine in our office. This turns our unruly 30GB video files (for a 30-minute video), into a little under 1GB. This file also ends up becoming the HD version of the video that members download/see on the site. From there, our off-site encoder chews through the video we give it and spits the result out onto the website.

When that's done, the original file from the producer (the "export") and the resulting compressed file (the "output") are copied to an external hard drive attached to the compressor machine. As before, when that hard drive fills up, we unplug it and label it "Exports ##." So now, we have three iterations of the video file (master, export, and output) in two places (Capture Archive and Export Archive drives). But THAT'S. NOT. ALL.

We still have all the files on the producer's machine! As you can imagine, a project like a Video Review takes up a LOT of hard drive space with all the footage we have to capture (sometimes upwards of 100GB). Technically, we could archive all that stuff, but in reality, when you're done with a project, there's really no reason to keep all that unused footage around. That's where "trimmed" projects come in.

 Media manager, making it look easy.
Final Cut Pro, the software we use to edit all our videos, has a feature called "Media Manager." Media Manager looks at all the clips you used in your project and copies only those sections of the original video file to a folder you designate. That means, if I have an hour-long gameplay video, but only use 15 seconds of it, Media Manager will only keep the 15 seconds, not the entire hour. This cuts down the project from a suicidal 100GB to a much more manageable 5-10GB. When Media Manager is done doing its thing, we copy the folder it makes to an external drive called "Trimmed Projects." The cool thing about trimmed projects is that they keep the Final Cut project files as well, allowing us to make small changes to the video after the fact if we find a typo or something.

 Our super-secret hard drive database software.
Now all our projects are now safe and secure in the Capture, Export, and Trimmed Projects archives. Nice, but how do you FIND something if you have to go back and dig something up? We here at Giant Bomb use an extremely powerful and highly technical database application known as Google Docs. When a drive fills up, we enter the file names of all the files on that drive into a Google Spreadsheet document, with each drive getting its own tab. That way, all we have to do is search the document for the game we want footage of, and we can see exactly where it is.

And there you have it! Now, hard drives aren't the best backup solution (they are subject to data corruption, hardware failure, and nuclear strikes, unlike cloud-based storage), but they are cheap and easy. Our archiving solution may not be the most secure and flexible, but it is cost-effective and easy to do. And best of all, it works for us!
 
 The BOBOD: Big ol' Box o' Drives.
65 Comments
66 Comments
  • 66 results
  • 1
  • 2
Posted by drewbert

Giant Bomb's Game of the Year awards are fast approaching, and you know what that means! Asset compilation! This is the time of year when we descend into the dusty catacombs of Giant Bomb's video archives and pull out all the relevant footage of games we're talking about. Since a few of you have requested insight into such topics, I thought now would be a good time to shed some light on how we manage our video data.

 Capturing some exciting gameplay footage.
The life of a Giant Bomb gameplay video, such as a Quick Look, begins at the capture stage. We plug our consoles into our capture gear, turn on the microphones, and hit "record" (it's a little more complicated than that, but that's for another post). The resulting video file is saved onto our capture machine, which resides in the control room. From there, the producer copies this "master" file over the network to their machine so they can work on it.

At the same time, the master is copied from the capture machine's local hard drive to an attached external hard drive. This is so we can delete the master from the local drive to save space (the local drive is only 500GB, which lasts us a week at most). When the external hard drive gets full, we unplug it, label it "Capture Archive ##," and set it on the shelf. A new hard drive goes in its place, ready to accept hours upon hours of game footage.

 The compressor machine's output log.
Meanwhile, the producer is toiling away on a video, using the master file brought over from the capture machine. Once the video is edited and complete (at this point, we call it an "export"), it gets copied, over the network, to our compressor machine, where magical things happen. Because we use an off-site compression solution (i.e., we send our videos across the Internet for someone else to compress), we first have to get our videos down to an Internet-friendly size. To do this, we do a preliminary compression using the compressor machine in our office. This turns our unruly 30GB video files (for a 30-minute video), into a little under 1GB. This file also ends up becoming the HD version of the video that members download/see on the site. From there, our off-site encoder chews through the video we give it and spits the result out onto the website.

When that's done, the original file from the producer (the "export") and the resulting compressed file (the "output") are copied to an external hard drive attached to the compressor machine. As before, when that hard drive fills up, we unplug it and label it "Exports ##." So now, we have three iterations of the video file (master, export, and output) in two places (Capture Archive and Export Archive drives). But THAT'S. NOT. ALL.

We still have all the files on the producer's machine! As you can imagine, a project like a Video Review takes up a LOT of hard drive space with all the footage we have to capture (sometimes upwards of 100GB). Technically, we could archive all that stuff, but in reality, when you're done with a project, there's really no reason to keep all that unused footage around. That's where "trimmed" projects come in.

 Media manager, making it look easy.
Final Cut Pro, the software we use to edit all our videos, has a feature called "Media Manager." Media Manager looks at all the clips you used in your project and copies only those sections of the original video file to a folder you designate. That means, if I have an hour-long gameplay video, but only use 15 seconds of it, Media Manager will only keep the 15 seconds, not the entire hour. This cuts down the project from a suicidal 100GB to a much more manageable 5-10GB. When Media Manager is done doing its thing, we copy the folder it makes to an external drive called "Trimmed Projects." The cool thing about trimmed projects is that they keep the Final Cut project files as well, allowing us to make small changes to the video after the fact if we find a typo or something.

 Our super-secret hard drive database software.
Now all our projects are now safe and secure in the Capture, Export, and Trimmed Projects archives. Nice, but how do you FIND something if you have to go back and dig something up? We here at Giant Bomb use an extremely powerful and highly technical database application known as Google Docs. When a drive fills up, we enter the file names of all the files on that drive into a Google Spreadsheet document, with each drive getting its own tab. That way, all we have to do is search the document for the game we want footage of, and we can see exactly where it is.

And there you have it! Now, hard drives aren't the best backup solution (they are subject to data corruption, hardware failure, and nuclear strikes, unlike cloud-based storage), but they are cheap and easy. Our archiving solution may not be the most secure and flexible, but it is cost-effective and easy to do. And best of all, it works for us!
 
 The BOBOD: Big ol' Box o' Drives.
Staff
Posted by norton123

This is awesome. I've actually been wondering how it all works. 

Edited by HawkSE

Well done. Thanks for letting us in on the secrets of GB

Posted by AndyPhifer

Sweet Christmas, that is insane. Also maybe why you guys have, hands down, the best non-Hollywood video content on the web.
Posted by TheAdmin

I love behind the scenes stuff... very neat to hear how it works. Do you guys just store the drives in a cool place till you need them again? Are you using just regular sata drives and an enclosure to make them into externals?

Posted by JJOR64

Can't wait till the GOTY videos are out.    :)
 
Thanks again for doing all that hard work.

Posted by ThatFrood

It doesn't look very wonderful.

Posted by mattjam3000

Very interesting as usual thanks Drew. Any chance I could get hold of one of those spare hard drives.

Posted by heatDrive88

That's pretty interesting. Considering you guys have so many people who do video production and touch all those files, I'm surprised you guys don't incorporate some sort of version and source control.

Posted by CharlesAlanRatliff

I find stuff like this fascinating, and it's cool to know how you guys handle all of this stuff.

Posted by dudeglove

Does the off site processing not end up a little costly?

Posted by MrHeist

Thanks for sharing all that!  It's fascinating to see how different houses rig together all the variables. 
 
Are drives segregated by WM site,  or does it all flow together depending on who's doing what?
 
Now that you're doing more and more videos with picture in picture (quick looks for kinect, move, etc) how much does that additional feed add to the storage size? And has it put a major increase on prep time for those videos? 
  
Have you thought about doing some sort of expandable RAID solution for backup (something like a bigass Drobo), or would you still tear through those too fast?

Posted by Merzalof

Thanks Drew, very cool to see how everything goes down on the back end of the video stuff.  Would love to see some more articles like this.

Posted by netfluffyG

Ah man, being a film editor has been my dream career for the past 6, almost 7 years and as unglamorous as it is, I can't help to stop and wish I could do what you do on a daily basis.

Posted by monsterwill

Yeah, you guys definitely have a smooth workflow for production. 

Posted by Hamz

Kudos to you for taking the time to share this with us. That's some interesting read right there. Great work! :)

Posted by kickinthehead

Thanks for the informative post! Sounds similar but more organized than the last place that I worked at :P

Posted by crusader8463

tl:dr. Make it a video next time. Those things are easy to make anyway. =)

Posted by MooseyMcMan

I'm not an expert on cloud stuff, but wouldn't a nuclear strike mess that up too?  
 
Great read though. Keep up the great work. 

Moderator
Edited by evilrazer

 Thank you, Drew, I was very curious about this and now I know the answer (:

Posted by Mesoian

Quote: 
"Final Cut Pro, the software we use to edit all our videos, has a feature called "Media Manager." Media Manager looks at all the clips you used in your project and copies only those sections of the original video file to a folder you designate. That means, if I have an hour-long gameplay video, but only use 15 seconds of it, Media Manager will only keep the 15 seconds, not the entire hour. This cuts down the project from a suicidal 100GB to a much more manageable 5-10GB. When Media Manager is done doing its thing, we copy the folder it makes to an external drive called "Trimmed Projects." The cool thing about trimmed projects is that they keep the Final Cut project files as well, allowing us to make small changes to the video after the fact if we find a typo or something. "
 
 
 
OH MY GOD! I Friggen hate Premiere's scratch disk system! I have MASSIVE files spread across 3 SATA external drives because I can't allow my primary HDD to get bogged down in work files. Media Manager sounds so much nicer...
 

Posted by russdog

Thanks for the info...very interesting to see how you handle this & compare to my strategies as a fellow videographer (on a much smaller scale.)

Posted by bluephoenixmedia

wow you guys are nuts to not use an online back up service....I've lost 2 500GB drives in the past because I was stupid and both contained years worth of video projects....I would immediately get a Carbonite account or something....damn....it's worth the investment trust me

Posted by ZmillA

I use google docs all the time

Posted by cooljammer00

I love busywork!  They should hire me.

Posted by LadyShayne

That was a genuinely interesting post. I appreciate the "How Giant Bomb Works" feature. It definitely puts everything you guys do into perspective.

Posted by DIABLOKU

nice post

Posted by damswedon

I actually feel smarter now.

Posted by Doctorchimp

Oh my god, all this is giving me bad flashbacks of me sitting in high school with hard drives all around compiling and compressing high school footage...god damn it... 
 
All that final cut pro knowledge I thought I would have dumped 2 years ago is all coming back.

Posted by Scooper

That was really interesting. It's insane how much space video takes up.

Posted by Shreik

Nice post Drew! I'm a sucker for everything behind the scenes,

Posted by Killroycantkill

*Takes note and steals all information from this blog*

Posted by countinhallways

Great post Drew. I have often found myself pondering these very subjects while browsing the site. Thanks.

Posted by Video_Game_King

Now that I have a rough idea of how you guys do videos, another question: any microphone recommendations?

Posted by Gaff

Hard questions must be asked: stacking all the drives on top of each other, which Whiskey Media staff measuring unit would we get?
A Chloe, an Ana, a Norm, a Brad or a Coonce?

Online
Posted by Gaff

Hard questions must be asked: stacking all the drives on top of each other, which Whiskey Media staff measuring unit would we get?
A Chloe, an Ana, a Norm, a Brad or a Coonce?

Online
Posted by Moloney

Interesting read. Cheers.
Posted by Brackynews

Fascinating. Love these "how-we-dos" for inspiration and general curiosity.
Though, lost opportunity not calling it the Wonderful Universe...
WU-DaMn

Posted by Winternet

I hope you're getting well paid, because you're doing the Lord's work.

Edited by Red12b
@Winternet said:

" I hope you're getting well paid, because you're doing the Lord's work. "

Never ask a man that, it's like asking a woman how old she is. 
 
1 gig?  
fuck... 
 
No wonder my monthly NZ gig allowance gets filled...
Posted by Red12b

Could you get Ryan to do one of these for the Bombcast?  
 
Because he edits that doesn't he? 

Posted by Death_Unicorn

That was super interesting, thanks!

Posted by Bigheart711

Nice. Very nice! :)

Edited by drewbert
@99X said:

" Are you using just regular sata drives and an enclosure to make them into externals? "

We use these. They make swapping drives all the time super easy.
 

@dudeglove

said:

" Does the off site processing not end up a little costly? "

Each video gets three separate compressions (Low, High, and HD), and we have five websites with video content, so it makes more sense to outsource it to a place that can process an unlimited number of videos at the same time. Since our office machine can only process a few videos at a time, the compression queue would fill up real quick.
 

@MrHeist

said:

" Are drives segregated by WM site,  or does it all flow together depending on who's doing what? Now that you're doing more and more videos with picture in picture (quick looks for kinect, move, etc) how much does that additional feed add to the storage size? And has it put a major increase on prep time for those videos?   Have you thought about doing some sort of expandable RAID solution for backup (something like a bigass Drobo), or would you still tear through those too fast? "

1. All Whiskey videos end up on the same Capture and Export drives. Trimmed drives are producer-specific.
2. The Quick Looks that use picture-in-picture/green screen stuff get composited in real time using our Tricaster system (yet another post for the future), so it doesn't require any post-processing and gets archived the same way a normal Quick Look would. The green screen on I Love Mondays, however, is done in post, so that ends up on the Trimmed drive. The green screen stuff does add to the turnaround time, but it's way faster with the Tricaster.
3. Right now, it's cheaper to keep pumping out hard drives than to have a huge RAID system. The initial purchase and setup cost for those things is significant, and we would still have to keep adding drives anyway.
 

@MooseyMcMan

said:

" I'm not an expert on cloud stuff, but wouldn't a nuclear strike mess that up too?  "

The way I understand it, a lot of cloud backup services duplicate their data across multiple data centers around the country. That way, if Seattle gets nuked, Atlanta still has a copy of your data.
 
@bluephoenixmedia said:

" wow you guys are nuts to not use an online back up service. "

We'd love to, but it's ridiculously expensive for how much data we generate.
 
@Video_Game_King said:

" any microphone recommendations? "

We use these.
 
@Gaff said:

" Hard questions must be asked: stacking all the drives on top of each other, which Whiskey Media staff measuring unit would we get? A Chloe, an Ana, a Norm, a Brad or a Coonce? "

Actually, each Export drive gets duplicated and stuck in a different box under Vinny's desk, so really we have about double the amount of hard drives you see in the picture. If we're assuming the hard drives are laying flat, I'm guessing about an Ana.
Staff
Posted by bluephoenixmedia
@drewbert:  Have you looked at Carbonite Pro yet? I believe it's roughly $100 a year or so and they support external drives for unlimited data. (at least they advertise it, not sure if there's an invisible cap) I'm pretty sure that's what leo laporte uses for his twit network.  I swear I'm not affiliated I just seriously have seen some terrible things happen to folks with data loss and I use a home media server myself with redundancy but I'm not dealing in the levels of data you guys are. As long as you don't mind the slow upload cap to their servers and run the system non stop for a few months you'll probably be able to have all of your drives backed up eventually.  I'm sure you guys could write if off in taxes as a business expense too.
Posted by Video_Game_King
@drewbert:
 
Shit. I'd ask for cheaper alternatives, but right now, I'm just concerned with how you get it on the computer, given that I don't see a USB plug or any cable that plugs into a computer. Balls don't plug into things.
Posted by drewbert
@bluephoenixmedia said:

" Have you looked at Carbonite Pro yet? "

According to this page, it's $250 per month for up to 500GB. Even if they could accommodate all our data, at that rate we'd be paying over $15,000 per month. Is there another option you're referring to? Services like that are tricky, too, since they often require the data to be resident on your computer at all times, which is impossible for us.
 
@Video_Game_King said:

" @drewbert:  Shit. I'd ask for cheaper alternatives, but right now, I'm just concerned with how you get it on the computer, given that I don't see a USB plug or any cable that plugs into a computer. Balls don't plug into things. "

Yeah, sorry, that's an XLR mic. It won't plug into a computer without something like this, but if you're serious about your audio, XLR is the way to go. I don't know much about USB mics, but I'm sure there are some armchair podcasters out there who can point you in the right direction.
Staff
Posted by lamiafusion

 I am doing this kind of stuff in school right now so it is awesome to see how you guys have streamlined the process for yourselves.     

Posted by bluephoenixmedia
@drewbert: wow, I assumed they had a flat rate...my bad...what about the amazon s3 service? I know shit gets expensive with video but maybe whiskey media could front the cost? what does tested use? the same method? 
Posted by Winternet
@Red12b: I didn't ask anything.
  • 66 results
  • 1
  • 2