Giant Bomb's Game of the Year awards are fast approaching, and you know what that means! Asset compilation! This is the time of year when we descend into the dusty catacombs of Giant Bomb's video archives and pull out all the relevant footage of games we're talking about. Since a few of you have requested insight into such topics, I thought now would be a good time to shed some light on how we manage our video data.
Capturing some exciting gameplay footage. The life of a Giant Bomb gameplay video, such as a Quick Look, begins at the capture stage. We plug our consoles into our capture gear, turn on the microphones, and hit "record" (it's a little more complicated than that, but that's for another post). The resulting video file is saved onto our capture machine, which resides in the control room. From there, the producer copies this "master" file over the network to their machine so they can work on it.
At the same time, the master is copied from the capture machine's local hard drive to an attached external hard drive. This is so we can delete the master from the local drive to save space (the local drive is only 500GB, which lasts us a week at most). When the external hard drive gets full, we unplug it, label it "Capture Archive ##," and set it on the shelf. A new hard drive goes in its place, ready to accept hours upon hours of game footage.
The compressor machine's output log. Meanwhile, the producer is toiling away on a video, using the master file brought over from the capture machine. Once the video is edited and complete (at this point, we call it an "export"), it gets copied, over the network, to our compressor machine, where magical things happen. Because we use an off-site compression solution (i.e., we send our videos across the Internet for someone else to compress), we first have to get our videos down to an Internet-friendly size. To do this, we do a preliminary compression using the compressor machine in our office. This turns our unruly 30GB video files (for a 30-minute video), into a little under 1GB. This file also ends up becoming the HD version of the video that members download/see on the site. From there, our off-site encoder chews through the video we give it and spits the result out onto the website.
When that's done, the original file from the producer (the "export") and the resulting compressed file (the "output") are copied to an external hard drive attached to the compressor machine. As before, when that hard drive fills up, we unplug it and label it "Exports ##." So now, we have three iterations of the video file (master, export, and output) in two places (Capture Archive and Export Archive drives). But THAT'S. NOT. ALL.
We still have all the files on the producer's machine! As you can imagine, a project like a Video Review takes up a LOT of hard drive space with all the footage we have to capture (sometimes upwards of 100GB). Technically, we could archive all that stuff, but in reality, when you're done with a project, there's really no reason to keep all that unused footage around. That's where "trimmed" projects come in.
Media manager, making it look easy. Final Cut Pro, the software we use to edit all our videos, has a feature called "Media Manager." Media Manager looks at all the clips you used in your project and copies only those sections of the original video file to a folder you designate. That means, if I have an hour-long gameplay video, but only use 15 seconds of it, Media Manager will only keep the 15 seconds, not the entire hour. This cuts down the project from a suicidal 100GB to a much more manageable 5-10GB. When Media Manager is done doing its thing, we copy the folder it makes to an external drive called "Trimmed Projects." The cool thing about trimmed projects is that they keep the Final Cut project files as well, allowing us to make small changes to the video after the fact if we find a typo or something.
Our super-secret hard drive database software. Now all our projects are now safe and secure in the Capture, Export, and Trimmed Projects archives. Nice, but how do you FIND something if you have to go back and dig something up? We here at Giant Bomb use an extremely powerful and highly technical database application known as Google Docs. When a drive fills up, we enter the file names of all the files on that drive into a Google Spreadsheet document, with each drive getting its own tab. That way, all we have to do is search the document for the game we want footage of, and we can see exactly where it is.
And there you have it! Now, hard drives aren't the best backup solution (they are subject to data corruption, hardware failure, and nuclear strikes, unlike cloud-based storage), but they are cheap and easy. Our archiving solution may not be the most secure and flexible, but it is cost-effective and easy to do. And best of all, it works for us! The BOBOD: Big ol' Box o' Drives. 65 Comments
Giant Bomb's Game of the Year awards are fast approaching, and you know what that means! Asset compilation! This is the time of year when we descend into the dusty catacombs of Giant Bomb's video archives and pull out all the relevant footage of games we're talking about. Since a few of you have requested insight into such topics, I thought now would be a good time to shed some light on how we manage our video data.
Capturing some exciting gameplay footage. The life of a Giant Bomb gameplay video, such as a Quick Look, begins at the capture stage. We plug our consoles into our capture gear, turn on the microphones, and hit "record" (it's a little more complicated than that, but that's for another post). The resulting video file is saved onto our capture machine, which resides in the control room. From there, the producer copies this "master" file over the network to their machine so they can work on it.
At the same time, the master is copied from the capture machine's local hard drive to an attached external hard drive. This is so we can delete the master from the local drive to save space (the local drive is only 500GB, which lasts us a week at most). When the external hard drive gets full, we unplug it, label it "Capture Archive ##," and set it on the shelf. A new hard drive goes in its place, ready to accept hours upon hours of game footage.
The compressor machine's output log. Meanwhile, the producer is toiling away on a video, using the master file brought over from the capture machine. Once the video is edited and complete (at this point, we call it an "export"), it gets copied, over the network, to our compressor machine, where magical things happen. Because we use an off-site compression solution (i.e., we send our videos across the Internet for someone else to compress), we first have to get our videos down to an Internet-friendly size. To do this, we do a preliminary compression using the compressor machine in our office. This turns our unruly 30GB video files (for a 30-minute video), into a little under 1GB. This file also ends up becoming the HD version of the video that members download/see on the site. From there, our off-site encoder chews through the video we give it and spits the result out onto the website.
When that's done, the original file from the producer (the "export") and the resulting compressed file (the "output") are copied to an external hard drive attached to the compressor machine. As before, when that hard drive fills up, we unplug it and label it "Exports ##." So now, we have three iterations of the video file (master, export, and output) in two places (Capture Archive and Export Archive drives). But THAT'S. NOT. ALL.
We still have all the files on the producer's machine! As you can imagine, a project like a Video Review takes up a LOT of hard drive space with all the footage we have to capture (sometimes upwards of 100GB). Technically, we could archive all that stuff, but in reality, when you're done with a project, there's really no reason to keep all that unused footage around. That's where "trimmed" projects come in.
Media manager, making it look easy. Final Cut Pro, the software we use to edit all our videos, has a feature called "Media Manager." Media Manager looks at all the clips you used in your project and copies only those sections of the original video file to a folder you designate. That means, if I have an hour-long gameplay video, but only use 15 seconds of it, Media Manager will only keep the 15 seconds, not the entire hour. This cuts down the project from a suicidal 100GB to a much more manageable 5-10GB. When Media Manager is done doing its thing, we copy the folder it makes to an external drive called "Trimmed Projects." The cool thing about trimmed projects is that they keep the Final Cut project files as well, allowing us to make small changes to the video after the fact if we find a typo or something.
Our super-secret hard drive database software. Now all our projects are now safe and secure in the Capture, Export, and Trimmed Projects archives. Nice, but how do you FIND something if you have to go back and dig something up? We here at Giant Bomb use an extremely powerful and highly technical database application known as Google Docs. When a drive fills up, we enter the file names of all the files on that drive into a Google Spreadsheet document, with each drive getting its own tab. That way, all we have to do is search the document for the game we want footage of, and we can see exactly where it is.
And there you have it! Now, hard drives aren't the best backup solution (they are subject to data corruption, hardware failure, and nuclear strikes, unlike cloud-based storage), but they are cheap and easy. Our archiving solution may not be the most secure and flexible, but it is cost-effective and easy to do. And best of all, it works for us! The BOBOD: Big ol' Box o' Drives.
I love behind the scenes stuff... very neat to hear how it works. Do you guys just store the drives in a cool place till you need them again? Are you using just regular sata drives and an enclosure to make them into externals?
That's pretty interesting. Considering you guys have so many people who do video production and touch all those files, I'm surprised you guys don't incorporate some sort of version and source control.
Thanks for sharing all that! It's fascinating to see how different houses rig together all the variables.
Are drives segregated by WM site, or does it all flow together depending on who's doing what?
Now that you're doing more and more videos with picture in picture (quick looks for kinect, move, etc) how much does that additional feed add to the storage size? And has it put a major increase on prep time for those videos?
Have you thought about doing some sort of expandable RAID solution for backup (something like a bigass Drobo), or would you still tear through those too fast?
Ah man, being a film editor has been my dream career for the past 6, almost 7 years and as unglamorous as it is, I can't help to stop and wish I could do what you do on a daily basis.
Quote: "Final Cut Pro, the software we use to edit all our videos, has a feature called "Media Manager." Media Manager looks at all the clips you used in your project and copies only those sections of the original video file to a folder you designate. That means, if I have an hour-long gameplay video, but only use 15 seconds of it, Media Manager will only keep the 15 seconds, not the entire hour. This cuts down the project from a suicidal 100GB to a much more manageable 5-10GB. When Media Manager is done doing its thing, we copy the folder it makes to an external drive called "Trimmed Projects." The cool thing about trimmed projects is that they keep the Final Cut project files as well, allowing us to make small changes to the video after the fact if we find a typo or something. "
OH MY GOD! I Friggen hate Premiere's scratch disk system! I have MASSIVE files spread across 3 SATA external drives because I can't allow my primary HDD to get bogged down in work files. Media Manager sounds so much nicer...
wow you guys are nuts to not use an online back up service....I've lost 2 500GB drives in the past because I was stupid and both contained years worth of video projects....I would immediately get a Carbonite account or something....damn....it's worth the investment trust me
Oh my god, all this is giving me bad flashbacks of me sitting in high school with hard drives all around compiling and compressing high school footage...god damn it...
All that final cut pro knowledge I thought I would have dumped 2 years ago is all coming back.
Hard questions must be asked: stacking all the drives on top of each other, which Whiskey Media staff measuring unit would we get? A Chloe, an Ana, a Norm, a Brad or a Coonce?
Hard questions must be asked: stacking all the drives on top of each other, which Whiskey Media staff measuring unit would we get? A Chloe, an Ana, a Norm, a Brad or a Coonce?
" Are you using just regular sata drives and an enclosure to make them into externals? "
We use these. They make swapping drives all the time super easy.
@dudeglove
said:
" Does the off site processing not end up a little costly? "
Each video gets three separate compressions (Low, High, and HD), and we have five websites with video content, so it makes more sense to outsource it to a place that can process an unlimited number of videos at the same time. Since our office machine can only process a few videos at a time, the compression queue would fill up real quick.
@MrHeist
said:
" Are drives segregated by WM site, or does it all flow together depending on who's doing what? Now that you're doing more and more videos with picture in picture (quick looks for kinect, move, etc) how much does that additional feed add to the storage size? And has it put a major increase on prep time for those videos? Have you thought about doing some sort of expandable RAID solution for backup (something like a bigass Drobo), or would you still tear through those too fast? "
1. All Whiskey videos end up on the same Capture and Export drives. Trimmed drives are producer-specific. 2. The Quick Looks that use picture-in-picture/green screen stuff get composited in real time using our Tricaster system (yet another post for the future), so it doesn't require any post-processing and gets archived the same way a normal Quick Look would. The green screen on I Love Mondays, however, is done in post, so that ends up on the Trimmed drive. The green screen stuff does add to the turnaround time, but it's way faster with the Tricaster. 3. Right now, it's cheaper to keep pumping out hard drives than to have a huge RAID system. The initial purchase and setup cost for those things is significant, and we would still have to keep adding drives anyway.
@MooseyMcMan
said:
" I'm not an expert on cloud stuff, but wouldn't a nuclear strike mess that up too? "
The way I understand it, a lot of cloud backup services duplicate their data across multiple data centers around the country. That way, if Seattle gets nuked, Atlanta still has a copy of your data.
@bluephoenixmedia said:
" wow you guys are nuts to not use an online back up service. "
We'd love to, but it's ridiculously expensive for how much data we generate.
" Hard questions must be asked: stacking all the drives on top of each other, which Whiskey Media staff measuring unit would we get? A Chloe, an Ana, a Norm, a Brad or a Coonce? "
Actually, each Export drive gets duplicated and stuck in a different box under Vinny's desk, so really we have about double the amount of hard drives you see in the picture. If we're assuming the hard drives are laying flat, I'm guessing about an Ana.
@drewbert: Have you looked at Carbonite Pro yet? I believe it's roughly $100 a year or so and they support external drives for unlimited data. (at least they advertise it, not sure if there's an invisible cap) I'm pretty sure that's what leo laporte uses for his twit network. I swear I'm not affiliated I just seriously have seen some terrible things happen to folks with data loss and I use a home media server myself with redundancy but I'm not dealing in the levels of data you guys are. As long as you don't mind the slow upload cap to their servers and run the system non stop for a few months you'll probably be able to have all of your drives backed up eventually. I'm sure you guys could write if off in taxes as a business expense too.
Shit. I'd ask for cheaper alternatives, but right now, I'm just concerned with how you get it on the computer, given that I don't see a USB plug or any cable that plugs into a computer. Balls don't plug into things.
According to this page, it's $250 per month for up to 500GB. Even if they could accommodate all our data, at that rate we'd be paying over $15,000 per month. Is there another option you're referring to? Services like that are tricky, too, since they often require the data to be resident on your computer at all times, which is impossible for us.
@Video_Game_King said:
" @drewbert: Shit. I'd ask for cheaper alternatives, but right now, I'm just concerned with how you get it on the computer, given that I don't see a USB plug or any cable that plugs into a computer. Balls don't plug into things. "
Yeah, sorry, that's an XLR mic. It won't plug into a computer without something like this, but if you're serious about your audio, XLR is the way to go. I don't know much about USB mics, but I'm sure there are some armchair podcasters out there who can point you in the right direction.
@drewbert: wow, I assumed they had a flat rate...my bad...what about the amazon s3 service? I know shit gets expensive with video but maybe whiskey media could front the cost? what does tested use? the same method?
66 Comments