Giant Bomb's Game of the Year awards are fast approaching, and you know what that means! Asset compilation! This is the time of year when we descend into the dusty catacombs of Giant Bomb's video archives and pull out all the relevant footage of games we're talking about. Since a few of you have requested insight into such topics, I thought now would be a good time to shed some light on how we manage our video data.
Capturing some exciting gameplay footage. The life of a Giant Bomb gameplay video, such as a Quick Look, begins at the capture stage. We plug our consoles into our capture gear, turn on the microphones, and hit "record" (it's a little more complicated than that, but that's for another post). The resulting video file is saved onto our capture machine, which resides in the control room. From there, the producer copies this "master" file over the network to their machine so they can work on it.
At the same time, the master is copied from the capture machine's local hard drive to an attached external hard drive. This is so we can delete the master from the local drive to save space (the local drive is only 500GB, which lasts us a week at most). When the external hard drive gets full, we unplug it, label it "Capture Archive ##," and set it on the shelf. A new hard drive goes in its place, ready to accept hours upon hours of game footage.
The compressor machine's output log. Meanwhile, the producer is toiling away on a video, using the master file brought over from the capture machine. Once the video is edited and complete (at this point, we call it an "export"), it gets copied, over the network, to our compressor machine, where magical things happen. Because we use an off-site compression solution (i.e., we send our videos across the Internet for someone else to compress), we first have to get our videos down to an Internet-friendly size. To do this, we do a preliminary compression using the compressor machine in our office. This turns our unruly 30GB video files (for a 30-minute video), into a little under 1GB. This file also ends up becoming the HD version of the video that members download/see on the site. From there, our off-site encoder chews through the video we give it and spits the result out onto the website.
When that's done, the original file from the producer (the "export") and the resulting compressed file (the "output") are copied to an external hard drive attached to the compressor machine. As before, when that hard drive fills up, we unplug it and label it "Exports ##." So now, we have three iterations of the video file (master, export, and output) in two places (Capture Archive and Export Archive drives). But THAT'S. NOT. ALL.
We still have all the files on the producer's machine! As you can imagine, a project like a Video Review takes up a LOT of hard drive space with all the footage we have to capture (sometimes upwards of 100GB). Technically, we could archive all that stuff, but in reality, when you're done with a project, there's really no reason to keep all that unused footage around. That's where "trimmed" projects come in.
Media manager, making it look easy. Final Cut Pro, the software we use to edit all our videos, has a feature called "Media Manager." Media Manager looks at all the clips you used in your project and copies only those sections of the original video file to a folder you designate. That means, if I have an hour-long gameplay video, but only use 15 seconds of it, Media Manager will only keep the 15 seconds, not the entire hour. This cuts down the project from a suicidal 100GB to a much more manageable 5-10GB. When Media Manager is done doing its thing, we copy the folder it makes to an external drive called "Trimmed Projects." The cool thing about trimmed projects is that they keep the Final Cut project files as well, allowing us to make small changes to the video after the fact if we find a typo or something.
Our super-secret hard drive database software. Now all our projects are now safe and secure in the Capture, Export, and Trimmed Projects archives. Nice, but how do you FIND something if you have to go back and dig something up? We here at Giant Bomb use an extremely powerful and highly technical database application known as Google Docs. When a drive fills up, we enter the file names of all the files on that drive into a Google Spreadsheet document, with each drive getting its own tab. That way, all we have to do is search the document for the game we want footage of, and we can see exactly where it is.
And there you have it! Now, hard drives aren't the best backup solution (they are subject to data corruption, hardware failure, and nuclear strikes, unlike cloud-based storage), but they are cheap and easy. Our archiving solution may not be the most secure and flexible, but it is cost-effective and easy to do. And best of all, it works for us! The BOBOD: Big ol' Box o' Drives. 65 Comments
Thanks for the info drew. As an aspiring designer/video producer I have so many questions I want to ask you and Vinny. All this stuff must be hell, but it looks like magic to the average user.
@drewbert: I wonder if I helped encouraged this. ; ) GoogleDocs haha, very nice.
@drewbert said:
" @bluephoenixmedia said:
" Have you looked at Carbonite Pro yet? "
According to this page, it's $250 per month for up to 500GB. Even if they could accommodate all our data, at that rate we'd be paying over $15,000 per month. Is there another option you're referring to? Services like that are tricky, too, since they often require the data to be resident on your computer at all times, which is impossible for us.
Yeah, I think there unlimited was for home use and all the data actually on the computer and externals.
I would be interested to know what your server set-up is like aswell, where are you hosted? Where do our video downloads come from? And members should back the video up at home. :P
Cool Drew. That was really interesting. It's nice to get a glimpse into what's going on in the "background" your site. Also. I'm surprised how many steps there are between recording the footage and the final product. I had imagined it would be a little easier than that.
Well, there goes my dream of recording gameplay videos of my own :P
This is a very interesting read Drew! A look into what's behind the video production such as this is always inspiring to me. Please do share more of the technical side of Whiskey Media's video production!
Awesome write up. I'd be great to see the evolution of the video work flow for whiskey media. Just some huge timeline with pictures of the gear and setups you have used from the inception of all this till now. I love gear porn.
Also use off site compression at work sometimes. Our computers are slow compared to our internet line that uploading it to a video hosting service, then downloading it and then uploading to our servers is quicker than doing it all locally. As well as freeing up the computer.
66 Comments