Giant Bomb's Game of the Year awards are fast approaching, and you know what that means! Asset compilation! This is the time of year when we descend into the dusty catacombs of Giant Bomb's video archives and pull out all the relevant footage of games we're talking about. Since a few of you have requested insight into such topics, I thought now would be a good time to shed some light on how we manage our video data.
Capturing some exciting gameplay footage. The life of a Giant Bomb gameplay video, such as a Quick Look, begins at the capture stage. We plug our consoles into our capture gear, turn on the microphones, and hit "record" (it's a little more complicated than that, but that's for another post). The resulting video file is saved onto our capture machine, which resides in the control room. From there, the producer copies this "master" file over the network to their machine so they can work on it.
At the same time, the master is copied from the capture machine's local hard drive to an attached external hard drive. This is so we can delete the master from the local drive to save space (the local drive is only 500GB, which lasts us a week at most). When the external hard drive gets full, we unplug it, label it "Capture Archive ##," and set it on the shelf. A new hard drive goes in its place, ready to accept hours upon hours of game footage.
The compressor machine's output log. Meanwhile, the producer is toiling away on a video, using the master file brought over from the capture machine. Once the video is edited and complete (at this point, we call it an "export"), it gets copied, over the network, to our compressor machine, where magical things happen. Because we use an off-site compression solution (i.e., we send our videos across the Internet for someone else to compress), we first have to get our videos down to an Internet-friendly size. To do this, we do a preliminary compression using the compressor machine in our office. This turns our unruly 30GB video files (for a 30-minute video), into a little under 1GB. This file also ends up becoming the HD version of the video that members download/see on the site. From there, our off-site encoder chews through the video we give it and spits the result out onto the website.
When that's done, the original file from the producer (the "export") and the resulting compressed file (the "output") are copied to an external hard drive attached to the compressor machine. As before, when that hard drive fills up, we unplug it and label it "Exports ##." So now, we have three iterations of the video file (master, export, and output) in two places (Capture Archive and Export Archive drives). But THAT'S. NOT. ALL.
We still have all the files on the producer's machine! As you can imagine, a project like a Video Review takes up a LOT of hard drive space with all the footage we have to capture (sometimes upwards of 100GB). Technically, we could archive all that stuff, but in reality, when you're done with a project, there's really no reason to keep all that unused footage around. That's where "trimmed" projects come in.
Media manager, making it look easy. Final Cut Pro, the software we use to edit all our videos, has a feature called "Media Manager." Media Manager looks at all the clips you used in your project and copies only those sections of the original video file to a folder you designate. That means, if I have an hour-long gameplay video, but only use 15 seconds of it, Media Manager will only keep the 15 seconds, not the entire hour. This cuts down the project from a suicidal 100GB to a much more manageable 5-10GB. When Media Manager is done doing its thing, we copy the folder it makes to an external drive called "Trimmed Projects." The cool thing about trimmed projects is that they keep the Final Cut project files as well, allowing us to make small changes to the video after the fact if we find a typo or something.
Our super-secret hard drive database software. Now all our projects are now safe and secure in the Capture, Export, and Trimmed Projects archives. Nice, but how do you FIND something if you have to go back and dig something up? We here at Giant Bomb use an extremely powerful and highly technical database application known as Google Docs. When a drive fills up, we enter the file names of all the files on that drive into a Google Spreadsheet document, with each drive getting its own tab. That way, all we have to do is search the document for the game we want footage of, and we can see exactly where it is.
And there you have it! Now, hard drives aren't the best backup solution (they are subject to data corruption, hardware failure, and nuclear strikes, unlike cloud-based storage), but they are cheap and easy. Our archiving solution may not be the most secure and flexible, but it is cost-effective and easy to do. And best of all, it works for us! The BOBOD: Big ol' Box o' Drives. 65 Comments
Ah man, being a film editor has been my dream career for the past 6, almost 7 years and as unglamorous as it is, I can't help to stop and wish I could do what you do on a daily basis.
Thanks for sharing all that! It's fascinating to see how different houses rig together all the variables.
Are drives segregated by WM site, or does it all flow together depending on who's doing what?
Now that you're doing more and more videos with picture in picture (quick looks for kinect, move, etc) how much does that additional feed add to the storage size? And has it put a major increase on prep time for those videos?
Have you thought about doing some sort of expandable RAID solution for backup (something like a bigass Drobo), or would you still tear through those too fast?
That's pretty interesting. Considering you guys have so many people who do video production and touch all those files, I'm surprised you guys don't incorporate some sort of version and source control.
I love behind the scenes stuff... very neat to hear how it works. Do you guys just store the drives in a cool place till you need them again? Are you using just regular sata drives and an enclosure to make them into externals?
Giant Bomb's Game of the Year awards are fast approaching, and you know what that means! Asset compilation! This is the time of year when we descend into the dusty catacombs of Giant Bomb's video archives and pull out all the relevant footage of games we're talking about. Since a few of you have requested insight into such topics, I thought now would be a good time to shed some light on how we manage our video data.
Capturing some exciting gameplay footage. The life of a Giant Bomb gameplay video, such as a Quick Look, begins at the capture stage. We plug our consoles into our capture gear, turn on the microphones, and hit "record" (it's a little more complicated than that, but that's for another post). The resulting video file is saved onto our capture machine, which resides in the control room. From there, the producer copies this "master" file over the network to their machine so they can work on it.
At the same time, the master is copied from the capture machine's local hard drive to an attached external hard drive. This is so we can delete the master from the local drive to save space (the local drive is only 500GB, which lasts us a week at most). When the external hard drive gets full, we unplug it, label it "Capture Archive ##," and set it on the shelf. A new hard drive goes in its place, ready to accept hours upon hours of game footage.
The compressor machine's output log. Meanwhile, the producer is toiling away on a video, using the master file brought over from the capture machine. Once the video is edited and complete (at this point, we call it an "export"), it gets copied, over the network, to our compressor machine, where magical things happen. Because we use an off-site compression solution (i.e., we send our videos across the Internet for someone else to compress), we first have to get our videos down to an Internet-friendly size. To do this, we do a preliminary compression using the compressor machine in our office. This turns our unruly 30GB video files (for a 30-minute video), into a little under 1GB. This file also ends up becoming the HD version of the video that members download/see on the site. From there, our off-site encoder chews through the video we give it and spits the result out onto the website.
When that's done, the original file from the producer (the "export") and the resulting compressed file (the "output") are copied to an external hard drive attached to the compressor machine. As before, when that hard drive fills up, we unplug it and label it "Exports ##." So now, we have three iterations of the video file (master, export, and output) in two places (Capture Archive and Export Archive drives). But THAT'S. NOT. ALL.
We still have all the files on the producer's machine! As you can imagine, a project like a Video Review takes up a LOT of hard drive space with all the footage we have to capture (sometimes upwards of 100GB). Technically, we could archive all that stuff, but in reality, when you're done with a project, there's really no reason to keep all that unused footage around. That's where "trimmed" projects come in.
Media manager, making it look easy. Final Cut Pro, the software we use to edit all our videos, has a feature called "Media Manager." Media Manager looks at all the clips you used in your project and copies only those sections of the original video file to a folder you designate. That means, if I have an hour-long gameplay video, but only use 15 seconds of it, Media Manager will only keep the 15 seconds, not the entire hour. This cuts down the project from a suicidal 100GB to a much more manageable 5-10GB. When Media Manager is done doing its thing, we copy the folder it makes to an external drive called "Trimmed Projects." The cool thing about trimmed projects is that they keep the Final Cut project files as well, allowing us to make small changes to the video after the fact if we find a typo or something.
Our super-secret hard drive database software. Now all our projects are now safe and secure in the Capture, Export, and Trimmed Projects archives. Nice, but how do you FIND something if you have to go back and dig something up? We here at Giant Bomb use an extremely powerful and highly technical database application known as Google Docs. When a drive fills up, we enter the file names of all the files on that drive into a Google Spreadsheet document, with each drive getting its own tab. That way, all we have to do is search the document for the game we want footage of, and we can see exactly where it is.
And there you have it! Now, hard drives aren't the best backup solution (they are subject to data corruption, hardware failure, and nuclear strikes, unlike cloud-based storage), but they are cheap and easy. Our archiving solution may not be the most secure and flexible, but it is cost-effective and easy to do. And best of all, it works for us! The BOBOD: Big ol' Box o' Drives.
66 Comments