Enough is never enough.

I'm the digital equivalent of a hoarder. Not in terms of the mess, as I do try to keep everything organized, but in terms of sheer quantity of media. If I've got the space for it, I refuse to delete it. I've kept backups of all my disk-based games, even the ones I haven't played in 15+ years. I've ripped (or am in the process of ripping) my blu-ray library, both for ease of access and categorization. I've kept entire runs of television shows, hundreds and hundreds of gigabytes of music, and at one point I had something like fifty fucking gigabytes of just wallpaper. Who the hell needs fifty gigabytes of wallpaper?

Well, I did. At least until I decided to prune my overripe collection of digital artifacts in order to meet the physical limitations of my hard drive. I'd determine what was necessary, what I used on a semi-regular basis, and toss the rest out. Windows doesn't play nice when your hard drive is scraping up against the limit, and I've long ago maxed my laptop's meagre storage capacity, so every now and then I'd repeat the purge, only to acquire more media and restart the cycle. Each and every time, the core collection of "essential" media (as essential as you can get when you are talking about things like movies, music, and games, of course) would grow a bit, and each purge would free up less and less space.

Well, I've finally hit the breaking point. I'm unwilling to cut things down any farther. I've hit an impasse.

So, what to do?

1. Man up and cut the media down again. Not going to happen; that's the shortest term of all short term solutions, and I'd rather avoid repeating this dilemma for a good while.
2. Slap another hard drive in there. Very doable; for $100 or so I could double or even triple the space I have. But there's some downsides; no redundancy (if a hard drive fails, it takes all the data with it), you're adding another drive letter (and will keep adding drive letters if you need more space), and it isn't truly a long term solution.
3. Go a bit crazy.

Now, before I explain what I mean by Option Three, I'm going to have to get a bit...technical.

RAIDing For Fun And Profit

When you start to talk about large amounts of hard drive space, you inevitably start looking at multiple disks. Even the largest hard drives tap out at four terabytes of space, and while that would serve most sane people for a good while, I just owned up to having fifty gigabytes of wallpaper. I think we know where I fall on the sane/crazy scatter plot. Anyway, you cannot look into multiple hard drives without hearing about RAID. RAID stands for Redundant Array of Independent Disks, and it's a way to expand the capacity of your system by pooling drives together in order to increase redundancy, performance, and space in a sort of balancing act depending on exactly what kind of RAID you choose:

1. RAID 0 is pure performance. All of your data is spread across all the drives in the array, meaning cranked up throughput, but also a big downside: if ANY hard drive fails, you lose ALL your data. Yep, doesn't matter if three out of four HDDs are fine, if number four decides to eat it, then all of your data goes bye-bye. Needless to say, not the solution I'm looking for.
2. RAID 1 is purely about safeguarding your data. It's usually used with just two drives, where the first mirrors the second and will stop you from losing any data in the event of a single hard drive failure. Needless to say, there are no performance benefits, and you lose the capacity of an entire drive (so if you install two 3TB drives, you only have access to 3TB, not 6TB, as one whole drive is just a mirror). As it provides no space increase over a single drive, it doesn't fit what I need.
3. RAID 10 usually uses four drives, and combines RAID 0 and 1. Your data is split across two hard drives and the remaining two are used as a mirror, so you get the protection of RAID 1 and the speed of RAID 0. Still, you lose out on half of your capacity, and that's a big deal when you start talking 4+ drives.
4. RAID 5 uses up one drive's worth of space to calculate parity data, meaning that you can survive the failure of any one hard drive. As long as you replace that drive before another dies, your system will integrate that new drive into the array and you'll be back to full strength again.
5. RAID 6 is the same as RAID 5, but uses two drive's worth of space for parity data, and therefore can take two hard drive failures before you lose any data.

There are other RAID levels, but those are the most common. Now, RAID usually requires a dedicated piece of hardware known as a RAID controller in order to handle the above; most motherboards support some RAID levels, but I wouldn't trust them to deliver the performance nor the safety of a dedicated RAID card. However, we have other options, and that requires us getting VERY technical.

ZFS: Zebras For Science!

Got your attention, didn't I?

ZFS actually stands for Zettabyte File System, which why I didn't use that as the paragraph head. Not nearly as exciting. I'm going to leave out a LOT of the details, partly because I find them boring, and partly because I don't quite understand all of them myself. The key points, however, can be laid out without much trouble:

1. ZFS includes a lot of the high level benefits that RAID does, and does them all in software.
2. It lets you pool multiple drives under one drive letter.
3. You will never hit the limit on how much data you can access; it would take enough energy to boil every ocean on Earth to fill up this file system.
4. It allows for expandability under the same letter (Run out of space? Add in a few more disks under the same pool.)
5. When configured for it, you can survive hard drive failures in the same way that certain RAID configurations can.

The downsides? First, it loves as much RAM as it can get, on the order of 1GB of RAM per TB of HDD. Also, you won't be running this under Windows, as it has no support for it. That means you're either working with a virtual machine in your main rig (too messy for my tastes), or you're using a separate box build just to house the drives and run whatever OS you choose. That leads straight into the next component of this build: a NAS.

Getting NASty

While doing the research for this build, I figured it would be best if this new machine could serve up the files to all my various devices across a network since it would be the new central repository for all media. That meant building a NAS, or Network Attached Storage. Now, the difference between a NAS and a server is basically nil, with a few exceptions:

1. NASes are usually designed to be run headless, i.e. no monitor. My particular NAS will be totally web administrated, so that means no keyboard or mouse once everything is up and running, either.
2. NASes usually have smaller or stripped down operating systems, which means less features but less maintenance. They also can fit right on a flash drive!
3. NASes are usually meant to do a small variety of tasks with low power usage, while servers are much more robust and usually consume more power.

So, I decided I wanted a box that I can move between my house and my apartment fairly easily, with no GPU or monitor necessary, and I wanted it to be able to act as a media center to my various devices. That influenced the part listing quite a bit, and after some extensive research, here's what I came up with:

The Build

Case: Fractal Design Node 304: $100. The Node was such a pain in the ass to get my hands on, you have no idea. I've been waiting since June for this thing to come it, and it goes out of stock as soon as I order it. I finally was able to procure one and it should be arriving on Saturday. Now, why the Node? For one, it's tiny. 10" x 8" x 15". Not all that much larger than a shoebox, really. This'll be a big help when moving it from location to location. And despite its size, it can fit six 3.5" (desktop sized) hard drives at once. Plus, with no external 5.25" bays for something like a useless CD/DVD drive, it also looks quite good. I knew from the moment it was announced that it would be the case for this build, and I stuck it out. Here's hoping I was right.

Motherboard: ASUS P8H77-I : $100. A mini-ITX case means a mini-ITX motherboard, and there are VERY few that have six SATA ports. This one fits the bill and came in at a very reasonable price point.

CPU: i3 3220: $130. Overkill of the highest order, really. This is equivalent to a processor you'd find in most NAS solutions that companies sell for two or three grand, not a little home box like what I'm building. So, why all the muscle? For one, expandability. I want to be able to toss three times the amount of drives I have in here and make sure the bottleneck doesn't fall back on the CPU. Next, power. It's an Ivy Bridge CPU so the TDP is way down thanks to the 22nm process. Finally, price vs performance. The "reasonable" choice of processor would cost around $70, and frankly $60 is so little when compared to the cost of the drives I'm putting in, I decided not to sweat the details and go for the extra oomph.

RAM: 16GB Corsair XMS3: $50. I got these on sale, and they're perfect fits. They're low profile which is great for the small case and the amount I bought wwill not go to waste, either now or later. ZFS loves as much ram as it can get.

PSU: Rosewill Capstone Modular 450W: $65. Newegg's house brand which is made by other reputable manufacturers for them, the Capstone 450W is compact, rated 80 PLUS Gold for top notch efficiency, and modular so I only need to include the cables that I....need. Plus, it's gotten some stellar reviews...wow, this reads like an ad. I should get a commission.

All told before hard drives: $445. When you look at comparable, pre-built NAS boxes from big names like QNAP or Synology, you are getting a four drive (vs my six), single core Atom CPU rated below 2Ghz (instead of my 3Ghz+ dual core i3), with barely 1GB of RAM (vs my 16, granted for ZFS). Honestly, it was a no brainer to do this myself.

Now, the drives:

6x Toshiba DT01ACA300 3TB: $870 ($139 per drive). So, there's the expense of it all. Brand new 3TB drives from Toshiba (who bought up what remained of Hitachi's 3.5" drive business after WD was forced to give it up) which are using Hitachi's factories and techniques meaning the reliability and RAID performance should be the same. One of the nicer things about these drives is the fact that they only have one platter per TB, meaning they should run cooler and with less power vs other models.

I'm going to be running six of these in a RAID-Z2 configuration (the ZFS software equivalent of RAID-6), which means I should wind up with (drum roll please) 12TB of usable space. In a system the size of a big shoebox. Oh yes.

I'm going to need a LOT more porn.

Building the build!

Uhhhh, haven't done this yet. Some of the parts as still in transit. Still, if anyone is interested, I'd be happy to post some pictures and a build log of my slapping the whole thing together and crying as 3/6 of the drives turn out to be DOA. Oh yeah, I've already mentally prepared for failure.

If anyone has any questions about why I chose what I did, any NAS/ZFS questions in general (or a desire for me to go more indepth, as much as I can), or has tips/pictures of their own NASes/home servers, feel free to post `im!

EDIT: I'd also love to shove this over to the PC forum instead of just general, if any mod can facilitate that.

I was thinking about doing something like that myself, but in the end I ended up just getting a 4 bay readyNAS box and sticking 4 3tb hitachi drives in it in a raid 5 setup. Very handy, as it supports iscsi and dlna which is mainly what I need it for.

Just rent some of that cloud, bro!

A build log would be dope. I love stuff like this (even if I can't afford it). ;(

I don't necessarily want a whole series on this, but take notes and report back on the trials, tribulations, and results of your attempted storage-space endeavour. This sounds pretty interesting and if I had a good amount of extra cash and time, I'd do this too.

I'm almost in need of something similar too, but since I can't organize my files for shit usually I'd just end up with a 12 TB large mess, and that's about 12 TB more than I want to clean up. I do have rather good organization on my external hard drive, but I don't know how long I could hold that up. 
 
The hardest part about organizing is when programs and/or games decide to just throw their files in random folders without letting you change it though, and that wouldn't be a problem with this I guess. 
 
Anyway, good luck with this and finding 12 TB worth of stuff to put in it!

@Amafi: It certainly would be simpler to go that route, that's for sure. As far as I can tell, FreeNAS has plugins for DLNA but iSCSI is a bit more up in the air.

@Ravenlight: I'd need a whole thunderstorm's worth to save what I need, and then I'd be limited by whatever internet connection I had at the time (I miss my 100Mb line D:).

@Laurentech: @believer258: Absolutely will do. I don't intent to make this a running thing as it should be fairly straightforward once I have all the parts (Newegg is understandably having some delays thanks to Sandy), but I'll make sure to post my impressions and some pictures since I seem to have some interest.

@Jack268: FreeNAS has plugins which, for instance, interface with your torrent client of choice and will automatically tag and reorder whatever you download. As you said, since this is a storage drive and not an installation drive, I'm hoping to keep the amount of program defined file locations down to a minimum.

Any updates on the build? Were you going to use this to install your games on as well, or just for your music & movies? If yes to games, does it actually perform well enough over the network for games to run as well as they would off an internal drive?

@psoplayer: I'm in the process of finishing the write up; it's taking longer then I expected due to a few factors outside of my control, plus I'm working on a second feature about how I'm upgrading all of the peripherals that my system uses as well (keyboard, microphone, speakers, etc). I should have it all together by sometime tonight. As for actually using it to install games, I haven't tried it but it should be possible. In fact, now I'm very curious; the NAS should provide more IOPS then a traditional single hard drive, so the only limiting factor is bandwidth. I'm going to give this a shot and will report back; networked gaming sounds pretty great.

@Mirado: Awesome! I'll be looking forward to it.

Alright, ladies and gents; part two.

Building A Better Mousetrap

So, the various parts in question finally arrive, and it's time to get down and dirty. I'm going to gloss over a bit of the normal build stuff ("Ok, then I connected a SATA cable to the third port and the third drive, then I connected a SATA cable to the fourth port....") as it is for all intents and purposes just like a normal build, but there are some special considerations when you are dealing with such a small form factor.

• Even before you start, you have to make sure everything is going to fit in your damn case. This is usually only on the fringes of your mind when you are doing an ATX build, as just about any mid or full tower case is completely unrestricted in terms of the dimensions of parts (with my HAF X more or less able to hold two PCs at once inside of it!), but you can totally screw yourself over if some critical component isn't going to play nice. Most of the time, this refers to GPUs (it even crops up in larger systems from time to time) but for this instance, the issue of the day was the PSU. I just, and I mean just squeaked by because I flunked the measurement portion of the ITX SAT, and I wound up getting real lucky. Seriously, I'm talking on the order of 10mm of clearance which I had to run wires through. Not ideal, but it worked and saved me the trouble of returning an otherwise awesome power supply.
• When you do a normal build, you can kinda get away with doing things in whatever order you like. Sure, it's probably a pain in the ass to try to fit RAM in under a CPU heatsink, but you don't really need to keep in mind exactly what cable to plug in next or map out which part has to be mounted before you can move on. With a mini-ITX build, you need to make a list of exactly what you are going to do, and you need to follow whatever instructions come with your case to the letter or you are going to wind up with a major headache. Want to mount the HDDs before the PSU? If your case is like mine, you're going to be pulling them all out again to fit it. Want to save the motherboard for last? You've lost half your room to maneuver thanks to the PSU and HDD, and good luck routing those cables!
• Speaking of cable management, oh man...if your PSU is modular, your best bet is to try and see if that company sells a set of shorter cabling, and if you're running multiple HDDs like me, you will want to look into finding the shortest SATA cables that you can which will still reach from your motherboard to the drives. I got a six pack of 10" ones that fit the bill, and I can't imagine trying to run standard, 2' or larger cables in a case the size of a shoebox. Because of the crazy space constrains (and the poor placement of the internal USB 3 header on the motherboard...thanks for that, ASUS) I was forced to leave the USB 3 ports on the front of the case inoperable, as the damn cable was just a bit too short and had to pass straight over the heatsink to work at all. Doesn't matter in the long run, but it is a bit annoying.
• This is a normal tip to key in mind on any build, but I feel it goes double for this one; keep track of your screws. A small case means small screws, but it doesn't imply a small amount of them, oh no. With the ability to mount six HDDs at once, three fans, and coupled with the PSU and mobo mounting screws, you are going to have a pile of slightly different sized (but just close enough to be indistinguishable until you are half way into wondering why this particular screw isn't turning when all its buddies did) bits of metal hell. Doesn't help that these things aren't individually packaged or inventoried in any way at the start, either.

With a few of those gripes out of the way, here are a few highlights:

• These are nice hard drive mounting brackets that came with the case, three of them in total. What wasn't nice was trying to screw the drives into them; vertical brackets with silicon O-rings, while great for noise and vibration dampening, are a royal pain to work with. Still, it was the only way six drives were ever getting into a case this small without piling them on the motherboard.
• Part of the nightmare of cables. This was further improved by some liberal twist tying, but there's only so much you can do when cable A has to cross B,C,D,E, and F to reach his destination. Not to mention that all of these drives needed to be wired for power as well, but luckily all of the SATA power adapters were fit onto two modular cables by the manufacturer. Eventually, everything was more or less tucked away for the maximum amount of airflow possible.
• The PSU before the HDD installation and before plugging anything in. Notice how I've got that one tiny little spot to run all of the cables out of the PSU (including the modular ones which stick out a bit farther, so those guys are turning almost 90 degrees out the gate). Still, I didn't have to RMA it and it runs like a dream, so it all worked out in the end.

So, a bit of sweat and tears (no blood, this time!) but I think you'll agree the end result is pretty sweet:

So, now what?

Off With His Head!

The whole point of this system is to run headless, i.e. without a monitor attached. Assuming nothing catches on fire when you first turn it on (and more to the point, that your router assigns your new NAS box a local IP address and that FreeNAS plays nice with whatever NIC comes onboard), you don't even need to plug it in for the first time setup. Of course, I did some tweaking beforehand (switching the HDDs into AHCI mode, setting up the boot device priority to default to the USB stick that holds the OS, things like that), but once I figured out why my router was being a jerk (I can't for the life of me remember why I limited it to handing out 10 IP addresses at the maximum), all keyboards, mice and video cables were unplugged, and all that remained was the power cord and the Ethernet.

Typing in my box's local IP on my main rig brings this up, which is the web interface for FreeNAS. Since this is just meant to be a kind of build log, and I'm no guru to begin with, I'm not going to go shot for shot on how I set up my box. However, here's the general idea:

1. The first thing I did was set up the actual volume that FreeNAS would later share out. I selected all of my disks (all were recognized by FreeNAS without any prompting) and put them in a RAID-Z2 configuration (4 data + 2 parity disks), giving me a total of 11TB of space with the capability of surviving two hard drive failures without losing any data.
2. Next I set up the user accounts, which honestly is a little more arcane then it needs to be. Since this was primarily going to be a single user system, and anyone who I give access to it is more or less trustworthy (hell, I'm more likely to fuck it up than they are), I set up a guest account which will be the main way of accessing the various shares.
3. Speaking of shares, FreeNAS offers three different ways of actually sharing the volume to different computers; AFP, NFS, and CIFS. AFP (Apple Filing Protocol) is the primary way to offer file services to Macs, and only Macs. NFS (Network File System, unfortunately, and not Need For Speed like I hoped) works with Macs, Linux, and other Unix-like systems, and kinda sorta works on some versions of Windows (which I will get to). CIFS (Common Internet File System) works on basically everything, but is limited to a single core, while NFS can take advantage of multi-core systems and should theoretically provide higher transfer rates then CIFS. Since my version of Windows was one of the few that supported it, I decided to go with that. As it turned out to be a horrible pain in the ass, I'll be coming back to how bad of a decision that was.
4. Next, I set up my plugins, which is the most confusing thing by far. FreeNAS is already a little weird; It's more or less just a Web GUI for the underlying FreeBSD operating system, which you don't have much interaction with unless you call up a shell. To install plugins, you first need to install what is known as a jail: a self contained FreeBSD installation which gets assigned its own IP address. You can't really directly interface with it (no SSHing into it, for instance), but it's where all of your plugin based software gets tossed. So, whenever you want to call up a plugin, you type in the IP address of the jail (along with whatever port is specific to that plugin), and that's basically the only way in. It's some weird form of operating system Inception, and I have to think there was an easier way to do it then that, but whatever. It works, so I'm not going to complain.
5. Finally, I checked the system performance. When running full bore, the system will use 12GB of RAM total, which is both kinda terrifying and also nice confirmation that the 16GB I packed in probably won't be wasted. None of the drives were throwing errors, the temps are more or less in range (a little on the toasty side but it's the fluctuations that'll kill you, not some extra heat), and the acoustics sound pretty good.

Everything had been running fairly smoothly up until this point, so something was bound to go wrong. And what do you know, it did!

Everything Goes Sideways

So, apparently Windows' implementation of NFS is horrible. I mean, atrocious. I spent hours trying to diagnose the litany of errors that it was throwing at me: permission denied, file locked by another user (what?), drive unavailable or offline (what??), it would write 0KB files, it would write half of a file then break or tell me the location was unavailable, or it would literally unmap itself from its assigned drive letter. I spent something like seven hours just reading up on Windows and NFS documentation, I did all sorts of weird command prompt magic, and I even redid all the user and sharing set up on the NAS side of things. The end result was that I finally got it working, but it was slow to respond, had a habit of locking up for five or six seconds every time I tried to access something, and the transfer rate was no better then 30 MB/s. Boo.

So I said fuck it, I set up a CIFS share, and I had everything up and running with guest access in like three clicks, Windows gave me no more problems, and I'm maxing out my Gigabit Ethernet connection (110 MB/s). It works with my Macs (which I gave AFP shares to anyway), and I did it all in under ten minutes. Moral of the story? If you have to dig around to enable something in Windows that is already taken care of by another service, don't waste your fucking time as it probably doesn't work that well anyway!

Pimp my Drive(s)

So, with the basics out of the way, I decided to try out some other awesome features that FreeNAS can grant you:

1. True remote administration, i.e. accessing the web GUI outside of my local network. This was enabled in just a few clicks, so there's really not much to say about it. The key thing is to set yourself up with either a static IP (a pain, really), or get into a free Dynamic DNS service which will take your IP and assign a named web address to it, so you don't have to keep track of your constantly shifting IP address. FreeNAS has a tool in it to automatically ping the service of your choice to keep your IP current, but my router has it built in as well so I just went through that. You also need to set up SSH service to your box, which (again) is spelled out via the lovely FreeNAS manual.
2. Remote plugin administration, which is way more complicated. The only thing I did this for is Transmission, and it was a bit of a nightmare of bad user authentication problems. If anyone would like more detail on this specifically, I can go into it as it took a bit of work, but the upshot is that now I can monitor my torrents from any PC or even my iPhone, no matter where I am, as long as my NAS is on and my home internet is working properly. It's pretty fucking awesome.

You want to know what else is fucking awesome? Network gaming!

Bend Steam To Your Will!

This deserve its own section because of how great this turned out. I'm probably going to do a whole blog post on it because of how it's been running so far, but here's a brief overview:

1. Steam doesn't like to run off of network drives. It throws all sorts of errors if you try to do it. If you find a away to get that to work, it'll more or less sabotage any installation you manage to make happen, by crashing all of the runtimes and redistributables that get installed the first time you run your game.
2. Some people have tried to get around this by leaving Steam on their main drive and just mapping the network drive to hold the main SteamApps folder, but that has a tendency of throwing up the same errors as above.

The solution that I found (through the hard work of this man), was to set up a NTFS junction, which I will admit is over even my head; I'm likening it to a shortcut that acts as though whatever it is linking actually exists where said shortcut is (so if you link /games/Steam to /networkdrive/gamesfolder, the OS thinks those files in gamesfolder actually reside in /games/Steam even though it's just a pointer). Is that right? No clue. As it works out, Steam is actually installed to my main PC, and the NAS is just holding the SteamApps folder. Frankly I'm still a bit fuzzy on why this isn't blowing up, but the upshot is that the fucking thing works, and it's amazing!

1. First off is the obvious benefit of being able to keep every damn game installed at the same time, which should go without saying. 11TB of space, after all.
2. Next, I've found that games actually load faster by using this setup! Keep in mind that I don't have a SSD (which I really should fix now that I have all this bulk storage) so it probably won't beat that, but it trounces your average (if overworked) 7200 RPM 1TB drive from about one and a half years ago. I've noticed a good jump in Dark Souls' load times, and no detrimental effect in multiplayer games like TF2. My theory is that the NAS provides way more IOPS then a single drive, which is the limiting factor in most loads. As most non-SSDs can't come close to hitting 110 MB/s even on an internal SATA setup when faced with random reads and writes, I don't think being on a network will be a factor.
3. Finally, that means my Steam library is totally portable, so if I take my (considerably tiny) NAS on the road, all I need to do is install steam on whatever machine I have on hand, point it to the NAS, and I've got access to every game I own on it. No reason to drag my big ass computer with me just because it has far more storage then my laptop; if I'm happy with my laptop's performance in whatever game I choose, it can just run right off of that without any installation!

I'll do some further testing and probably make a blog post about it in the future, but if you have the opportunity to move your Steam games onto a fileserver which can hit the transfer limit of GigE, fucking go for it. I'm nabbing my whole library (nearly 200 games) as I speak.

So, What's next?

The next phase it to rip my Blu-Ray collection onto it, and then set it up as a streaming box. As it stands I can stream whatever's on it through the regular file browser, but I'd rather have tighter iTunes integration and perhaps something along the lines of XBMC or DLNA, all of which can be provided by adding new plugins. From there, I'm going to finish grabbing my Steam library, probably download every video Giant Bomb has to offer, and then....I don't know? Lots of porn? 11TB is a big number to hit, and something tells me I'm going to run out of ideas before I run out of space...

...and if I do, nothing stops me from getting a bigger case and six more drives! :D

I'm going to eventually read through all of this (acronyms like NAS, ZFS, etc make me happy), but I did read through to the part where you mentioned your build. I read x6 3TB drives and almost had a heart attack - holy ballfuck, man. I thought I had a problem with hoarding, but I've been humbled.

Also, now I know where my next non-hardware update funds are going. Maybe not 18TB of unformatted space, but this concept is really cool. But I'd be remiss if I didn't say fuck you, sir.

So if you go visit some relatives on the other side of the country and you carry a fairly decent laptop with you, you can play some slightly older games at lower settings? Don't have to leave those late-night TF2 or CS1.6/S/GO sessions at home? And when you decide to get a new build you don't have to redownload and reinstall every game?

Awesome! I've gotta do this at some point.

@SirOptimusPrime: You know, it's a funny thing. I debated on the size of the drives for a good three months (I had to wait for the case I wanted to come out, so I had plenty of time to deliberate), and I was going to go with 1TB or 2TB drives all the way, when I ran into a few issues:

1. Most of the more affordable 2TB drives were "green" drives, which usually love to do absolutely performance and reliability destroying shit like spinning down after 10 seconds and parking the drive heads. Now, most times a drive that spins down when it is idle isn't necessarily a bad thing as it helps to save on power (and if its just a bulk media drive that will be unused for a good period of time each day, no reason to keep it spinning), but it is absolutely ruinous in a multi disk striped NAS; since the data is spread across all disks, one might not be accessed in the window before it spins down, and if it holds the last part of a file, the system might report that the drive is dead (when it cannot quickly access it) and drop the drive from the array, which is a nightmare waiting to happen. In the case of head parking, some people have recorded their drives parking and unparking their heads literally a hundred thousand times or more in just one or two weeks because of the same issue as the idle spindown. While there are tools to fix this (which mess with the firmware), I didn't want the hassle.
2. The remainder of the drives at a decent price have reports of failures across the board in RAIDs, with very few manufactures coming out unscathed (Hitachi was one of the few). While all drives can fail and a RAID is a sort of torture test, very few of the cheaper models (non-enterprise 1-2TB SATA drives) stood out like their 3TB brothers.
3. I got these drives for a song. $130 a piece. Yeah, that's still $810 in hard drives alone, but they're a) from Toshiba, the company that got all of Hitachi's 3.5" HDD division (after WD was forced to sell it) b) One platter per TB, which helps keep heat and power consumption down C) Probably the last major storage upgrade I'm ever going to need. Since I only bother to build my own PCs (with an occasional MBP thrown in), I can now go with a smaller SSD right from the get-go, as all bulk storage of games, music, and movies...of anything that can't take advantage of a SSD's massive IOPS, basically, will be on the NAS.

It's nuts, but it's a kind of crazy that fits my usage pattern. Hell, I just bought a $100 microphone and I don't really do anything that can take advantage of it at all (at least not yet). Now THAT'S a purchase I'm going to have trouble defending.

As for doing it yourself; go for it. As you say it doesn't have to be 18TB, but don't go out and buy a pre-built NAS box; the base machine was $400 and it tears apart commercial products in the $1k+ range. If you scale back the drives (say 4x2TB in RAID Z1 (so 6TB unformatted with one disk redundancy), you can also go back on the base hardware, and easily get this to $700 or less, drives included. It's amazing how flexible you can make it.

@believer258: Well, you still have to take the NAS with you, obviously (unless you have a gigabit internet connection on both ends of your journey and latency of 20ms or less, in which case I guess the skies the limit), but yeah; the box will store your whole games library and your laptop can just play right off of it. You don't even need a router if you want to direct connect an Ethernet line, and if you do have one you theoretically can load everything wirelessly on a home network assuming the connection is strong enough. So if you laptop is short on storage space, this'll keep everything preinstalled and ready to go as long as you set up that NTFS junction correctly, and like you said there's nothing stopping you from reformatting old machines or building new ones without the fear of losing your games (assuming you also backup the saves) or the tedium of reinstalling them. As I see it, this thing will make sure I never have to delete or reinstall anything that I don't want to keep on just a single machine. Hell, since it works for Steam, I could hazard a guess to say I could install practically everything to it and just run them in much the same manner, but I'm not that crazy...yet.

Very cool stuff. I might have to get around to building one of these for myself one of these days. For the time being it's enough work just trying to keep things working with programs spanning between my SSD and old 1TB drive. I've already used those NTFS junction points extensively to avoid having to move music libraries and configuration files around.

@psoplayer: Never having to juggle between multiple drives, I didn't even know that they existed until now. The weird thing with them and networked drives comes from how specific you need to be; even if you mapped the drive to X:, pointing it to "X:\Stuff" will fail while pointing it to the IP ("192.168.X.X:\Stuff") works just fine, for whatever reason. Must of made sense to someone when they decided that.

@Ravenlight: It really is a pretty case; the decision to drop any external 5.25" bays (for CD/DVD drives) and letting the fans pull in air from the top, just behind the solid metal front, is a major thumbs up in my book. It also has a bit of acoustic dampening in it (so...foam, I guess? But "acoustic dampening material" sounds a lot cooler), which keeps the noise nice and low.

I could of had this build done in July but I stuck it out for this specific case (which was a real pain to find even after they finally released it; only NCIX had it and it proceeded to go straight out of stock) and I think I made the right move.

@Mirado:

I think you made the right choice to hold out for that case. Thing looks like it's from the future.

Excuse my lack of re-reading, but how much did the whole unit cost to build? And does it have a cupholder?

It's funny, I actually have the opposite problem of too much space and me refusing to fill it. So much of my life was spent with a PC that had a 20 GB hard drive when most people had 120 GB ones, a 320GB when most had 1TB and stuff like that. So until my current PC, 120GB SSD and a 2TB, I have always had far less space then what was considered the norm for the time and as a result I always had to be super strict on what I kept installed or saved. Now that I have the space however I'm trying to fill it up but I can't break these years of habit that I have ingrained in me to uninstall games once I'm done with them, or to keep all my media on an external hard drive and stuff like that.

I have 37 games currently installed on Steam, a couple 25GB-30GB MMO's that are not on Steam, several entire seasons of TV shows, about 20 movies and I still have 1.25 TB of space left on my big hard drive. Before this I had a 320GB hard drive and a 500gb external one and I kept bumping up against the limits of those but apparently I just needed a little more.

@Ravenlight: Build total came to about $1300, $850 or so was purely HDDs. No cup holder but it has really solid rubber feet meaning it won't slide around, so you can kind of use it as a tiny serving tray.

@crusader8463: I think I stated it in the blog, but I probably could have gotten away with just slapping another 1-2TB drive in there and calling it a day, but this should prevent me from ever having to do that in the future. In a way, you can think of it as an uber-external; it'll store your media but also serve it up and download it for you. Lately I've found that my issue is the amount of physical media I own (mainly Blu-Rays, I'm a movie nut), and every time I bounce between my home and my apartment I'm forced to take only a portion of my collection with me, and I inevitably leave behind the one movie that we get a craving for right at that moment. This will let me rip them all to one location which I can then serve up all over to multiple devices at once, if necessary.

Honestly, I'm the type of person that would uninstall games when I'm done with them, but the videos and music started to become too large, and now I don't even have to uninstall the game anymore. I've been downloading them all as I type! :D

@Mirado: Ya, once I realized just how much space I had with my 2 TB hard drive I went crazy and just downloaded a crap ton of games I had no intention of ever playing just because I could. As for the movie thing I can't really relate as I'm the type to watch a movie or TV show one time then never watch it again. Once I know the story I just find rewatching a movie uninteresting and tedious for some reason now. So I delete them as soon as I'm done watching them. The only time I rewatch a movie these days is if it's something I watched a very long time ago and can't remember all the parts.

But ya, I can certainly see the benefits of your situation. Try not to go insane sitting there ripping all of those disks. If you got a few laying around maybe speed things up by installing a couple extra drives so you can be ripping 2-3 at a time. :)

@crusader8463: There's always another bottleneck. First I don't have enough space, now I don't have enough Blu-Ray drives, next I'll be trying to surgically attach more limbs so I can feed the machine faster.

@Mirado: I'm already quite familiar with symbolic links from UNIX systems, and I can confirm that the accusations made in that article you linked are fair. Contrasted against the somewhat simplistic but completely sane world of UNIX file systems and permissions, trying to do anything more complicated than running a single C:\ drive with NTFS is somewhat maddening.

I've never been happy with mounting network volumes to a drive letter in windows because it never seems to stick around after rebooting. I wasn't aware that junction points could use a network address as a target, but that's a really handy trick that I'll have to start using.

@psoplayer: So far, I've been able to make it hang on, but time will tell if that remains the case. I've come to the conclusion that most computers have Alzheimer's and selectively forget or remember based on some formula that I don't quite understand, and I have a comp sci degree. To the uninitiated or untrained, this must be a swirling nightmare of terminology contrasting what seems like a simple problem on the surface (using two drives together), whereas even for those of us with a background, it merely reduces it to a throbbing migraine.

I can't get over how shitty NFS is on Windows. Why even waste the time to implement such a fucked up solution? That above all else was dropping the drive letter; CIFS seems to be a bit more stable.

@Mirado: Well, thanks for the explanation. I pretty much figured you either were, a) backed into a purchasing corner because of shitty RAID drives like you mentioned or b) future-proofing your storage for the next N years (actually, I don't know the math but I wonder how many years of constant, cumulative hoarding would it take for you to fill up all 10.6TB).

Yeah I haven't bothered buying prebuilt anything since I was, like, 12. I usually research before spending any amount of money, so my assumption would have been founded anyways. I'll keep this bookmarked since you have a couple hot links that I might want to reference in the future - I can already feel the 1TB HDD cramping my style. Too bad I'm jobless and need a new GPU before anything else, but like I said once I get things settled down I can foresee spending about ~$900 on something like this.

God damn it, I just said that didn't I? Fuck, guess we're living on noodles for the next month or so.

@SirOptimusPrime: I once lived an entire month on more or less nothing but Halloween candy. In March. College was weird.

How long would it take to fill it all up? If I was determined, less then six months. I'd only be limited by my connection speed and whatever physical media I ripped, so I honestly think I could do it in less then half a year. I'm not going to, because at that point I'm just getting stuff to get stuff (contrary to what most people think when they see this much media, I've seen or heard it all at least once, although I do have a lot of games to play on my list) which is kinda...well, stupid, I guess. As it stands, I think I'll get a good five or six years of usage out of it before I start looking at expanding it, unless they start putting out films on disks at 2K or 4K resolution. (God, I hope they don't do that.)

I tell you what. If I woke up in bed with that sweet thang, I definitely wouldn't kick it out.

Excellent work! MONKEY!

I was thinking of messing around with junctions to run games off my NAS but I decided to just make a 1tb isci target on there and mount that as a local disk instead. Means I only have 7tb left for storage and backup, but I'll manage somehow.

Now I just need to set up some kind of automated offsite backup from the NAS to my VPS for some pictures and whatnot.

Dat case...

Those SATA headers...

I'll be in my bunk.