What is a Ghetto-SAN?
It's a SAN built on the cheap with whatever you can get your hands on.
In my situation, it's not sacrificing any type of reliability or speed, it's simply putting together a lot of parts that may not typically exist in enterprise SAN deployments.
I built a 6 TB FreeBSD/Samba server roughly two years ago when 1.5 TB drives first came out, but we've long since outgrown that. I could build another 12 TB easily enough, but that may not last me the year, and I don't want to start piling more servers around the office. I need consolidation as well as massive storage.
I project that we need at least 24 TB to make it two more years, and 36 TB may be preferable. Some of my data recovery projects are chewing up 3-5 TB and the complex ones involving law firms as clients sometimes mean I need to hold on to that data for 3 months before I can delete it.
I need a SAN, but I can't afford even the "affordable" SAN's from EMC, EqualLogic, etc. These things start at $100k for the features and storage density that I need today.
So I decided to build my own, basing it on opensource software (thanks guys!), ZFS, SATA drives, and whatever else I needed to make this thing the foundation of our little data centre. I'm flip-flopping on FreeBSD/Solaris/Nexenta as my SAN OS, but that's another topic.
It's been over 6 months since I started building and testing the environment, and I've learned a lot. This is not a task for the faint-at-heart, nor for those who don't like to do a lot of testing.
I plan on posting more information about what I've learned, my SAN specs, etc. over the next few months, but first I wanted to quickly comment on what I feel are two dead-end paths for some people trying to design their own Ghetto-SAN's.
Mistake #1 - Getting hung up with Controller/Disk bandwidth when you don't have the network for it
You need to realize that a 6 drive FreeBSD/ZFS raidz on a 3 year old Core 2 system with Intel's ICH 7 drive chipset, and 4 gig of RAM can saturate a 1 Gb network port - At least mine could.
This was my test bench for early SAN performance stats. It's when I knew I needed to look into faster network performance (and I eventually settled on 10 Gb Ethernet over FiberChannel and InfiniBand. That's also another post.)
If you're not talking about a 10Gbe network as the backbone for your SAN, don't worry about your drive performance. There's no need to buy seperate SAS controllers, SAS drives, and make sure everything is connected end-to-end.
Just get a simple SAS HBA for ~$200, hook it into a SAS backplane (and the old 3.0 Gb/sec ones are dropping nicely in price now) and you're going to be able to run that 1 Gb network port into the ground... and it will probably be faster than a basic 4 drive RAID-10 Direct Attached Storage (DAS) for a lot of things, thanks to ZFS having a great caching system. It beats the hell out of a PERC6 or H700 4 drive RAID-10 with 300 gig 6.0 Gb/sec SAS drives. You do have more latency, but the throughput is at least double in my tests, which well makes up for the 100 ms access time. Every VM i've moved to my test SAN has performed better, and "felt" better than the DAS units.
So you have 2 1 Gbe pots that you're going to team? Same story, I bet you'll be running ethernet into the ground before your disks are starving for I/O.
With 4 Ethernet ports, you may have cause - but you still need to do some testing. I decided on a single SAS 6.0 Gb/s HBA and SAS 6.0 Gb/s backplane for my SAN, and it's working out great for me.
Mistake #2 - Putting all your eggs in one basket
Everything fails. Everything.
If you have just one of anything that you depend on, you need to go read that last line again. I've got over 20 years experience in IT, and trust me when I say that the only thing you can count on with technology is it's eventual failure.
I have dual everything in my data centre. Dual UPS's connected to dual battery banks, feeding dual PDU's to the servers that all have dual power supplies, so each server is fed from two UPS' in case one of them fails under load when the power cuts out and the generator hasn't started yet. Dual firewalls, dual ESX servers, etc. You get the picture.
Same for your SAN. It's far better to have two cheaper SAN boxes than one more expensive one.
I've gone with a fairly expensive Primary SAN (redundant back planes, power, network, etc). It will be my main go-to box, and will deliver the day-to-day performance that I need to stay sane.
I will also have a Secondary SAN that will be run off an older server that will save my ass if the primary goes down. It won't be very fast at all, will be tight on storage, and low on RAM - BUT it will hold a copy of all my critical data, and a very recent snapshot backup of the main critical servers so I can get my network up and running within 15 minutes in case of failure in the main SAN.
This is completely redundant hardware, done on the cheap - and you need something like this if you care about your data. I just need to stuff more drives into an old case, setup some zfs send/recv and snapshot jobs, and I'm done.
I'm currently lucky - Not losing data is my primary concern. Quick repair is my second, but my clients will tolerate a rare incident that may result in their data access being interrupted for 30 minutes, followed by a day of slow service until we get the primary back on-line. Not everyone may be that lucky, so make sure you take the worst-case scenario into account when you start designing your SAN.
--
I hope to followup on this more regularly as my SAN timidly transitions from testing into full production. Comments and opinions welcome.
No comments:
Post a Comment