Jump to content
Ashlar

ZFS secondary storage server for work

Recommended Posts

Hey fellas,

 

I'm looking at putting together a ZFS server for work (pending approval) which will be used mainly as a secondary storage system (archival target for the most part) with an existing NetApp system as the primary.

 

I've sent off some specs to our sourcing guys to get a quote for the actual server, something (actually a lot!) along the lines of this:

 

1 - 2 Dual or Quad-core Xeon ~3Ghz

DDR2 4GB

6* 500GB SATA drives -or- 4* 750GB

Option of an Intel SSD, maybe 40 – 80GB

At least 2 NICs, 4 would be better

 

No OS (Solaris 10 is a free download, but we can buy a support subscription through Sun)

 

Server must fit with the Solaris Hardware Compatibility List. Systems are listed here:

http://www.sun.com/bigadmin/hcl/data/sol/s...ults.page1.html

Firstly, does that spec look OK?

 

Secondly, have any of you guys implemented ZFS (actually, I know some of you have ;p ) but particularly, have you implemented one to go into a business environment?

 

Any pitfalls or gotchas I should be aware of?

 

Another other comments?

 

Actually, I'm also interested in recommendations in the way of archival software (information LCM), though I guess that's another story (but again, if you have experience, please by all means throw in your 5c worth).

 

Thanks in advance, mates...

 

 

-A-

 

[Edit: extra highlighting / clarity]

Edited by Ashlar

Share this post


Link to post
Share on other sites

Spec looks fine.

 

ZFS in implemented in a business environment.

 

I'm not giving any advice on working with your NetApp stuff though.

Share this post


Link to post
Share on other sites

LoL... yeah, I do recall you having some sort of beef with NetApp, heh.

No worries.

 

Cool, thanks for input.

 

So no other gotchas about configuring or working with ZFS that you know off the top of your head?

 

Cheers.

 

 

-A-

Share this post


Link to post
Share on other sites

I don't have a beef with NetApp.

 

I just don't use it in a business environment, I'm not going to give advice on it.

 

Big thing: Ensure compression is on before you start. Use bzip2 -9 compression, you'll have plenty of cycles to handle it, and it should help your IO too. Magic!

Share this post


Link to post
Share on other sites

LoL... yeah, I do recall you having some sort of beef with NetApp, heh.

No worries.

 

Cool, thanks for input.

 

So no other gotchas about configuring or working with ZFS that you know off the top of your head?

 

Cheers.

 

 

-A-

A few, that you need to be aware of, off the top of my head:

 

1. If using a large RAID controller construct underneath ZFS (which sort of makes it a moot point), make sure controller cache passthrough/bypass is enabled

2. Be forewarned, making a ZRAID1 is a fairly fixed thing. Make the ZRAID1 pool the size that you intend for it to live, for quite some time. Expanding zraid1 pools is still a non trivial task

3. If you're going to logzilla or readzilla, an Intel SSD is a bit of a toy. Be careful with it. Make sure you get the "E" series. You'll blow that SSD straight into the next life, otherwise.

4. If you are doing high database transaction locking (we're talking thousands of ops per second), you might need to do some fairly evil ZFS tuning, for cache, L1/L2ARC so it gets the balance between DRAM bursts/writes correct.

 

 

z

Share this post


Link to post
Share on other sites

I don't have a beef with NetApp.

 

I just don't use it in a business environment, I'm not going to give advice on it.

 

Big thing: Ensure compression is on before you start. Use bzip2 -9 compression, you'll have plenty of cycles to handle it, and it should help your IO too. Magic!

Ah ok, maybe I just read into some other previous comment too much. Can't remember exactly what it was and CBF looking it up.

Fair enough... no need for NetApp advice, I've plenty of resources/avenues there. ;)

 

Thanks for the tip about compression... and yeah, I did read that. Pretty neat idea really, given enough CPU grunt. (Maybe I better check he is getting 2 CPUs -- I told him 1 quad or 2 dual/quad, depending on price.)

 

 

-A-

Share this post


Link to post
Share on other sites

A few, that you need to be aware of, off the top of my head:

 

1. If using a large RAID controller construct underneath ZFS (which sort of makes it a moot point), make sure controller cache passthrough/bypass is enabled

2. Be forewarned, making a ZRAID1 is a fairly fixed thing. Make the ZRAID1 pool the size that you intend for it to live, for quite some time. Expanding zraid1 pools is still a non trivial task

3. If you're going to logzilla or readzilla, an Intel SSD is a bit of a toy. Be careful with it. Make sure you get the "E" series. You'll blow that SSD straight into the next life, otherwise.

4. If you are doing high database transaction locking (we're talking thousands of ops per second), you might need to do some fairly evil ZFS tuning, for cache, L1/L2ARC so it gets the balance between DRAM bursts/writes correct.

 

 

z

1. AFAIK, I'll just be using JBOD and giving those disks to ZFS, correct? I have the Atomic tute, and of course others online which I'll be following. May start out with a POC if I can get my hands on boxen with decent number of disks -- otherwise, fuck it, straight into the real deal (if it gets approved, which I'm thinking is quite likely).

 

2. ZRAID1? Is that actually RAID-1 or RAID-Z? Though I'd be using RAID-Z... Maybe I should start with more disks.. hmm.

 

3. What's the alternative? I mean the intel are like $7-800 as it is, from what I've seen (though we'd get a discount). But ok, "E" series it is!

 

4. No, don't think we'll getting into the thousands of OPS.

 

Awesome... thanks a bunch for that, Zebra.

I'll let ya's know how it goes. =)

 

 

-A-

Share this post


Link to post
Share on other sites

1. If using a controller, yes, you don't want to make RAID sets, but you also want to bypass the controller cache on the controller, as ZFS uses in kernel stuff for caching, which can have a negative effect if the RAID controller cache is still on.

 

2. RAIDZ1. It's RAID-Z, with a single parity drive, ie, x+1 raid set.

 

3. Not using an SSD.

 

4. Then don't stress too much.

Share this post


Link to post
Share on other sites

1. If using a controller, yes, you don't want to make RAID sets, but you also want to bypass the controller cache on the controller, as ZFS uses in kernel stuff for caching, which can have a negative effect if the RAID controller cache is still on.

 

2. RAIDZ1. It's RAID-Z, with a single parity drive, ie, x+1 raid set.

 

3. Not using an SSD.

 

4. Then don't stress too much.

Ok awesome.

Thanks again, mate!

 

 

-A-

Share this post


Link to post
Share on other sites

two types of RAID under ZFS. RAIDZ and RAIDZ2. RAIDZ sometimes called RAIDZ1 or a rearangemnt of is a kin to RAID5. RAIDZ2 is akin to RAID6. If it is supposed to be a backup then RAIDZ2 is probably the one you want.

 

I think Solaris 10 still only offers LZMA compression as its most hardcore.

 

one of the gotchas with ZFS is its write ahead mecanism is not great with slow moving data - I hear. For example slow written logs will not get compressed mach because of the slow tricle of the write across the network. Create a new filesystem turn on the bect compression you have on it and periodically move your log files to this new filesystem. Moving at disk speed to this new location will compress the files quite a bit.

 

Compression is not applied retroactively - turn it on before you dump any data to your server.

Edited by Sparky

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×