Jump to content
Sign in to follow this  
Master_Scythe

moving to ZFS plan

Recommended Posts

am i right, in reading this: If i go with RaidZ1, no matter how many disks I have, i can lose one? So I could have 50disks in the one pool, and still only HAVE to have one 'wasted' disk?

If so, i may go a 10 disk RaidZ2......

also, if disks in the pool are different size, and the largest fails, how could it recover? (250gb, 250gb, 1TB, and the 1TB falls over).

 

Now my question is thus; it seems I cant 'grow' a pool of RaidZ once its created, is this correct?

Because ive realised I COULD have 50% of my disks spare if i shuffle data, but I'd like all disks to be in a 10 disk raidZ2 in the end.

 

So am I able to take 5 disks, move the data off those 5, then add the now blank 5? From what I read I'd preffer a 1X 10disk-Z2 than 2X 5Disk-Z1

 

Advice?

 

 

still googling for myself, just fill me in if you beat me to it :P

 

*braces for flying striped horse attack*

Share this post


Link to post
Share on other sites

I think you could create a zpool, migrate your data, then create a vdev from the remaining disks.

You should then be able to add that vdev to your existing zpool if you plan it right.

I've not done this but with some research and planning you should be able to achieve something like this.

 

All drives, or at least partitions in any vdev need to be the same size. But you can build z pools up from dissimilar vdevs I think.

 

I've probably just demonstrated my almost complete lack of knowledge in the above.

I've only made two zpools so far, both using only identical sized disks. So I'm far from an expert here, PLEASE do your own research BEFORE committing your data.

Share this post


Link to post
Share on other sites

I would, that's why I've only built zpools from similar sized drives.

Though in your place I'd probably start with a more modest array with the disks you have.

 

Time to do some googling sport...

 

Well it's late and I'm off to bed.

Share this post


Link to post
Share on other sites

NexentaStor 3.x Community ISO Features

*blah blah blah*

•The ability to automatically expand pools

 

 

By expand, do they mean switch one disk out for a bigger one? (with RaidZ2 could i just yank up to two at a time and replace?)

Or does it mean add more disks like i want?

 

Nexentastor is looking like the winner. It doesnt look like it can run an IRC, itunes, bittorrent, or games server of any sort.... But I can build another device for that, I have a microITX board lying around somewhere which could handle this.

 

Though in your place I'd probably start with a more modest array with the disks you have.

If im doing this, i'm going all out!

Share this post


Link to post
Share on other sites

I believe in nexenta's case it's only the adding to a pool. You have always been able to add disks to a pool. It is a realatively new thing to add a drive to a raidz vdev. I'm a little unclear on the actual physical process as the feature is advetised as 'single disk expansion' or something similar. I guess you would need to expand the array 5 times to do it rather than in one big go to add 5 disks. But then thet also say that you can add a disk and go from raidz to raidz2.

 

I'm pretty sure nexenta stor is based on build 134b still. Tht means your not running a version of zfs new enough to have the single disk ezpansion feature. OpenIndiana or Solaris 11 express are your only options there. you can add napp-it to them for web admin.

Share this post


Link to post
Share on other sites

Is it just RAIDZ that stops me adding disks to the same 'group' (whatever word i mean; pool? vdev?).

 

Can I effectively make a ZFS - JBOD, then convert that to RAIDZ without losing data?

 

 

 

 

EDIT: lets be realistic for a moment.

Id like to buy a bunch of large drives (1TB-2TB); probably 6-10 of them.

Have them all set up in RaidZ2.

 

And besides inital creation, and configuring shares, I never want to have to touch it.

I want it to handle bit-rot automatically, repair corruption automatically.

 

The only interaction I want, is when (if) a drive dies. And even then, I'd like to just yank it out and replace it, like a happy RAID1 'standalone nas' device i could buy from dlink.

 

I'll stop talking and listen: What do i need?

Edited by Master_Scythe

Share this post


Link to post
Share on other sites

You could swap out the disks for larger ones.

For example : If you have a zpool of 1Gb drives, you could swap one drive out for a 2GB drive , attach the new drive to the zpool, wait for the re-silver to complete, repeat for each drive in the zpool. At the end of that process you'd have a zpool approximately twice the size of the original. You would only see the increase in space after the final drive is swapped out.

 

The very latest ZFS version in OpenIndiana allows adding drives to raidz1 pools I believe, though i don't think that is considered stable as yet.

 

I tried Nexentastor a couple of weeks ago and whilst it worked OK there were a few things I didn't like. For example they use non standard terminology in the web interface ('Folders' instead of 'file systems' is one such), no big deal but I've ended up going to OpenIndiana and setting it up by hand - didn't take very long as I've done it before on OpenSolaris B134. I personally find Solaris a bit of a bear to work with but as all I'm doing with it now is ZFS file serving I can live with that. That's actually the reason why I migrate my whole server to a virtualised ESXi based platform. Now if I want other services such as mail serving I can spool up a dedicated VM for the task and use another OS that I'm more comfortable with for that task.

 

What I found is that once your Zpool is running, it's easy to change the underlying OS later if you need to. I can '# zpool export mypool' the storage zpool, run up another ZFS capable OS (Making sure it can read my ZFS version, at the moment I'm using ZFS version 22), then import the zpool into the new OS with a simple '# zpool import mypool' command. So I'm not locked into any one OS, I can swap freely between OpenIndiana, Solaris Express, Nexenta, FreeBSD and just import my existing zpool/s.

Share this post


Link to post
Share on other sites

I dont mind work in the initial setup; just once its working I want to never hear from it again.

 

 

Why wouldnt i see a size increase untill the last disk? I thought ZFS pool (including raidZ) coiuld handle mixed size drives?

Share this post


Link to post
Share on other sites

I don't know why you find it funny that I mentioned zebra, the storage god incarnate.

i didnt. it was more a 'where is he?' comment

Share this post


Link to post
Share on other sites

Ive you just throw all your disks at a pool then it can handle any size disks and use all of the available capacity in a weird kind of raid0-jbod type of franken filesystem. but you get zero redundancy. Not a happy place for 10tb of data to rest.

 

If you want redundancy (Mirror, Raidz, Raidz2, Raidz3) then you need same size disks.

 

As far as i can see the only way your going to build an array of 10 disks in raidz2 at this time with the majority of zfs implementations out there is to do it in one go. If you use openindiana or Solaris 11 Express then you can do the expansion.

 

If you are strugling you could just build the array from ten disks, grab anything you have lying around borrow some of friends and then as you dump data off of your currrent disks you can expand the array by using bigger disks. The raidz2 array will always be the size of the smallest disk * how many disks you have, so just swap out the smallest and let it expand.

Share this post


Link to post
Share on other sites

You rang?

 

 

am i right, in reading this: If i go with RaidZ1, no matter how many disks I have, i can lose one? So I could have 50disks in the one pool, and still only HAVE to have one 'wasted' disk?

If so, i may go a 10 disk RaidZ2......

In a 10 disk set, I'd heavily consider a RAIDZ2, yes - and purely on the basis of the BLER/Uncorrectable error and MTBF of the spindes PLUS the MTTL of that number of spindles. Do the math. Look at the chances of a failure in a set that big. Sure, you'll lose two disks - but you'll gain satisfaction in knowing you can with stand two full module failures AND have another ~160 odd IOPS of throughput as a result of those two "unusable" spindle spaces.

 

also, if disks in the pool are different size, and the largest fails, how could it recover? (250gb, 250gb, 1TB, and the 1TB falls over).

IIRC - you'll need equal vdev sizes for it all to make sense, but if you try to RAIDZ a 250GB, 250GB, 1TB and 1TB, the largest drive alloc size on a vdev will be 250GB. It'll protect you from yourself, and you won't ever see the 1TB.

 

Now my question is thus; it seems I cant 'grow' a pool of RaidZ once its created, is this correct?

Because ive realised I COULD have 50% of my disks spare if i shuffle data, but I'd like all disks to be in a 10 disk raidZ2 in the end.

There was a pile of talk about this being implemented correctly in OI and Sol 11 Ex, but it still hasn't occurred. The ONLY way to grow a device set is to attach ANOTHER device set to it. You still can't grow disk by disk.

 

So am I able to take 5 disks, move the data off those 5, then add the now blank 5? From what I read I'd preffer a 1X 10disk-Z2 than 2X 5Disk-Z1

You're able to take 5 disks. Copy the data off. Nuke them. Make a raidz1, then copy all the data back on. You can't add to these and make that one heterogeneous pool bigger, though.

 

Best practice:

 

You want a 10 disk RAIDZ2, with a hot spare hanging off the side, with zfs compression=on, an SSD cache for L2ARC and zfs dedup=on.

 

My home rig is an 8 disk RAIDZ1 with 1TB drives, an OCZ vertex SSD acting as L2ARC, dedupe turned on, with a Q6600 CPU and 8GB of RAM taking care of...

 

Cheers.

 

z

Edited by zebra

Share this post


Link to post
Share on other sites

On a home fileserver I would not reccomend the last option. Dedupe. Think about it this way majority of what you store will be highly compressed DVD and Bluray images. and it will be in a fairly organised structure. Unless you plan on having doubles of stuff (for what ever reason), then dedupe is not going to do much but chew through a bunch of ram, and that in and of itself is pretty much a requirement for having a ssd l2arc.

 

Don't get me wrong dedupe can be handy I would turn it on for my downloads directory. (i have a habit of downloading Nvidia drivers many times) dedupe would fix my tardyness. as for compression, just turn it on, its not going to do anything for your highly compressed stuff but should have an effect on your flac music collection (at a guess). either way it doesn't chew much in the way of resourses and will do something when it can.

 

As for l2arc, it's a cherry. In my theory an l2arc with the zfs prefetch option on should be able to dump an entire piece of media to the ssd then let all the drives idle while you watch, that might be good for power and noice. Have i done it - not yet. Also the ssd should end up with the directory structure of the drive on it, this could make brousing the file hierachy a bit quicker.

 

But as for the massive con about an l2arc. just like ram all of the data is deleted on a reboot. so if you plan on turning the server off before bed then this might be an issue, since your cache will always be cold.

 

ssd are getting cheap enough that im going to bang one in just to see what happens.

 

also since the l2arc stores metadata in the arc (RAM) you need enough ram for the size of l2arc you plan on using. a 400gb l2arc will consume arround 15gb of ram. so 100gb will nail 4gb of ram. and you want atleast some of your ram free for an os and other programs. so a 60gb or a 40gb drive might be the good options. Personally im going with 8gb of ram and a 60gb ssd.

Share this post


Link to post
Share on other sites

Yeah I wont have doubles worth noting.

I was worried, as i only have a dual core E1400+ and 1 or 2GB of ram, not sure.....

It doesnt seem it needs that much power, ZFS as used by FreeNaS only requires 690mb of ram minimum, and a 600mhz+ processor.

I'll be the only one accessing said files, nd only doing one thing at a time.

 

Didnt atomic have a 'setting up ZFS' guide? I have the mag SOMEWHERE, can someone scan me a page or 2?

 

Meanwhile i'll find a guide online. (or use nexentastor, it seems to have a lot of praise)

 

If i cant mismatch drive sizes, I have 2X 1.5TB, so i may go and buy a bunch more of 1.5's.

 

Then again, i have 4X 1TB drives, so that'd work out cheaper...... i dunno. Im gonna go see how much data I can move to a single drive.

Share this post


Link to post
Share on other sites

After reading everything, and seeing how difficult it all is :P

FreeNAS is looking promising again. Ive used it before, and it seems BSD 8.0's ZFS is pretty stable.

Is there a really compelling reason NOT to?

 

When I say difficult, I mean it seems to lack the same level of support, I cant run many additional services (itunes server, torrents, IRC server etc), well, i know i can, but they all have to be compiled.

 

EDIT: FreeNAS 8.0 is running ZFS version 14.

 

The only thing it seems to be failing, is if you format the OS, it seems to fail at recovering the old pool (am i using the right terms?)

 

Though I assume ZFS is universal, so worst case scenario, liveboot OpenSolaris and recover it that way?

Edited by Master_Scythe

Share this post


Link to post
Share on other sites

You can move forwards pool versions, you can't go backwards.

 

So if you're running ZFS version 30, don't expect Solaris 10 to do anything with it. But if you're running version 14, Solaris 11 will pick it up no worries (providing it isn't horribly broken in some other way).

 

Anyway, run Solaris 11 Express. It's fucking dead simple. It's even got a vaguely normal looking package manager.

Share this post


Link to post
Share on other sites

ill give solaris 11 express a go :)

 

can everything be checked on via GUI?

 

I dont mind config via command line, in fact, i navigate and issue most commands on my windows machine from a cmd prompt, but when i want to know if all is OK, I like to have it shown to me, plain and simple.

 

Do you know a good location for a 'complete noobs guide to open sol and ZFS' brains? even a walkthrough would be nice.

Share this post


Link to post
Share on other sites

napp-it. its a web interface like nexentastor. not as pretty though, and i havent used it asside from the trial you can find running on the site.

Share this post


Link to post
Share on other sites

that'd do it!

 

OK Ive realised, because im running RAID1 right now, i can backup all my data to my desktop + a single 2TB drive.

 

What would you guys vote? just buy a few more 1TB and enjoy the 4TB i already have?

Say... up to 5TB+? (8X 1TB drives in RaidZ2)

Share this post


Link to post
Share on other sites

Well I'd say that you will be limited by the amount of sata ports you have. which is usually six. LSI are the sas cards of choice for sun so you get great solaris support for them but they don't come cheap. I went with a lsi sas card and 12 drives in raidz2.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×