Jump to content
p0is0n

File Servers & NAS

Recommended Posts

God I love ZFS.

I pulled and replaced the failed drive.

Then

# zpool replace mypool c3t1d0 c3t1d0
and this was the result.

Error on openindiana from 14.01.2013 21:45

  pool: mypool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Mon Jan 14 21:22:38 2013
	1.02G scanned out of 1.68T at 797K/s, 629h37m to go
	251M resilvered, 0.06% done
config:

	NAME			  STATE	 READ WRITE CKSUM
	mypool			DEGRADED	 0	 0	 0
	  raidz1-0		DEGRADED	 0	 0	 0
		c3t3d0		ONLINE	   0	 0	 0
		c3t0d0		ONLINE	   0	 0	 0
		c3t2d0		ONLINE	   0	 0	 0
		replacing-3   DEGRADED	 0	 0	 0
		  c3t1d0/old  FAULTED	  0	 0	 0  corrupted data
		  c3t1d0	  ONLINE	   0	 0	 0  (resilvering)

errors: No known data errors
The actual time taken to complete the resilver was about 6 hours not the 629 or so reported, the reported time varies throughout the process according the the data currently being resilvered.

 

Of course in my instance the tricky part of this whole operation was removing the old RDM pass through and adding the new RDM on the ESXi Hypervisor that my ZFS server is virtualized on top of. That was done with much help from this excellent howto and some of very careful identification, copy and pasting of vml strings. These are not nearly as sweet to work with as ZFS. I suppose that is only to be expected as this 'feature' is not officially supported on the free version of ESXi I'm using and thus is not implemented in the ESXi GUI.

 

For those who don't know RDM passthrough on ESXi allows a virtual machine to have direct access to a physical device. Normally vm's don't get that kind of control of physical disks. In my case the four drives making up the mypool vdev are connected using this method.

Posted Image

Edited by CptnChrysler

Share this post


Link to post
Share on other sites

I am looking at getting a HP Microserver as well as some RED drives to turn it into a NAS. But I am completely green in this field.

 

What would the best OS be to run it (freeNAS? Ubuntu?).

 

Also, if I only put 3 drives in and RAID 5 them, then want to put in a 4th, is there any troubles expanding the drive?

Share this post


Link to post
Share on other sites

Ubuntu would probably be your best bet following p0is0n's advice as I have found adding extra features seems to be easier on Ubuntu than many other distros.

 

You could try CentOS first (following my post above) - download the PDF and see if it's easy enough to follow. The scripts I have provided make it very easy to get up and running and as far as I can tell, I have managed to provide enough functionality to make it worthwhile. I am about to embark on some IP camera monitoring functionality next actually. If you do go down the CentOS route with my scripts I am happy to give you a hand where I can should you encounter issues. I kind of want to see where someone random gets with it all rather than me doing it for people.

 

As for Raid5 - I wouldn't bother. ZFS will give you the same sort of setup but it works really well and offers some advantages over Raid5.

 

As for expanding Raid5 - no you can't. It's a backup, blow away and recreate exercise. Expansion on ZFS is possible but is limited.

 

http://serverfault.com/questions/190207/ho...ing-raidz-zpool

 

To reiterate, you cant just add a single disk drive to a raidz1 or raidz2 pool. This is a known limitation (you can check with SUN ZFS docs). If you have an existing raidz and you MUST increase that particular pool's storage capabilities, you have 3 options:

 

1) Add a raidz of the same configuration to the pool (think 3 disk raidz + 3 disk raidz or 5 + 5, for example)

 

2) Replace each (and every) disk in your raidz pool one by one, letting it resilver after inserting each upgraded disk

 

3) Backup your data, destroy your pool and create a new raidz pool with a bigger amount of disks.

One thing to keep in mind - whatever you decide to do, you should also think about backup. The more disks you throw in a NAS, the more you will likely need to backup. Your strategy may vary. If your NAS is storing less than 3TB, an external 3TB USB Hard Drive is a simple backup. Two of them alternating to offsite locations is a simple but effective backup strategy. If your storage needs exceed 3TB, you will be looking for either some 2 bay NASs or building another NAS with enough drives to sync with.

 

If your internet connection is great, cloud based services may be an option. Crashplan has excellent Linux support and is pretty easy to install.

Edited by The Tick

Share this post


Link to post
Share on other sites
$ sudo zfs list

NAME USED AVAIL REFER MOUNTPOINT

storage 2.59T 4.54T 117G /storage

storage/downloads 486G 4.54T 486G /storage/downloads

storage/movies 861G 4.54T 861G /storage/movies

storage/music 189G 4.54T 189G /storage/music

storage/tv 996G 4.54T 996G /storage/tv

How do you create sub directories under your zpool?

When i create them using mkdir, they don't show up under "zfs list" like yours have.

I just destroyed and recreated my zpool as it wouldn't mount. It showed being mounted as /storage, but it turned out copying data to /storage filled up my OS drive, which isn't in the zpool.

I think I'm doing something wrong.

Share this post


Link to post
Share on other sites

Your storage pool mounts itself once you have started the zfs service.

 

You don't need to create a mount point and mount it like a traditional Raid array or single drive.

Share this post


Link to post
Share on other sites

NOT freeNAS, somehow people like that abomination.

the webGUI breaks things, and can often never be resolved without command line trickery.

 

Make a software raid 1 or 5 (or Z), and share it. Ubuntu will do fine.

Share this post


Link to post
Share on other sites

Who said anything about freeNAS?

 

mudjimba is following p0is0n's tutorial referring to ZFS under Ubuntu. I assume that ZFS for linux works the same under Ubuntu as it does CentOS in which case, creating your pool with the zfs service running is all you need to do. It will mount the pool for you.

 

mudjimba - once your zfs pool has been created, zfs list should show you the main mount point. That should be enough to create directories under there using mkdir.

 

If you want to create additional mount points, you need to use the zfs create command.

 

zfs create storage/homes

zfs create storage/whatever

 

I'm not sure what the advantage is to doing this over simply creating your volume and then using mkdir as usual although some enlightened soul may provide more information.

Edited by The Tick

Share this post


Link to post
Share on other sites

I'm going to throw in Windows just because it's a MicroServer Owners Club, not a nix fan fest :P

Share this post


Link to post
Share on other sites

If you want to create additional mount points, you need to use the zfs create command.

 

zfs create storage/homes

zfs create storage/whatever

 

I'm not sure what the advantage is to doing this over simply creating your volume and then using mkdir as usual although some enlightened soul may provide more information.

My understanding is the benefits of zfs filesystems are you can set different attributes on each filesystem, which you can't do with straight directories.

 

For example:

storage/media

-compression=off

-copies=1

storage/photos

-copies=2

-compression=gzip

Share this post


Link to post
Share on other sites

Aw cheers for the info guys.

It seems to be working ok now, not sure what happened the first time.

Replacing the fan tomorrow, will post how it performs acoustic wise.

Share this post


Link to post
Share on other sites

Who said anything about freeNAS?

Bowlen did in his last post

 

My apologies.

 

FWIW it does help when you quote the text of the post you are replying to when it isn't immediately above your reply and actually dates your reply by a few days.

 

 

I'm not sure what the advantage is to doing this over simply creating your volume and then using mkdir as usual although some enlightened soul may provide more information.

My understanding is the benefits of zfs filesystems are you can set different attributes on each filesystem, which you can't do with straight directories.

 

For example:

storage/media

-compression=off

-copies=1

storage/photos

-copies=2

-compression=gzip

 

 

:D

Share this post


Link to post
Share on other sites

Just wanted to add my experience to something here.

I'm a huge fan of whatever windows does to handle Software RAID5's.

 

Its the only one I cant break. Ive tried pulling power mid rebuild, changing drive controllers mid rebuild (BSOD, reboot, continues OK), change brand of drive, change from SATA drive to PATA....

 

NOTHING seems to stop a windows Raid5 rebuild. And it worked EVERY time. I tested the hell out of that solution.

Every other OS and drive redundancy solution I tried broke if I changed drive controller mid rebuild. OR if I tried to use a PATA drive when it was previously a SATA one.

What was I testing this for? Well for a home solution, I wanted my data to be 100% hardware independent. Windows Raid5 proved it could.

ZFS and Linux Raid probably can too, but it was no where near as idiot proof.

 

The only reason it didn't stick around for me was because we have common 4 hour blackouts, and I had a 'dumb' cheap UPS, so I'd be rebuilding almost daily.

I ended up with manual bi-daily Syncs, rather than any form of RAID, but I'm special like that :P

 

 

ANYWAY, if I had a more reliable place, I'd try ZFS again for sure. I used it under FUSE for a while and it was pretty sweet.....

 

Let us kow how it performs mudjimba

Edited by Master_Scythe

Share this post


Link to post
Share on other sites

So I ran into a previous problem again after setting up samba.

ZFS doesn't mount the zpool on startup.

Running "zfs mount -a" mounts it fine, so will I need to run that as a launchdaemon to get around any auto restart/power failure issues?

 

Side note I ordered a 140mm fan instead of a 120mm, waiting for the later to ship.

I bought a bitfenix PWN fan, it's virtually silent to my ears from 5~ inches, air pressure felt very much the same.

The new fan didn't affect the microservers noise output at all though. The 4x deskstars i have in there are putting out more noise then the stock fan did. There is also a slight electrical whine from the PSU.

Not worth the hassle!

Share this post


Link to post
Share on other sites

If anyone had downloaded the scripts I previously posted, I have updated them slightly to reflect the newer version of ZFS.

 

They changed some of the download paths so the older version of the installer won't work.

 

Download from here

Share this post


Link to post
Share on other sites

So I ran into a previous problem again after setting up samba.

ZFS doesn't mount the zpool on startup.

Running "zfs mount -a" mounts it fine, so will I need to run that as a launchdaemon to get around any auto restart/power failure issues?

 

Side note I ordered a 140mm fan instead of a 120mm, waiting for the later to ship.

I bought a bitfenix PWN fan, it's virtually silent to my ears from 5~ inches, air pressure felt very much the same.

The new fan didn't affect the microservers noise output at all though. The 4x deskstars i have in there are putting out more noise then the stock fan did. There is also a slight electrical whine from the PSU.

Not worth the hassle!

Haven't been around here much recently, started a new job where I have much less time to post.

 

Something I might have forgotten to add to the guide.. I edited /etc/rc.local and added this:

 

smbd stop
nmbd stop
zfs mount -a
smbd start
nmbd start

Never had any issues with ZFS not mounting, apprently it can be an issue with ubuntu 12.04+

Hope that helps

Share this post


Link to post
Share on other sites

Quick IMO on topics like this.

If ZFS is just so damn good, why is there not a decent GUI front end for people who hate scripting?

I love scripts and will use them for everything EXCEPT file management. Just bad experiences. One wrong letter can spell disaster.

 

You'd think the community would be excited enough that when you find a tech you KNOW is the best, a new GUI is made.

Take LAME for example. Tons of great GUI.

Share this post


Link to post
Share on other sites

If ZFS is just so damn good, why is there not a decent GUI front end for people who hate scripting?

Why must one imply the other?

Share this post


Link to post
Share on other sites

Because a complete package is a sign of quality and..... completion.

 

If its truly the best, it would be the most used. Right now, it lets itself down in one area, usability.

I can talk first hand, making a windows RAID5 is much MUCH easier than making an Ubuntu RaidZ

 

To take the usual comparison of cars to computers argument; best performance, best economy, but the steering wheel is behind your head so you'll have to learn to drive with your arms above your shoulders without going numb.

The smart person who felt that car was truly ready for the world because its the best, would move the steering wheel to a user friendly location.

Share this post


Link to post
Share on other sites

to setup my raidz pool on ubuntu took about 30 seconds, to both install zfs then make a pool. i could then take snapshots, copy data etc. nothing more to be done. one line is all it takes to create the pool 'sudo zpool create -f -o ashift=12 storage raidz /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde'. how long does it take to setup raid5 in windows M_S? is there any kind of performance hit or do you get decent read/write speeds?

 

i have a script i wrote which simplifies things like snapshots and scrubbing, both of which are very simple commands, it would take longer to open a gui and click a button then to type the 3 or 4 words necessary eg. 'zpool scrub storage'.

 

take a snapshot? 'zfs snapshot -r /storage/directory@snapshotname' or look at a snapshot? 'zfs list -t snapshot'.

 

want to add a spare disk? 'zpool add storage spare /dev/sd[x]' need to export the pool, maybe move to another machine or another os? 'zpool export storage'

 

as you can see, it's so simple there isn't much need for a gui. if you know what you want to do, it's almost certainly quicker to just type 3 short words then navigate a gui.

 

all of that said, there are options if a gui is required. nexenta comes to mind, there is a free community version and paid commercial version, it is based on open solaris and simplifies things greatly, the terminal commands will auto-complete or help you complete the command, and it has a web based gui which can be used to manage pools. there are add ins available too such as HA cluster which can be managed from this interface.

 

I am not sure if there is anything for linux, never looked. you can find my script somewhere in this thread to simplify basic administrative tasks.

Share this post


Link to post
Share on other sites

Because a complete package is a sign of quality and..... completion.

 

If its truly the best, it would be the most used. Right now, it lets itself down in one area, usability.

I can talk first hand, making a windows RAID5 is much MUCH easier than making an Ubuntu RaidZ

 

To take the usual comparison of cars to computers argument; best performance, best economy, but the steering wheel is behind your head so you'll have to learn to drive with your arms above your shoulders without going numb.

The smart person who felt that car was truly ready for the world because its the best, would move the steering wheel to a user friendly location.

Your analogy doesn't stack up. Your car one ... well ... out of a sene of being polite I won't relay the exact phrase that went through my head when I read it.

 

It's simpler for me to ssh remotely into a linux server and complete tasks then it is to go through VNC and launch gui tools. You don't have to script anything if you know the commands which, let's face it, are very very basic.

 

A web based control would probably do the trick too. FreeNAS gives you that if that is what you are after - a much simpler experience for people who aren't as cluey when it comes to Linux even though I recall you sneered at it earlier. There is probably, somewhere a plug for webmin too if that floats your boat.

 

ZFS seems to provide a shitload of advantages over software Raid. Accept it or not, that's entirely up to you. To imply it's shit because it doesn't have a GUI? Please.

 

In any event, considering it's history and that zfsonlinux (which is what I am playing with) seems to be backed by no particular industry or corporation - I'll cut them some slack for not producing a GUI for it.

Edited by The Tick

Share this post


Link to post
Share on other sites

to setup my raidz pool on ubuntu took about 30 seconds, to both install zfs then make a pool. i could then take snapshots, copy data etc. nothing more to be done. one line is all it takes to create the pool 'sudo zpool create -f -o ashift=12 storage raidz /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde'. how long does it take to setup raid5 in windows M_S? is there any kind of performance hit or do you get decent read/write speeds?

Yeah probably about the same. Open disk manager, new software raid, next, next, next, done.

From a clean windows install probably 15 seconds of interaction, 1 minute of 'quick format' and you're going.

I didn't notice any performance hit. My Raid still played back 1080p fine, and I was able to dump CD rips just as fast as my burner could rip them.

I was also able to download to the network share; so its well above my internet connection speed.

IMO it passed all the tests I could possibly throw at it as a home user wanting a 10tb redundant drive.

 

It's simpler for me to ssh remotely into a linux server and complete tasks then it is to go through VNC and launch gui tools. You don't have to script anything if you know the commands which, let's face it, are very very basic.

I wont argue that; but do you really consider yourself to be 'the average Joe'? you seem, like most people here, to also be a 'geek'.

I dont think thats the norm honestly. I'd need to download an SSH client and enable SSH at the least. And god knows what that'd do to my front end while i'm playing back stuff.

 

ZFS seems to provide a shitload of advantages over software Raid. Accept it or not, that's entirely up to you. To imply it's shit because it doesn't have a GUI? Please.

It does provide a lot. I didnt think i'd tried to say otherwise?

Who implied it was shit? I'll go re read my posts, but I thought I suggested it was 'hard' and felt unfinished. But i'll go reread. I often forget or change my opinion without notice :P

 

A web based control would probably do the trick too. FreeNAS gives you that if that is what you are after - a much simpler experience for people who aren't as cluey when it comes to Linux even though I recall you sneered at it earlier. There is probably, somewhere a plug for webmin too if that floats your boat.

I'd try a webmin yeah. FreeNAS breaks ZFS. To me a good tool needs to be slightly robust in its construction.

Using freenas, if i simulated blackouts or interrupted commands (by sending more before it was finished), i'd be able to 'break it' very quickly.

Something normal linux or windows raid survived fine.

 

nexenta comes to mind, there is a free community version and paid commercial version

Nexcenta is cool. I used it a bit. I was still able to break it by pulling drives during a resilver (once again, something that won me on Windows RAID, I literally could NOT break it.

Though I'll happily admit, i'd be OK with nexcenta, it was at least hard to break. I just couldn't get Xbox Media Centre to run on it, and hardware decoding for my GPU seemed non existant which was a requirement for my server. (my server got a GPU added and became one of my clients too).

Interestingly, at least with most of my mates, this is a common scenario. their HTPC gets extra drives and becomes both the client for the TV and a server for the PC's elsewhere.

 

________________________________________________________________________________

 

 

 

Anyway Tick, I think you missed my point.

To my view, society dictates whats best via a number of different aspects.

The ones to my mind, are: performance, reliability, ease of use (and accessibility), simplicity (using and understanding is different), and cost. If you manage to tick all of those, to me, your product is perfect.

If one is lacking too much, it can pull the plug on all the others.

 

What I'm suggesting, is when a technology becomes the best, people get excited, and development on all the above fronts accelerates to incredible speed. (polarization in eyeware is a good example. The Engine is another. Hell, computers themselves).

ZFS has a huge list of features I'd love on a desktop, a server, anywhere in fact. So why don't I use it? More to my point, why doesn't my mum use it? a truly revolutionary technology should take the world by storm. and IMO its because the ease of use is lacking.

Using a spoken sentence or two, away from a PC, i can explain to someone how to format something to FAT, and they'll likely remember. Its not so easy with ZFS imo. strings of commands are much harder to talk verbally or memorise.

 

Examples of progress?

compression was cool; someone made winzip.

MP3s were fun, someone made winamp\napster

2.4GHZ RF is cool, someone made peripherals using it.

 

I'm saying that someone, maybe a certain type of person? the artsy type? Is missing from the equation.

No developer has gotten excited on the front of usability and uptake. wanting to spread it to the world! No pretty icons. No 'breakproof' gui.

 

It's something the Open Source community does a lot in my experience.

They develop truly amazing things, but make it too hard for the non IT savvy to access them. Its a big weakness in uptake of something people apparently feel is revolutionary and 'the best'.

LAME is still the best example, and IMO, the GUI tools available had a big part in the MP3 survivng against OGG.

 

When I get a new toy (phone? mp3 player?), its taken out everywhere and shown to everyone, and explained to everyone.

Just last week I was at a coffee shop with some friends and we all got excited talking about flash drive formatting and the 'free space' bit map of ExFAT and how this affected our smartphones and arcade machines. We all knew about it because it was presented so readily (it just... 'appeared' in windows and mac; visually). I was the only person who knew about ZFS.

 

Its very late, I hope I made sense. Im so tired. Im hitting bed.

Edited by Master_Scythe

Share this post


Link to post
Share on other sites

No, thats what I thought was a very obvious exaggeration, my bad, shoulda had a ':p' after it.

On a less exaggerated note, my 'gamer' friends do though. None of them have ever heard of ZFS either (i asked em last night). They had heard of "Windows raid" and "Liniux raid" as well as "hardware raid".

 

Redundancy is fast becoming a requirement. So far my views into the end user are that the winners are 'the cloud' and windows RAID.

I agree it should be ZFS, 100X over, but I honestly believe usability, and availability is why its not.

Great tech, hard to use, and to obtain (limited to types of *nix).

Too bad the OSX thing never worked out.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×