Jump to content
Sign in to follow this  
lew~

ZFS NAS

Recommended Posts

Hi Guys,

 

No longer require a desktop computer any more. Don’t really play any games my MacBook Pro can’t play. My desktop computer currently has 1.5TB fake-Raid5, 1TB fake-Raid5, and 3.5TB external single disks giving a total of 6TB non-redundant storage across multiple volumes.

Looking to replace this with at least 6TB of expandable ZFS storage for sharing over CIFS/NFS to multiple clients, i.e. HTPC, MacBook Pro, etc.

 

 

Requirements:

1. 6TB initial ZFS storage for mainly HD movies with room to grow.

2. Headless server.

3. Reasonably portable, would like to be able to take to the occasional LAN.

4. As low power as possible without replacing existing hardware (below) until it dies.

a. Disk spin down when not in use

b. Hibernate/Sleep during the day with scheduled wake up/WoL

5. Reasonable performance. ZFS will take care of itself in this respect really. As long as I get GigE out of it I’m happy.

6. As quiet as possible.

 

 

Existing Hardware:

• Q6600

• 4GB RAM - will upgrade to 4x2GB for 8GB total.

• 8800GT - get rid of this to free up a PCI-E slot, PCI video card or similar.

• P35-DS3P - 2x PCI-E x16 slots (x16 & x4), 3x PCI-E x1, 2x PCI-E

• Misc. HDD's - 3x750GB, 1x640GB, 2x500GB, 1x1.5TB, 1x2TB

• Antec P182 - maximum 14 3.5" bays with 5.25" adaptors

• Corsair HX-520

 

 

Bought:

1. 4x2TB Hitachi

2. LSI SAS 3442E-R

a. Expandable with a SAS expander, like the HP ones.

 

 

 

To Buy:

1. Thinking of buying the NZXT Whisper to replace the P182 for a potential maximum of 18 3.5" bays (utilising 5.25" bays with brackets). Only concern is HDD cooling if the 5.25" bays were utilised, although I could probably rig a fan to blow over them if required.

Another option is to get a proper chassis like the Norco 4220 with 20x hot swap bays. Biggest problem is I don't really have anywhere clean to store it. At least a desktop case can go in a corner with a switch and router. Also, not very portable. And loud (pre-modification/fan replacement). And expensive.

2. Dual-port GigE PCI-E 1x Intel NIC or 2x single-port GigE PCI-E 1x Intel NICs.

I understand PCI-E 1x slots have 250MB/s bandwidth, so 2GigE should be fine. Set up link aggregation to easily saturate GigE clients.

 

OS:

FreeBSD 8.0 or OpenSolaris, ZFS version 14.

Leaning towards FreeBSD at this stage. In my mind, the ease of use of FreeBSD with ports, improved hardware support and ease of configuration outweighs any lag in ZFS releases. The array should be able to saturate GigE anyway.

Because important data would be backed up, also considering running a more recent release of FreeBSD/OSol for (mainly) ZFS dedup.

 

Data Structure:

4x2TB Hitachi's in a Raidz1 pool. 6TB capacity, expandable with another 4x2TB Raidz1 pool to 12TB, etc.

Once the pools are up I’ll do testing with dummy data to see how the Q6600 copes with varying levels of compression (gzip-9 etc), and what sort of performance I can expect out of it (to determine whether I’ll even use 2xGigE Intel NICs.

 

 

Backups:

Thinking important data would be backed up to external drive(s) made up from the old existing drives.

 

 

 

Questions:

 

OS – FreeBSD vs. OpenSolaris. Thoughts? Is the future of ZFS on OSol a concern with Oracle? Is ZFS implemented well enough on FreeBSD in terms of features/stability/performance to be a worthy alternative?

Anyone running dev releases of ZFS? Worth it for dedup considering it’ll mainly be HD movies?

 

Data Structure – With the current proposal of a 4 disk Raidz1 pool, 4 disks are used to read/write data. When this expands with another 4 disks all 8 disks would be used to read/write data, correct?

Is there a more power efficient way to structure my pools/disks?

 

General comments/thoughts?

Edited by lew~

Share this post


Link to post
Share on other sites

*Waits for Zebra*.

 

I'm really interested in how this goes. Pitty I cant help you as I'm in the same boat.

 

 

Where are you getting the LSI SAS 3442E-R from?

Share this post


Link to post
Share on other sites

Got the HBA from a friend who bought it from eBay, wasn't compatible with his expander unfortunately.

 

There's quite a few on eBay actually, including other HBA's like the Super Micro SASLP-MV8 & USAS-L8i. Save quite a bit of money buying online. Fan-out cables too, they're $80~ locally or $20~ on eBay.

Share this post


Link to post
Share on other sites

Might have to grab one off ebay then

 

 

Thanks man ^_^

Share this post


Link to post
Share on other sites

Use OpenSolaris.

 

It's not dying.

 

Make sure your hardware is on the OpenSolaris HCL. I had to buy an additional GbE network card, my onboard wasn't supported when I built the original one.

 

1) raidz1 will cover this

2) headless works fine with opensol, vnc / ssh available

3) that just depends how big the case is

4) you can probably get these things sorted, mine stays on 24/7 though doing other tasks

5) you can / will saturate gige.

6) mostly covered by the same answer as 3, replace big with quiet.

 

I'm currently running mine with 6*1TB and 4*1.5TB raidz1 sets. Thumps along. Life is sweet.

Share this post


Link to post
Share on other sites

Thanks brains,

 

The HBA isn't listed on the HCL, but LSI have Solaris drivers on their site so I might give OSol a whirl first.

 

What release are you running brains?

Do you have dedup turned on for any media pools? If so, what ratios are you seeing?

Share this post


Link to post
Share on other sites

I've got an LSI 3081ER in mine. I may have got it because Sun OEM it >_> but if drivers exist on the LSI site, don't stress.

 

I'm currently running opensol b131. I did have dedup on, but I wasn't getting any kind of noticable saving (due to the kind of data that was being dropped on there) and it tended to kill zpool scrub speeds, so it was removed.

 

Scrubbing your pools is important.

 

I think that about covers it.

Share this post


Link to post
Share on other sites

Just put a real osol release on there to start with to see how a test pool handles. if all is well then upgrade to build 134. Esentially build 135 should be the real next opensol release. but currently oracle are hunting through the repository and all code releated to the 'Sun' brand and rebranding the entire thing before a real release. 134 should be reasonably final in that regard. however i expect a lot of things will be broken between the two builds. Package names will all change and this is going to hurt anyone with build 134 and below running in production.

 

as for compression - just turn it on. my single core athlon 64 3200+ can according to file bench bang out 398.2mb/s with compression on (4 * 750gb Samsung F1) it's very light. But in the majority of home user cases its not going to do anything. All our data is usually jpg, mpeg. mkv or whatnot, inother words already compressed. In this respect ZFS is pretty smart - it will only try so hard to compress something before it gives up and just writes the native file to disk. Dedup can only be had with the dev tree (build 112+) and dedup didn't apear till something like build 128 or thereabouts.

 

use some of these.

http://www.pccasegear.com/index.php?main_p...roducts_id=5630

 

or that type of thing should keep drive within operating temp.

 

Buy a intel gigabit card its easier than screwing around with getting a onboard to work only to find the performance shite.

 

And for your final question it will only work like that if you drop them into the same pool. and even then i believe other tricky things happen - stuff i don't understand yet. however you are right in the same pool both arrays will be used - just creat two pools. movies and music or what not. ZFS is not great in this regard it naturally wants to spread its load, and the ecomentalists among us who are watching watts pile up on our bill want it to contain itself.

 

An LSI card is perfect they are the chosen supplier for Sun system, you will find them in all of their boxes. in other words well supported.

 

oh and incase you didn't know solaris will not install to a raid array (unless its raid1 - mirror) as a home server just use a single disk for the os, rpool mirror is not worth it. Just backup your /etc directory to your storage array. if your os disk fails pluck a new one from somewhere reinstall and throw your smb.conf file in and you are pretty much done aside from internet connectivity. Mirroring isn't worth the extra power consumption unless you make substantial changes to the os or need serious uptime.

 

NextentaStor is possibly something you want to try. Don't bother with freebsd its setup is not as good a osol, and the first time you use live upgrade you will 'jizz in your pants'

Share this post


Link to post
Share on other sites

Hi.

 

brains and sparky have covered off on all of this really well I think. Gosh. It brings a tear to my eye. We've all come such a long way together.

 

Anyway...

 

A couple of my comments.

 

Data Structure:

4x2TB Hitachi's in a Raidz1 pool. 6TB capacity, expandable with another 4x2TB Raidz1 pool to 12TB, etc.

Once the pools are up I’ll do testing with dummy data to see how the Q6600 copes with varying levels of compression (gzip-9 etc), and what sort of performance I can expect out of it (to determine whether I’ll even use 2xGigE Intel NICs.

OK. Of worthy note here are the performance expectations you'll see from a Q6600. I have, as does brains, a Q6600, 6GB RAM in mine. I run standard compression on mine (ljzw), with zfs compression=on, zfs dedup=on. If I'm piling data at it over single GbE, or doing a bulk copy on the array from place to place, I can seriously warm my load averages to near 4.00, 4.00, 4.00. So - you probably know your UNIX enough to know that this means you're effectively throwing enough floating point work through your processor to give it exactly zero idle time/100% utilisation of procs on and off the pipeline. Does it harm performance? Seemingly no - but stuff can sure as hell get busy if you start busting out all of OSOL's powerful shiz.

 

Backups:

Thinking important data would be backed up to external drive(s) made up from the old existing drives.

Nexenta crew are releasing their ZFS tape backing store tool soon....should be freaking epic.

 

OS – FreeBSD vs. OpenSolaris. Thoughts? Is the future of ZFS on OSol a concern with Oracle? Is ZFS implemented well enough on FreeBSD in terms of features/stability/performance to be a worthy alternative?

Anyone running dev releases of ZFS? Worth it for dedup considering it’ll mainly be HD movies?

OSOL all the way. It's not going anywhere as far as we understand it. Too much rides on it - like Oracle's 7000 series FISHWorks box. FreeBSD's current ZFS to get working as a root boot is an immense PITA. Not too mention, doesn't have current ZFS functionality. I'm running b134, just like the other two - and frankly, nothing could be finer. For dedup, like brains said - this kind of data won't benefit too much. Lemme show you something:

 

zebra@rapoosev5:/storage/media# du -sh video
611G	video

So, 611GB of HD/SD Video mostly in XViD/mkv/.x264 stuff there.

 

zebra@rapoosev5:/storage/media# zpool list storage

NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT

storage 3.62T 2.06T 1.56T 56% 1.07x ONLINE -

...and

 

zebra@rapoosev5:/storage/media# zfs get compressratio storage

NAME PROPERTY VALUE SOURCE

storage compressratio 1.03x -

So, I make a small saving with dedupe. Brains and I fight about this shiz all the time, but, in context, I agree with him, and think it's prolly not worth the performance penalty if this is your primary data type.

 

Having said that. If you slip an SSD in the mix and zpool attach cache /dev/ssd/device/here/ to your pool - you'll see some stunning gains in your dedup performance and scrub times. It gets _mental_. The HSP or Hybrid Storage Pool is a v. powerful thing.

 

Data Structure – With the current proposal of a 4 disk Raidz1 pool, 4 disks are used to read/write data. When this expands with another 4 disks all 8 disks would be used to read/write data, correct?

Is there a more power efficient way to structure my pools/disks?

If you remake the array with all the disks, yes. Expansion on a per disk basis like this isn't really working like a normal volume manager yet. Some say in b150 something special will make it work like magic.

 

As for power - not really, but I can offer this tidbit:

 

For drives controlled by the ata driver natively (which yours may or may not be, given the LSI controller), uncomment and set the standby setting in /kernel/drv/ata.conf, and reboot. This will allow "proper" spin down for you.

 

Oh...and one more thing. If when you're testing your ATA activity behaviour because the disks JUST WONT SPIN DOWN, you can check to see 100% if there is activity across your ATA or not with dtrace. Something like this spins to mind:

 

dtrace -n fbt::ata_disk_transport:entry

Hope that helps!

 

Cheers.

 

z

Edited by zebra

Share this post


Link to post
Share on other sites

Really awesome replies, thank you. I’ve taken it all on board.

 

 

Just put a real osol release on there to start with to see how a test pool handles. if all is well then upgrade to build 134.

I’ll do this tonight. I’ve backed up my fake-RAID arrays, blowing it away tonight and rebuilding.

 

however i expect a lot of things will be broken between the two builds. Package names will all change and this is going to hurt anyone with build 134 and below running in production.

Sounds nasty, but as long as you guys are in the same boat with me I’m not too worried.

 

I wish I’d seen these before I bought the Scythe bay. Thanks, that’s a great option.

The Zalman ones look pretty awesome too, http://www.pccasegear.com/index.php?main_p...oducts_id=13883.

 

 

NICs

I’ll give the build a go with the on board initially but plan on grabbing at least 1x Intel GigE NIC.

 

Power consumption

If I’m honest with myself, I would have got green drives if I was serious about power saving. But I decided I didn’t want girly disks and bought Hitachi’s, so I guess it’s not a huge deal if they’re all spinning.

I’ll take a look at the possibility of spin down through ata.conf/other.

 

Scrubbing

How often do you guys scrub your pools? And how long does it take?

Especially interested in your storage pool zeb, seeing as brains mentioned dedup kills speeds.

 

 

Compression & Dedup

Because the data on the media pool will already be compressed (as you guys have pointed out) I’m not expecting any amazing compression or deduplication, but sounds like it won’t hurt too much. And if the occasional raw BR may benefits somewhat, all the better.

 

Having said that. If you slip an SSD in the mix and zpool attach cache /dev/ssd/device/here/ to your pool - you'll see some stunning gains in your dedup performance and scrub times. It gets _mental_. The HSP or Hybrid Storage Pool is a v. powerful thing.

How do cache devices specifically improve dedup and scrub performance?

I’m thinking… if the scrub is going through the pool sequentially doing its thing, with the cache device acting as a buffer of data. Wouldn’t the scrub process eventually run out of data on the SSD, and again be limited by the performance of the pool?

I think I’m on the wrong track, but can’t get my head around it. Any articles you can point me to?

 

 

D-trace looks really powerful, and not something I’ve really played with yet. Something to look into I think.

 

The Speedlabs storage section is kinda slow… :( Is there anywhere online you guys hang/recommend for NAS/OSOL/ZFS discussions?

HardOCP forums have been really useful.

 

 

I haven’t done any work yet today, this is way too exciting. Tempted by a career change now.

 

Cheers guys.

Share this post


Link to post
Share on other sites

You better do a build log! Would nice to have a guide for this kind of setup.

Share this post


Link to post
Share on other sites

You better do a build log! Would nice to have a guide for this kind of setup.

I could... the install won't be interesting though, and zpool creation is so simple... maybe the benchmarks might be interesting. Possibly. :)

Share this post


Link to post
Share on other sites

I'm keen on the bench marks for those drives because im also looking at getting 4 of those 2tb Hitachi drives.

 

So once you've got it up and going it would be great to post some screenies :)

Share this post


Link to post
Share on other sites

Yeah i just want a job working with some sun/oracle hardware.......... Zebra?

 

Just use a real release ie 2009.06 or what not to see that there are not any oddities with your hardware. Sun have sometimes broken things pretty badly in the dev builds and not fixed it for a long time. ie about 12 builds did not work properly on AMD phenom 1 cpu's.

 

I expect things to be broken however i believe that oracle is trying their best to not have broken stuff in there and that is why build 134 is coming up 3 months old when usually they release every two weeks. alot of the package changes happened with build 133 so I think they are trying to build some compatibility with the previous release. it would make sence considering sun had always touted it binary compatibility with sol 10, 9, 8, 7 apps.

 

I'm going to pick up the silverstone version of the 4-3 mount.

 

Scrub your consumer disks once a week, takes a few hours. but the array is still usable. zpool status will give you an estimated completion time after you tell it to go.

 

Oh and while im running an old release on the actual server with a lot of MKV files on there i have a compression ratio of 1.01x and I'm pretty sure the 1% comes from the RAW files i dumped on there from my digital camera. If you have a bunch of text files you can watch your compression ration go through the roof. but most of our media stuff just has bugger all benefit.

 

Now when you give the real release a run and create a test pool. give it a bit of a run like that when you upgrade to the dev tree you need to upgrade that zfs upgrade and zpool upgrade - (might be update) to move to the latest versions test these thoghly because if you dump your data onto one of these upgraded arrays you can not downgrade them and its likely a real release wont be able to use them. A common prctice seems to be to use a real release to build your pool the upgrade to the dev tree and don't upgrade your ZFS. You still have most of the awesome but you will miss out on dedup. L2ARC, compression, ZIL are all in the last release. if you run this config and run into trouble with the dev release then hug ZFS because of something called live upgrade and its backend beadm. when you pump up your system from real to dev esentially what it does is move the filesystem to a new rpool then create symlinks in the old location and update all the filels in the new release and place the old file in a usable location. now in reality this is achieve by carefull placement of a couple of snapshots and not synlinks. what it gives you is a new os that takes up about 4gb of hdd space and an old one that only takes up a few kb of disk space because its only the chages between the too os's. and it will give the option of booting either of them at the grub menu. id you don't like your new environment then us beadm (boot environment) to blast away the new environment and make the old bootable. Nice!!

Edited by Sparky

Share this post


Link to post
Share on other sites

Update...

 

 

Everything migrated to the NZXT Whisper using Zalman 5.25" drive bays.

Intel Gig CT Desktop Adaptor didn't get picked up by osol automatically, removed it for now so I can at least continue setting it up.

 

lou@nas:~# uname -a
SunOS nas 5.11 snv_134 i86pc i386 i86pc
lou@nas:~# zpool list
NAME	SIZE  ALLOC   FREE	CAP  DEDUP  HEALTH  ALTROOT
rpool   596G  5.64G   590G	 0%  1.00x  ONLINE  -
tank   7.25T   245K  7.25T	 0%  1.00x  DEGRADED  -

 

 

 

Haven't installed the other drives for the second 3TB raidz pool. For some reason one of the 2TB drives isn't being picked up by the HBA, but the 2TB drive works on another sata port, and another drive works on that HBA port, haven't looked at that properly yet.

 

lou@nas:~# zfs get sharesmb tank/media
NAME		PROPERTY  VALUE			   SOURCE
tank/media  sharesmb  rw=@192.168.1.0/24  loca
l

 

I've done this:

lou@nas:~# tail -1 /etc/pam.conf 
other   password required	   pam_smb_passwd.so.1	 nowarnlou@nas:~#
And reset the passwd for account 'lou'.

 

I can connect to smb://192.168.1.2, authenticate and the server appears in Finder but I can't write any files to the media share. OS X prompts for a username and password: http://i49.tinypic.com/6jh8g9.png then stops with this http://i47.tinypic.com/bi0195.png.

 

 

I grabbed the OpenSolaris Bible, just working through that and google to try and get stuff working.

 

I rebuilt a few times before moving everything to the new case, I found that if osol didn't shutdown cleanly it would boot indefinitely (osol graphical loading screen) which was kind of disappointing. I'm not good enough with it to work out why.

 

 

 

 

Once I've got smb sharing working, and happy with the stability and performance of the system and pools, I'll start migrating my data on to it.

Share this post


Link to post
Share on other sites

I cant help with the SMB stuff since i still have not moved over from Samba yet. Since you say 'finder' I take it you are on OS X, so then I assume you could try NFS.

 

Curious about the intel nic not being detected, they usually come up through the e1000g driver and play nice.

 

Osol not shutting down correctly, have not had that one before, ive killed it on boot :) but never on shutdown.

Share this post


Link to post
Share on other sites

Thanks Sparky,

 

This thread seems to indicate it should work fine, as I would expect from an Intel product. Might give it another try when I get a chance.

 

Regarding the SMB stuff, eventually it needs to be accessed by Windows hosts as well. If NFS performs better for my OS X host then I'll use that as well though.

 

I'll work through this link when I get a chance, hopefully help me get read/write working.

 

 

Need to fix, NIC, SMB RW, HBA Drive detection and I can finally start stress/perf testing!

Share this post


Link to post
Share on other sites

I had a lot of trouble with the built in SMB sharing of ZFS, it'd work fine for a week or so, then it'd just stop and all 'zfs' command operations would hang, only a reboot would fix it.

 

So, I got a SAMBA service going, and it's worked non stop for: 12:13pm up 75 day(s), 19:46, 1 user, load average: 0.05, 0.04, 0.03

 

I installed via pkg-get and the blastwave repos and there's a few tricks to getting SAMBA running,

 

- you have to create a config file at /etc/sfw/samba/smb.conf

 

then stop the zfs sharing:

 

zfs set sharesmb=off bucket/Media

svcadm disable smb/server

 

add a password for the relevant user: smbpasswd -a strageattractor

 

and finally, start the samba server: svcadm enable samba

 

 

Then you can mount with full privileges using the username/password configured above.

 

 

I'm using an old version of Solaris Express, build 116 I think, as at the time, it was the only distribution that had support for my sunix SATA cards.

Share this post


Link to post
Share on other sites

I was going to suggest something similar, but I wasn't sure how OS and samba integrate.

Share this post


Link to post
Share on other sites

Another cool and interesting option is to export iSCSI LUN's over COMSTAR and simply plop them onto your target host, formatted as whatever native FS you like, using ZFS as the backing store.

 

Epic performance, even on home networking!

 

z

Edited by zebra

Share this post


Link to post
Share on other sites

I'm half considering this at the moment so that I can have ZFS storage as well as virtulisation. Only issue is I'd need separate boxes for both :(

Share this post


Link to post
Share on other sites

I'm half considering this at the moment so that I can have ZFS storage as well as virtulisation. Only issue is I'd need separate boxes for both :(

I did exactly this until very recently. Had vbox running headless on the ZFS file server, and had uTorrent, ircd et al running in it, atop ZFS pools. Axed it in the end and broke free. Running pure unix IRC and UNIX BT client now on OSOL (Transmission). Compiled from source without *too* much hassle...

 

z

Share this post


Link to post
Share on other sites

Having used both OpenSolaris (builds 111, 120-129, 131) and FreeBSD (builds 7.0, 7.1 and 8.0), for a home server with ZFS, I've found FreeBSD to be much nicer.

 

Sure, FreeBSD 8.0 Stable doesn't have the latest version of ZFS. You miss out on dedup, iSCSI and automatic sharing via NFS and SMB (not sure how reliable the SMB is on OSol anyway!), none of which really bothered me. What I did find was a lot of stuff broke between dev builds, and I found myself eagerly waiting for the next development release, not for the Next Cool New Thing, but to simply to fix what the last had broken. I was often disappointed.

 

OpenSolaris's system update is totally awesome, and FreeBSD 8 isn't anywhere near that yet. However, I feel that the trade off of having more ports / packages available, and not having to muck around installing extra stuff worth it, and FreeBSD wins for me in that regard.

 

In summary, OpenSolaris is great if all you're doing is ZFS and you want every single thing ZFS has to offer, but for everything else, it's much more frustrating. I got sick of dealing with that frustration for my home server :)

Edited by wilsontc

Share this post


Link to post
Share on other sites

Found some time..

 

- b134 running

- smb r/w working

- Intel NIC still doesn't even detect as far I can tell - either in osol or FreeBSD. Intend to try Windows, it may be DOA.

- SAS HBA writes maxing out at 7MB/s in osol, FreeBSD and Windows 7. Plan to update the firmware to try and fix this.

 

I'm not getting very good performance out of my 4x 2TB disk raidz1 pool. Currently the pool is made up of 3x 2TB Hitachi's and 1x 2TB Seagate. The drives are directly attached to the motherboard in AHCI mode.

 

tank iozone results:

 

 root@nas:~# /usr/benchmarks/iozone/iozone -R -l 5 -u 5 -r 4k -s 100m -F /tank/f1 /tank/f2 /tank/f3 /tank/f4 /tank/f5
		Iozone: Performance Test of File I/O
				Version $Revision: 3.323 $
				Compiled for 32 bit mode.
				Build: Solaris10cc-64 

		Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
					 Al Slater, Scott Rhine, Mike Wisner, Ken Goss
					 Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
					 Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
					 Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
					 Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

		Run began: Wed Jul 21 18:33:13 2010

		Excel chart generation enabled
		Record Size 4 KB
		File size set to 102400 KB
		Command line used: /usr/benchmarks/iozone/iozone -R -l 5 -u 5 -r 4k -s 100m -F /tank/f1 /tank/f2 /tank/f3 /tank/f4 /tank/f5
		Output is in Kbytes/sec
		Time Resolution = 0.000001 seconds.
		Processor cache size set to 1024 Kbytes.
		Processor cache line size set to 32 bytes.
		File stride size set to 17 * record size.
		Min process = 5 
		Max process = 5 
		Throughput test with 5 processes
		Each process writes a 102400 Kbyte file in 4 Kbyte records

		Children see throughput for  5 initial writers  =  238286.51 KB/sec
		Parent sees throughput for  5 initial writers   =   64108.48 KB/sec
		Min throughput per process					  =   40849.69 KB/sec 
		Max throughput per process					  =   52981.06 KB/sec
		Avg throughput per process					  =   47657.30 KB/sec
		Min xfer										=   79508.00 KB

		Children see throughput for  5 rewriters		=  632109.74 KB/sec
		Parent sees throughput for  5 rewriters		 =   78932.90 KB/sec
		Min throughput per process					  =  109697.19 KB/sec 
		Max throughput per process					  =  138991.03 KB/sec
		Avg throughput per process					  =  126421.95 KB/sec
		Min xfer										=   77180.00 KB

		Children see throughput for  5 readers		  = 1286008.22 KB/sec
		Parent sees throughput for  5 readers		   = 1269233.50 KB/sec
		Min throughput per process					  =  141605.52 KB/sec 
		Max throughput per process					  =  350586.47 KB/sec
		Avg throughput per process					  =  257201.64 KB/sec
		Min xfer										=   39952.00 KB

		Children see throughput for 5 re-readers		= 1755400.53 KB/sec
		Parent sees throughput for 5 re-readers		 = 1703278.51 KB/sec
		Min throughput per process					  =  237918.66 KB/sec 
		Max throughput per process					  =  435313.03 KB/sec
		Avg throughput per process					  =  351080.11 KB/sec
		Min xfer										=   58248.00 KB

		Children see throughput for 5 reverse readers   = 1614811.05 KB/sec
		Parent sees throughput for 5 reverse readers	= 1552448.11 KB/sec
		Min throughput per process					  =  244936.61 KB/sec 
		Max throughput per process					  =  394238.88 KB/sec
		Avg throughput per process					  =  322962.21 KB/sec
		Min xfer										=   58360.00 KB

		Children see throughput for 5 stride readers	= 1366732.18 KB/sec
		Parent sees throughput for 5 stride readers	 = 1307209.88 KB/sec
		Min throughput per process					  =   38253.55 KB/sec 
		Max throughput per process					  =  377621.47 KB/sec
		Avg throughput per process					  =  273346.44 KB/sec
		Min xfer										=   10604.00 KB

		Children see throughput for 5 random readers	= 1228025.02 KB/sec
		Parent sees throughput for 5 random readers	 = 1200078.19 KB/sec
		Min throughput per process					  =  138531.33 KB/sec 
		Max throughput per process					  =  292320.06 KB/sec
		Avg throughput per process					  =  245605.00 KB/sec
		Min xfer										=   48736.00 KB

		Children see throughput for 5 mixed workload	= 1158675.28 KB/sec
		Parent sees throughput for 5 mixed workload	 =  155662.08 KB/sec
		Min throughput per process					  =   95152.00 KB/sec 
		Max throughput per process					  =  291330.44 KB/sec
		Avg throughput per process					  =  231735.06 KB/sec
		Min xfer										=   33892.00 KB

		Children see throughput for 5 random writers	=  751709.68 KB/sec
		Parent sees throughput for 5 random writers	 =   51340.24 KB/sec
		Min throughput per process					  =   45656.27 KB/sec 
		Max throughput per process					  =  206802.72 KB/sec
		Avg throughput per process					  =  150341.94 KB/sec
		Min xfer										=   22736.00 KB



"Throughput report Y-axis is type of test X-axis is number of processes"
"Record size = 4 Kbytes "
"Output is in Kbytes/sec"

"  Initial write "  238286.51 

"		Rewrite "  632109.74 

"		   Read " 1286008.22 

"		Re-read " 1755400.53 

"   Reverse Read " 1614811.05 

"	Stride read " 1366732.18 

"	Random read " 1228025.02 

" Mixed workload " 1158675.28 

"   Random write "  751709.68

 

 

 

 

while copying over the Gig network from MBP to osol via smb writes seem to be pausing. I can hear the disks when they're writing. The entire time the finder copy dialog is progressing at a few MB/s.

 

SMB:

 

$ pfexec zpool iostat tank 1
			   capacity	 operations	bandwidth
pool		alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank		1.97G  7.25T	  0	 31  1.14K  3.09M
tank		1.97G  7.25T	  0	  0	  0	  0
tank		1.97G  7.25T	  0	  0	  0	  0
tank		1.97G  7.25T	  0	141	  0  17.2M
tank		1.97G  7.25T	  0  1.29K	  0   163M
tank		1.51G  7.25T	  0  2.15K	  0  13.6M
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	404	  0  49.8M
tank		1.51G  7.25T	  0	528	  0  63.8M
tank		1.76G  7.25T	  0  2.69K	  0  81.2M
tank		1.76G  7.25T	  0	  0	  0	  0
tank		1.76G  7.25T	  0	  0	  0	  0
tank		1.76G  7.25T	  0	  0	  0	  0
tank		1.76G  7.25T	  0	  0	  0	  0
tank		1.76G  7.25T	  0	  0	  0	  0
^C
last pid:   910;  load avg:  1.54,  1.21,  0.86;  up 0+00:36:15
59 processes: 58 sleeping, 1 on cpu
CPU states: 38.4% idle,  0.0% user, 61.6% kernel,  0.0% iowait,  0.0% swap
Kernel: 1637 ctxsw, 3383 intr, 56 syscall
Memory: 8190M phys mem, 5682M free mem, 2047M total swap, 2047M free swap

   PID USERNAME NLWP PRI NICE  SIZE   RES STATE	TIME	CPU COMMAND
   534 root	   17  59	0   16M 9236K sleep	0:02  0.25% smbd
	65 root		1  59	0 8272K 1800K sleep	0:00  0.01% dhcpagent
   698 root		1  59	0 3868K 2188K cpu/3	0:00  0.01% top

AFP seems to be better though:

 

$ pfexec zpool iostat tank 1
			   capacity	 operations	bandwidth
pool		alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank		3.36G  7.25T	  0	 41  1.02K  3.49M
tank		3.36G  7.25T	  0	  0	  0	  0
tank		3.36G  7.25T	  0	  0	  0	  0
tank		3.36G  7.25T	  0  2.89K	  0  4.07M
tank		3.37G  7.25T	  0	191	  0  22.5M
tank		3.40G  7.25T	  0	877	  0  19.6M
tank		3.42G  7.25T	  0	624	  0  2.99M
tank		3.42G  7.25T	  0	  0	  0	  0
tank		3.42G  7.25T	  0	  0	  0	  0
tank		3.42G  7.25T	  0	133	  0  16.0M
tank		1.13G  7.25T	  0  1.55K	  0  45.7M
tank		1.13G  7.25T	  0	831	  0  21.6M
tank		1.16G  7.25T	  0	717	  0  18.3M
tank		1.18G  7.25T	  0	  0	  0	  0
tank		1.18G  7.25T	  0	  0	  0	  0
tank		1.18G  7.25T	  0	113	  0  13.5M
tank		1.18G  7.25T	  0	229	  0  28.6M
tank		1.25G  7.25T	  0  1.39K	  0  26.2M
tank		1.25G  7.25T	  0	884	  0  15.8M
tank		1.29G  7.25T	  0	726	  0  18.5M
tank		1.31G  7.25T	  0	  0	  0	  0
tank		1.31G  7.25T	  0	  0	  0	  0
tank		1.31G  7.25T	  0	135	  0  16.4M
tank		1.31G  7.25T	  0  1.15K	  0  29.5M
tank		1.37G  7.25T	  0	278	  0  23.3M
tank		1.41G  7.25T	  0  1.39K	  0  20.5M
tank		1.42G  7.25T	  0	  0	  0	  0
tank		1.42G  7.25T	  0	  0	  0	  0
tank		1.42G  7.25T	  0	141	  0  17.7M
tank		1.42G  7.25T	  0	522	  0  64.9M
tank		1.54G  7.25T	  0  1.26K	  0  25.0M
tank		1.51G  7.25T	  0	874	  0  7.37M
tank		1.51G  7.25T	  0	  0	  0	  0
tank		1.51G  7.25T	  0	 90	  0  11.2M
tank		1.55G  7.25T	  0  1.10K	  0  29.7M
tank		1.57G  7.25T	  0	636	  0  17.8M
tank		1.58G  7.25T	  0	404	  0   693K
tank		1.58G  7.25T	  0	  0	  0	  0
tank		1.58G  7.25T	  0	  0	  0	  0
tank		1.58G  7.25T	  0	370	  0  46.1M
tank		1.65G  7.25T	  0  1.26K	  0  19.7M
tank		1.68G  7.25T	  0	783	  0  16.7M
tank		1.69G  7.25T	  0	339	  0   585K
tank		1.69G  7.25T	  0	  0	  0	  0
tank		1.69G  7.25T	  0	276	  0  34.2M
tank		1.74G  7.25T	  0  1.08K	  0  17.7M
tank		1.76G  7.25T	  0   1004	  0  15.2M
tank		1.77G  7.25T	  0	  0	  0	  0
tank		1.77G  7.25T	  0	  0	  0	  0
tank		1.77G  7.25T	  0	797	  0  45.6M
tank		1.77G  7.25T	  0	638	  0  1.11M
tank		1.77G  7.25T	  0	  0	  0	  0
tank		1.77G  7.25T	  0	  0	  0	  0
tank		1.77G  7.25T	  0	  0	  0	  0
tank		1.77G  7.25T	  0	  0	  0	  0
tank		1.83G  7.25T	  0	473	  0  58.6M
tank		1.83G  7.25T	  0  1.45K	  0  14.4M
tank		1.93G  7.25T	  0	864	  0  24.7M
tank		1.96G  7.25T	  0	 24	  0  33.8K
tank		1.96G  7.25T	  0	  0	  0	  0
tank		1.96G  7.25T	  0	256	  0  31.6M
tank		1.96G  7.25T	  0	419	  0  49.3M
tank		2.01G  7.25T	  0  1.07K	  0  12.1M
tank		2.07G  7.25T	  0	911	  0  11.6M
tank		2.09G  7.25T	  0	  0	  0	  0
tank		2.09G  7.25T	  0	  0	  0	  0
tank		2.09G  7.25T	  0	  0	  0	  0
^C

 

If you want any other information or tests, let me know.

 

 

Cheers

Edited by lew~

Share this post


Link to post
Share on other sites

Hrrrrm....

 

What kind of speeds are you getting on a zpool scrub?

 

z

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×