Jump to content
p0is0n

File Servers & NAS

Recommended Posts

Thanks for taking an interest guys, glad some of you found it useful. :) Also thanks to Nich... for renaming the thread at my request.

 

@Mac Dude - there are some nice GUIs available for configuring samba as an alternative to editing config files, I haven't really looked into them yet, but when I do I will be sure to update the posts here. It might be an easier way to achieve the same or similar results. If you are going down the ubuntu path, you will find most of your questions are just a google away from being answered which I love, but feel free to ask here too, I am happy to share what i know so far, as are others which I am sure you know ;)

 

One of the best things about linux is that someone has made a tool or a program to make almost anything easier or simpler, or to do almost anything you could think of. Sometimes it's just finding the right one.

Share this post


Link to post
Share on other sites

Another quick update, I found a great torrent client with a web-ui so I've installed it on my server too. Tested it over the weekend and works well IMO. Things have come a long way since I first started visiting uncle Torrence with BitComet and then BitTornado about 9 or so years ago. I use this mostly for music, as I haven't been able to find a lot of what I like listen to on Usenet, but am a member of a very good private tracker :) Same goes for older TV shows. I've often heard people raving about how good it is compared to torrents, so maybe I'm doing it wrong. It is great for most things I want, but there is a much larger library of files available via torrent IMO. I believe sickbeard is also capable of searching for torrents so I will look at integrating that eventually, and see how well it works.

 

The torrent client I've installed is called Deluge. It's available for multiple OS, not just linux. I found that there are less options on Linux than I would like, uTorrent was my preference previously but it has very limited Linux support. An alpha that seems to be 2 years old, so I decided to look around.

 

Deluge is a bit different from most torrent clients, like uTorrent, in that it can either be used with a GUI like a normal client, or as a thin client primarily running as a daemon in the background, and you can connect to it with a number of interfaces such as a gui client, command line or web-gui. I opted for web-gui since I access my server from a number of machines, I don't want to have to configure a client on each one. It is pretty much identical to the GUI and does everything I require. There is a repository available for ubuntu, so it was just a case of installing from there.

 

Does anyone actually find the terminal commands I've been typing out useful? I know when I was new to linux it was of great benefit, even if just as a reminder but now I am getting a bit more confident, I find it less informative. I will continue to do so, but let me know what you guys think.

 

This will install the core deluge package (inc. gui), the daemon (deluged) and the web-ui. For those who don't know, a 'd' on the end of a program usually means it is a daemon. This is something I didn't learn until I started using linux.

sudo apt-get update
sudo apt-get install deluge deluged deluge-web
I then run the daemon 'deluged' and the web-ui 'deluge-web' from my terminal and I can access it via a browser. It's that simple. The default port is 8112 but I changed this to 8084 as I am already using 8081-8083 for other stuff. First thing that pops up is connection manager, you should see the address of the machine running the daemon, you might have to click connect and/or start daemon then you're good to go.

 

I setup a folder to auto-add torrents from, that I can drop .torrent files into, or alternatively, it's also possible to add by the url. I ran into one small issue where it wasn't adding my account information into some torrents from private trackers and complaining I wasn't seeding, so I had to edit the tracker URL but it has worked fine for a few others.

 

Here is a sample of the GUI, not from my machine (i'm at work and my desktop is off) but it's close enough.

Posted Image

 

Everything else has been running fine. I wasn't sure if ZFS on linux might cause any issues especially once I really started using it, but so far it's been flawless and actually pretty impressed with it. I haven't needed to use snapshots or anything yet, so you may see a rage post down the track if that doesn't work, but by all accounts it should work fine.

 

That's all for now, cheers.

Edited by p0is0n

Share this post


Link to post
Share on other sites

@Mac Dude - there are some nice GUIs available for configuring samba as an alternative to editing config files,

Webmin (as you have already shown how to install earlier) can do it pretty well.

Share this post


Link to post
Share on other sites

@Mac Dude - there are some nice GUIs available for configuring samba as an alternative to editing config files,

Webmin (as you have already shown how to install earlier) can do it pretty well.

 

Actually didn't know that about Webmin, thanks Tick :) Will make a note to check it out more thoroughly when I have a chance.

 

Edit: Sure enough, has a nice little interface for samba, and also openssh which i've installed. Seems to be able to tell what's installed and offers to manage it.

Posted Image

Edited by p0is0n

Share this post


Link to post
Share on other sites

I'm in much the same position as you p0is0n, with the added benefit of having done classes with Linux so I've had a more structured approach to my learning. I LOVE the way you write out your tutorials, having read a lot over the years and finding some easy and some completely beyond my ability to comprehend, yours are the BEST I've read yet.

 

You convey the concepts clearly, with plenty of expansion on the subject when required. You litter your posts with neat little factoids and tidbits, which is awesome, and maybe it's just me and my current state of learning linux, but I think it's great you keep the commands in your post. I believe it gives a nice tutorial feel to the posts, that if you want to read the entire post than you can. If you want a quick and dirty get it running now read, it's very quick to pick out those commands and do it.

 

10/10 from me !

Share this post


Link to post
Share on other sites

Thanks Devilsmurf, that makes me really glad :) I am not much of a writer so it is really good to get some positive feedback. Muchas Gracias

 

Edit: Next up I will be looking at implementing backups from windows (aka win7) machines to my storage array, another duty a file server should be able to perform. I will also be looking at scheduling automatic ZFS snapshots for recovery purposes.

Edited by p0is0n

Share this post


Link to post
Share on other sites

In the interest of furthering my knowledge, I've started refreshing my memory on scripting from quite a few years ago. I am quite comfortable with programming structure and syntax, however, need a refresher on a few things. I thought to begin with I would try to simplify some basic administrative functions. Here is a simple script I created to list/create/delete snapshots and a few other things. All ZFS related at this stage.

 

Let me know what you guys think and if there is anything I should add to it at this stage. It was really just a few simple commands to get me started and to make things a bit easier for me :)

 

If you wish to try it, save it as something.sh and then make it executable (chmod 755 something.sh) and then type ./something.sh to run.

I've tested it thoroughly tonight and seems to work well.

 

It is really really basic right now, but I will look at adding some more complicated stuff into it eventually.. this is about all I need to regularly do at the moment.

 

#!/bin/bash

# ZFS Administrative Tools
# 06.11.2012

# My first attempt at scripting. A simple menu designed to provide quick and easy access
# to some basic ZFS commands, so that they do not need to be repeated regularly.
# This may be elaborated on with time.

# The name of my pool may differ from yours, so I have added this line for pool name.
echo "Please enter the name of your zpool: "
read pool

# Loop script until exit
while true; do

# Clear screen
# clear

# Menu of options
echo "- - - - - - - - - - - - - - - - - - -"
echo "	   Simple ZFS Tools"
echo "- - - - - - - - - - - - - - - - - - -"
echo "A) List ZFS datasets"
echo "B) Check zpool/scrub status"
echo "C) View a list of available snapshots"
echo "D) Create a new snapshot"
echo "E) Rollback to snapshot"
echo "F) Destroy a snapshot"
echo "G) Initiate scrub"
echo "H) Stop scrub"
echo "I) View disk space consumed by snapshots"
echo "X) Quit"
echo "Please make a selection: "

# Read input from the user
read choice

# Do something based on the input provided
# Allows for upper or lower case characters
case $choice in
		a|A) sudo zfs list -r $pool;;

		b|B) sudo zpool status -v $pool;;

		c|C) sudo zfs list -t snapshot;;

		d|D) echo "Please enter a name for new snapshot: "
			 read new
			 sudo zfs snapshot -r $pool@$new;;

		e|E) echo "Please enter the exact name of the snapshot you wish to roll back to (Eg. storage/downloads@snap1): "
			 read restore
			 echo "WARNING: This will erase any snapshot more recent than the one selected."
			 echo "Press Y to continue..."
			 read confirm
			 if [ $confirm = Y -o $confirm = y ]; then sudo zfs rollback -r $restore; else echo "Aborted..."; fi;;

		f|F) echo "Enter the name of the snapshot to destroy: "
			 read destr
			 sudo zfs destroy -r $pool@$destr;;

		g|G) sudo zpool scrub $pool;;

		h|H) sudo zpool scrub -s $pool;;

		i|I) sudo zfs list -o space;;

		x|X) echo "Exiting..."
			 break;;

		*) echo "Invalid Choice";;
esac
echo "Press any key to continue..."
read key

# Ends the loop and exits
done
exit 0

Edit: Updated to include restoring snapshots.

Edited by p0is0n

Share this post


Link to post
Share on other sites

So creating and deleting snapshots, along with everything but restoring snapshots seems to work fine.

The snapshot restore "appears" to work, but I am not convinced it actually does anything.

 

I created a test snapshot, created new files, then restored to a previous snapshot to no apparent effect... perhaps I don't quite understand how snapshots work.

Upon further testing, i created a file, made a snapshot, duplicated the file with a new name, deleted the original, restored the snapshot, and things were back as they were.

 

I was under the assumption zfs rollback -r storage@snapshot would restore the subfolders as well, but it seems you must be very specific, so I have altered my script as such.

 

Seems to be working as desired now.

Share this post


Link to post
Share on other sites

In your script, you have clear screen commented followed by clear. I'm not 100% about scripting but should clear be commented as well, I don't think it should ?

 

I love the detail you do put into commenting, makes it easy for scripting/programming noobs such as myself to quickly grasp what you are doing.

 

So due to my financial situation I'm not able to splash out a lot of money on new hardware. What I have been able to do is pickup some hardware on the cheap.

 

I've grabbed a DELL server, not 100% sure on the model but it's a dual processor with 1 2.4ghz xeon P4 era, 2gb ram with a 6 port Megaraid SATA 150 card. I wouldn't mind grabbing a 2nd Xeon if the price was right.

 

Much stuff arseing about has enabled me to get the onboard SATA ports (2 of them ) out of raid mode and into ATA mode.

 

The issue here is that it's not set in bios but through a DOS program. It has a DVD drive and USB boot support but I couldn't get either to work. I got myself a couple PXE/TFTP servers going and I've since been able to load DOS through PXE and do a Win2K3 install on there.

 

I'd ideally like to flash the Megaraid to be a HBA card, but while later LSI cards seem to be quite easy to do this I can't see the same options on this card. Maybe I sell it and look for a cheap PCI-X SATA card ?

 

I've also just bought a 775 mobo/E2140/2gb ram/Vista combo for cheap, I'm going to throw together a quick wooden box since I can't get my hands on a PC case easy and cheap that I can make do what I can do with wood. Plus it's a project I've wanted to do for years.

 

So I'm thinking I'll two servers concurrently. The reason for this is I want to run up Linux (perhaps with ZFS) for a server, but past personal experience has taught me that I shouldn't be trusting my data on a test server, especially stuff I'm working on learning. I haven't decided the precise setup right now, but I do have 4 different PC's I can throw at the 2 server setup.

 

So I'm thinking a Linux build very much along the lines of the one used in this thread, because :

 

a) It seems to be working great guns for the OP ;

b) It's a lot easier to follow the tutorial style of this thread for me personally, and I've got a great fallback person to ask questions about why this is happening, or have a dialog to try and diagnose an issue,

c) I like the setup the OP has gone with.

 

Next post I'll detail the hardware I have at my disposal and see what people think what should be used where.

Share this post


Link to post
Share on other sites

thanks for your feedback mate, always supportive :)

 

first up with the 'clear' line I had it in there, but then when i was messing around with snapshots, it was more useful to print a list of snapshots, then look at it to restore or delete one, so i commented out the clear line. it's there if I want it I guess...

 

sounds like you're on the right track to a nice little file server, just be careful of older raid cards or HBAs not accepting larder capacity HDDs. You can get an IBM M1015 which is a 6gb/s SAS HBA with 2 internal ports for about $90 or so these days, which would handle up to 8 drives so may be an option but not sure on your budget.. for around $200 you could just get a microserver :)

 

the 775 setup might be the better way to go, but depends on exactly what you want to do with it and how much storage you need.

Share this post


Link to post
Share on other sites

I'd prefer not to have to spend $90 on a controller, because as you pointed out I'm almost half way to a microserver.

 

With that in mind, I'm going to be simply fileserver but media sharing to the house too, which often involves transcoding files to fit machines like the Xbox 360. I've got 3 kids, and the Xbox setup is easy enough the 3 yr old is learning how to pop on a movie if he likes.

 

So I'm not sure if the microserver will have enough grunt to handle that in the future.

 

 

So the 4 PC's I have currently at my disposal is as follows :

 

The main/gaming PC :

 

P4 3.2Ghz

3 Gb Ram (for some reason it won't recognise a 4th stick, no matter what I do )

64gb Samsung SSD

1tb Western Digital HDD

ATI 6870 1gb GPU

Windows 7

 

The media server :

 

Dell Inspirion 530S

Core 2 CPU (2.4ghz I think, it's currently switched off )

2gb Ram

500gb HDD

Windows 7

 

File Server :

 

Dell Full Size Tower

 

P4 Xeon 2.4ghz

4 gb ram

2tb Samsung

2tb Samsung

Win2K3

Pc to be picked up :

 

Abit P35 motherboard

Core Duo 2 E2140 CPU

2 gb ram

Windows Vista

 

So I've got a real hodgepodge of hard drives lying around too, several 2tb hard drives, several 1 tb hard drives, and a couple smaller. Files are ALL over the place, I need to consolidate, delete and organise about 5-6 years of downloads, personal files, OS installs across almost every drive. I'm hoping to eradicate/consolidate at least 1 tb of data by the time I'm done, maybe more. 3tb Toshiba hard drives are also on the radar, so I can sell off these ones and move to less hard drives.

 

I couldn't resist a couple 300gb Velociraptors either, I can use them in the gaming PC as my application drives in Raid 0 for a bit more speed. That way I can move my Steam install off the SSD and make some room there too.

 

I want to run 2 file servers (1 Windows (production), 1 Linux (testing ) with the purpose of serving files mainly for media sharing. I might do some backups of personal photos/docs to both servers as well as on the main pc and an offsite dropbox+facebook so I don't lose any photos (already had one catastrophic loss on that front, hence my hesitation to put my faith in systems I'm still learning). Transcoding/encoding will be on the cards too, and eventually tv capture to try and replace some austar PVR's I have to put up with.

 

Eventually I want to consolidate and move to much newer, smaller, power friendly hardware too. Trying to sell off this stuff can be difficult though if you don't use Ebay, once I have a solid plan moving forward I'll know what to keep and what to sell/junk.

 

A big issue really comes down to computer cases and power supplies in this situation. The Inspirion is arguably the fastest, but it's also noisy and the case is shit. I might relocated that hardware to my new case I'll build, and then I can use more HD's and minimise some noise with larger fans (I literally have a box full of them ). 7v fan mods are on the cards too, got plenty of molexs I picked up from Jaycar so I can rewire the fans completely if I need to.

 

Things to do :

 

1. Formulate a solid plan

2. GET IT DONE

3. ...........

4. Profit.

 

Big couple days coming up, I'm hoping to post back Friday night/Saturday morning and report that I've at least got the plan together and started to get my drives in order.

Share this post


Link to post
Share on other sites

Things to do :

 

1. Formulate a solid plan

2. GET IT DONE

3. ...........

4. Profit.

I like this bit the most... sounds like my plans most of the time. That or 1. Make it up as you go along. 2. Correct any mistakes.

 

One of the attractions to ZFS for some is the ability to pool a mismatch of various drives, either into one big pool (with no redundancy) or into smaller mirrors or raidz (like raid5) configurations. When 2 or more drives are paired, total usable space is equivalent to the capacity of the smallest drive. Given your variety of drives this may be of some benefit, although you could end up losing some capacity. If you're planning on getting rid of them it may not be the best option.

 

Will be keeping an eye on your progress :)

 

EDIT: Today only.. N40L with 250GB HDD is going for $199. Tempting to grab another.

http://www.ozbargain.com.au/node/84092

Edit2: Couldn't resist.. bought another one :) Too good a price, have a spare hdd and RAM i removed from my other one. Don't even know what I will use it for..

Edited by p0is0n

Share this post


Link to post
Share on other sites

We've spent WAY too much money this week, the sad thing is I had of held back a bit I MIGHT have been able to grab that N40L...Fail.

 

I'd love to do ZFS right now, but that stupid Raid controller is standing in my way of attaching enough drives in IT mode.

Share this post


Link to post
Share on other sites

I have been thinking of re-configuring my zpool into 2 raidz vdev's of 3 disks each, so I decide to re-run my initial benchmarks so I will have something to compare it to if I do.

Was surprised at the increased write performance over the last tests, read speed seems a little lower then before, but I ran it a few times and it's at least consistent. I think overall, it's an improvement.

 

Write:

dd if=/dev/zero of=/storage/downloads/ddtest bs=1024000 count=25000
25000+0 records in
25000+0 records out
25600000000 bytes (26 GB) copied, 153.629 s, 167 MB/s

dd if=/dev/zero of=/storage/downloads/ddtest bs=1024000 count=100000
100000+0 records in
100000+0 records out
102400000000 bytes (102 GB) copied, 607.315 s, 169 MB/s

Read:

dd if=/storage/downloads/ddtest of=/dev/null bs=1024000
25000+0 records in
25000+0 records out
25600000000 bytes (26 GB) copied, 97.8555 s, 262 MB/s

dd if=/storage/downloads/ddtest of=/dev/null bs=1024000
100000+0 records in
100000+0 records out
102400000000 bytes (102 GB) copied, 446.267 s, 229 MB/s

Not sure what to attribute this to, might try and run some different testing, see if I can produce similar results, any suggestions?

Share this post


Link to post
Share on other sites

Different spots on the hard drive platter ?

 

I don't know much about the workings of ZFS, so I really don't know.

 

Are you going with 2 pools for a performance increase, organisation, or easier backups ?

Share this post


Link to post
Share on other sites

Probably right about different spots on the disk now, it's a lot more full. ZFS is apparently very careful about how it lays out data which may account for the improved performance. I've started filling it quite a bit.

 

I had a few reasons for wanting to switch to 2 smaller vdevs, it should theoretically improve performance a little, but I'm not 100% on this.. it will be easier to increase capacity if I run out, in that I will only have to replace 3 drives instead of 5 to grow. I'm not sure if it is worth the effort though... The other choice which probably offers higher MTTDL would be 6 disk raidz2, same number of data and parity disks, but safer data, slightly lower performance perhaps. Given it's only connected by gigabit anything above 100MB/s should be sufficient I suppose.

 

I also just like to tinker and try new things. Will probably just end up leaving it for now.. if it isn't broke and all.. :)

Share this post


Link to post
Share on other sites

Have you tried network performance testing to see what speeds your network is getting ?

 

Since everything I have is frankenstein, I also popped a router I had into service for the bedroom and all the devices so I wasn't running 4 cat5 cables across the hall. After some head scratching over low network speeds I realised that the the router is only 100Mbs. Dammit ! So another thing to add to my list of needed hardware, since only the Xbox 360 is 100mbs limited.

Share this post


Link to post
Share on other sites

Have you tried network performance testing to see what speeds your network is getting ?

I only have a simple $30 gigabit switch, but just briefly copying a few dvd image (1.5GB to 3.0GB) files to and from the server I am consistently getting transfer speeds of around 85MB/s to 90MB/s which is about full speed for gigabit or close to it.

I then copied a bunch of smaller files of <1M to ~5MB totalling about 2GB and was getting speeds between 30MB/s and 50MB/s so the smaller files are a bit slower, but it's to be expected, one long file is much easier to handle.

 

Not sure if there are any utilities I can use to measure this more accurately but I'm happy with the result. In my use over the last 2 months I haven't encountered any real issues with performance, it's been working very well.

Share this post


Link to post
Share on other sites

So as you said, anything over 100mb/s is good for disk transfer speeds. Unless you are moving files around frequently that is.

Share this post


Link to post
Share on other sites

A mate of mine has a simple 4 disk RAIDed NAS using the same WD drives used in the OP.

10 days after purchase, he discovered 1 drive wasn't working, then discovered a burn mark in the case and then discovered a pretty little brown mark on the drive

 

Posted Image

 

The store swapped it out no problems.

I am not saying the model of drive is bad, this could happen to any drive, any component at all really. Just unlucky. Lucky it was RAIDed so nothing lost.

 

When he installed the replacement drive, he went to work while it was doing the process of rebuilding the array and when he got home it said it was incomplete. It said all 4 drives were offline. His 2 year old daughter had ejected all 4 hard disks while it was rebuilding. haha.

  • Like 1

Share this post


Link to post
Share on other sites

A mate of mine has a simple 4 disk RAIDed NAS using the same WD drives used in the OP.

10 days after purchase, he discovered 1 drive wasn't working, then discovered a burn mark in the case and then discovered a pretty little brown mark on the drive

 

The store swapped it out no problems.

I am not saying the model of drive is bad, this could happen to any drive, any component at all really. Just unlucky. Lucky it was RAIDed so nothing lost.

 

When he installed the replacement drive, he went to work while it was doing the process of rebuilding the array and when he got home it said it was incomplete. It said all 4 drives were offline. His 2 year old daughter had ejected all 4 hard disks while it was rebuilding. haha.

Ouch.. hope it completed the rebuild when he put them back in.. :( I'm glad the N40L has a lock on the drive bays not that there are ever kids in my house.

Share this post


Link to post
Share on other sites

Poor bastard.

 

I've been fruitlessly doing battle with a Seagate 7200.2 Barracuda. The infamouse BSY bug. Fuck. My. Life.

 

I've had enough. I'm listing it and it's brother drive (which I tried to do a PCB swap, and I've since lost the screws).

 

I'm off to Trademart now, they've pissed me off no end.

Share this post


Link to post
Share on other sites

so can non hp file servers pop in?

just purchased a celeron g550 samsung 830 64g wd gree 3tb stuck them in with some spare parts p67-ud4 2x2g 1600mhz 8-8-8 1.35v in a rusty old p2 case

have two 2tb samsungs that i will probably set in raid1 for more important data

and currently my seasonic x1000w which is about to go back for rma due to it buzzing not starting most the time without flicking its power switch like a mad man while my main rig is running on a corsair tx750 v2 which is a rma replacement for a v1

using gtx480 to set it up but will pull it when done

with the gtx480 it idles at 75w

without any gpu\igp it idles ~45w before i do any tweaking but thats only with the ssd

without the 1kw psu power use id guess the pc is using ~30w be nice if they made good quality 150w psu with 80+ gold or platinum but since they dont maybe i can settle for a 300w gold

 

i was hoping to give linux another go for the first time in ~10 years but i want to run a automation program made for xp that wont work on our vista pc due to it blocking the com port somehow and im not sure if it will work reliably if at all but i guess its worth a go

any suggestions for what type of linux i should try? thanks

Share this post


Link to post
Share on other sites

Give ubuntu a go Dasa. Anything file server and Nas related can be posted in here.

 

 

I'm still wanting to do a write up of my 22tb NAS but i'm not finding myself with that kind of time right now.

Share this post


Link to post
Share on other sites

decided to order fsp aurum 400w $88 deliverd from skycomp as it looks to have very good low load efficency and seasonic 300w gold tfx are not avalible in au

http://translate.google.com/translate?hl=e...hreadid%3D31516

 

edit took quick snap in its current stage of development which is just testing really

Posted Image

Edited by Dasa

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×