RAID arrays in (Puppy) Linux

How to do things, solutions, recipes, tutorials
Message
Author
tempestuous
Posts: 5464
Joined: Fri 10 Jun 2005, 05:12
Location: Australia

#31 Post by tempestuous »

toronado wrote:ROX seems to show empty directories in /mnt/ for partitions that don't exist. So, if I don't mount md0, there is a (empty) directory there anyway. I found many other "dummy" directories there for previously created partitions and what not that no longer exist.
Yes, it seems to be a quirk of the mdadm application that it creates additional "dummy" or "ghost" device nodes, typically of the format "/dev/md_d0" or similar. You can manually stop these dummy arrays as such -

Code: Select all

mdadm --stop /dev/whatever
but as long as these devices/arrays don't interfere with the correct device/array, there's no problem, and you can just ignore them.
toronado wrote:I just added the commands to assemble and mount the RAID array to my Samba Auto-Start script and it works great.
Sure, that startup script is fine, but generally the correct place to such additional commands is /etc/rc.d/rc.local
I have just updated the instructions in the third post.

tempestuous
Posts: 5464
Joined: Fri 10 Jun 2005, 05:12
Location: Australia

#32 Post by tempestuous »

And in the final wash up ... you have Puppy running with a functioning Linux software RAID-0 array.
Bravo.

toronado
Posts: 95
Joined: Wed 04 Sep 2013, 21:09

#33 Post by toronado »

tempestuous wrote:
toronado wrote:ROX seems to show empty directories in /mnt/ for partitions that don't exist. So, if I don't mount md0, there is a (empty) directory there anyway. I found many other "dummy" directories there for previously created partitions and what not that no longer exist.
Yes, it seems to be a quirk of the mdadm application that it creates additional "dummy" or "ghost" device nodes, typically of the format "/dev/md_d0" or similar. You can manually stop these dummy arrays as such -

Code: Select all

mdadm --stop /dev/whatever
but as long as these devices/arrays don't interfere with the correct device/array, there's no problem, and you can just ignore them.
This is probably getting off-topic for this thread, but I don't think this issue pertains to mdadm. (It might not have anything to do with ROX either.) For example, well before installing the mdadm pet I noticed empty directories in /mnt/ for partitions such as sda2 sdb2 sdc4 sdd2 etc. and none of these partitions actually existed (or at least they didn't exist at the time I was browsing /mnt/). And while some of these partitions I actually created in the past and subsequently deleted (such as sdc4), others (like sda2, sdb2, sdd2) I don't recall ever creating in the first place. It's an old computer, so maybe it's haunted. :-)
tempestuous wrote:
toronado wrote:I just added the commands to assemble and mount the RAID array to my Samba Auto-Start script and it works great.
Sure, that startup script is fine, but generally the correct place to such additional commands is /etc/rc.d/rc.local
I have just updated the instructions in the third post.
OK thanks.

carenrose
Posts: 36
Joined: Tue 11 Dec 2012, 03:22

#34 Post by carenrose »

Sorry, I know this thread is old, but I figured this is probably the best place to put this question.

I have succeeded in setting up my RAID (1, btw). I'm now trying - and failing - to install Grub.

From the menu entry gui program, when I get to the step, something like "which disk or whatever do you want to put it in on?" and it suggests "/dev/sda(1?)" I tried both /dev/md0 and /dev/sda (and/or /dev/sda1) and /dev/sdb (and same).
I am not currently on that computer so I don't remember if that step asked for the disk or the partition, but whichever it asked for I put, ok? :D
It said that /dev/md0 "is not a valid Linux" something or other, and it couldn't mount sda or sdb to do its stuff there - which I figured as much.

Anyways, I tried command line. The below is approximation of what it said: (Yes, I know I'm being overly descriptive.)

Code: Select all

# grub
Blah blah something about BIOS this will take a while. ...

grub > find /boot/grub/stage1

Error 15: File not found
Ok, so I copied the files from /usr/sbin/grub (or wherever they are) to md0's /boot/grub that I created ... so it should be present on *both physical drives* right? And grub won't find when I do the above yet again.

What am I missing? Am I just totally lost?

tempestuous
Posts: 5464
Joined: Fri 10 Jun 2005, 05:12
Location: Australia

#35 Post by tempestuous »

Ah, you're trying to boot from a RAID array, and that's complicated. Personally, I would avoid this, and install Puppy on a separate (non-RAID) drive - even a small USB flash drive. Then just use the RAID array for your user-data.

But if you're determined to persist, you will need to do some research and experimentation, and then rebuild Puppy's initrd.
Let me explain it in principle: the software RAID array can generally only be understood, and thus accessible, to a running operating system. It's difficult (but not impossible) to access files on the array at bootup.

What you need to do is include all necessary drivers, utilities, and configuration logic into Puppy's initial ramdisk.
In your case, that means rebuilding the initrd image to include the mdadm application, raid1 kernel module, and also modify the initrd startup scripts to assemble and mount the RAID array right at the start of the boot sequence.
I have no experience in this, so cannot help with the fine details.

Once this is all achieved, yes, you can put the grub configuration files onto the array ...
but as I understand it, the Master Boot Record must still be installed onto a single physical drive. I don't believe it's possible to share the MBR on an array - that would effectively be sharing two boot sectors. I can't imagine that any motherboard bios would be able to recognise this.

toronado
Posts: 95
Joined: Wed 04 Sep 2013, 21:09

#36 Post by toronado »

I've installed a new version of Puppy (PhatSlacko 5.5.02 for its easy Samba config) and I installed mdadm-3.2.5 from the PPM and rebooted but no md0 in /mnt.

I tried:

Code: Select all

# mdadm --assemble /dev/md0
mdadm: /dev/md0 not identified in config file.
So before I go messing things up further, what should I try next?

tempestuous
Posts: 5464
Joined: Fri 10 Jun 2005, 05:12
Location: Australia

#37 Post by tempestuous »

So as I understand it, you have installed a new version of Puppy, but you want to access the software RAID array that you previously created?
toronado wrote:

Code: Select all

mdadm: /dev/md0 not identified in config file.
Oops, it sounds like you didn't keep a copy of your configuration file (from your earlier installation) - /etc/mdadm.conf

toronado
Posts: 95
Joined: Wed 04 Sep 2013, 21:09

#38 Post by toronado »

tempestuous wrote:So as I understand it, you have installed a new version of Puppy, but you want to access the software RAID array that you previously created?
toronado wrote:

Code: Select all

mdadm: /dev/md0 not identified in config file.
Oops, it sounds like you didn't keep a copy of your configuration file (from your earlier installation) - /etc/mdadm.conf
I haven't erased the previous install (it's on a separate partition). Are you saying all I need to do is copy over the config file?

tempestuous
Posts: 5464
Joined: Fri 10 Jun 2005, 05:12
Location: Australia

#39 Post by tempestuous »

I think so, yes.

toronado
Posts: 95
Joined: Wed 04 Sep 2013, 21:09

#40 Post by toronado »

Thanks, it worked.

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

Linux Software RAID

#41 Post by rufwoof »

tempestuous wrote:3. LINUX SOFTWARE RAID

If your RAID setup is running only under Linux, and there's no need to dual boot into Windows, this is the better RAID solution than bios-RAID.
For motherboards with onboard RAID (Host RAID) as mentioned in the previous section, disable the RAID function in bios. Yes, this sounds a bit strange, but Linux will take over all RAID configuration.
There's a HOWTO here -
https://raid.wiki.kernel.org/index.php/ ... _operation
Here's a summary. Let's assume you want RAID1 (mirroring for redundancy)
using /dev/sdb1 and /dev/sdc1 (your drives must be partitioned with GParted, first)
RAID0 support is built directly into the Puppy kernel, but RAID1 support is via the external kernel module "raid1", so we first need to load this module -

Code: Select all

modprobe raid1
Install the mdadm dotpet package attached to this post. Then run this command to configure the RAID array

Code: Select all

mdadm --create --verbose /dev/md0 --level=raid1 --raid-devices=2 /dev/sdb1 /dev/sdc1
You should hear the drives working as the array is initialised. Once this is complete, the new RAID configuration must be saved, with this command -

Code: Select all

mdadm --detail --scan >> /etc/mdadm.conf
(It's worth keeping a copy of this configuration file on a USB dongle, for example, in case your system drive fails in the future, and you need to be able to access the RAID array on a different installation.)
Your new RAID array is /dev/md0. Go ahead and format it with ext3 (alternative filesystems may be considered)

Code: Select all

mkfs.ext3 -v -m .1 -b 4096 -E stride=32,stripe-width=64 /dev/md0
All those extra formatting options I just included are explained in the RAID wiki I mentioned earlier.
Now mount the RAID array -

Code: Select all

mkdir /mnt/md0
mount /dev/md0 /mnt/md0
Now you can browse to /mnt/md0 with ROX, and test it by copying some files to/from.

That's it, but we also need to restore this configuration at each bootup. My mdadm dotpet includes a special mdadm udev rule which, in theory, will auto-detect software-raid arrays at bootup, and restore their configuration ... unfortunately testing under Puppy 5.28 has shown that this udev function fails!!
So you need to add some extra commands to Puppy's startup scripts - open /etc/rc.d/rc.local in Geany, and add these 3 lines -

Code: Select all

modprobe raid1
mdadm --assemble /dev/md0
mount /dev/md0 /mnt/md0
Save.
Now each time you boot Puppy, your RAID array will be ready to use at /mnt/md0
Just to add from a new-user angle, the

Code: Select all

mdadm --create --verbose /dev/md0 --level=raid1 --raid-devices=2 /dev/sdb1 /dev/sdc1
stage took over a hour for me (pair of 45GB USB/HDD). The command when entered 'finished' instantly, but the disks whirred for ages. Found a command that shows the progress of that stage

cat /proc/mdstat

that provides % done and estimated time to completion details.

Another useful command is

mdadm -D /dev/md0

to see the status of the RAID mirror. The standard Linux thing appears to be that the monitor will email root when one of a pair of RAID1 fails, but a Puppy doesn't have standard linux mail I guess you wouldn't find out that one disk had perhaps failed until the second one also failed - which isn't what is desired. Periodic manual checks using the above might indicate that one disk had failed enabling you to make a backup of the /mnt/md0 content before replacing the failed disk.

I've setup a software Raid1 for the purpose of storing one-time-pad keys for encrypted documents http://www.murga-linux.com/puppy/viewto ... 999#881999 that I store on googledrive, as the loss of the key is paramount to loss of data. Having encrypted data in the cloud, keys stored on USB, laptop/pc using both to decrypt/work/reencrypt without storing copies unencrypted means that if the laptop is lost/stolen data is still safe (thief can't access personal data without the key). Assuming cloud data is safe/backed up, but unreadable (encrypted) by the host, and two pairs of USB's hold the keys, then all-round that's pretty secure/safe.

On another note, whilst reading around I found the attached youtube report where a guy plugged in a multiple number of usb's into a usb hub and set them up as stripped arrays. The speed of two stripped USB's was significantly quicker than one USB, but then the speed benefit tailed off pretty quickly thereafter for 3, 4 .... 8.
Attachments
i.png
(68.1 KiB) Downloaded 292 times
[size=75]( ͡° ͜ʖ ͡°) :wq[/size]
[url=http://murga-linux.com/puppy/viewtopic.php?p=1028256#1028256][size=75]Fatdog multi-session usb[/url][/size]
[size=75][url=https://hashbang.sh]echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh[/url][/size]

Post Reply