Linux mdadm Reliability

S

Skud

Gawd
Joined
Sep 4, 2002
Messages
587
Quick question..

I'm putting together a low-powered atom based "server" that will be in a hosted location not easily accessible. It won't be anything critical, but I'm going to have two 500GB SATA drives which I'd like to mirror. I have a great deal of experience with various hardware RAID controllers (Dell PERC, HP SmartArray, etc.) and with those, if a drive in a RAID 1 fails it's completely transparent to the OS. You can keep working while the drive is failed and you swap in a replacement.

Is this the same with Linux? If I have two drives (or I guess in Linux their partitions) mirrored and say the "primary" disk fails does it seamlessly fail over to the mirror? I think it should because it appears the disks are abstracted to the OS through the RAID device /dev/md0 (or whatever it may be).

Also, how does this work for booting? I see that I need to setup GRUB to boot the RAID volumes, but what if say a disk starts throwing errors and doesn't die completely? A hardware based controller will knock that disk offline and prevent it from being used. I could see GRUB trying to boot off that disk if it were the primary. If that happens, the machine would fail to boot..

Thoughts?

Thanks!!
Riley
 
Is this the same with Linux?
Click to expand...

Yes.

Also, how does this work for booting? I see that I need to setup GRUB to boot the RAID volumes, but what if say a disk starts throwing errors and doesn't die completely?
Click to expand...

I setup each drive to boot with grub independently. I use a separate /boot from the os. If the first drive does not work the bios automatically chooses the next drive.
 
Skud said:
Is this the same with Linux? If I have two drives (or I guess in Linux their partitions) mirrored and say the "primary" disk fails does it seamlessly fail over to the mirror? I think it should because it appears the disks are abstracted to the OS through the RAID device /dev/md0 (or whatever it may be).

Also, how does this work for booting? I see that I need to setup GRUB to boot the RAID volumes, but what if say a disk starts throwing errors and doesn't die completely? A hardware based controller will knock that disk offline and prevent it from being used. I could see GRUB trying to boot off that disk if it were the primary. If that happens, the machine would fail to boot..

Thoughts?

Thanks!!
Riley
Click to expand...

It's better in my opinion. When your server is remote, you can't just login to the PERC controller and check the health of your RAID? You only know when you're screwed as a LED starts blinking on the faulty drive. With 'mdadm', you can simply SSH into your server and add / remove / stop / monitor your RAID volume from with in the O.S.

I have a small home server running two 1 TB drives in a mirror with 'mdadm' and it has been amazing. I have even tested it with forcing a drive to fail (removed the power plug to the SATA adapter on drive 1) and the system kept on going.

As for Grub, I simply greated my partitions as follows in identical sizes:

sda1 = 4 GB (/boot)
sda2 = The rest of the drive as 'fd' (RAID)

sdb1 = 4 GB (Swap)
sdb2 = The test of the drive as 'fd' (RAID)

Then create your RAID with sda2 & sdb2 and mount /dev/md0 as /.
 
Great, thanks guys..

So, what happens with a semi-failed drive on a reboot? For example..

A drive fails or starts throwing errors and is marked failed by mdadm. Then, I reboot the system. What happens in the following scenarios?

1) The drive is *really* dead and doesn't respond. I assume I need to setup the second drive in the boot order and make sure GRUB is installed on it and can boot?

2) The drive isn't *really* (just about to die) dead and starts responding again on a reboot.. At which point does mdadm load up and I get my redundancy back?

Riley
 
When I had a live system up in RAID1 running for over 1 month. I decided to pull the power plug from one of the drives. I was notified immediately that I had a drive failure. Then 'mdadm' automatically removed the failed drive from the mirror. I then rebooted the server and it came back up perfect working on one SATA drive. I did see it notify me that the RAID status was 'degraded' and was unable to synchronize with a second drive. I then powered down and re-attached the power cable to the SATA drive. I simply then used the 'mdadm' command to add my /dev/sdb2 partition back into the /dev/md0 array. It synchronized the drive with the master and everything worked great. I love 'Software RAID' but it all depends on your experience. Many people hate it simply because they don't understand it and how to do the commands via CLI (Command Line Interface). To me hardware RAID (especially PERC) are single point of failure and I have seen many many RAID cards fail leaving me stuck in the mud.

The thing to understand is that when a drive throws errors or fails completely, mdadm just assumes its a dead drive and removes it from the RAID. If it starts working again or not is no longer a concern of mdadm because it's no longer it's problem after bring removed from the RAID. Once the drive is replaced or working again, mdadm expects you as the administrator to manually re-add it back into the mirror using a simple command. Then it will handle synchronization and everything else.
 
Here is my work PC running 'mdadm'. It's been rock solid running RAID 5:

Code: 
[root@tuna ~]# mdadm --detail /dev/md0
/dev/md0:
        Version : 0.90
  Creation Time : [B][COLOR="Yellow"]Wed Jan 30 05:30:58 2008[/COLOR][/B]
     Raid Level : raid5
     Array Size : 720587328 (687.21 GiB 737.88 GB)
  Used Dev Size : 240195776 (229.07 GiB 245.96 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Mon Feb 22 09:47:46 2010
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 316c7753:c0aa0c51:0cb5fda2:0e8c6e4e
         Events : 0.213675

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       18        1      active sync   /dev/sdb2
       2       8       34        2      active sync   /dev/sdc2
       3       8       50        3      active sync   /dev/sdd2
 
I have two mdadm RAID5 arrays running with 8 disks apiece in the same machine, no problems at all. Never tried RAID1 though.
 
I have over 75 hard drives involved in linux software raid using mdadm at work. I believe the current count is 12 arrays with most of them being raid5 or raid6. Some of these were added in 2004 and are still running 24/7/365. I have had to hot swap out a few drives and replace a few powersupplies but for the most part very little maintenance. I do use raid1 but only for the boot where in some cases it may be a 10 drive mirror. Yes you can raid1 with 10 disks and all disks will be a mirror of each other..
 
You must log in or register to reply here.
Back
Top