Another successful Linux MD RAID recovery

So there I was, minding my own business, not bothering anyone, working over the weekend again on my Linux desktop, trying to get some stuff ready for Monday.

My entire "/" partition is on a MD RAID10 array with 6 disks -- 4 disks inside the computer chassis, two in an external Sans Digital enclosure connected via eSATA. Boot partition is on a USB flash drive, since Linux can't boot from a MD RAID10 array at this time. I used to create a small partition on each disk and run MD RAID1 on all of them since /boot can be on a RAID1 and using the larger partitions for RAID10, but then decided it was a waste of disk space and switched to putting /boot onto a USB flash drive and using the entire capacity of the disks for the RAID10.

So, anyway, there I was, working away, when my 2 year old comes in with something important to show me. I ask her to hang out for a sec while I get to a point where I can stop, then step away from the computer for a moment. When I get back, the monitor is dark, moved the mouse expecting the password prompt to unlock the screensaver, the cursor moved, then froze. Damn, not good, I had some unsaved work. Thought maybe X crashed, but couldn't shell in from another machine, so that's not it. Then noticed the eSATA light on the array was dark and the cable was unplugged. My daughter must have pulled out the eSATA cable while hanging out behind the office desk. Ok, not the end of the world -- RAID10 with 6 disks can easily handle a loss of two disks, but this didn't explain the weird behavior. Not knowing what else to do, I accepted the loss of a few hours of work and rebooted.

BIOS took forever to POST, then complained that two drives are missing. OS won't boot. Strange. Should still be able to boot. Poked around the BIOS and it seems the two disks that were missing were two of the four inside the chassis, not the two on the external array that got unplugged. Very strange. Unplugged one disk at a time and rebooted until I identified which disks are at fault. Booted without them, POST was quick, but couldn't boot into the OS. Crap. Booted to the Gentoo SystemRescueCd. After unplugging/replugging disks and running "rescan-scsi-bus.sh" and "lsscsi", identified that one of the internal disks would cause the system to hang for a while when attempting to access with "fdisk -l /dev/sdd" or "smartctl -i /dev/sdd". Ok, so looks like it's gone bad. Interesting timing, but I still should have 5 good disks, why won't it boot?

After examining the RAID superblock on each disk with "mdadm -E", the explanation began to emerge. When the eSATA cable got unplugged, mdadm marked the two disks in it as "faulty removed" and that's the information that was contained in the superblock of the first 3 disks (/dev/sd[abc]1). According to the superblock of the two disks in the enclosure, they thought everything was fine, which makes sense as that was the last thing they new knew when they got so rudely disconnected. So then, because, when it rains it pours, in addition to the two disks that went missing due to the disconnected enclosure, one of the four remaining disks died and because I got unlucky (of course) the wrong disk failed so the array was left with three working disks and choked.

To test this, I attempted to start the array with:

mdadm --assemble -v --scan --uuid=a60b30f2:741aa1ab:e368bf24:bd0fce41

It would report (among other things, since I'm running with the "verbose" flag) that:

mdadm: /dev/md/1 assembled from 3 drives - not enough to start the array.

Ok, so this is why it couldn't boot. AFAIK, a RAID10 array could survive loosing up to half of its disks so long as the right disks fail. Guess in my case, wrong disks were missing. 😉 Anyway, was able to force the array to start with:

mdadm --assemble -v --scan --force --run --uuid=a60b30f2:741aa1ab:e368bf24:bd0fce41

Got:

mdadm: /dev/md/1 has been started with 5 drives (out of 6).

Sweet.

cat /proc/mdstat 
md1 : active raid10 sda1[0] sdg1[5] sdf1[4] sdc1[2] sdb1[1]
      1465151808 blocks 64K chunks 2 near-copies [6/5] [UUU_UU]
mdadm -D /dev/md1
/dev/md1:
        Version : 0.90
  Creation Time : Wed Dec  8 23:10:08 2010
     Raid Level : raid10
     Array Size : 1465151808 (1397.28 GiB 1500.32 GB)
  Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
   Raid Devices : 6
  Total Devices : 5
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Sun Jan 23 19:27:02 2011
          State : clean, degraded
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 64K

           UUID : a60b30f2:741aa1ab:e368bf24:bd0fce41
         Events : 0.48

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1
       3       0        0        3      removed
       4       8       81        4      active sync   /dev/sdf1
       5       8       97        5      active sync   /dev/sdg1

Replaced the failed disk inside the chassis and added to the array with:

mdadm --add /dev/md1 /dev/sdd1
mdadm -D /dev/md1
/dev/md1:
        Version : 0.90
  Creation Time : Wed Dec  8 23:10:08 2010
     Raid Level : raid10
     Array Size : 1465151808 (1397.28 GiB 1500.32 GB)
  Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Sun Jan 23 19:32:54 2011
          State : clean, degraded, recovering
 Active Devices : 5
Working Devices : 6
 Failed Devices : 0
  Spare Devices : 1

         Layout : near=2
     Chunk Size : 64K

 Rebuild Status : 0% complete

           UUID : a60b30f2:741aa1ab:e368bf24:bd0fce41
         Events : 0.51

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1
       6       8       49        3      spare rebuilding   /dev/sdd1
       4       8       81        4      active sync   /dev/sdf1
       5       8       97        5      active sync   /dev/sdg1

cat /proc/mdstat 
md1 : active raid10 sdd1[6] sda1[0] sdg1[5] sdf1[4] sdc1[2] sdb1[1]
      1465151808 blocks 64K chunks 2 near-copies [6/5] [UUU_UU]
      [>....................]  recovery =  0.7% (3813760/488383936) finish=91.5min speed=88198K/sec

Now I wait. Might as well write a blog post about it.

So, aside from loosing about a day due to unsaved work and to the time it took to troubleshoot and fix, I feel good about not loosing the entire array, not having to rebuild the machine and restore from backup. I've used a number of RAID implementations over the years and Linux MD is my favorite to date due to great documentation, amazing flexibility, great resiliency and good speed (when using RAID10).

1 Comment

  • 1. icius replies at 7th November 2011, 12:15 pm :

    Thanks so much for sharing your experience. I had a 4 disk RAID 10 array fail today and couldn’t figure out how to get it started in degraded mode after one of the disks failed. Thought I had lost everything until I tried the command:

    mdadm –assemble -v –scan –force –run –uuid=a60b30f2:741aa1ab:e368bf24:bd0fce41

    Now I’m frantically backing up the contents of the array to another machine.

    You saved my bacon 🙂

Leave a comment

NOTE: Enclose quotes in <blockquote></blockquote>. Enclose code in <pre lang="LANG"></pre> (where LANG is one of these).