Today mdadm send me a mail to warn that one of my hard drive (/dev/hdd1) was ejected from my RAID-5 array. After some manipulations (no writes, just reads on the file system to get informations) and reboots, I ended up with a file system in a strange state: the folder structure was totally messed up and lots of files disappeared.
Assuming that this situation was about an inconsistent file index, I decided to reset the superblocks of the remaining physical disks:
mdadm --zero-superblock /dev/hdc1 mdadm --zero-superblock /dev/hdb1
I don’t know why I decided to do so, but it was the stupidest idea of the week. After such a violent treatment, my array refused to start:
[root@localhost ~]$ mdadm --assemble /dev/md0 --auto --scan --update=summaries --verbose mdadm: looking for devices for /dev/md0 mdadm: no RAID superblock on /dev/hdc1 mdadm: /dev/hdc1 has wrong raid level. mdadm: no RAID superblock on /dev/hdb1 mdadm: /dev/hdb1 has wrong raid level. mdadm: no devices found for /dev/md0
At this moment I was sure that all my data assets were lost. I was desperate. My only alternative was to ask Google. So I did.
I spend several minutes browsing the web without hope. I finally found someone in the same situation as mine (sorry, in french) on debian-user-french mailing list.
The solution was to recreate the RAID array. This sound counter-intuitive: if we recreate a raid array over an existing one, it will be erased ! Right ? Wrong ! As it is said on debian-user-french, mdadm is smart enough to “see” that HDD of the new array were elements of a previous one. Knowing that, mdadm will try to do its best (i.e. if parameters match the previous array configuration) and rebuild the new array upon the previous one in a non-destructive way, by keeping HDD content.
So, here is how I finally recovered my RAID array:
[root@localhost ~]$ mdadm --create /dev/md0 --verbose --level=5 --raid-devices=3 /dev/hdc1 missing /dev/hdb1 mdadm: layout defaults to left-symmetric mdadm: chunk size defaults to 64K mdadm: size set to 312568576K mdadm: array /dev/md0 started.
Of course this doesn’t solve my initial problem about the /dev/md0 file system: it is still in an altered state. Maybe it’s too late to recover data. But at least I reverted all my today’s mistakes, and the situation will not deteriorate until I power up my RAID ! 
Thank god I wasn’t the only one who did that. Thanks so much for putting this out there – I’d rather not think of how much time you saved with this!
There’s nothing quite as bad as having to try to restore original research data from a hundred different places! Thanks again!
Tim
Thanks!!!
I fixed RAID0 after upgrading kernel..
So has anyone gotten this to work on a raid 5 and seen the data afterwords? I have 5 drives that would love to back up an running again
Only one of my superblocks is corrupted, (as might be the drive). Is there a way to get four drives out of a a five drive array up and running again when mdadm has forgotten they exist?
Thanks very much.
I didn’t have your problems, but your tutorial helped me getting up a raid which superblock was not found.
Thanks for sharing this tip!
You may have saved me hours of recovering from backup!
Thanks so much for this! You just save me 750 GB of data!
It’s hard to believe that this simple fact (that the create option does not overwrite data) has been left so poorly documented. In fact, as good as mdadm is, it is very poorly documented.
It seems every Fedora upgrade I go through it’s a battle to bring in my old existing RAID 1 arrays. With Core 8 it would have been impossible if not for this write up.
i have a different situation if any one can help,
i have this NAS it is the intel ss4000-e. when i changed the authentication from local to active directory, a message showed up saying that all data will be destroyed,i clicked ok by mistake. i have very important data on this array and need to get it back. can anyone help please?
Thank you
))
With your french link I just rebuild my RAID1 that *lost* superblocks.
I did
and I have all my data now.
Note:
I do not understand why
--zero-superblockon/dev/sda, but it worked.(It’s not a productive server, just a fresh install, so I dared to do so)
can this
recover RAID level zero SUPER-BLOCK after new installation of my linux
NOTE: I was use ubuntu and my new installation was Fedora9
-l0is correct for raid level 0.But no idea if
will work
man mdadm -l, --level= Set raid level. When used with --create, options are: linear, raid0, 0, stripe, raid1, 1, mirror, raid4, 4, raid5, 5, raid6, 6, raid10, 10, multipath, mp, faulty. Obviously some of these are synonymous. When used with --build, only linear, stripe, raid0, 0, raid1, multipath, mp, and faulty are valid. Not yet supported with --grow.Thanks so much. Short and and just to the point, excellent. Helped me a lot
.
Hello,
I am hoping this thread is still somewhat alive.
I have a four drive raid5 mdadm array. I recently tried to move the raid array to a new computer and somehow lost 3 out of the 4 superblocks in the process.
I am wondering if someone could answer a question for me.
If I run
mdadm --createon this array, but the physical order of the drives is different than it was (ie.sdbmay now be connected tosdc), will mdadm properly reconstruct this array?hi there,
i use a 4 drive raid 5 wich now is almost fucked up. i just zeroed my superblocks becouse of trying to recover a bad array. now i created a array with same parameters as used in old.
but now i get in
mdadm --detail /dev/md0i had no spare in previous array. is everything over now? i cannot mount
/dev/md0becausemount: wrong fs type, bad option, bad superblock on /dev/md0, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or soand fsck reports “unsupported features”?!?
Hi,
This also works if you have a two disk RAID1, one disk goes faulty, the other drops out of the array and you re-add the one that drops out of the array as a spare thinking that it is the faulty one…
Long story short – data is back
Dude, awesome!
Been looking for hours for help with this.
Had a RAID1 array where one of the disk started failing, and the other one was marked as spare, therefore not allowing the md-device to be created. It seems to be working now. Great undocumented feature of mdadm!
I recreated the array from just the working disk (with “missing”) and then added another drive.
Greets from Sweden
This is an interesting find… the mdadm superblocks exist (at least partially) to allow you to reconstruct the array without knowing all of the exact details about which components live where. If you have a set of RAID5 components where you have all drives except one AND all the superblock info is present on those components, you can just
and mdadm will have enough info to know where all the data lives. However, if you do NOT have the superblock info, mdadm has to be explicitly told where each component drive exists in the array.
If you’re recreating a RAID5 array, the key piece of information here is that if you do NOT have the superblocks, the order of the components in the NEW array has to be the same as the order of the components in the OLD array.
So if you originally did
but lost all the superblocks, as long as you can run the same create command with the same drives (or the “missing” keyword in place of one of them) then mdadm will create the same superblock/slot assignment and you’re good to go.
Only in this case can mdadm read the data that is on the components of the array. This implies that once you have a functioning RAID5 array, you should get the output of
and store that somewhere, because it will tell you which “slot” the various components go into.
I’m trying currently to rebuild two arrays that somehow got messed up. In one case, the array initially just showed one drive missing. Since
smartctlshowed no actual drive problems, I added it back into the array. Once it finished resynching,mdadmimmediately threw out a massive number of error messages. After that the array showed two drives missing.My other array was corrupted in the same incident that took out the one above. The difference is, it always showed two drives missing.
I tried getting the drives to re-assemble but they won’t, so I wrote a script to cycle through the permutations of drives, with and without one missing, to see if I could create a mountable volume.
When that failed, I tried it again with a
--zero-superblockbefore slotting in a drive. That also failed. Each time I try to mount, I get:mount: wrong fs type, bad option, bad superblock on /dev/md2, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or soI’m guessing there is some corruption in the file system. However I don’t want to try fixing it until I can get the drive order correct and there are a lot of choices. I don’t want to even think about what a
fsckmight do if the drive order was wrong.Can anyone suggest a method for determining the correct drive order in a RAID 5 array other than attempting to mount a file system?
the problem i had some posts above was my controler. i build up a new pc with my existing devices and used a SATA 1 Controller with 4×1.5 TB. the sad thing was, that my controller accepted the devices and mdadm could rebuild my array, but all data was smashed with CRC error. the TOC was okay and i could see all fileinformation, but 3/4 files had CRC errors. shame on me cuz i used a SATA 1 controller and shame on the controller to accept my devices. i lost all data!
Nevermind, it turns out that the issue was trying mount an ext3 file system as ext2. I’d always thought that you could do it, but apparently that is not true. When I reran my script with the mount going as ext3, I got a successful mount.
Thanks for the info. It was very helpful