Been asked to setup a software raid of 12TB on a minimal CentOS 6.4 installation with 5 disks of 3TB each. Never played with raid nor lvm before so the challenge was great!
Creating of 3TB partitions on the physical disks
# parted /dev/sdX # (parted) mklabel gpt # (parted) unit TB # (parted) mkpart primary 0.00TB 3.00TB # (parted) print
Creating the raid5 array with all the prepared disks
# mdadm --create /dev/md0 --level=raid5 --raid-devices=5 /dev/sdX# /dev/sdX# /dev/sdX# /dev/sdX# /dev/sd#
Viewing the state of creation of the new array
# watch cat /proc/mdstat
Once the array is successfully created you have to store it into the config file
# echo "DEVICE partitions" > /etc/mdadm.conf # echo "MAILADDR root@localhost" >> /etc/mdadm.conf # mdadm --detail --scan >> /etc/mdadm.conf
so we can start creating lvm volumes on top of them
# pvcreate /dev/md0 # vgcreate vg_NAME /dev/md0 # lvcreate --name lv_NAME -l 100%FREE vg_NAME
We now created a physical volume, a volume group and a logical volume which can easily be resized and moved on top of the raid5 setup
To start using the volume we finally have to create a file system on it and check if everything went alright
# mkfs.ext4 /dev/vg_NAME/lv_NAME # fsck.ext4 -f /dev/vg_NAME/lv_NAME
After the succesfull file system chack I came to a working raid setup. Nevertheless I figured out that by rebooting the machine the raid array wasn't initializing as I expected. Instead of the md0 as configured the raid array was coming up as a read-only one named md127. I did some research and found a usefull topic on it on the redhat bugzilla forum.
There I found out that this can be reactivated manually by stopping the read-only instance and reassambling the array based on the /etc/mdadm.conf file:
# mdadm --stop /dev/md172 # mdadm --assemble --scan
Although that's not really the best solution because you have to do this by every reboot of your system. So I looked a bit further and found out you can regenerate your initramfs image using your mdadm.conf file using dracut:
# mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.old # dracut --mdadmconf --force /boot/initramfs-$(uname -r).img $(uname -r)
Once that images is build I successfully rebooted the system and the md0 came up without any problem.
The final step is to get the new logical volume mounted automatically at boot, therefore you have to add something in your /etc/fstab file:
# /dev/mapper/vg_NAME-lv_NAME /var/NAME ext4 defaults 1 1
Some useful commands
## Stop raid array # mdadm --stop /dev/md0 ## Start raid array # mdadm --assemble --scan