Difference between revisions of "RAID"

From RHS Wiki
Jump to navigation Jump to search
 
(16 intermediate revisions by the same user not shown)
Line 24: Line 24:
 
sr0    1024M        rom  </nowiki>
 
sr0    1024M        rom  </nowiki>
  
−
TODO: Disks order
+
blkid
 +
<nowiki>/dev/sdh: UUID="45dcb1c7-5dcd-8d8e-7830-0cdc8da5755c" UUID_SUB="c52532c2-fa0b-e5f2-9a71-166b20d63084" LABEL="Morpheus:0" TYPE="linux_raid_member"
 +
/dev/sdj: UUID="45dcb1c7-5dcd-8d8e-7830-0cdc8da5755c" UUID_SUB="524b7cad-dd3d-cd84-2f4b-228f691fb660" LABEL="Morpheus:0" TYPE="linux_raid_member"
 +
/dev/sdk: UUID="45dcb1c7-5dcd-8d8e-7830-0cdc8da5755c" UUID_SUB="f7105152-6eb2-3f7c-28a2-61f0f8c966d8" LABEL="Morpheus:0" TYPE="linux_raid_member"
 +
/dev/sdl: UUID="45dcb1c7-5dcd-8d8e-7830-0cdc8da5755c" UUID_SUB="a7dad691-d287-f1e2-17e2-76cbdd543b27" LABEL="Morpheus:0" TYPE="linux_raid_member"</nowiki>
  
 +
mdadm -D /dev/md0
 +
 +
<nowiki>/dev/md0:
 +
          Version : 1.2
 +
    Creation Time : Mon Sep 10 20:22:45 2018
 +
        Raid Level : raid5
 +
    Used Dev Size : 18446744073709551615
 +
      Raid Devices : 4
 +
    Total Devices : 4
 +
      Persistence : Superblock is persistent
 +
 +
      Update Time : Fri Sep 21 03:16:52 2018
 +
            State : active, FAILED, Not Started
 +
    Active Devices : 3
 +
  Working Devices : 4
 +
    Failed Devices : 0
 +
    Spare Devices : 1
 +
 +
            Layout : left-symmetric
 +
        Chunk Size : 512K
 +
 +
Consistency Policy : unknown
 +
 +
              Name : Morpheus:0  (local to host Morpheus)
 +
              UUID : 45dcb1c7:5dcd8d8e:78300cdc:8da5755c
 +
            Events : 11291
 +
 +
    Number  Major  Minor  RaidDevice State
 +
      -      0        0        0      removed
 +
      -      0        0        1      removed
 +
      -      0        0        2      removed
 +
      -      0        0        3      removed
 +
 +
      -      8      176        0      sync  /dev/sdl
 +
      -      8      144        3      spare rebuilding  /dev/sdj
 +
      -      8      112        1      sync  /dev/sdh
 +
      -      8      160        2      sync  /dev/sdk
 +
</nowiki>
 
=== Create RAID 5 ===
 
=== Create RAID 5 ===
 
  sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sdi /dev/sdj /dev/sdk /dev/sdl
 
  sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sdi /dev/sdj /dev/sdk /dev/sdl
Line 37: Line 79:
 
       bitmap: 0/30 pages [0KB], 65536KB chunk
 
       bitmap: 0/30 pages [0KB], 65536KB chunk
  
 +
unused devices: <none>
 +
</nowiki>
 +
 +
<nowiki>rafa@Morpheus:~$ cat /proc/mdstat
 +
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
 +
md0 : inactive sdk[2] sdj[4] sdh[1] sdl[0]
 +
      15627545950 blocks super 1.2
 +
     
 
unused devices: <none>
 
unused devices: <none>
 
</nowiki>
 
</nowiki>
Line 45: Line 95:
 
Enable the array at boot time
 
Enable the array at boot time
 
  sudo update-initramfs -u
 
  sudo update-initramfs -u
−
 
+
== Create Filesystem ==
 
=== Using LVM ===
 
=== Using LVM ===
−
  pvcreate /dev/md0
+
  sudo pvcreate /dev/md0
−
  vgcreate lvm_group_name /dev/md0
+
  sudo vgcreate lvm_group_name /dev/md0
−
  lvcreate -L30G -nlogical_volume_group_name lvm_group_name
+
  sudo lvcreate -L30G -nlogical_volume_group_name lvm_group_name
−
  mkfs.ext4 /dev/lvm_group_name/logical_volume_group_name
+
  sudo mkfs.ext4 /dev/lvm_group_name/logical_volume_group_name
−
  echo '/dev/lvm_group_name/logical_volume_group_name /mnt/mountdir ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab
+
  sudo echo '/dev/lvm_group_name/logical_volume_group_name /mnt/mountdir ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab
  
 
=== Using EXT4 ===
 
=== Using EXT4 ===
Line 66: Line 116:
  
  
−
=== LVM and RAID Cheatsheet ===
+
== LVM and RAID Cheatsheet ==
 
https://gist.github.com/plepe/52ecc9f18efb32c68d18
 
https://gist.github.com/plepe/52ecc9f18efb32c68d18
 +
== Add HOT SPARE drive ==
 +
mdadm --add /dev/md0 /dev/sdh1
 +
hdparm -S 180 /dev/[sh]d[a-z]  # makes disks to stop spining after 15 mins of inactivity
 +
Add this command to the end of the file /etc/rc.d/rc.local to ensure that it is executed every time the system is booted:
 +
 +
<source lang=bash>#!/bin/sh
 +
#
 +
# This script will be executed *after* all the other init scripts.
 +
# You can put your own initialization stuff in here if you don't
 +
# want to do the full Sys V style init stuff.
 +
 +
touch /var/lock/subsys/local
 +
hdparm -S 180 /dev/[sh]d[a-z]</source>
 +
 +
== Replace Failed Drive ==
 +
* Identify the failed drive with mdadm -D /dev/md0
 +
if=/dev/disk/by-id/scsi-drive-that-is-dying of=/dev/null
 +
* Remove the failed drive
 +
mdadm /dev/md0 --remove /dev/sda
 +
* Add it again
 +
mdadm --manage /dev/mdN -a /dev/sdd1
 +
If a RAID is operated with a spare disk, it will jump in for the disk set to faulty. A rebuild is performed automatically. The disk set to faulty appears in the output of mdadm -D /dev/mdN as faulty spare.
 +
 +
To put it back into the array as a spare disk, it must first be removed using
 +
mdadm --manage /dev/mdN -r /dev/sdX1
 +
and then added again
 +
mdadm --manage /dev/mdN -a /dev/sdd1
 +
 +
root@morpheus:~# ls /dev/mapper/
 +
    control  lvm-music  lvm-transmission  lvm-video
 +
root@morpheus:~# dmsetup remove lvm-music
 +
root@morpheus:~# dmsetup remove lvm-transmission
 +
root@morpheus:~# dmsetup remove lvm-video
 +
root@morpheus:~# mdadm --stop /dev/md0
 +
    mdadm: stopped /dev/md0
 +
* Copy the partition table to the new disk
 +
https://www.howtoforge.com/tutorial/linux-raid-replace-failed-harddisk/
 +
 +
== Force manual resync ==
 +
sudo mdadm --stop /dev/md0
 +
sudo mdadm --assemble --run --force --update=resync /dev/md0 /dev/sdl /dev/sdh /dev/sdj /dev/sdk
  
 
== Check RAID status ==
 
== Check RAID status ==
 
  mdadm -D /dev/md0
 
  mdadm -D /dev/md0
 +
 +
== Simulate disk failure ==
 +
sudo mdadm /dev/md0 --fail /dev/sdXX
 +
sudo mdadm /dev/md0 --remove /dev/sdXX
 +
sudo mdadm /dev/md0 --add /dev/sdXX
 +
== Recover ARRAY ==
 +
sudo mdadm --assemble --scan
 +
 +
== Send email alerts ==
 +
Edit /etc/mdadm.conf file:
 +
MAILADDR dustymabe@gmail.com
 +
 
== References ==  
 
== References ==  
 
https://www.digitalocean.com/community/tutorials/how-to-create-raid-arrays-with-mdadm-on-ubuntu-16-04#top<br />
 
https://www.digitalocean.com/community/tutorials/how-to-create-raid-arrays-with-mdadm-on-ubuntu-16-04#top<br />

Latest revision as of 19:32, 7 February 2019

Create RAID 5 Ubuntu[edit]

Disk Information[edit]

lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT 

NAME     SIZE FSTYPE TYPE MOUNTPOINT
sda    931.5G        disk 
└─sda1 931.5G ntfs   part /var/lib/transmission-daemon/downloads/HM_30
sdb    931.5G        disk 
└─sdb1 931.5G ntfs   part 
sdc    931.5G        disk 
└─sdc1 931.5G ntfs   part /var/lib/transmission-daemon/downloads/HM_40
sdd    931.5G        disk 
├─sdd1   100M ntfs   part 
├─sdd2   293G ntfs   part 
├─sdd3    14G ntfs   part 
├─sdd4     1K        part 
├─sdd5    28G ext4   part /
├─sdd6  93.1G ext4   part /home
├─sdd7  11.2G swap   part [SWAP]
└─sdd8   1.9G ext4   part /var/log
sdi      3.7T        disk 
sdj      3.7T        disk 
sdk      3.7T        disk 
sdl      3.7T        disk 
sr0     1024M        rom

blkid 

/dev/sdh: UUID="45dcb1c7-5dcd-8d8e-7830-0cdc8da5755c" UUID_SUB="c52532c2-fa0b-e5f2-9a71-166b20d63084" LABEL="Morpheus:0" TYPE="linux_raid_member"
/dev/sdj: UUID="45dcb1c7-5dcd-8d8e-7830-0cdc8da5755c" UUID_SUB="524b7cad-dd3d-cd84-2f4b-228f691fb660" LABEL="Morpheus:0" TYPE="linux_raid_member"
/dev/sdk: UUID="45dcb1c7-5dcd-8d8e-7830-0cdc8da5755c" UUID_SUB="f7105152-6eb2-3f7c-28a2-61f0f8c966d8" LABEL="Morpheus:0" TYPE="linux_raid_member"
/dev/sdl: UUID="45dcb1c7-5dcd-8d8e-7830-0cdc8da5755c" UUID_SUB="a7dad691-d287-f1e2-17e2-76cbdd543b27" LABEL="Morpheus:0" TYPE="linux_raid_member"

mdadm -D /dev/md0

/dev/md0:
           Version : 1.2
     Creation Time : Mon Sep 10 20:22:45 2018
        Raid Level : raid5
     Used Dev Size : 18446744073709551615
      Raid Devices : 4
     Total Devices : 4
       Persistence : Superblock is persistent

       Update Time : Fri Sep 21 03:16:52 2018
             State : active, FAILED, Not Started 
    Active Devices : 3
   Working Devices : 4
    Failed Devices : 0
     Spare Devices : 1

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : unknown

              Name : Morpheus:0  (local to host Morpheus)
              UUID : 45dcb1c7:5dcd8d8e:78300cdc:8da5755c
            Events : 11291

    Number   Major   Minor   RaidDevice State
       -       0        0        0      removed
       -       0        0        1      removed
       -       0        0        2      removed
       -       0        0        3      removed

       -       8      176        0      sync   /dev/sdl
       -       8      144        3      spare rebuilding   /dev/sdj
       -       8      112        1      sync   /dev/sdh
       -       8      160        2      sync   /dev/sdk

Create RAID 5[edit]

sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sdi /dev/sdj /dev/sdk /dev/sdl

Monitor the array progress creation with 

cat /proc/mdstat

Output: 

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sdl[4] sdk[2] sdj[1] sdi[0]
      11720658432 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_]
      [>....................]  recovery =  0.0% (2091700/3906886144) finish=7753.1min speed=8393K/sec
      bitmap: 0/30 pages [0KB], 65536KB chunk

unused devices: <none>


rafa@Morpheus:~$ cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] 
md0 : inactive sdk[2] sdj[4] sdh[1] sdl[0]
      15627545950 blocks super 1.2
       
unused devices: <none>

Save the array layout[edit]

Wait until the array creation is finished and run: 

sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

Enable the array at boot time 

sudo update-initramfs -u

Create Filesystem[edit]

Using LVM[edit]

sudo pvcreate /dev/md0
sudo vgcreate lvm_group_name /dev/md0
sudo lvcreate -L30G -nlogical_volume_group_name lvm_group_name
sudo mkfs.ext4 /dev/lvm_group_name/logical_volume_group_name
sudo echo '/dev/lvm_group_name/logical_volume_group_name /mnt/mountdir ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab

Using EXT4[edit]

Create the filesystem[edit]
sudo mkfs.ext4 -F /dev/md0
Mount the filesystem[edit]
sudo mount /dev/md0 /mnt/md0

Add mount option to fstab 

echo '/dev/md0 /mnt/md0 ext4 defaults,nofail,discard 0 0' | sudo tee -a /etc/fstab

TODO: Add a disk to RAID5 with LVM[edit]

https://ubuntuforums.org/showthread.php?t=713936


LVM and RAID Cheatsheet[edit]

https://gist.github.com/plepe/52ecc9f18efb32c68d18

Add HOT SPARE drive[edit]

mdadm --add /dev/md0 /dev/sdh1
hdparm -S 180 /dev/[sh]d[a-z]  # makes disks to stop spining after 15 mins of inactivity

Add this command to the end of the file /etc/rc.d/rc.local to ensure that it is executed every time the system is booted: 

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local
hdparm -S 180 /dev/[sh]d[a-z]

Replace Failed Drive[edit]

  • Identify the failed drive with mdadm -D /dev/md0
if=/dev/disk/by-id/scsi-drive-that-is-dying of=/dev/null
  • Remove the failed drive
mdadm /dev/md0 --remove /dev/sda
  • Add it again
mdadm --manage /dev/mdN -a /dev/sdd1

If a RAID is operated with a spare disk, it will jump in for the disk set to faulty. A rebuild is performed automatically. The disk set to faulty appears in the output of mdadm -D /dev/mdN as faulty spare.

To put it back into the array as a spare disk, it must first be removed using 

mdadm --manage /dev/mdN -r /dev/sdX1

and then added again 

mdadm --manage /dev/mdN -a /dev/sdd1

root@morpheus:~# ls /dev/mapper/
    control  lvm-music  lvm-transmission  lvm-video
root@morpheus:~# dmsetup remove lvm-music
root@morpheus:~# dmsetup remove lvm-transmission
root@morpheus:~# dmsetup remove lvm-video
root@morpheus:~# mdadm --stop /dev/md0
    mdadm: stopped /dev/md0
  • Copy the partition table to the new disk

https://www.howtoforge.com/tutorial/linux-raid-replace-failed-harddisk/

Force manual resync[edit]

sudo mdadm --stop /dev/md0
sudo mdadm --assemble --run --force --update=resync /dev/md0 /dev/sdl /dev/sdh /dev/sdj /dev/sdk

Check RAID status[edit]

mdadm -D /dev/md0

Simulate disk failure[edit]

sudo mdadm /dev/md0 --fail /dev/sdXX
sudo mdadm /dev/md0 --remove /dev/sdXX
sudo mdadm /dev/md0 --add /dev/sdXX

Recover ARRAY[edit]

sudo mdadm --assemble --scan

Send email alerts[edit]

Edit /etc/mdadm.conf file: 

MAILADDR dustymabe@gmail.com

References[edit]

https://www.digitalocean.com/community/tutorials/how-to-create-raid-arrays-with-mdadm-on-ubuntu-16-04#top
https://unix.stackexchange.com/questions/88654/what-does-mdadms-spare-number-mean
https://wiki.bandaancha.st/RAID_y_LVM_en_Linux

https://zackreed.me/software-raid-5-in-debian-with-mdadm/
https://zackreed.me/adding-an-extra-disk-to-an-mdadm-array/

Retrieved from "https://wiki.herrerosolis.com/index.php?title=RAID&oldid=1724"