Ruan Bekker's Blog

From a Curious mind to Posts on Github

Create a RAID5 Array With Mdadm on Linux

setup-raid5-array-ubuntu-linux

In this tutorial we will setup a RAID5 array, which is striping across multiple drives with distributed paritiy, which is good for redundancy. We will be using Ubuntu for our Linux Distribution, but the technique applies to other Linux Distributions as well.

What are we trying to achieve

We will run a server with one root disk and 6 extra disks, where we will first create our raid5 array with three disks, then I will show you how to expand your raid5 array by adding three other disks.

Things fail all the time, and it’s not fun when hard drives breaks, therefore we want to do our best to prevent our applications from going down due to hardware failures. To achieve data redundancy, we want to use three hard drives, which we want to add into a raid configuration that will proviide us:

  • striping, which is the technique of segmenting logically sequential data, so that consecutive segments are stored on different physical storage devices.
  • distributed parity, where parity data are distributed between the physical disks, where there is only one parity block per disk, this provide protection against one physical disk failure, where the minimum number of disks are three.

This is how a RAID5 array looks like (image from diskpart.com):

raid5

Hardware Overview

We will have a Linux server with one root disk and six extra disks:

1
2
3
4
5
6
7
8
9
10

$ lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0    0    8G  0 disk
└─xvda1 202:1    0    8G  0 part /
xvdb    202:16   0   10G  0 disk
xvdc    202:32   0   10G  0 disk
xvdd    202:48   0   10G  0 disk
xvde    202:64   0   10G  0 disk
xvdf    202:80   0   10G  0 disk
xvdg    202:96   0   10G  0 disk

Dependencies

We require mdadm to create our raid configuration:

1
2

$ sudo apt update
$ sudo apt install mdadm -y

Format Disks

First we will format and partition the following disks: /dev/xvdb, /dev/xvdc, /dev/xvdd, I will demonstrate the process for one disk, but repeat them for the other as well:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

$ fdisk /dev/xvdc

Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

The old ext4 signature will be removed by a write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x26a2d2f6.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-20971519, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-20971519, default 20971519):

Created a new partition 1 of type 'Linux' and of size 10 GiB.

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Create RAID5 Array

Using mdadm, create the /dev/md0 device, by specifying the raid level and the disks that we want to add to the array:

1
2
3

$ mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/xvdb1 /dev/xvdc1 /dev/xvdd1
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

Now that our device has been added, we can monitor the process:

1
2
3
4
5
6
7

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 xvdd1[3] xvdc1[1] xvdb1[0]
      20951040 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
      [==>..................]  recovery = 11.5% (1212732/10475520) finish=4.7min speed=32103K/sec

unused devices: <none>

As you can see, currently its at 11.5%, give it some time to let it complete, you should treat the following as a completed state:

1
2
3
4
5
6

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 xvdd1[3] xvdc1[1] xvdb1[0]
      20951040 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]

unused devices: <none>

We can also inspect devices with mdadm:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

$ mdadm -E /dev/xvd[b-d]1
/dev/xvdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : ea997bce:a530519c:ae41022e:0f4306bf
           Name : ip-172-31-3-57:0  (local to host ip-172-31-3-57)
  Creation Time : Wed Jan 12 13:36:39 2022
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 20951040 (9.99 GiB 10.73 GB)
     Array Size : 20951040 (19.98 GiB 21.45 GB)
    Data Offset : 18432 sectors
   Super Offset : 8 sectors
   Unused Space : before=18280 sectors, after=0 sectors
          State : clean
    Device UUID : 8305a179:3ef96520:6c7b41dd:bdc7401f

    Update Time : Wed Jan 12 13:42:14 2022
  Bad Block Log : 512 entries available at offset 136 sectors
       Checksum : 1f9b4887 - correct
         Events : 18

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)

To get information about your raid5 device:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

$ mdadm --detail /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Wed Jan 12 13:36:39 2022
        Raid Level : raid5
        Array Size : 20951040 (19.98 GiB 21.45 GB)
     Used Dev Size : 10475520 (9.99 GiB 10.73 GB)
      Raid Devices : 3
     Total Devices : 3
       Persistence : Superblock is persistent

       Update Time : Wed Jan 12 13:42:14 2022
             State : clean
    Active Devices : 3
   Working Devices : 3
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : resync

              Name : ip-172-31-3-57:0  (local to host ip-172-31-3-57)
              UUID : ea997bce:a530519c:ae41022e:0f4306bf
            Events : 18

    Number   Major   Minor   RaidDevice State
       0     202       17        0      active sync   /dev/xvdb1
       1     202       33        1      active sync   /dev/xvdc1
       3     202       49        2      active sync   /dev/xvdd1

Create Filesystems

We will use our /dev/md0 device and create a ext4 filesystem:

1
2
3
4
5
6
7
8
9
10
11
12

$ mkfs.ext4 /dev/md0
mke2fs 1.45.5 (07-Jan-2020)
Creating filesystem with 5237760 4k blocks and 1310720 inodes
Filesystem UUID: 579f045e-d270-4ff2-b36b-8dc506c27c5f
Superblock backups stored on blocks:
  32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
  4096000

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

We can then verify that by looking at our block devices using lsblk:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

$ lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
xvda    202:0    0    8G  0 disk
└─xvda1 202:1    0    8G  0 part  /
xvdb    202:16   0   10G  0 disk
└─xvdb1 202:17   0   10G  0 part
  └─md0   9:0    0   20G  0 raid5
xvdc    202:32   0   10G  0 disk
└─xvdc1 202:33   0   10G  0 part
  └─md0   9:0    0   20G  0 raid5
xvdd    202:48   0   10G  0 disk
└─xvdd1 202:49   0   10G  0 part
  └─md0   9:0    0   20G  0 raid5
xvde    202:64   0   10G  0 disk
xvdf    202:80   0   10G  0 disk
xvdg    202:96   0   10G  0 disk

Now we can mount our device to /mnt:

1

$ mount /dev/md0 /mnt

We can verify that the device is mounted by using df:

1
2
3
4

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.7G  1.5G  6.3G  19% /
/dev/md0         20G   45M   19G   1% /mnt

To persist the device across reboots, add it to the /etc/fstab file:

1
2

$ cat /etc/fstab
/dev/md0                /mnt     ext4   defaults                0 0

Now our filesystem which is mounted at /mnt is ready to be used.

RAID Configuration (across reboots)

By default RAID doesn’t have a config file, therefore we need to save it manually. If this step is not followed RAID device will not be in md0, but perhaps something else.

So, we must have to save the configuration to persist across reboots, when it reboot it gets loaded to the kernel and RAID will also get loaded.

1

$ mdadm --detail --scan --verbose >> /etc/mdadm.conf

Note: Saving the configuration will keep the RAID level stable in the md0 device.

Adding Spare Devices

Earlier I mentioned that we have spare disks that we can use to expand our raid device. After they have been formatted we can add them as spare devices to our raid setup:

1
2
3
4

$ mdadm --add /dev/md0 /dev/xvde1 /dev/xvdf1 /dev/xvdg1
mdadm: added /dev/xvde1
mdadm: added /dev/xvdf1
mdadm: added /dev/xvdg1

Verify our change by viewing the detail of our device:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

$ mdadm --detail /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Wed Jan 12 13:36:39 2022
        Raid Level : raid5
        Array Size : 20951040 (19.98 GiB 21.45 GB)
     Used Dev Size : 10475520 (9.99 GiB 10.73 GB)
      Raid Devices : 3
     Total Devices : 6
       Persistence : Superblock is persistent

       Update Time : Wed Jan 12 14:28:23 2022
             State : clean
    Active Devices : 3
   Working Devices : 6
    Failed Devices : 0
     Spare Devices : 3

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : resync

              Name : ip-172-31-3-57:0  (local to host ip-172-31-3-57)
              UUID : ea997bce:a530519c:ae41022e:0f4306bf
            Events : 27

    Number   Major   Minor   RaidDevice State
       0     202       17        0      active sync   /dev/xvdb1
       1     202       33        1      active sync   /dev/xvdc1
       3     202       49        2      active sync   /dev/xvdd1

       4     202       65        -      spare   /dev/xvde1
       5     202       81        -      spare   /dev/xvdf1
       6     202       97        -      spare   /dev/xvdg1

As you can see it’s only spares at this moment, we can use the spares for data storage, by growing our device:

1

$ mdadm --grow --raid-devices=6 /dev/md0

Verify:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

$ mdadm --detail /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Wed Jan 12 13:36:39 2022
        Raid Level : raid5
        Array Size : 20951040 (19.98 GiB 21.45 GB)
     Used Dev Size : 10475520 (9.99 GiB 10.73 GB)
      Raid Devices : 6
     Total Devices : 6
       Persistence : Superblock is persistent

       Update Time : Wed Jan 12 15:15:31 2022
             State : clean, reshaping
    Active Devices : 6
   Working Devices : 6
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : resync

    Reshape Status : 0% complete
     Delta Devices : 3, (3->6)

              Name : ip-172-31-3-57:0  (local to host ip-172-31-3-57)
              UUID : ea997bce:a530519c:ae41022e:0f4306bf
            Events : 36

    Number   Major   Minor   RaidDevice State
       0     202       17        0      active sync   /dev/xvdb1
       1     202       33        1      active sync   /dev/xvdc1
       3     202       49        2      active sync   /dev/xvdd1
       6     202       97        3      active sync   /dev/xvdg1
       5     202       81        4      active sync   /dev/xvdf1
       4     202       65        5      active sync   /dev/xvde1

Wait for the raid to rebuild, by viewing the mdstat::

1
2
3
4
5
6
7

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 xvdg1[6] xvdf1[5] xvde1[4] xvdd1[3] xvdc1[1] xvdb1[0]
      20951040 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6] [UUUUUU]
      [>....................]  reshape =  0.7% (76772/10475520) finish=18.0min speed=9596K/sec

unused devices: <none>

Resizing our Filesystem

Once we added the spares and growed our device, we need to run integrity checks, then we can resize the volume. But first, we need to unmount our filesystem:

1

$ umount /mnt

Run a integrity check:

1
2
3
4
5
6
7
8

$ e2fsck -f /dev/md0
e2fsck 1.45.5 (07-Jan-2020)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/md0: 12/1310720 files (0.0% non-contiguous), 126323/5237760 blocks

Once that has passed, resize the file system:

1
2
3
4

$ resize2fs /dev/md0
resize2fs 1.45.5 (07-Jan-2020)
Resizing the filesystem on /dev/md0 to 13094400 (4k) blocks.
The filesystem on /dev/md0 is now 13094400 (4k) blocks long.

Then we remount our filesystem:

1

$ mount /dev/md0 /mnt

After the filesystem has been mounted, we can view the disk size and confirm that the size increased:

1
2
3

$ df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0         50G   52M   47G   1% /mnt

Thank You

Thanks for reading, feel free to check out my website, feel free to subscribe to my newsletter or follow me at @ruanbekker on Twitter.