Linux Software RAID 1(미러링) 구성

※ Linux 시스템에서 mdadm을 이용한 Soft RAID 구성
최초작성 - 2006.08.01. / 1차개정 - 2013.11.11. / 2차개정 - 2016.02.11

 

RAID 의 레벨은 대표적으로 0, 1, 5, 6, 1+0, 0+1 등 다양한 레벨이 존재하며 본 게시물에서는 보편적이면서 많이 사용되는 RAID Level 1 (미러링 모드) 에 대해서 알아보고자 합니다.

RAID 레벨에 대한 보다 자세한 정의는 구글등 검색엔진을 활용하시기 바랍니다.

 
다음 과정은 시스템을 초기 설치시 RAID 1 레벨을 구성하는것이 아닌 이미 설치된 시스템에 디스크 두개를 추가하여 구성하는 방식입니다.

또한 일반적으로 많이 알려진 raidtools를 이용한 raid 구성이 아닌 mdadm을 이용한 구성입니다.

 

RAID 구성에 사용되는 mdadm은 Linux kernel 기반에서 작동하는 raid software로서 아래와 같은 패키지가 설치되어있어야 구현 가능합니다.

 

[root@TEST01 ~]# [root@TEST01 ~]# yum provides /sbin/mdadm Loaded plugins: fastestmirror, priorities, security Loading mirror speeds from cached hostfile * base: data.nicehosting.co.kr * extras: data.nicehosting.co.kr * rpmforge: ftp.riken.jp * updates: data.nicehosting.co.kr mdadm-2.6.9-5.el5.x86_64 : mdadm controls Linux md devices (software RAID arrays) Repo : base Matched from: Filename : /sbin/mdadm   mdadm-2.6.9-3.el5.x86_64 : mdadm controls Linux md devices (software RAID arrays) Repo : installed Matched from: Other : Provides-match: /sbin/mdadm   [root@TEST01 ~]# rpm -ql mdadm /etc/cron.weekly/99-raid-check /etc/rc.d/init.d/mdmonitor /etc/rc.d/init.d/mdmpd /etc/sysconfig/raid-check /sbin/mdadm /sbin/mdadm.static /sbin/mdassemble.static /sbin/mdmpd /usr/share/doc/mdadm-2.6.9 /usr/share/doc/mdadm-2.6.9/ANNOUNCE-2.0 /usr/share/doc/mdadm-2.6.9/ANNOUNCE-2.1 /usr/share/doc/mdadm-2.6.9/ANNOUNCE-2.2 /usr/share/doc/mdadm-2.6.9/ANNOUNCE-2.3 /usr/share/doc/mdadm-2.6.9/ANNOUNCE-2.3.1 /usr/share/doc/mdadm-2.6.9/ANNOUNCE-2.4 . . 생략 . .

 

 

▶ fdisk 명령으로 레이드를 구성할 디스크 디바이스 확인 및 /dev/sdb와 sdc를 파티셔닝 작업

[root@TEST01 ~]# fdisk -l Disk /dev/sda: 21.4 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 19 152586 83 Linux /dev/sda2 20 83 514080 83 Linux /dev/sda3 84 1103 8193150 83 Linux /dev/sda4 1104 2610 12104977+ 5 Extended /dev/sda5 1104 1234 1052226 82 Linux swap / Solaris /dev/sda6 1235 1361 1020096 83 Linux /dev/sda7 1362 1488 1020096 83 Linux /dev/sda8 1489 1552 514048+ 83 Linux /dev/sda9 1553 2610 8498353+ 83 Linux Disk /dev/sdb: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System Disk /dev/sdc: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/sdc doesn't contain a valid partition table [root@TEST01 ~]# [root@TEST01 ~]# fdisk /dev/sd sda sda1 sda2 sda3 sda4 sda5 sda6 sda7 sda8 sda9 sdb sdc [root@TEST01 ~]# fdisk /dev/sd sda sda1 sda2 sda3 sda4 sda5 sda6 sda7 sda8 sda9 sdb sdc [root@TEST01 ~]# fdisk /dev/sdb Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-261, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-261, default 261): Using default value 261 Command (m for help): t Selected partition 1 Hex code (type L to list codes): fd Changed system type of partition 1 to fd (Linux raid autodetect) Command (m for help): l 0 Empty 1e Hidden W95 FAT1 80 Old Minix be Solaris boot 1 FAT12 24 NEC DOS 81 Minix / old Lin bf Solaris 2 XENIX root 39 Plan 9 82 Linux swap / So c1 DRDOS/sec (FAT- 3 XENIX usr 3c PartitionMagic 83 Linux c4 DRDOS/sec (FAT- 4 FAT16 <32M 40 Venix 80286 84 OS/2 hidden C: c6 DRDOS/sec (FAT- 5 Extended 41 PPC PReP Boot 85 Linux extended c7 Syrinx 6 FAT16 42 SFS 86 NTFS volume set da Non-FS data 7 HPFS/NTFS 4d QNX4.x 87 NTFS volume set db CP/M / CTOS / . 8 AIX 4e QNX4.x 2nd part 88 Linux plaintext de Dell Utility 9 AIX bootable 4f QNX4.x 3rd part 8e Linux LVM df BootIt a OS/2 Boot Manag 50 OnTrack DM 93 Amoeba e1 DOS access b W95 FAT32 51 OnTrack DM6 Aux 94 Amoeba BBT e3 DOS R/O c W95 FAT32 (LBA) 52 CP/M 9f BSD/OS e4 SpeedStor e W95 FAT16 (LBA) 53 OnTrack DM6 Aux a0 IBM Thinkpad hi eb BeOS fs f W95 Ext'd (LBA) 54 OnTrackDM6 a5 FreeBSD ee EFI GPT 10 OPUS 55 EZ-Drive a6 OpenBSD ef EFI (FAT-12/16/ 11 Hidden FAT12 56 Golden Bow a7 NeXTSTEP f0 Linux/PA-RISC b 12 Compaq diagnost 5c Priam Edisk a8 Darwin UFS f1 SpeedStor 14 Hidden FAT16 <3 61 SpeedStor a9 NetBSD f4 SpeedStor 16 Hidden FAT16 63 GNU HURD or Sys ab Darwin boot f2 DOS secondary 17 Hidden HPFS/NTF 64 Novell Netware b7 BSDI fs fd Linux raid auto 18 AST SmartSleep 65 Novell Netware b8 BSDI swap fe LANstep 1b Hidden W95 FAT3 70 DiskSecure Mult bb Boot Wizard hid ff BBT 1c Hidden W95 FAT3 75 PC/IX Command (m for help): p Disk /dev/sdb: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 1 261 2096451 fd Linux raid autodetect Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# fdisk /dev/sdc Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-261, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-261, default 261): Using default value 261 Command (m for help): t Selected partition 1 Hex code (type L to list codes): fd Changed system type of partition 1 to fd (Linux raid autodetect) Command (m for help): p Disk /dev/sdc: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdc1 1 261 2096451 fd Linux raid autodetect Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.

 

 

▶ 시스템에 현재 구성된 레이드 노드를 확인

 

[root@TEST01 ~]# [root@TEST01 ~]# export LANG=C [root@TEST01 ~]# ls -la /dev/md ls: /dev/md: No such file or directory [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# ls -la /dev/md* brw-r----- 1 root disk 9, 0 Dec 21 08:27 /dev/md0

 

 

▶ 시스템에 현재 /dev/md0 라는 레이드 노드가 이미 구성되어 있기 때문에 rm 명령으로 해당 디바이스명을 삭제하거나 혹은 다음과 같이 추가

 

[root@TEST01 ~]# mknod /dev/md1 b 9 1 [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]#

 

※ 명령해설 : mknod는 장치 Device 파일등의 특수 파일을 만드는 명령으로서 아래와 같이 mknod / 장치형식 / Major Number / Minor Number 형식으로 생성하며, 초기값으로 만들어지는 파일의 권한은 Umask 0666 값으로 생성된다.

 

p:  FIFO  b: 블럭 특수 파일  c 또는 u: 캐릭터 특수 파일(character special file)

 

※ Linux에 예약 및 생성된 Device Major / Minor Number 확인 : Linux System은 각 장치들의 고유식별을 위해 Major Number와 Minor Number를 사용하며, 해당 내역은 아래와 같이 확인 가능하다.

 

1) 각 장치 유형별 예약된 Major Number 확인

 

[root@centos7 ~]# cat /proc/devices Character devices:   1 mem   4 /dev/vc/0   4 tty   4 ttyS   5 /dev/tty   5 /dev/console   5 /dev/ptmx   7 vcs  10 misc  13 input  14 sound  21 sg  29 fb  99 ppdev 116 alsa 128 ptm 136 pts 162 raw 180 usb 188 ttyUSB 189 usb_device 202 cpu/msr 203 cpu/cpuid 226 drm 249 hidraw 250 usbmon 251 bsg 252 watchdog 253 rtc 254 tpm   Block devices: 259 blkext   8 sd   9 md  11 sr  65 sd  66 sd  67 sd  68 sd  69 sd  70 sd  71 sd 128 sd 129 sd 130 sd 131 sd 132 sd 133 sd 134 sd 135 sd 253 device-mapper 254 mdp

 

 

2) OS 이하 기 생성된 장치별 Major / Minor Number 확인

 

[root@centos7 ~]# [root@centos7 ~]# cat /proc/diskstats     8       0 sda 13183 40 1294536 27820 1133 70 20941 2123 0 10636 29326    8       1 sda1 306 0 55525 561 257 0 4096 52 0 302 608    8       2 sda2 12787 40 1237411 27218 746 70 16845 1965 0 10471 28575    8      16 sdb 787 4 70766 1346 282 27 2425 317 0 770 1639    8      32 sdc 231 0 3176 42 0 0 0 0 0 34 37    8      48 sdd 231 0 3176 45 0 0 0 0 0 36 39   11       0 sr0 21 0 132 5 0 0 0 0 0 4 4  253       0 dm-0 147 0 2584 217 0 0 0 0 0 195 217  253       1 dm-1 13259 0 1303545 28585 995 0 19270 2503 0 11039 31160 [root@centos7 ~]# [root@centos7 ~]# [root@centos7 ~]#

 

 

 

▶ mdadm 명령으로 레이드 레벨을 구성하고 해당 디바이스를 구성

 

[root@TEST01 ~]# mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1 mdadm: /dev/sdb1 appears to contain an ext2fs file system size=2096448K mtime=Wed Dec 21 07:42:55 2005 Continue creating array? Continue creating array? (y/n) Continue creating array? (y/n) y mdadm: array /dev/md1 started. [root@TEST01 ~]# [root@TEST01 ~]#

 

▶ mdadm 명령으로 해당 디바이스가 잘 구성되었는지 스캔

 

[root@TEST01 ~]# mdadm --detail --scan ARRAY /dev/md1 level=raid1 num-devices=2 UUID=1a87ce48:f0917a06:4120df55:7dfe1c95 devices=/dev/sdb1,/dev/sdc1 [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# clear [root@TEST01 ~]#

 

 

 

 

▶ RAID 1 레벨로 구성된 레이드 노드 디바이스를 ext3로 포멧

 

[root@TEST01 ~]# [root@TEST01 ~]# mkfs.ext3 /dev/md1 mke2fs 1.37 (21-Mar-2005) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) 262144 inodes, 524096 blocks 26204 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=536870912 16 block groups 32768 blocks per group, 32768 fragments per group 16384 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912 Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 32 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. [root@TEST01 ~]# [root@TEST01 ~]#

 

 

▶ 레이드가 구성된 /dev/md1 디바이스를 raid_disk 라는 디렉토리로 mount

 

[root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# mkdir /raid_disk [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# mount /dev/md1 /raid_disk [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda7 988088 217296 719788 24% / /dev/sda1 147764 11319 128816 9% /boot /dev/shm 62892 0 62892 0% /dev/shm /dev/sda9 8232040 51548 7755576 1% /home /dev/sda6 988088 17676 919408 2% /opt /dev/sda2 497861 11576 460581 3% /tmp /dev/sda3 7936288 6772748 753884 90% /usr /dev/sda8 497829 160064 312063 34% /var /dev/md1 2063440 35880 1922744 2% /raid_disk [root@TEST01 ~]# [root@TEST01 ~]#

 

▶ 추가된 노드 디바이스와 마운트 정보를 fstab 에 기록

 

[root@TEST01 ~]# [root@TEST01 ~]# vi /etc/fstab # This file is edited by fstab-sync - see 'man fstab-sync' for details LABEL=/ / ext3 defaults 1 1 LABEL=/boot /boot ext3 defaults 1 2 /dev/devpts /dev/pts devpts gid=5,mode=620 0 0 /dev/shm /dev/shm tmpfs defaults 0 0 LABEL=/home /home ext3 defaults 1 2 /dev/md1 /raid_disk ext3 defaults 1 2 LABEL=/opt /opt ext3 defaults 1 2 /dev/proc /proc proc defaults 0 0 /dev/sys /sys sysfs defaults 0 0 LABEL=/tmp /tmp ext3 defaults 1 2 LABEL=/usr /usr ext3 defaults 1 2 LABEL=/var /var ext3 defaults 1 2 LABEL=SWAP-sda5 swap swap defaults 0 0 /dev/fd0 /media/floppy auto pamconsole,exec,noauto,managed 0 0 "/etc/fstab" 16L, 1215C written [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# clear [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# init 6 [root@TEST01 ~]#

 

▶ 디스크를 하나 제거후 status 메시지 확인후 다시 디스크를 추가하는 작업

 

[root@TEST01 ~]# mdadm --detail --scan ARRAY /dev/md1 level=raid1 num-devices=2 UUID=1a87ce48:f0917a06:4120df55:7dfe1c95 devices=/dev/sdb1 [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# cat /proc/mdstat Personalities : [raid1] md1 : active raid1 sdb1[0] 2096384 blocks [2/1] [U_] unused devices: <none> [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda7 965M 213M 703M 24% / /dev/sda1 145M 12M 126M 9% /boot /dev/shm 62M 0 62M 0% /dev/shm /dev/sda9 7.9G 51M 7.4G 1% /home /dev/md1 2.0G 36M 1.9G 2% /raid_disk /dev/sda6 965M 18M 898M 2% /opt /dev/sda2 487M 12M 450M 3% /tmp /dev/sda3 7.6G 6.5G 737M 90% /usr /dev/sda8 487M 157M 305M 34% /var [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# mdadm --detail --scan ARRAY /dev/md1 level=raid1 num-devices=2 UUID=1a87ce48:f0917a06:4120df55:7dfe1c95 devices=/dev/sdb1 [root@TEST01 ~]#

 

 

▶ 추가된 새 디스크를 새로 파티셔닝 작업

 

[root@TEST01 ~]# fdisk /dev/sdc Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-261, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-261, default 261): Using default value 261 Command (m for help): t Selected partition 1 Hex code (type L to list codes): fd Changed system type of partition 1 to fd (Linux raid autodetect) Command (m for help): l 0 Empty 1e Hidden W95 FAT1 80 Old Minix be Solaris boot 1 FAT12 24 NEC DOS 81 Minix / old Lin bf Solaris 2 XENIX root 39 Plan 9 82 Linux swap / So c1 DRDOS/sec (FAT- 3 XENIX usr 3c PartitionMagic 83 Linux c4 DRDOS/sec (FAT- 4 FAT16 <32M 40 Venix 80286 84 OS/2 hidden C: c6 DRDOS/sec (FAT- 5 Extended 41 PPC PReP Boot 85 Linux extended c7 Syrinx 6 FAT16 42 SFS 86 NTFS volume set da Non-FS data 7 HPFS/NTFS 4d QNX4.x 87 NTFS volume set db CP/M / CTOS / . 8 AIX 4e QNX4.x 2nd part 88 Linux plaintext de Dell Utility 9 AIX bootable 4f QNX4.x 3rd part 8e Linux LVM df BootIt a OS/2 Boot Manag 50 OnTrack DM 93 Amoeba e1 DOS access b W95 FAT32 51 OnTrack DM6 Aux 94 Amoeba BBT e3 DOS R/O c W95 FAT32 (LBA) 52 CP/M 9f BSD/OS e4 SpeedStor e W95 FAT16 (LBA) 53 OnTrack DM6 Aux a0 IBM Thinkpad hi eb BeOS fs f W95 Ext'd (LBA) 54 OnTrackDM6 a5 FreeBSD ee EFI GPT 10 OPUS 55 EZ-Drive a6 OpenBSD ef EFI (FAT-12/16/ 11 Hidden FAT12 56 Golden Bow a7 NeXTSTEP f0 Linux/PA-RISC b 12 Compaq diagnost 5c Priam Edisk a8 Darwin UFS f1 SpeedStor 14 Hidden FAT16 <3 61 SpeedStor a9 NetBSD f4 SpeedStor 16 Hidden FAT16 63 GNU HURD or Sys ab Darwin boot f2 DOS secondary 17 Hidden HPFS/NTF 64 Novell Netware b7 BSDI fs fd Linux raid auto 18 AST SmartSleep 65 Novell Netware b8 BSDI swap fe LANstep 1b Hidden W95 FAT3 70 DiskSecure Mult bb Boot Wizard hid ff BBT 1c Hidden W95 FAT3 75 PC/IX Command (m for help): p Disk /dev/sdc: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdc1 1 261 2096451 fd Linux raid autodetect Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]#

 

 

▶ mdadm 명령으로 새로 추가된 디스크를 노드에 재 삽입

 

[root@TEST01 ~]# [root@TEST01 ~]# mdadm /dev/md1 --add /dev/sdc1 mdadm: hot added /dev/sdc1 [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda7 965M 213M 703M 24% / /dev/sda1 145M 12M 126M 9% /boot /dev/shm 62M 0 62M 0% /dev/shm /dev/sda9 7.9G 51M 7.4G 1% /home /dev/md1 2.0G 36M 1.9G 2% /raid_disk /dev/sda6 965M 18M 898M 2% /opt /dev/sda2 487M 12M 450M 3% /tmp /dev/sda3 7.6G 6.5G 737M 90% /usr /dev/sda8 487M 157M 305M 34% /var [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]# mdadm --detail --scan ARRAY /dev/md1 level=raid1 num-devices=2 spares=1 UUID=1a87ce48:f0917a06:4120df55:7dfe1c95 devices=/dev/sdb1,/dev/sdc1 [root@TEST01 ~]# [root@TEST01 ~]# [root@TEST01 ~]#

 

 

 

▶ 데이터가 제대로 기록되는지 확인

 

[root@TEST01 ~]# [root@TEST01 ~]# touch /raid_disk/test [root@TEST01 ~]# cd /raid_disk/ [root@TEST01 raid_disk]# ll 합계 20 drwx------ 2 root root 16384 12월 21 08:43 lost+found -rw-r--r-- 1 root root 0 12월 21 09:14 test [root@TEST01 raid_disk]# [root@TEST01 raid_disk]# [root@TEST01 raid_disk]# export LANG=C [root@TEST01 raid_disk]# [root@TEST01 raid_disk]# ll total 20 drwx------ 2 root root 16384 Dec 21 08:43 lost+found -rw-r--r-- 1 root root 0 Dec 21 09:14 test [root@TEST01 raid_disk]# clear [root@TEST01 raid_disk]# ls -la total 32 drwxr-xr-x 3 root root 4096 Dec 21 09:14 . drwxr-xr-x 26 root root 4096 Dec 21 09:04 .. drwx------ 2 root root 16384 Dec 21 08:43 lost+found -rw-r--r-- 1 root root 0 Dec 21 09:14 test [root@TEST01 raid_disk]#

 

 

※ SoftRaid1 Device 벤치마킹 테스트 내역