I have a CentOS 6.10 Server with 4 x 2TB Drives. One was failing (bad sectors and causing major disk IO bottlenecks). This is a headless server in a remote data center. Technician is going to replace the bad hard drive shortly. I figured out the bad drive (/dev/sdb) and marked the partitions with the drive as faulty, then removed.

mdadm /dev/md2 -f /dev/sdb2
mdadm /dev/md3 -f /dev/sdb3
mdadm /dev/md5 -f /dev/sdb5

next, I used the -r flag to remove the drive from the set:

mdadm /dev/md2 -r /dev/sdb2
mdadm /dev/md3 -r /dev/sdb3
mdadm /dev/md5 -r /dev/sdb5

So now, here's my cat /proc/mdstat:

Personalities : [raid1] [raid10]
md2 : active raid1 sdd2[3] sda2[0] sdc2[2]
      523200 blocks [4/3] [U_UU]

md3 : active raid1 sda3[0] sdd3[3] sdc3[2]
      102398912 blocks [4/3] [U_UU]

md5 : active raid10 sdd5[3] sda5[0] sdc5[2]
      3699072000 blocks 512K chunks 2 near-copies [4/3] [U_UU]
      bitmap: 27/28 pages [108KB], 65536KB chunk

unused devices: <none>

Here is the details of my various partitions:

mdadm --detail /dev/md2

        Version : 0.90
  Creation Time : Wed Nov 27 19:55:48 2019
     Raid Level : raid1
     Array Size : 523200 (510.94 MiB 535.76 MB)
  Used Dev Size : 523200 (510.94 MiB 535.76 MB)
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Thu Apr  9 01:07:31 2020
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

           UUID : 7c0d92f0:0c872155:a4d2adc2:26fd5302
         Events : 0.65

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       2       0        0        2      removed
       2       8       34        2      active sync   /dev/sdc2
       3       8       50        3      active sync   /dev/sdd2

mdadm --detail /dev/md3

/dev/md3:
        Version : 0.90
  Creation Time : Wed Nov 27 19:55:49 2019
     Raid Level : raid1
     Array Size : 102398912 (97.66 GiB 104.86 GB)
  Used Dev Size : 102398912 (97.66 GiB 104.86 GB)
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 3
    Persistence : Superblock is persistent

    Update Time : Thu Apr  9 23:25:07 2020
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

           UUID : 87473072:b4d350c2:a4d2adc2:26fd5302
         Events : 0.22932

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       2       0        0        2      removed
       2       8       35        2      active sync   /dev/sdc3
       3       8       51        3      active sync   /dev/sdd3

mdadm --detail /dev/md5

/dev/md5:
        Version : 0.90
  Creation Time : Wed Nov 27 19:55:51 2019
     Raid Level : raid10
     Array Size : 3699072000 (3527.71 GiB 3787.85 GB)
  Used Dev Size : 1849536000 (1763.85 GiB 1893.92 GB)
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 5
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Thu Apr  9 23:24:47 2020
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 512K

           UUID : 256cd09b:989c40e7:a4d2adc2:26fd5302
         Events : 0.246450

    Number   Major   Minor   RaidDevice State
       0       8        5        0      active sync set-A   /dev/sda5
       2       0        0        2      removed
       2       8       37        2      active sync set-A   /dev/sdc5
       3       8       53        3      active sync set-B   /dev/sdd5

fdisk -l /dev/sda

fdisk -l /dev/sda

WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1      243202  1953514583+  ee  GPT

Notes:

/dev/md2  Partition: /boot
/dev/md3  Partition: /
/dev/md5  Partition: /vz  (This is an OpevVZ Server running SolusVM)

Need to make I put this drive back in service correctly.

This was the correct procedure. it's worth noting the order of the sgdisk -R command. It's destination first, then source.

Copy partition Table:

sgdisk -R /dev/sdb /dev/sda

Randomize GUID

sgdisk -G /dev/sdb

Check if partition tables are the same:

sgdisk -p /dev/sda
sgdisk -p /dev/sdb
sgdisk -p /dev/sdc
sgdisk -p /dev/sdd

Add disk back:

mdadm --manage /dev/md2 --add /dev/sdb2
mdadm --manage /dev/md3 --add /dev/sdb3
mdadm --manage /dev/md5 --add /dev/sdb5