Failure Description

Expand the Linux file system according to the customer's demand. 

During the implementation process, after mapping the hard disk to the host, creating the PV and adding it to the VG, 

the system prompts "unknown device".

[root@KMS-Svr cache]# pvs

  Incorrect metadata area header checksum on /dev/sdb1 at offset 4096

  Couldn't find device with uuid ZPy1sa-fXhe-qrcQ-HFhi-eazU-4Mg3-toY3Xv.

  PV             VG            Fmt  Attr PSize   PFree

  /dev/sda2      vg_testkmssvr lvm2 a--  149.51g      0

  /dev/sdb1                    lvm2 a--  499.99g 499.99g

  /dev/sdc1      vg_testkmssvr lvm2 a--  100.00g      0

  /dev/sdd       vm2t          lvm2 a--    2.00t      0

  /dev/sde       vm2t          lvm2 a--    2.20t      0

  /dev/sdf1      vm2t          lvm2 a--    2.00t      0

  /dev/sdg       vm2t          lvm2 a--    2.00t      0

  /dev/sdh       vm2t          lvm2 a--    2.00t      0

  /dev/sdi       vm2t          lvm2 a--    2.00t      0

  /dev/sdj       vm2t          lvm2 a--    2.00t      0

  /dev/sdk1      vm2t          lvm2 a--    1.86t  74.36g

  /dev/sdk2      vg_testkmssvr lvm2 a--  140.64g  19.63g

  unknown device vm2t          lvm2 a-m    2.00t   2.00t

Troubleshooting

2.1 Log Analysis

There are no special error messages in the messages log. 

dmesg logs are as follows:

sd 2:0:12:0: [sdl] Very big device. Trying to use READ CAPACITY(16).

sd 2:0:12:0: [sdl] Cache data unavailable

sd 2:0:12:0: [sdl] Assuming drive cache: write through

 sdl: sdl1

sd 2:0:12:0: [sdl] Very big device. Trying to use READ CAPACITY(16).

sd 2:0:12:0: [sdl] Cache data unavailable

sd 2:0:12:0: [sdl] Assuming drive cache: write through

 sdl: sdl1

EXT4-fs (dm-2): warning: checktime reached, running e2fsck is recommended

EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts:

With the output of the dmesg logs, you can see that the newly added disk was successfully recognized at the OS level. Communicating with the customer, 

we learned that this disk was previously used on other hosts, and the old host had been taken offline. The engineer speculates that 

the VG/PV information of the old host is stored on the disk and that it needs to be further processed.

2.2 Culling problem disks

Run the command:

vgreduce vm2t --removemissing

[root@KMS-Svr cache]# vgreduce vm2t --removemissing

  Incorrect metadata area header checksum on /dev/sdb1 at offset 4096

  Couldn't find device with uuid ZPy1sa-fXhe-qrcQ-HFhi-eazU-4Mg3-toY3Xv.

  Wrote out consistent volume group vm2t

[root@KMS-Svr cache]# vgs

  Incorrect metadata area header checksum on /dev/sdb1 at offset 4096

  VG            #PV #LV #SN Attr   VSize   VFree

  vg_testkmssvr   3   3   0 wz--n- 390.14g 19.63g

  vm2t            8   1   0 wz--n-  16.06t 74.36g

[root@KMS-Svr cache]# pvs

  Incorrect metadata area header checksum on /dev/sdb1 at offset 4096

  PV         VG            Fmt  Attr PSize   PFree

  /dev/sda2  vg_testkmssvr lvm2 a--  149.51g      0

  /dev/sdb1                lvm2 a--  499.99g 499.99g

  /dev/sdc1  vg_testkmssvr lvm2 a--  100.00g      0

  /dev/sdd   vm2t          lvm2 a--    2.00t      0

  /dev/sde   vm2t          lvm2 a--    2.20t      0

  /dev/sdf1  vm2t          lvm2 a--    2.00t      0

  /dev/sdg   vm2t          lvm2 a--    2.00t      0

  /dev/sdh   vm2t          lvm2 a--    2.00t      0

  /dev/sdi   vm2t          lvm2 a--    2.00t      0

  /dev/sdj   vm2t          lvm2 a--    2.00t      0

  /dev/sdk1  vm2t          lvm2 a--    1.86t  74.36g

  /dev/sdk2  vg_testkmssvr lvm2 a--  140.64g  19.63g

You can find that the disk status is normal at this point, but you can't query it in the pvs command output. Try to query it by fdisk -l command:

Disk /dev/sdl: 2199.0 GB, 2199023255552 bytes

255 heads, 63 sectors/track, 267349 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x9706bba2

   Device Boot      Start         End      Blocks   Id  System

/dev/sdl1               1      267349  2147480811   8e  Linux LVM

Combined with the lsblk command query, at this time there is sdl disk at the OS level, and there is already partition sdl1, 

you need to re-create the PV and add it to the specified VG.

2.3 Re-create PV

Run the command： pvcreate /dev/sdl1 --force

[root@KMS-Svr cache]# pvcreate /dev/sdl1 --force

  Physical volume "/dev/sdl1" successfully created

[root@KMS-Svr cache]# pvs

  Incorrect metadata area header checksum on /dev/sdb1 at offset 4096

  PV         VG            Fmt  Attr PSize   PFree

  /dev/sda2  vg_testkmssvr lvm2 a--  149.51g      0

  /dev/sdb1                lvm2 a--  499.99g 499.99g

  /dev/sdc1  vg_testkmssvr lvm2 a--  100.00g      0

  /dev/sdd   vm2t          lvm2 a--    2.00t      0

  /dev/sde   vm2t          lvm2 a--    2.20t      0

  /dev/sdf1  vm2t          lvm2 a--    2.00t      0

  /dev/sdg   vm2t          lvm2 a--    2.00t      0

  /dev/sdh   vm2t          lvm2 a--    2.00t      0

  /dev/sdi   vm2t          lvm2 a--    2.00t      0

  /dev/sdj   vm2t          lvm2 a--    2.00t      0

  /dev/sdk1  vm2t          lvm2 a--    1.86t  74.36g

  /dev/sdk2  vg_testkmssvr lvm2 a--  140.64g  19.63g

  /dev/sdl1                lvm2 a--    2.00t   2.00t

Run command：vgextend vm2t /dev/sdl1

 [root@KMS-Svr cache]# vgs

  Incorrect metadata area header checksum on /dev/sdb1 at offset 4096

  VG            #PV #LV #SN Attr   VSize   VFree

  vg_testkmssvr   3   3   0 wz--n- 390.14g 19.63g

  vm2t            9   1   0 wz--n-  18.06t  2.07t

[root@KMS-Svr cache]# pvs

  Incorrect metadata area header checksum on /dev/sdb1 at offset 4096

  PV         VG            Fmt  Attr PSize   PFree

  /dev/sda2  vg_testkmssvr lvm2 a--  149.51g      0

  /dev/sdb1                lvm2 a--  499.99g 499.99g

  /dev/sdc1  vg_testkmssvr lvm2 a--  100.00g      0

  /dev/sdd   vm2t          lvm2 a--    2.00t      0

  /dev/sde   vm2t          lvm2 a--    2.20t      0

  /dev/sdf1  vm2t          lvm2 a--    2.00t      0

  /dev/sdg   vm2t          lvm2 a--    2.00t      0

  /dev/sdh   vm2t          lvm2 a--    2.00t      0

  /dev/sdi   vm2t          lvm2 a--    2.00t      0

  /dev/sdj   vm2t          lvm2 a--    2.00t      0

  /dev/sdk1  vm2t          lvm2 a--    1.86t  74.36g

  /dev/sdk2  vg_testkmssvr lvm2 a--  140.64g  19.63g

  /dev/sdl1  vm2t          lvm2 a--    2.00t   2.00t

Add the disk "/dev/sdl1" in the PV to the VG of the specified vm2t and use PVS to check the success of the addition.

Lesson Learned

When the LVM block device status is abnormal or "unknown device" appears, 

you can use vgscan to rescan all supported LVM block devices in the system, 

and exclude the problematic pvs from the VGs, so as to get the normal status of the VGs.

Knowledge Expansion

4.1 Vgscan Command Usage

vgscan [ option_args ]

vgscan

[ –cache ]  Scan all devices and send metadata to lvmetad daemon
[ –ignorelockingfailure ]  Allows the command to continue to execute as a read-only metadata operation after the lock fails.

[ -mknodes ] Also checks for special LVM files in /dev needed for active LVs and creates any missing files and removes unused files.

[ -notifydbus ] Sends a notification to D-Bus. This command exits with an error if LVMs supporting D-Bus notification are not built, or if the notify_dbus configuration setting is disabled.

[ -reportformat basic|json ] Overrides the current output format of the report, which is defined globally by the report/output_format setting in lvm.conf. basic is the original format containing columns and rows.

4.2 LVM File System Size Limits

When the file system is larger than 16TB, the resize2fs command will fail. The reason is that the kernel version of the operating system is too low, 

and you need to upgrade the operating system kernel if you need a larger file system.

The following figure shows that the resize2fs command fails:

4.3 LVM and System Kernel Relationships

The maximum space for a single LVM is limited by CPU architecture and Linux kernel version:

a, Linux kernel version 2.4.x limits the maximum file system to 2TB;

b, Some earlier Linux kernels before 2.4.x limited the file system to a maximum of 1TB;

c, The combination of a 32-bit CPU and Linux kernel version 2.6.x limits the file system to a maximum of 16TB;

d, For Linux kernel 2.6.x running on a 64-bit CPU, the maximum file system limit is 8EB.

For more information, please visit Antute's official website:7yx3.fredrimonta.com