ACOS 6.3.0

Arcfra Cloud Operating System>
CLI command>
Managing block storage services

Managing extents

An extent is the basic unit managed by the MetaCluster. A volume is divided into multiple fixed-size extents, which are stored on the ChunkServer. Data recovery, migration, copy-on-write (COW), and other operations are all performed at the extent level. Technical personnel can view and debug the cluster's extents through LExtents and PExtents. LExtents are logical extents, and PExtents are the physical extents stored on disk.

Viewing extents

Viewing detailed information about an LExtent

Procedure

Run the following command on any node in the cluster to view detailed information about a specific LExtent:

zbs-meta lextent show <lextent_id>

Output example

-------------  -------
Logical ID     8
Logical Epoch  8431008
Perf PID       None
Perf Epoch     None
Cap PID        24348
Cap Epoch      8742712
Prioritized    False
-------------  -------

Output note

Parameter	Description
`Logical ID`	The LExtent ID.
`Logical Epoch`	The LExtent version number.
`Perf PID`	The performance tier PExtent ID.
`Perf Epoch`	The performance tier PExtent version number.
`Cap PID`	The capacity tier PExtent ID.
`Cap Epoch`	The capacity tier PExtent version number.
`Prioritized`	Whether it is referenced by a pinned volume.

Viewing the volume that an LExtent belongs to

Procedure

Run the following command on any node in the cluster to view the volume that the LExtent belongs to:

zbs-meta lextent getref <lextent_id>

Output example

-----------------------  ------------------------------------
Pool Name                zbs-images
Volume ID                ef5e94a3-2215-4ed9-814e-d08e393d7648
Volume Name              ef5e94a3-2215-4ed9
Volume Is Snapshot       False
Volume Snapshot Pool ID
Snapshot Name
-----------------------  ------------------------------------

Output note

Parameter	Description
`Pool Name`	The datastore that the LExtent belongs to.
`Volume ID`	The ID of the volume that the LExtent belongs to.
`Volume Name`	The name of the volume that the LExtent belongs to.
`Volume Is Snapshot`	Whether the volume that the LExtent belongs to is a snapshot.
`Volume Snapshot Pool ID`	The ID of the datastore to which the snapshot containing the LExtent belongs.
`Snapshot Name`	The name of the snapshot to which the LExtent belongs.

Viewing PExtents in the cluster

Procedure

Run the following command on any node in the cluster to view the PExtents within the specified range:

zbs-meta pextent list [start_id] [end_id] [--show_non_exist] [--show_cap_only] [--show_perf_only]

Parameter	Description
`start_id`	The starting PExtent ID for the search.
`end_id`	The ending PExtent ID for the search.
`--show_non_exist`	Displays PExtent IDs that have not been allocated.
`--show_cap_only`	Displays only PExtent IDs in the capacity tier.
`--show_perf_only`	Displays only PExtent IDs in the performance tier.

Note:

As the cluster scales, this operation can be very time-consuming.

Output example

[root@9f29fde8244f zbs-client-py]# zbs-meta pextent list 1 548 --show_non_exist --show_cap_only
  ID  Replica    Alive Replica    Temporary Replica    Ever Exist    Is Garbage      Origin PExtent    Origin Epoch    Expected Replica num    Epoch    Prefer Local  Allocated Space    Resiliency Type    PExtent Type    EC Param    Is Sinkable
----  ---------  ---------------  -------------------  ------------  ------------  ----------------  --------------  ----------------------  -------  --------------  -----------------  -----------------  --------------  ----------  -------------
   1  [4, 2, 7]  [4, 2, 7]        []                   True          False                        0               0                       3        1               2  5.25 MiB           RT_REPLICA         PT_CAP
   2  []         []               []                   False         False                        0               0                       3        2               2  0.00 B             RT_REPLICA         PT_CAP
   3  []         []               []                   False         False                        0               0                       3        3               2  0.00 B             RT_REPLICA         PT_CAP
   4  []         []               []                   False         False                        0               0                       3        4               2  0.00 B             RT_REPLICA         PT_CAP
...

Output note

Parameter	Description
`Replica`	The replica location.
`Alive Replica`	The location of active replicas.
`Ever Exist`	Whether the data actually exists.
`Temporary Replica`	For replicated PExtents, a temporary replica is created when a replica fails.
`Is Garbage`	Whether the data needs to be recycled.
`Origin PExtent`	The source PExtent.
`Origin Epoch`	The `Epoch` of the source PExtent.
`Expected Replica num`	The expected replication factor.
`Epoch`	The version number.
`Prefer Local`	The default chunk ID where the PExtent is stored.
`Allocated Space`	The allocated physical space.
`Resiliency Type`	The type of redundancy policy.
`PExtent Type`	Whether it is a capacity PExtent or a performance PExtent.
`EC Param`	The EC policy attributes of the EC redundancy policy.
`Is Sinkable`	Whether it can be demoted.

Viewing detailed information about a PExtent

Procedure

Run the following command on any node in the cluster to view detailed information about a specific PExtent:

zbs-meta pextent show <pextent_id>

Output example

--------------------  ----------
ID                    5
Replica               [3, 1, 2]
Alive Replica         [3, 1, 2]
Temporary Replica     []
Ever Exist            True
Is Garbage            False
Origin PExtent        0
Origin Epoch          0
Expected Replica num  3
Epoch                 5
Prefer Local          3
Allocated Space       2.25 MiB
Resiliency Type       RT_REPLICA
PExtent Type          PT_PERF
EC Param
Is Sinkable           True
--------------------  ----------

Output note

Parameter	Description
`Replica`	The replica location.
`Alive Replica`	The location of active replicas.
`Ever Exist`	Whether the data actually exists.
`Temporary Replica`	For replicated PExtents, a temporary replica is created when a replica fails.
`Is Garbage`	Whether the data needs to be recycled.
`Origin PExtent`	The source PExtent.
`Origin Epoch`	The `Epoch` of the source PExtent.
`Expected Replica num`	The expected replication factor.
`Epoch`	The version number.
`Prefer Local`	The default chunk ID where the PExtent is stored.
`Allocated Space`	The allocated physical space.
`Resiliency Type`	The type of redundancy policy.
`PExtent Type`	Whether it is a capacity PExtent or a performance PExtent.
`EC Param`	The EC policy attributes of the EC redundancy policy.
`Is Sinkable`	Whether it can be demoted.

Viewing the volume that an extent belongs to

Procedure

Run the following command on any node in the cluster to view the volume that the extent belongs to:

zbs-meta pextent getref <pextent_id> [--show_lextent_only]

Parameter	Description
`--show_lextent_only`	Show details only of the associated LExtent. Default: `False`.

Output example

[root@9f29fde8244f zbs-client-py]# zbs-meta pextent getref 1
-------------  -------
Logical ID           1
Logical Epoch        1
Perf PID       5485175
Perf Epoch     8997306
Cap PID              1
Cap Epoch            1
Prioritized      False
-------------  -------
-----------------------  ------------------------------------
Pool Name                ee8e1b5d-be53-4c7d-80df-d651579c5dd4
Volume ID                f08a4c60-e5b9-4046-b388-ecc23fa8e06c
Volume Name              f08a4c60-e5b9-4046-b388-ecc23fa8e06c
Volume Is Snapshot       False
Volume Snapshot Pool ID
Snapshot Name
-----------------------  ------------------------------------

Output note

Parameter	Description
`Logical ID`	The LExtent ID.
`Logical Epoch`	The LExtent version number.
`Perf PID`	The performance tier PExtent ID.
`Perf Epoch`	The performance tier PExtent version number.
`Cap PID`	The capacity tier PExtent ID.
`Cap Epoch`	The capacity tier PExtent version number.
`Prioritized`	Whether it is referenced by a pinned volume.
`Pool Name`	The datastore that the extent belongs to.
`Volume ID`	The ID of the volume that the extent belongs to.
`Volume Name`	The name of the volume that the extent belongs to.
`Volume Is Snapshot`	Whether the volume that the extent belongs to is a snapshot.
`Volume Snapshot Pool ID`	The ID of the datastore to which the snapshot containing the extent belongs.
`Snapshot Name`	The name of the snapshot to which the extent belongs.

Viewing all sub-extents of a specified extent

Procedure

Run the following command on any node in the cluster:

zbs-meta pextent list_children <pextent_id>

Output example

    ID  Replica    Alive Replica    Temporary Replica    Ever Exist    Is Garbage      Origin PExtent    Origin Epoch    Expected Replica num    Epoch    Prefer Local  Allocated Space    Unique Space    Shared Space    Resiliency Type    PExtent Type    EC Param    Is Sinkable
------  ---------  ---------------  -------------------  ------------  ------------  ----------------  --------------  ----------------------  -------  --------------  -----------------  --------------  --------------  -----------------  --------------  ----------  -------------
917439  []         []               []                   False         False                   917343          917343                       2   917439               2  0.00 B             0.00 B          0.00 B          RT_REPLICA         PT_CAP

Output note

Parameter	Description
`Replica`	The replica location.
`Alive Replica`	The location of active replicas.
`Ever Exist`	Whether the data actually exists.
`Temporary Replica`	For replicated PExtents, a temporary replica is created when a replica fails.
`Is Garbage`	Whether the data needs to be recycled.
`Origin PExtent`	The source PExtent.
`Origin Epoch`	The `Epoch` of the source PExtent.
`Expected Replica num`	The expected replication factor.
`Epoch`	The version number.
`Prefer Local`	The default chunk ID where the PExtent is stored.
`Allocated Space`	The allocated physical space.
`Resiliency Type`	The type of redundancy policy.
`PExtent Type`	Whether it is a capacity PExtent or a performance PExtent.
`EC Param`	The EC policy attributes of the EC redundancy policy.
`Is Sinkable`	Whether it can be demoted.

Viewing the virtual machines associated with an extent

Procedure

Run the following command on any node to view the virtual machines associated with a PExtent.

zbs-tool elf get_vm_by_pid <pid>

Parameter	Description
`pid`	The PExtent ID.

Output example

In an ACOS (AVE) cluster, corresponding virtual machine names will be displayed. In an ACOS (VMware ESXi) cluster, related file names will be displayed. You can use file names to further locate related virtual machines in the VMware management interface.

     ID  Replica    Alive Replica    Temporary Replica    Ever Exist    Affected ZBS Volumes                      Affected VMs
-------  ---------  ---------------  -------------------  ------------  ----------------------------------------  ----------------------------------------------
111111  [2, 1]     [2, 1]           []                   False         ['3848f8a2-dd36-43dc-bd5e-90094f46990x']  ['vm-281bb7c2-5ada-4730-ae7b-9c7da2758091-2']

Output note

Parameter	Description
`ID`	The PExtent ID.
`Replica`	The expected PExtent replicas.
`Alive Replica`	The active PExtent replicas.
`Temporary Replica`	For replicated PExtents, a temporary replica is created when a replica fails.
`Ever Exist`	Whether the PExtent has ever been allocated (thin-provisioned PExtents exist only after reads/writes).
`Affected ZBS Volumes`	The IDs of virtual volumes affected by the PExtent. Multiple IDs may be displayed.
`Affected VMs`	The IDs of virtual machines affected by the PExtent. Multiple IDs may be displayed.
`Affected ZBS NFS Files`	The NFS file names corresponding to the virtual volumes affected by the PExtent. Displayed only when the NFS access protocol is used; otherwise, the value is empty.

Searching for PExtents in a specific state

Procedure

Run the following command on any node in the cluster to find PExtents in a specific state:

zbs-meta pextent find <pextent_status> [--start START] [--length Length]

Parameter	Description
`pextent_status`	Searches for PExtents in a specified state: `dead`: PExtents whose replicas are all inaccessible. `need_recover`: PExtents that require data recovery and are currently being recovered. `may_recover`: PExtents available for potential data recovery. They are not currently being recovered, but recovery is pending. `garbage`: PExtents marked for reclamation. `need_drain`: PExtents that require data demotion.
`--start START`	The starting PExtent ID for the search. Default: `1`.
`--length LENGTH`	The number of PExtents to return. Default and maximum: `1024`.

Usage example

$ zbs-meta pextent find need_recover

Output example

   ID  Replica    Alive Replica    Temporary Replica    Ever Exist    Is Garbage      Origin PExtent    Origin Epoch    Expected Replica num    Epoch    Prefer Local  Allocated Space    Resiliency Type    PExtent Type    EC Param    Is Sinkable
----  ---------  ---------------  -------------------  ------------  ------------  ----------------  --------------  ----------------------  -------  --------------  -----------------  -----------------  --------------  ----------  -------------
  22  [3, 2]     [3, 2]           []                   True          False                        0               0                       3       14               0  512.00 MiB         RT_REPLICA         PT_PERF                     True
  23  [3, 2]     [3, 2]           []                   False         False                        0               0                       3       15               3  0.00 B             RT_REPLICA         PT_PERF                     True
  24  [3, 2]     [3, 2]           []                   False         False                        0               0                       3       16               3  0.00 B             RT_REPLICA         PT_PERF                     True
  25  [3, 2]     [3, 2]           []                   False         False                        0               0                       3       17               3  0.00 B             RT_REPLICA         PT_PERF                     True

Searching for dead extents and their impact

When a physical disk fails or a node goes offline, some PExtents may lose all active replicas. These PExtents are then marked as dead extents and become inaccessible. Users can use commands to identify dead extents and the affected virtual volumes, virtual machines, etc.

Procedure

Run the following command on any node in the cluster:

zbs-tool abnormal_detect find_dead_extent [--start START] [--length Length]

Parameter	Description
`--start START`	The starting PExtent ID for the search. Default: `1`.
`--length LENGTH`	The number of PExtents to return. Default and maximum: `1024`.

Output example

The output differs depending on the virtualization platform:

AVE platform

You can view dead extents, affected virtual volumes, and associated NFS file names, and query the affected virtual machines.

Example output:

[root@444541dcac3f zbs-client-py]# zbs-tool abnormal_detect find_dead_extent
ID  Replica    Alive Replica    Temporary Replica    Ever Exist    Is Garbage      Origin PExtent    Origin Epoch    Expected Replica num    Epoch    Prefer Local  Allocated Space    Resiliency Type    PExtent Type    EC Param    Is Sinkable    Affected ZBS Volumes                      Affected ZBS NFS Files    Affected VMs
----  ---------  ---------------  -------------------  ------------  ------------  ----------------  --------------  ----------------------  -------  --------------  -----------------  -----------------  --------------  ----------  -------------  ----------------------------------------  ------------------------  --------------
139  [1, 3]     []               []                   True          False                        0               0                       2      139               1  0.00 B             RT_REPLICA         PT_PERF                     True           ['e6fd591c-2176-47ed-b50c-51a59cfe4a67']  []                        [example_vm]

From the above output, you can identify the following key information:

The PID of the dead extent is 139.
The virtual volume ID affected by the dead extent is e6fd591c-2176-47ed-b50c-51a59cfe4a67;
The affected virtual machine is example_vm.

VMware ESXi platform

You can view dead extents, affected virtual volumes, and corresponding NFS file names. You can further locate the affected virtual machines on VMware ESXi using the NFS file names. Example output:

[root@444541dcac3f zbs-client-py]# zbs-tool abnormal_detect find_dead_extent
ID  Replica    Alive Replica    Temporary Replica    Ever Exist    Is Garbage      Origin PExtent    Origin Epoch    Expected Replica num    Epoch    Prefer Local  Allocated Space    Resiliency Type    PExtent Type    EC Param    Is Sinkable    Affected ZBS Volumes                      Affected ZBS NFS Files
----  ---------  ---------------  -------------------  ------------  ------------  ----------------  --------------  ----------------------  -------  --------------  -----------------  -----------------  --------------  ----------  -------------  ----------------------------------------  ------------------------
105  [3, 2]     []               []                   True          False                        0               0                       2      105               3  1024.00 KiB        RT_REPLICA         PT_PERF                     True           ['cd74ead0-58a8-42d6-81e3-d1057275046a']  [example_vm.vmdk]

From the above output, you can identify the following key information:

The PID of the dead extent is 105.
The virtual volume ID affected by the dead extent is cd74ead0-58a8-42d6-81e3-d1057275046a;
The affected virtual volume corresponds to the NFS file example_vm.vmdk. Use this filename to further locate the affected virtual machine on the VMware ESXi virtualization.

Output note

Parameter	Description
`ID`	The PID of the dead extent.
`Replica`	The expected replicas of the dead extent.
`Alive Replica`	The alive replicas of the dead extent.
`Temporary Replica`	For replicated PExtents, a temporary replica is created when a replica fails.
`Ever Exist`	Whether the dead extent has ever been allocated (thin-provisioned extents exist only after reads or writes).
`Is Garbage`	Whether the dead extent is marked as Garbage. Garbage indicates that the associated virtual volume and snapshots are fully deleted, making this extent no longer useful.
`Origin PExtent`	Whether the dead extent is copied from another extent. If copied, displays the source extent's PID; otherwise, displays as `0`.
`Origin Epoch`	The `Epoch` of the source PExtent.
`Expected Replica num`	The expected replication factor for the dead extent is determined by its parent object's (virtual volume or snapshot) configuration.
`Epoch`	Used together with PID to exclusively identify the attribute value of an extent.
`Prefer Local`	The node ID where the dead extent replica is expected to be allocated, as determined by the replica allocation policy.
`Allocated Space`	The allocated physical space.
`Resiliency Type`	The type of redundancy policy.
`PExtent Type`	Whether it is a capacity PExtent or a performance PExtent.
`EC Param`	The EC policy attributes of the EC redundancy policy.
`Is Sinkable`	Whether it can be demoted.
`Affected ZBS Volumes`	The IDs of virtual volumes affected by the dead extent. Multiple IDs may be displayed.
`Affected ZBS NFS Files`	The NFS file names corresponding to the virtual volumes affected by the dead extent. Displayed only when the NFS access protocol is used; otherwise, the value is empty.
`Affected VMs`	The virtual machines affected by the dead extent. Displayed only on AVE platform; multiple virtual machine names may be displayed.

Writing data read from a PExtent

Procedure

Run the following command on any node in the cluster to write data from the specified PExtent_id into output_file, starting at offset and for a length of length:

zbs-meta pextent read -o <output_file> <pextent_id> <offset> <length>

Parameter	Description
`output_file`	The file in which to store the read data.
`offset`	The starting offset for reading data (relative to the virtual volume), which must be a multiple of 512.
`length`	The length of data to read, which must be a multiple of 512.

Viewing PExtent properties on a chunk

Procedure

Run the following command on the current node to view the properties of a PExtent located on the chunk.

zbs-chunk extent show <pid>

Output example

pid: 5
epoch: 5
generation: 1783
status: 1
private_blob_num: 3
shared_blob_num: 0
thick_provision: False
perf: True

In this article

Viewing extents
Searching for PExtents in a specific state
Searching for dead extents and their impact
Writing data read from a PExtent
Viewing PExtent properties on a chunk