ACOS 6.3.0

Arcfra Cloud Operating System>
CLI command>
Managing chunks

Viewing chunks

Viewing the list of connected chunks in a cluster

Procedure

Run the following command on any node in the cluster:

zbs-meta chunk list [--show_details]

Parameter description

Parameter	Description
`--show_details`	Optional. Specifies whether to display more detailed chunk information.

Output example

  ID  IP            Host Name                                         Port  Storage Pool    Use State    Link Status        Data Capacity    Valid Space    Allocated Space    Failure Data Space    Perf Data Capacity    Perf Valid Space    Perf Allocated Space    Perf Failure Data Space    Registered Date      LSM Version    Zone     Maintenance Mode
----  ------------  ----------------------------------------------  ------  --------------  -----------  -----------------  ---------------  -------------  -----------------  --------------------  --------------------  ------------------  ----------------------  -------------------------  -------------------  -------------  -------  ------------------
   1  10.2.234.198  acos-6-3-0-b45-el7X0617125715X3   10200  system          IN_USE       CONNECTED_ERROR    602.34 GiB       602.34 GiB     0.00 B             0.00 B                20.00 GiB             20.00 GiB           768.00 KiB              0.00 B                     2024-06-17 14:06:12  2.4.1          default  False
   2  10.2.234.196  acos-6-3-0-b45-el7X0617125715X1   10200  system          IN_USE       CONNECTED_HEALTHY  602.34 GiB       602.34 GiB     0.00 B             0.00 B                20.00 GiB             20.00 GiB           256.75 MiB              0.00 B                     2024-06-17 14:06:13  2.4.1          default  False
   3  10.2.234.197  acos-6-3-0-b45-el7X0617125715X2   10200  system          IN_USE       CONNECTED_HEALTHY  602.34 GiB       602.34 GiB     0.00 B             0.00 B                20.00 GiB             20.00 GiB           256.75 MiB              0.00 B                     2024-06-17 14:06:13  2.4.1          default  False

Output note

Parameter	Description
`ID`	The chunk ID.
`IP`	The chunk storage IP.
`HostName`	The chunk node hostname.
`Port`	The chunk RPC port.
`Storage Pool`	The associated storage pool.
`Use State`	The usage status.
`Link Status`	The connection status.
`Data Capacity`	The total data capacity provided by the capacity tier.
`Valid Space`	The valid data space in the capacity tier.
`Allocated Space`	The allocated data space in the capacity tier (space recognized as used by meta).
`Failure Data Space`	The failure data space in the capacity tier.
`Perf Data Capacity`	The total data capacity provided by the performance tier.
`Perf Valid Space`	The valid data space in the performance tier.
`Perf Allocated Space`	The allocated data space in the performance tier (space recognized as used by meta).
`Perf Failure Data Space`	The failure data space in the performance tier.
`Registered Date`	The registration time.
`LSM Version`	The LSM version.
`Zone`	The associated availability zone.
`Maintenance Mode`	Whether it is in maintenance mode.

Viewing the list of chunks on a cluster node

Procedure

Run the following command on any cluster node to view the list of chunks on a specified node in the cluster:

zbs-meta node show <data_ip> [--show_details]

You can also run the following command on any cluster node to view the list of chunks on all nodes in the cluster:

zbs-meta node list [--show_details]

Parameter description

Parameter	Description
`data_ip`	The node storage IP address.
`--show_details`	Optional. Specifies whether to display more detailed node information.

Output example

Node: 10.0.128.55, Host: Node55
  INS ID    CID    Data Port  Status               Cap Size    Cap Valid    Cap Alloc    Perf Size    Perf Valid    Perf Alloc    Register Date
--------  -----  -----------  -------------------  ----------  -----------  -----------  -----------  ------------  ------------  -------------------
       1      2        10201  🟢 CONNECTED_HEALTHY  4.70 TiB    4.70 TiB     10.00 GiB    536.00 GiB   536.00 GiB    270.60 GiB    2024-12-24 11:20:43
       2      4        12201  🟢 CONNECTED_HEALTHY  5.05 TiB    5.05 TiB     0.00 B       475.20 GiB   475.20 GiB    85.52 GiB     2025-01-15 14:32:34
       3      5        14201  🟢 CONNECTED_HEALTHY  5.20 TiB    5.20 TiB     5.00 GiB     594.00 GiB   594.00 GiB    186.90 GiB    2025-01-15 14:32:37
       4      6        16201  🟢 CONNECTED_HEALTHY  5.20 TiB    5.20 TiB     5.00 GiB     594.00 GiB   594.00 GiB    186.90 GiB    2025-01-15 14:37:50

Output note

Parameter	Description
`Node`	The node storage IP.
`Host`	The node hostname.
`INS ID`	The physical disk pool ID.
`CID`	The chunk ID.
`Data Port`	The chunk data communication port.
`Status`	The connection status.
`Cap Size`	The total data capacity provided by the capacity tier.
`Cap Valid`	The valid data space in the capacity tier.
`Cap Alloc`	The allocated data space in the capacity tier (space recognized as used by meta).
`Perf Size`	The total data capacity provided by the performance tier.
`Perf Valid`	The valid data space in the performance tier.
`Perf Alloc`	The allocated data space in the performance tier (space recognized as used by meta).
`Registered Date`	The registration time.

Checking data link connectivity between healthy chunks in the cluster

Procedure

Run the following command on any node in the cluster to check the data link connectivity between chunks that are normally connected to meta:

zbs-meta cluster get_chunk_connectivities

Output example

cid 1: {1: CONNECTED, 2: CONNECTED, 3: CONNECTED, 4: DISCONNECTED}
cid 2: {1: CONNECTED, 2: CONNECTED, 3: CONNECTED, 4: CONNECTED}
cid 3: {1: CONNECTED, 2: CONNECTED, 3: CONNECTED, 4: CONNECTED}
cid 4: {1: DISCONNECTED, 2: CONNECTED, 3: CONNECTED, 4: CONNECTED}

Viewing data link connectivity between a chunk and other chunks in the cluster

Procedure

Run the following command on any node in the cluster to check the data link connectivity between a chunk and other chunks that are normally connected to meta:

zbs-meta cluster show_chunk_connectivity <cid>

Parameter	Description
`cid`	The chunk ID required to obtain data link connectivity with other chunks.

Output example

dst cid: 1, CONNECTED
dst cid: 2, CONNECTED
dst cid: 3, CONNECTED
dst cid: 4, DISCONNECTED

Viewing detailed information about a node chunk

Procedure

Run the following command on any node in the cluster:

zbs-meta chunk show <data_ip> <data_port>

Parameter	Description
`data_ip`	The node storage IP address.
`data_port`	The port used by the chunk.

Output example

-----------------------  ----------------------------------------------
ID                       2
IP                       10.2.234.196
Host Name                5-6-0-b45-el7X0617125715X1
Port                     10200
Storage Pool             system
Use State                IN_USE
Link Status              CONNECTED_HEALTHY
Data Capacity            602.34 GiB
Valid Space              602.34 GiB
Allocated Space          0.00 B
Failure Data Space       0.00 B
Perf Data Capacity       20.00 GiB
Perf Valid Space         20.00 GiB
Perf Allocated Space     256.75 MiB
Perf Failure Data Space  0.00 B
Registered Date          2024-06-17 14:06:13
LSM Version              2.4.1
Zone                     default
Maintenance Mode         False
-----------------------  ----------------------------------------------

Output note

Parameter	Description
`Storage Pool`	The associated storage pool.
`Use State`	The usage status.
`Link Status`	The connection status.
`Data Capacity`	The total data capacity provided by the capacity tier.
`Valid Space`	The valid data space in the capacity tier.
`Allocated Space`	The allocated data space in the capacity tier (space recognized as used by meta).
`Failure Data Space`	The failure data space in the capacity tier.
`Perf Data Capacity`	The total data capacity provided by the performance tier.
`Perf Valid Space`	The valid data space in the performance tier.
`Perf Allocated Space`	The allocated data space in the performance tier (space recognized as used by meta).
`Perf Failure Data Space`	The failure data space in the performance tier.
`Registered Date`	The registration time.
`LSM Version`	The LSM version.
`Zone`	The associated availability zone.
`Maintenance Mode`	Whether it is in maintenance mode.

Viewing the PID information about a node

Procedure

Run the following command on any node in the cluster:

zbs-meta chunk list_pid <chunk_id>

Output example

[root@localhost ~]# zbs-meta chunk list_pid 1
cap pid num: 11
cap thin pid num: 5
cap thick pid num: 6
cap rx pid num: 0
cap tx pid num: 4
cap recover src pid num: 5
cap new thin pid num: 0
cap thin reserved pid num: 0
cap thick reserved pid num: 0

perf pid num: 0
perf thin pid num: 0
perf thick pid num: 0
perf rx pid num: 0
perf tx pid num: 0
perf recover src pid num: 0
perf new thin pid num: 0
perf thin reserved pid num: 0
perf thick reserved pid num: 0

cap pid:                  8930,8938,9107,9112,10193,11230,11231,11234,11246,11250,11254
cap thin pid:             8930,8938,9107,9112,10193
cap thick pid:            11230,11231,11234,11246,11250,11254
cap rx pid:
cap tx pid:               12107,13426,14619,14628
cap recover src pid:      12107,13426,14619,14628,14654
cap new thin pid:
cap thin reserved pid:
cap thick reserved pid:

perf pid:
perf thin pid:
perf thick pid:
perf rx pid:
perf tx pid:
perf recover src pid:
perf new thin pid:
perf thin reserved pid:
perf thick reserved pid:

Output note

Parameter	Description
`cap pid`	The list of PIDs in the capacity tier managed by the chunk.
`cap thin pid`	The thin-provisioned PIDs in the capacity tier.
`cap thick pid`	The thick-provisioned PIDs in the capacity tier.
`cap rx pid`	The PIDs being received in the capacity tier.
`cap tx pid`	The PIDs being sent in the capacity tier.
`perf pid`	The list of PIDs in the performance tier managed by the chunk.
`perf thin pid`	The thin-provisioned PIDs in the performance tier.
`perf thick pid`	The thick-provisioned PIDs in the performance tier.
`perf rx pid`	The PIDs being received in the performance tier.
`perf tx pid`	The PIDs being sent in the performance tier.

Viewing the space utilization of all chunks in a cluster

Procedure

Run the following command on any node in the cluster:

zbs-meta chunk list_space

Output example

  ID  Cap Valid    Cap Alloc    Cap Ratio    Cap Total    Cap Failure    Cap Inherited    Perf Thin/Thick Valid    Perf Thin/Thick Alloc    Perf Thin/Thick Ratio    Perf Valid    Perf Alloc    Perf Total    Perf Failure    Perf Inherited    Prior Percent    Prior Planned    Prior Alloc
----  -----------  -----------  -----------  -----------  -------------  ---------------  -----------------------  -----------------------  -----------------------  ------------  ------------  ------------  --------------  ----------------  ---------------  ---------------  -------------
   1  109.57 TiB   8.48 TiB     7.70%        109.57 TiB   0.00 B         4.81 GiB         (8.82 TiB, 475.33 GiB)   (4.13 TiB, 226.25 GiB)   (46.85%, 47.60%)         9.28 TiB      4.60 TiB      9.28 TiB      0.00 B          0.00 B            5%               475.33 GiB       226.25 GiB
   2  95.87 TiB    12.82 TiB    13.40%       95.87 TiB    0.00 B         19.66 GiB        (7.84 TiB, 422.53 GiB)   (3.54 TiB, 303.25 GiB)   (45.22%, 71.77%)         8.25 TiB      3.96 TiB      8.25 TiB      0.00 B          0.00 B            5%               422.53 GiB       303.25 GiB
   3  41.09 TiB    7.82 TiB     19.00%       41.09 TiB    0.00 B         7.50 GiB         (7.84 TiB, 422.53 GiB)   (3.88 TiB, 305.00 GiB)   (49.43%, 72.18%)         8.25 TiB      4.29 TiB      8.25 TiB      0.00 B          0.00 B            5%               422.53 GiB       305.00 GiB
   4  47.94 TiB    5.42 TiB     11.30%       47.94 TiB    0.00 B         184.25 MiB       (3.92 TiB, 211.27 GiB)   (1.91 TiB, 131.00 GiB)   (48.84%, 62.01%)         4.13 TiB      2.12 TiB      4.13 TiB      0.00 B          0.00 B            5%               211.27 GiB       131.00 GiB
   5  41.09 TiB    898.48 GiB   2.10%        41.09 TiB    0.00 B         0.00 B           (7.84 TiB, 422.53 GiB)   (0.00 B, 507.50 MiB)     (0.00%, 0.12%)           8.25 TiB      422.53 GiB    8.25 TiB      0.00 B          0.00 B            5%               422.53 GiB       0.00 B
   6  109.57 TiB   6.39 TiB     5.80%        109.57 TiB   0.00 B         18.50 MiB        (8.82 TiB, 475.33 GiB)   (4.03 TiB, 223.75 GiB)   (45.69%, 47.07%)         9.28 TiB      4.49 TiB      9.28 TiB      0.00 B          0.00 B            5%               475.33 GiB       223.75 GiB
   7  47.94 TiB    5.08 TiB     10.60%       47.94 TiB    0.00 B         333.50 MiB       (3.92 TiB, 211.27 GiB)   (1.80 TiB, 152.00 GiB)   (46.01%, 71.95%)         4.13 TiB      2.01 TiB      4.13 TiB      0.00 B          0.00 B            5%               211.27 GiB       152.00 GiB

Viewing the chunk information in storage maintenance mode

Obtain information on chunks in storage maintenance mode, including chunk IDs and timeouts.

Procedure

Run the following command on any node in the cluster:

zbs-meta chunk get_maintenance

Output example

maintenance cid: 1
expire time left: 98 s

Output note

Parameter	Description
`maintenance cid`	The ID of the chunk in maintenance mode.
`expire time left`	The maintenance mode expiration time.

Viewing chunk information

View information related to chunks, including supported features, the LSM version, and I/O rates.

Procedure

Run the following command on a cluster node to view the information about a specified physical disk pool chunk:

zbs-chunk [--ins_id <id>] summary

Parameter	Description
`--ins_id <id>`	Optional. The physical disk pool ID. This parameter can be used to filter and display information for a specific physical disk pool in environments with multiple physical disk pools.

Output example

LSM Version:
============================================================
lsm_state                     : LSM_READY
lsm_version                   : 2.4.3
ctime(Instance 1)             : 2025-06-07T14:31:56Z
lsm_uuid(Instance 1)          : 7a1da998-1e48-472c-a7c7-298e8de2cc3d
ctime(Instance 2)             : 2025-06-07T14:31:56Z
lsm_uuid(Instance 2)          : be837875-fe8b-4b03-87a4-3c8712ab3db9

LSM Capability:
============================================================
LSM_CAP_DISK_SAFE_UMOUNT      : True
LSM_CAP_DISK_REJECT_UNHEALTHY : True
LSM_CAP_PARTITION_ISOLATE     : True
LSM_CAP_COMPARE_EXTENT        : True
LSM_CAP_CACHE_ISOLATE         : True

LSM Speed(All instances):
============================================================
LSM IOPS                      : 0 /s
LSM BW                        : 0.00 B/s(0.00 B/s)
Writeback IOPS                : 0 /s
Writeback BW                  : 0.00 B/s(0.00 B/s)
Unmap IOPS                    : 0 /s
Unmap BW                      : 0.00 B/s(0.00 B/s)
Waiting Reclaim Items         : 0

Data Channel Server:
============================================================
Use RDMA                      : False

chunk Capability:
============================================================
Agile Recover                 : True

Storage Server Status:
============================================================
iSCSI                         : Running
NFS                           : Running
Vhost                         : Running
NVMe-oF                       : Running

Flags:
============================================================
metric_on                     : True
trace_on                      : False
adaptive_trace_on             : True
accel_copy_mode               : ACCEL_COPY_DISABLED
chunk_instances_num           : 2

Output note

Parameter	Description
`lsm_state`	The LSM status.
`ctime`	The LSM creation time.
`lsm_uuid`	The LSM UUID.
`lsm_version`	The LSM version.
`LSM_CAP_DISK_SAFE_UMOUNT`	`True` indicates that LSM can safely unmount physical disks.
`LSM_CAP_DISK_REJECT_UNHEALTHY`	`True` indicates that LSM rejects unhealthy physical disks.
`LSM_CAP_PARTITION_ISOLATE`	`True` indicates that LSM can isolate physical data disks.
`LSM_CAP_COMPARE_EXTENT`	`True` indicates that LSM can compare extent data.
`LSM_CAP_CACHE_ISOLATE`	`True` indicates that LSM can isolate physical cache disks.
`LSM IOPS`	The current LSM IOPS.
`LSM BW`	The current LSM bandwidth.
`Writeback IOPS`	The LSM writeback IOPS from the cache partition to the data partition.
`Writeback BW`	The LSM writeback BW from the cache partition to the data partition.
`Unmap IOPS`	The current Unmap IOPS.
`Unmap BW`	The current Unmap bandwidth.
`Waiting Reclaim Items`	The number of extents waiting to be reclaimed.
`Use RDMA`	Whether RDMA is enabled.
`Agile Recover`	Whether agile recovery is enabled. After the chunk exits maintenance mode, agile recovery can complete data recovery more efficiently.
`iSCSI`	The iSCSI server running status.
`NFS`	The NFS server running status.
`Vhost`	The Vhost server running status.
`NVMe-oF`	The NVMe-oF server running status.
`metric_on`	`True` indicates that metric collection is enabled.
`trace_on`	`True` indicates that I/O trace collection is enabled.
`adaptive_trace_on`	`True` indicates that dynamic I/O trace collection is enabled.
`accel_copy_mode`	The status of the hardware-accelerated copy. `ACCEL_COPY_DISABLED` indicates that hardware-accelerated copy is disabled. `ACCEL_COPY_BY_DSA` indicates that hardware-accelerated copy is enabled. `ACCEL_COPY_BY_SOFTWARE` indicates that hardware-accelerated copy failed to be enabled due to a configuration error.
`chunk_instances_num`	Number of physical disk pools on the node.

Viewing chunk counters

The impact of read and write operations on the chunk is temporarily recorded in the chunk's counters.

Procedure

Run the following command on a cluster node to view all counters:

zbs-chunk counter list '*'

Output example

=================== ========== ============= ============== ==============

============ =========== ========== =========== ========== =========

NAME DURATION TOTAL VALUE TOTAL TIME TOTAL CNT

LAST VALUE LAST TIME LAST CNT CUR VALUE CUR TIME CUR CNT

=================== ========== ============= ============== ==============

============ =========== ========== =========== ========== =========

chunk read counter 1 1.66235e+11 1.45566e+14 4.01898e+07

0 0 0 0 0 0

chunk write counter 1 3.30623e+11 5.42942e+14 4.08241e+07

0 0 0 0 0 0

migrate 1 0 0 0

0 0 0 0 0 0

recover 1 2.68435e+08 1024 1024

0 0 0 0 0 0

=================== ========== ============= ============== ==============

============ =========== ========== =========== ========== =========

Viewing node recovery speed

Procedure

Run the following command on a cluster node to view the recovery speed of all or specific physical disk pools on the current node:

zbs-chunk [--ins_id <id>] recover list

Parameter	Description
`--ins_id <id>`	Optional. The physical disk pool ID. This parameter can be used to filter and display information for a specific physical disk pool in environments with multiple physical disk pools.

Output example

Total Recover Speed: 0.00 B/s(0.00 B/s)
From Local Speed: 0.00 B/s(0.00 B/s)
From Remote Speed: 0.00 B/s(0.00 B/s)

Viewing node migration speed

Procedure

Run the following command on a cluster node to view the migration speed of all or specific physical disk pools on the current node:

zbs-chunk [--ins_id <id>] migrate list

Parameter	Description
`--ins_id <id>`	Optional. The physical disk pool ID. This parameter can be used to filter and display information for a specific physical disk pool in environments with multiple physical disk pools.

Output example

Total Migrate Speed: 0.00 B/s(0.00 B/s)
From Local Speed: 0.00 B/s(0.00 B/s)
From Remote Speed: 0.00 B/s(0.00 B/s)

Viewing extents of a local chunk

Procedure

Run the following command on a cluster node to view extents in a specified range within a physical disk pool chunk:

zbs-chunk [--ins_id <id>] extent list [--start <START_PID>] [--length <LENGTH>]

Parameter	Description
`--ins_id <id>`	Optional. The physical disk pool ID. The default value is `1`. This parameter can be used to filter and display information for a specific physical disk pool in environments with multiple physical disk pools.
`--start <START_PID>`	Optional. The starting PID of the extent. The default value is `0`.
`--length <LENGTH>`	Optional. The number of extents. The default and maximum value is `1024`.

Output example

=====  ========  ==============  =========  ===========  ============  ========  =======  =====  =================  ======
  PID    STATUS    NUM CHILDREN    PART ID    EXTENT NO    ORIGIN PID  BITMAP      EPOCH    GEN  THICK PROVISION    PERF
=====  ========  ==============  =========  ===========  ============  ========  =======  =====  =================  ======
    5         1               0          0            0             0                  5   1783  False              True
   22         1               0          0            0             0                 14   1025  False              True
   23         1               0          0            0             0                 15      0  False              True
   24         1               0          0            0             0                 16      0  False              True
   25         1               0          0            0             0                 17      0  False              True
=====  ========  ==============  =========  ===========  ============  ========  =======  =====  =================  ======

Viewing all extents of a local chunk

Procedure

Run the following command on a cluster node to view all the extents in a specified physical disk pool chunk:

zbs-chunk [--ins_id <id>] extent listall --output_file <file>

Parameter	Description
`--ins_id <id>`	Optional. The physical disk pool ID. The default value is `1`. This parameter can be used to filter and display information for a specific physical disk pool in environments with multiple physical disk pools.
`--output_file <file>`	Required. The path of the output file.

Output example

List all extents done.

Viewing the address configuration of local chunks

Procedure

Run the following command on a cluster node to view the address configuration information about all or specific physical disk pools' chunks on the current node:

zbs-chunk [--ins_id <id>] address show

Parameter	Description
`--ins_id <id>`	Optional. The physical disk pool ID. This parameter can be used to filter and display information for a specific physical disk pool in environments with multiple physical disk pools.

Output example

rpc_ip: "10.0.128.25"
rpc_port: 10200
data_ip: "10.0.128.25"
data_port: 10201
data_unix_path: ""
meta_ip: "10.0.128.25"
meta_port: 10100
chunk_id: 4
secondary_data_ip: "20.0.128.25"
scvm_mode_host_data_ip: ""
instance_id: 1

rpc_ip: "10.0.128.25"
rpc_port: 10200
data_ip: "10.0.128.25"
data_port: 12201
data_unix_path: ""
meta_ip: "10.0.128.25"
meta_port: 10100
chunk_id: 9
secondary_data_ip: "20.0.128.25"
scvm_mode_host_data_ip: ""
instance_id: 2

Viewing real-time I/O metric data of a chunk

Procedure

Run the following command on a cluster node to view the I/O Metrics:

zbs-chunk [--ins_id <id>] metric <METRIC_TYPE>

Parameter	Description
`--ins_id <id>`	Optional. The physical disk pool ID. The default value is `1`. This parameter can be used to filter and display information for a specific physical disk pool in environments with multiple physical disk pools.
`METRIC_TYPE`	Specifies the I/O type for which to view real-time metric data. Optional values: `app` (application I/O), `sink` (demotion I/O), `reposition` (recovery and migration I/O), and `fc` (flow control on the performance tier).

Output example

Viewing the metrics of application I/O (METRIC_TYPE = app)

-----------------------------  -----------------------------------------------------------
Access Perf Read               IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Perf Write              IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap EC Read             IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap EC Write            IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap Replica Read        IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap Replica Write       IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Perf Promote Write      IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap EC Promote Read     IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap Rep Promote Read    IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Perf Read          IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Perf Write         IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Cap EC Read        IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Cap EC Write       IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Cap Replica Read   IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Cap Replica Write  IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Perf Read           IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Perf Write          IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Cap EC Read         IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Cap EC Write        IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Cap Replica Read    IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Cap Replica Write   IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Help                           App IO
-----------------------------  -----------------------------------------------------------

Viewing the metrics of demotion I/O (METRIC_TYPE = sink)

-----------------------------  ------------------------------------------------------------------------------------------------
Access Perf Read               IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Perf Write              IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap EC Read             IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap EC Write            IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap Replica Read        IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap Replica Write       IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Perf Promote Write      IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap EC Promote Read     IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap Rep Promote Read    IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Perf Read          IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Perf Write         IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Cap EC Read        IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Cap EC Write       IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Cap Replica Read   IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Cap Replica Write  IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Perf Read           IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Perf Write          IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Cap EC Read         IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Cap EC Write        IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Cap Replica Read    IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Cap Replica Write   IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Help                           -----------------------------------------------------------------------------------------------
                               * Local: sink io triggered by this chunk; Remote: sink io from by other chunk.

                               * perf write:                    promote during sink & unmap
                               * perf read:                     sink speed
                               * cap ec read:                   ec read during block-granularity update + promote for ec block-size alignment.
                               * cap replica read:              should always be 0.
                               * cap write (both ec & replica):         sink speed with write amplification
                               * ec perf promote write:                 promote for ec block-size alignment (write part).
                               * cap ec promote read:           promote for ec block-size alignment (read part).
                               * cap replica promote read:      should always be 0.
-----------------------------  ------------------------------------------------------------------------------------------------

Viewing the metrics of recovery and migration I/O (METRIC_TYPE = reposition)

-----------------------------  -----------------------------------------------------------
Access Perf Read               IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Perf Write              IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap EC Read             IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap EC Write            IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap Replica Read        IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap Replica Write       IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Perf Promote Write      IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap EC Promote Read     IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Access Cap Rep Promote Read    IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Perf Read          IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Perf Write         IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Cap EC Read        IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Cap EC Write       IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Cap Replica Read   IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Remote Cap Replica Write  IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Perf Read           IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Perf Write          IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Cap EC Read         IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Cap EC Write        IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Cap Replica Read    IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
From Local Cap Replica Write   IOPS: 0, BW: 0.000 MiB/s, Lat: 0.000 ms, ReqSize: 0.000 KiB
Help                           Recover or Migrate IO
-----------------------------  -----------------------------------------------------------

Viewing the metrics of performance tier flow control (METRIC_TYPE = fc)

[Flow Manager]
Flow control enable: False
Avail tokens: 2
Perf thin used ratio: 1.0%
Perf thin free ratio: 98.0%
Flow Controller      Requested Tokens    Used Tokens    Over Used Tokens
-----------------  ------------------  -------------  ------------------
1 -> 1                              0              0                   0
2 -> 1                              0              0                   0
3 -> 1                              0              0                   0
4 -> 1                              0              0                   0

[Flow Controller]
Flow Controller    Dest Perf Thin Not Free Ratio      Avail Tokens    Used Tokens No Wait    Used Tokens After Wait    Over Used Tokens    Avg Wait token Lat    Avg Wait token Num
-----------------  -------------------------------  --------------  ---------------------  ------------------------  ------------------  --------------------  --------------------
1 -> 1             2.0%                                   10000000                      0                         0                   0                     0                     0
1 -> 2             7.0%                                   10000000                      0                         0                   0                     0                     0
1 -> 3             7.0%                                   10000000                      0                         0                   0                     0                     0
1 -> 4             2.0%                                   10000000                      0                         0                   0                     0                     0

Output note

I/O metrics output

Parameter	Description
`Access Perf Read`	The performance tier read I/O initiated by access.
`Access Perf Write`	The performance tier write I/O initiated by access.
`Access Cap EC Read`	The capacity tier EC read I/O initiated by access.
`Access Cap EC Write`	The capacity tier EC write I/O initiated by access.
`Access Cap Replica Read`	The capacity tier replica read I/O initiated by access.
`Access Cap Replica Write`	The capacity tier replica write I/O initiated by access.
`Access Perf Promote Write`	The promote I/O initiated by access to match a specific I/O length size, writing to the performance tier portion.
`Access Cap EC Promote Read`	The promote I/O initiated by access to match a specific I/O length size, reading from the capacity tier EC portion.
`Access Cap Rep Promote Read`	The promote I/O initiated by access to match a specific I/O length size, reading from the capacity tier replica portion.
`From Remote Perf Read`	The performance tier read I/O from remote access.
`From Remote Perf Write`	The performance tier write I/O from remote access.
`From Remote Cap EC Read`	The capacity tier EC read I/O from remote access.
`From Remote Cap EC Write`	The capacity tier EC write I/O from remote access.
`From Remote Cap Replica Read`	The capacity tier replica read I/O from remote access.
`From Remote Cap Replica Write`	The capacity tier replica write I/O from remote access.
`From Local Perf Read`	The performance tier read I/O from local access.
`From Local Perf Write`	The performance tier write I/O from local access.
`From Local Cap EC Read`	The capacity tier EC read I/O from local access.
`From Local Cap EC Write`	The capacity tier EC write I/O from local access.
`From Local Cap Replica Read`	The capacity tier replica read I/O from local access.
`From Local Cap Replica Write`	The capacity tier replica write I/O from local access.

Flow manager metrics of performance tier flow control

Parameter	Description
`Flow Manager`	The module that allocates tokens: source chunk (allocates tokens) → destination chunk (requests tokens).
`Flow control enable`	Whether performance tier flow control is enabled.
`Avail token`	The number of currently available tokens.
`Perf thin used ratios`	The current usage ratio of the performance tier.
`Perf thin free ratio`	The current free ratio of the performance tier.
`Flow Controller`	The module that requests tokens: source chunk (requests tokens) → destination chunk (allocates tokens).
`Requested Tokens`	The number of tokens requested in the last 1 second (mostly for reference, can generally be ignored).
`Used Tokens`	The number of tokens consumed in the last 1 second.
`Over Used Tokens`	The number of replica I/Os issued in the last 1 second without obtaining a token, typically due to DataChannel disconnection, token acquisition timeout, or similar issues.

Flow controller metrics of performance tier flow control

Parameter	Description
`Flow Controller`	The module that requests tokens: source chunk (requests tokens) → destination chunk (allocates tokens).
`Dest Perf Thin Not Free Ratio`	The usage ratio of the performance tier space on the destination chunk.
`Avail Tokens`	The tokens currently held for the destination node. When flow control is not enabled, the value is fixed at `10000000`.
`Used Tokens No Wait`	The number of replica I/Os in the last 1 second that obtained tokens without waiting.
`Used Tokens After Wait`	The number of replica I/Os in the last 1 second that obtained tokens after waiting.
`Over Used Tokens`	The number of replica I/Os issued without obtaining a token, typically due to DataChannel disconnection, token acquisition timeout, or similar issues.
`Avg Wait Token Lat`	The average latency of waiting for tokens in the last 1 second.
`Avg Wait Token Num`	The number of replica I/Os in the last 1 second that wait for tokens.

Viewing the demotion information about a local chunk

Run the following command on a cluster node to view the demotion information:

zbs-chunk [--ins_id <id>] sink list

Parameter	Description
`--ins_id <id>`	Optional. The physical disk pool ID. This parameter can be used to filter and display information for a specific physical disk pool in environments with multiple physical disk pools.

Output example

Sink mode: SINK_HIGH
Inactive interval: 900 s
Sink inactive lease: True
Cap directly write policy: CAP_DIO_ALL_THROTTLED
Sinkable block lru info:
  Active blocks: 4914422
  Inactive blocks: 631664
  Clean blocks: 0
  Reserve active blocks: 6556339
Potential sinkable block lru info:
  Active blocks: 0
  Inactive blocks: 0
  Clean blocks: 0
  Reserve active blocks: 6556339
Accelerate cids: [4]
Accelerate blocks: 179
Inactive extents: 11125
Max sink task num: 32
Max sink io concurrency: 32
    lid    lepoch    perf_pid    perf_epoch  perf_location      cur_block_no  type
-------  --------  ----------  ------------  ---------------  --------------  -----------
2729295  13723839     2758531      39830494  [2, 3]                      454  SINK_EXTENT
2900678  13896894     4661280      41724371  [2, 4]                      241  SINK_BLOCK
2961581  14890778     8191744      44708358  [2, 1]                      116  DRAIN_EXTENT

Output note

Parameter	Description
`Sink mode`	The demotion policy level on the current node.
`Inactive interval`	The time threshold to determine whether an extent is inactive.
`Sink inactive lease`	Whether to demote inactive extents.
`Cap directly write policy`	The direct-write policy of the capacity tier on the current node.
`Sinkable block lru info`	The LRU information about demotable blocks on the current node.
`Potential sinkable block lru info`	The LRU information about potentially demotable blocks on the current node.
`Accelerate cids`	The list of chunks that require accelerated demotion as identified by the current node.
`Accelerate blocks`	The number of blocks that require accelerated demotion as identified by the current node.
`Inactive extents`	The number of inactive extents on the current node.
`Max sink task num`	The maximum number of demotion tasks on the current node.
`Max sink io concurrency`	The maximum I/O concurrency for demotion on the current node.

Viewing the data link information about a local chunk

Run the following command on a cluster node to view its data link information as a client connected to other nodes:

zbs-chunk dc get_manager_info

Parameter	Description
`--dst_ip`	Optional. Specifies whether to display only data links for the specified destination IP address.
`--type`	Optional. Specifies whether to display only data links for the storage network or the access network.
`--show_details`	Optional. Specifies whether to display detailed information.
`--show_pids`	Optional. Specifies whether to display all PID information for each data link.
`--summary`	Optional. Specifies whether to display only summary information.

Output example

Version: DATA_CHANNEL_V3
Src                  Dst                  Type      Up Time (sec)
-------------------  -------------------  ------  ---------------
10.10.130.72:32830   10.10.130.73:11201   RDMA              52374
10.10.130.72:37114   10.10.130.74:11201   RDMA              54065
10.10.130.72:45090   10.10.130.72:11201   RDMA              54065
10.10.130.72:46660   10.10.130.71:11201   RDMA              53334
10.199.130.72:46482  10.199.130.71:10201  TCP               53334
10.199.130.72:57680  10.199.130.74:10201  TCP               54067
10.199.130.72:59436  10.199.130.73:10201  TCP               52374

Output note

Parameter	Description
`Version`	The version of the data link.
`Src`	The IP address and port used by the local node for the data link.
`Dst`	The IP address and port used by the peer node for the data link.
`Type`	The data link type, which can be TCP or RDMA.
`Up Time (sec)`	The duration for which the data link has been active.

Run the following command on a cluster node to view its data link information as a server connected to other nodes:

zbs-chunk dc get_server_info

Parameter	Description
`--show_details`	Optional. Specifies whether to display detailed information.

Output example

Version: DATA_CHANNEL_V3
Src                  Dst                  Type      Up Time (sec)
-------------------  -------------------  ------  ---------------
10.10.130.72:10201   10.10.130.72:52768   TCP                5551
10.10.130.72:11201   10.10.130.73:45276   RDMA                 26
10.10.130.72:11201   10.10.130.74:46626   RDMA                411
10.10.130.72:11201   10.10.130.72:45090   RDMA              54559
10.10.130.72:11201   10.10.130.71:52162   RDMA              53825
10.199.130.72:10201  10.199.130.73:40152  TCP                  27
10.199.130.72:10201  10.199.130.74:56334  TCP                 413
10.199.130.72:10201  10.199.130.71:55514  TCP               53828

Output note

Parameter	Description
`Version`	The version of the data link.
`Src`	The IP address and port used by the local node for the data link.
`Dst`	The IP address and port used by the peer node for the data link.
`Type`	The data link type, which can be TCP or RDMA.
`Up Time (sec)`	The duration for which the data link has been active.

In this article

Viewing the list of connected chunks in a cluster
Viewing the list of chunks on a cluster node
Checking data link connectivity between healthy chunks in the cluster
Viewing data link connectivity between a chunk and other chunks in the cluster
Viewing detailed information about a node chunk
Viewing the PID information about a node
Viewing the space utilization of all chunks in a cluster
Viewing the chunk information in storage maintenance mode
Viewing chunk information
Viewing chunk counters
Viewing node recovery speed
Viewing node migration speed
Viewing extents of a local chunk
Viewing all extents of a local chunk
Viewing the address configuration of local chunks
Viewing real-time I/O metric data of a chunk
Viewing the demotion information about a local chunk
Viewing the data link information about a local chunk