Procedure
Run the following command on any node in the cluster to create a session with the specified name.
zbs-perf-tools trace session create [--session <session_name>]
| Parameter | Description |
|---|---|
--session <session_name> | The session name. |
After executing the command, a folder named after the session name will be generated in the /root/zbs-trace/ directory.
Output note
If the command succeeds, no output is displayed.
Procedure
Run the following command on any node in the cluster to view existing sessions.
zbs-perf-tools trace session list
Output example
------------------
name session1
started false
------------------Procedure
Run the following command in the /root/zbs-trace/<session name> directory on any node in the cluster to add a target to the session:
zbs-perf-tools trace session add-target <TARGET NAME> <TARGET HOST> [--event]
| Parameter | Description |
|---|---|
TARGET NAME | The target name. Target names within the same session must be unique. |
TARGET HOST | The IP address of the node where the target is located. |
--event | An array-type parameter. The event names to trace. Currently supported values are zbs_client:*, access:*, and lsm2:*. |
Output note
If the command succeeds, no output is displayed.
When collecting data for a specified volume, you can add multiple targets using a JSON file.
Procedure
Run the following command in the /root/zbs-trace/<session name> directory on any node in the cluster:
zbs-perf-tools trace session add-target --from-json <JSON FILE>
| Parameter | Description |
|---|---|
--from-json | Adds targets using a JSON file. |
JSON FILE | The JSON file name. |
Output note
If the command succeeds, no output is displayed.
Procedure
Run the following command in the /root/zbs-trace/<session name> directory on any node in the cluster to update the target of a session:
zbs-perf-tools trace session update-target <TARGET NAME> [--remove-event <stringArray>] [--add-event <stringArray>]
| Parameter | Description |
|---|---|
--remove-event <stringArray> | An array-type parameter. Removes events. |
--add-event <stringArray> | An array-type parameter. Adds events. |
Output note
If the command succeeds, no output is displayed.
Procedure
Run the following command in the /root/zbs-trace/<session name> directory on any node in the cluster to remove the target from a session:
zbs-perf-tools trace session remove-target <TARGET NAME>
Output note
If the command succeeds, no output is displayed.
Procedure
Run the following command in the /root/zbs-trace/<session name> directory on any node in the cluster to start the session:
zbs-perf-tools trace session start [-d]
If you specify the -d option, the session will start in the background.
After stopping the session, the target trace data will be generated in the /root/zbs-trace/<session name>/trace-data/ directory.
Output example
$zbs-perf-tools trace session start
session started...Procedure
Run the following command on any node in the cluster:
zbs-perf-tools trace session destroy
Output note
If the command succeeds, no output is displayed.
Procedure
Run the following command on any node in the cluster to output the collected performance data to a specified directory:
zbs-perf-tools trace analyze parse <MODULE> <PATH> <OUTPUT DIR> [--time_begin] [--time_end] [--volume] [--slow_io_latency]
| Parameter | Description |
|---|---|
MODULE | The module to analyze. It can be client, access, lsm, or all, where all represents all modules. |
PATH | The trace data directory. Needs to specify the time level. |
OUTPUT DIR | The output directory. |
--time_begin | The starting time for parsing. |
--time_end | The end time for parsing. Must be used with --time_begin. |
--volume | Specifies the JSON file generated by zbs-perf-tools volume gentrace [VOLUME ID]. |
--slow_io_latency | Outputs I/Os that exceed the specified latency to the [OUTPUT DIR]/[MODULE].slow_io file. The unit: ms. Default: 200. |
Output note
If the command succeeds, no output is displayed.
Procedure
Run the following command on any node in the cluster to view the performance data analysis results of ABS modules:
zbs-perf-tools trace analyze report <MODULE> <DIR>
| Parameter | Description |
|---|---|
MODULE | The module to analyze. It can be client, access, lsm, or all, where all represents all modules. |
DIR | The parsed data directory. |
Output example
Analyze data for the Client module
zbs-perf-tools trace analyze report client cid-2
CLIENT CID 2
---------------------------------------------------------------------------
OP AVG P50 P95 P99 MAX N
---------------------------------------------------------------------------
read 416.10 US 280.58 US 544.77 US 11.60 MS 11.65 MS 82
write 1.21 MS 544.77 US 6.13 MS 12.65 MS 87.86 MS 1378248
readwrite 1.21 MS 544.77 US 6.13 MS 12.65 MS 87.86 MS 1378330
---------------------------------------------------------------------------
TO LEASE OWNER
-----------------------------------------------------------------------------------
TO CID OP AVG P50 P95 P99 MAX N
-----------------------------------------------------------------------------------
2 read 416.10 US 280.58 US 544.77 US 11.60 MS 11.65 MS 82
write 1.21 MS 544.77 US 6.13 MS 12.65 MS 87.86 MS 1378248
readwrite 1.21 MS 544.77 US 6.13 MS 12.65 MS 87.86 MS 1378330
-----------------------------------------------------------------------------------
FAILED COUNT: 0.000000Analyze data for the access module
$zbs-perf-tools trace analyze report access cid-2
ACCESS
----------------------------------------------------------------------------------------
AVG P50 P95 P99 MAX N
----------------------------------------------------------------------------------------
read 422.59 US 244.74 US 561.15 US 11.47 MS 11.53 MS 88
write 1.21 MS 544.77 US 6.13 MS 12.52 MS 87.86 MS 1378249
readwrite 1.21 MS 544.77 US 6.13 MS 12.52 MS 87.86 MS 1378337
sync_gen 912.89 US 468.99 US 3.88 MS 6.91 MS 11.38 MS 239
wait_recover 625.62 NS 540.00 NS 924.00 NS 1.18 US 429.49 US 1378629
caw 544.34 NS 490.00 NS 724.00 NS 900.00 NS 120.19 US 1378630
replica_io_read 284.56 US 236.54 US 544.77 US 2.69 MS 2.69 MS 88
replica_io_write 815.45 US 399.36 US 3.42 MS 10.29 MS 87.86 MS 2756640
replica_io_readwrite 815.44 US 399.36 US 3.42 MS 10.29 MS 87.86 MS 2756728
----------------------------------------------------------------------------------------
TO REPLICA
-----------------------------------------------------------------------------------
TO CID OP AVG P50 P95 P99 MAX N
-----------------------------------------------------------------------------------
1 read 0.00 NS 0.00 NS 0.00 NS 0.00 NS 0.00 NS 0
write 1.19 MS 536.58 US 6.13 MS 12.52 MS 17.79 MS 1378241
readwrite 1.19 MS 536.58 US 6.13 MS 12.52 MS 17.79 MS 1378241
2 read 284.56 US 236.54 US 544.77 US 2.69 MS 2.69 MS 88
write 443.16 US 264.19 US 1.66 MS 3.65 MS 87.86 MS 1378399
readwrite 443.15 US 264.19 US 1.66 MS 3.65 MS 87.86 MS 1378487
-----------------------------------------------------------------------------------Analyze data for the LSM module
$zbs-perf-tools trace analyze report lsm cid-2
LSM
---------------------------------------------------------------------------------------
AVG P50 P95 P99 MAX N
---------------------------------------------------------------------------------------
read 207.65 US 162.82 US 444.42 US 733.18 US 737.24 US 88
write 276.36 US 234.50 US 468.99 US 626.69 US 87.98 MS 1380604
readwrite 276.36 US 234.50 US 468.99 US 626.69 US 87.98 MS 1380692
throttle 2.29 US 1.82 US 5.09 US 9.02 US 129.51 US 1380819
lock_range 1.37 US 700.00 NS 2.22 US 5.86 US 10.46 MS 1380699
reserve_journal 984.42 NS 764.00 NS 2.51 US 5.98 US 129.63 US 1380611
tx_commit 206.83 US 175.10 US 399.36 US 528.38 US 87.85 MS 1380608
cache_read 203.65 US 158.72 US 440.32 US 733.18 US 733.73 US 88
cache_write 63.49 US 42.24 US 123.39 US 205.82 US 25.86 MS 1380604
cache_readwrite 63.50 US 42.24 US 123.39 US 205.82 US 25.86 MS 1380692
partition_read 0.00 NS 0.00 NS 0.00 NS 0.00 NS 0.00 NS 0
partition_write 0.00 NS 0.00 NS 0.00 NS 0.00 NS 0.00 NS 0
partition_readwrite 0.00 NS 0.00 NS 0.00 NS 0.00 NS 0.00 NS 0
---------------------------------------------------------------------------------------
TX COMMIT
--------------------------------------------------------------------------------
PATH AVG P50 P95 P99 MAX N
--------------------------------------------------------------------------------
/dev/sdc1 265.18 US 240.64 US 481.28 US 577.54 US 2.03 MS 414317
/dev/nvme1n1p3 189.75 US 148.48 US 313.34 US 448.51 US 87.85 MS 438224
/dev/nvme0n1p3 175.24 US 150.53 US 317.44 US 428.03 US 65.05 MS 528067
--------------------------------------------------------------------------------
CACHE
--------------------------------------------------------------------------------------------
PATH OP AVG P50 P95 P99 MAX N
--------------------------------------------------------------------------------------------
/dev/nvme0n1p4 read 237.58 US 203.78 US 432.13 US 460.80 US 462.31 US 26
write 57.98 US 42.24 US 122.37 US 195.58 US 19.65 MS 678635
readwrite 57.98 US 42.24 US 122.37 US 195.58 US 19.65 MS 678661
/dev/nvme1n1p4 read 189.42 US 134.14 US 440.32 US 733.18 US 733.73 US 62
write 68.82 US 42.24 US 123.39 US 234.50 US 25.86 MS 701969
readwrite 68.83 US 42.24 US 123.39 US 234.50 US 25.86 MS 702031
--------------------------------------------------------------------------------------------
FAILED: 0
COW PARERENT: 0
PROMOTION BLOCK: 0
RECOVER: 0
SYNCGEN: 120
READWRITE CACHE HIT RATIO: 1
READ CACHE HIT RATIO: 1
WRITE CACHE HIT RATIO: 1Procedure
Run the following command on any node in the cluster to output the latency distribution histogram between paired events. Paired events are usually the start and end of an I/O event, or two consecutive events.
zbs-perf-tools trace analyze latdist <TRACE DATA DIR> <TRACE_BEGIN> <TRACE_END> <TRACE_ID>
| Parameter | Description |
|---|---|
TRACE DATA DIR | The trace data directory created at the start of the session. |
TRACE_BEGIN | The name of the first trace event in the paired trace events. |
TRACE_END | The name of the second trace event in the paired trace events. |
TRACE_ID | The field name used to correlate TRACE_BEGIN and TRACE_END. |
Output example
$zbs-perf-tools trace analyze latdist trace-data/cid-2-10-168-57-62/2022-04-06-083408+0800/ zbs_client:io_begin zbs_client:io_end trace_id
lat : count distribution
0 -> 64.00 : 3 |* |
64.00 -> 128.00 : 62 |* |
128.00 -> 256.00 : 8730 |* |
256.00 -> 512.00 : 567272 |***************************** |
512.00 -> 1024.00 : 639739 |********************************|
1024.00 -> 2048.00 : 18695 |* |
2048.00 -> 4096.00 : 35172 |** |
4096.00 -> 8192.00 : 63875 |**** |
8192.00 -> 16384.00 : 44440 |*** |
16384.00 -> 32768.00 : 198 |* |
32768.00 -> +inf : 144 |* |