Shutting down or restarting nodes

Shut down or restart one or more nodes.

Note:

Before shutting down or restarting nodes, make sure to log-in to AOC. Access the virtual machine list of the cluster, and confirm that the target host either has no virtual machines or that the virtual machines are in the Shutdown state.

Procedure

zbs-cluster shutdown_hosts [--hosts <HOST_IPS> --action {reboot|poweroff}] [--network {mgt|storage} --reason <TEXT>]

Parameter	Description	Example
`--hosts`	The IP address of the target node. To operate on multiple nodes simultaneously, enter multiple node IP addresses separated by commas (`,`). If you want to shut down all nodes, use `all` instead of all host IPs.	`--hosts 10.172.11.111,10.172.11.112`
`--action`	The power management action. `reboot` indicates restarting the hosts; `poweroff` indicates shutting down the hosts.	`--action reboot`
`--network`	Enter the network type corresponding to the node IP address. `mgt` indicates the management network, and `storage` indicates the storage network. Defaults to the storage network.	`--network mgt`
`--reason`	The reason for the shutdown or restart. This will be recorded in the audit log.	`--reason upgrade kernel`

Operation example

Restarting a node

$ zbs-cluster shutdown_hosts --hosts=10.172.20.111 --action=reboot --network=storage

Shutting down multiple nodes

$ zbs-cluster shutdown_hosts --hosts=192.168.22.111,192.168.22.112 --action=poweroff --network=mgt

Shutting down the entire cluster

$ zbs-cluster shutdown_hosts --hosts=all --action=poweroff --network=storage

Output example

$sudo zbs-cluster shutdown_hosts --hosts=10.168.22.112 --action=reboot
2023-04-17 17:24:06,241 cmdline.py 169 [2697] [INFO]
2023-04-17 17:24:06,251 cmdline.py 169 [2697] [INFO] PLAY [10.168.22.112] ***********************************************************
2023-04-17 17:24:06,252 cmdline.py 169 [2697] [INFO]
2023-04-17 17:24:06,253 cmdline.py 169 [2697] [INFO] TASK [fail] ********************************************************************
2023-04-17 17:24:06,312 cmdline.py 169 [2697] [INFO] skipping: [10.168.22.112]
2023-04-17 17:24:06,313 cmdline.py 169 [2697] [INFO]
2023-04-17 17:24:06,313 cmdline.py 169 [2697] [INFO] TASK [gather host ip address facts] ********************************************
2023-04-17 17:24:10,728 cmdline.py 169 [2697] [INFO] ok: [10.168.22.112]
2023-04-17 17:24:10,728 cmdline.py 169 [2697] [INFO]
2023-04-17 17:24:10,728 cmdline.py 169 [2697] [INFO] TASK [set_fact] ****************************************************************
2023-04-17 17:24:10,793 cmdline.py 169 [2697] [INFO] ok: [10.168.22.112]
2023-04-17 17:24:10,793 cmdline.py 169 [2697] [INFO]
2023-04-17 17:24:10,793 cmdline.py 169 [2697] [INFO] TASK [stop vipservice for migrate vip to other hosts before shutdown] **********
2023-04-17 17:24:14,663 cmdline.py 169 [2697] [INFO] changed: [10.168.22.112]
2023-04-17 17:24:14,664 cmdline.py 169 [2697] [INFO]
2023-04-17 17:24:14,664 cmdline.py 169 [2697] [INFO] TASK [stop non-zbs services] ***************************************************
2023-04-17 17:25:35,378 cmdline.py 169 [2697] [INFO] changed: [10.168.22.112]
2023-04-17 17:25:35,379 cmdline.py 169 [2697] [INFO]
2023-04-17 17:25:35,379 cmdline.py 169 [2697] [INFO] TASK [gather running vm list on the target host] *******************************
2023-04-17 17:25:38,037 cmdline.py 169 [2697] [INFO] fatal: [10.168.22.112]: FAILED! => {"changed": true, "cmd": ["timeout", "-k", "300", "300", "virsh", "list", "--all", "--state-running", "--uuid"], "delta": "0:00:00.028824", "end": "2023-04-17 17:25:37.586861", "msg": "non-zero return code", "rc": 1, "start": "2023-04-17 17:25:37.558037", "stderr": "error: failed to connect to the hypervisor\nerror: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory", "stderr_lines": ["error: failed to connect to the hypervisor", "error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory"], "stdout": "", "stdout_lines": []}
2023-04-17 17:25:38,037 cmdline.py 169 [2697] [INFO] ...ignoring
2023-04-17 17:25:38,037 cmdline.py 169 [2697] [INFO]
2023-04-17 17:25:38,037 cmdline.py 169 [2697] [INFO] TASK [stop all running vm by virsh shutdown] ***********************************
2023-04-17 17:25:38,076 cmdline.py 169 [2697] [INFO] skipping: [10.168.22.112]
2023-04-17 17:25:38,076 cmdline.py 169 [2697] [INFO]
2023-04-17 17:25:38,076 cmdline.py 169 [2697] [INFO] TASK [virsh shutdown result] ***************************************************
2023-04-17 17:25:38,132 cmdline.py 169 [2697] [INFO] ok: [10.168.22.112] => {
2023-04-17 17:25:38,133 cmdline.py 169 [2697] [INFO]     "msg": {
2023-04-17 17:25:38,133 cmdline.py 169 [2697] [INFO]         "changed": false,
2023-04-17 17:25:38,133 cmdline.py 169 [2697] [INFO]         "skip_reason": "Conditional result was False",
2023-04-17 17:25:38,133 cmdline.py 169 [2697] [INFO]         "skipped": true
2023-04-17 17:25:38,133 cmdline.py 169 [2697] [INFO]     }
2023-04-17 17:25:38,133 cmdline.py 169 [2697] [INFO] }
2023-04-17 17:25:38,133 cmdline.py 169 [2697] [INFO]
2023-04-17 17:25:38,133 cmdline.py 169 [2697] [INFO] TASK [stop all running vm by virsh destroy] ************************************
2023-04-17 17:25:38,171 cmdline.py 169 [2697] [INFO] skipping: [10.168.22.112]
2023-04-17 17:25:38,171 cmdline.py 169 [2697] [INFO]
2023-04-17 17:25:38,171 cmdline.py 169 [2697] [INFO] TASK [virsh destroy result] ****************************************************
2023-04-17 17:25:38,231 cmdline.py 169 [2697] [INFO] ok: [10.168.22.112] => {
2023-04-17 17:25:38,231 cmdline.py 169 [2697] [INFO]     "msg": {
2023-04-17 17:25:38,231 cmdline.py 169 [2697] [INFO]         "changed": false,
2023-04-17 17:25:38,231 cmdline.py 169 [2697] [INFO]         "skip_reason": "Conditional result was False",
2023-04-17 17:25:38,232 cmdline.py 169 [2697] [INFO]         "skipped": true
2023-04-17 17:25:38,232 cmdline.py 169 [2697] [INFO]     }
2023-04-17 17:25:38,232 cmdline.py 169 [2697] [INFO] }
2023-04-17 17:25:38,232 cmdline.py 169 [2697] [INFO]
2023-04-17 17:25:38,232 cmdline.py 169 [2697] [INFO] TASK [umount nfs mount point] **************************************************
2023-04-17 17:25:40,979 cmdline.py 169 [2697] [INFO] changed: [10.168.22.112]
2023-04-17 17:25:40,979 cmdline.py 169 [2697] [INFO]
2023-04-17 17:25:40,979 cmdline.py 169 [2697] [INFO] TASK [stop all acos system services] *****************************************
2023-04-17 17:25:59,166 cmdline.py 169 [2697] [INFO] changed: [10.168.22.112]
2023-04-17 17:25:59,166 cmdline.py 169 [2697] [INFO]
2023-04-17 17:25:59,167 cmdline.py 169 [2697] [INFO] TASK [shutdown system by reboot on 10.168.22.112] ******************************
2023-04-17 17:26:03,978 cmdline.py 169 [2697] [INFO] fatal: [10.168.22.112]: FAILED! => {"changed": false, "module_stderr": "Connection to 10.168.22.112 closed.\r\n", "module_stdout": "/bin/sh: line 1: 25671 Terminated              sudo -H -S -n -u root /bin/sh -c 'echo BECOME-SUCCESS-tarxkpmgivpmzdmewmkzqjbjfjlyjhbf ; /usr/bin/python /home/arcfra/.ansible/tmp/ansible-tmp-1681723559.5943196-3544-107651429146572/AnsiballZ_command.py'\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 143}
2023-04-17 17:26:03,978 cmdline.py 169 [2697] [INFO] ...ignoring
2023-04-17 17:26:03,978 cmdline.py 169 [2697] [INFO]
2023-04-17 17:26:03,978 cmdline.py 169 [2697] [INFO] TASK [wait for system shutdown] ************************************************
2023-04-17 17:26:09,563 cmdline.py 169 [2697] [INFO] ok: [10.168.22.112 -> localhost]
2023-04-17 17:26:09,564 cmdline.py 169 [2697] [INFO]
2023-04-17 17:26:09,564 cmdline.py 169 [2697] [INFO] TASK [wait for system finished reboot] ****************************************
2023-04-17 17:27:32,801 cmdline.py 169 [2697] [INFO] FAILED - RETRYING: [10.168.22.112 -> localhost]: wait for system finished reboot (120 retries left).
2023-04-17 17:27:32,802 cmdline.py 169 [2697] [INFO] FAILED - RETRYING: [10.168.22.112 -> localhost]: wait for system finished reboot (119 retries left).
2023-04-17 17:27:32,802 cmdline.py 169 [2697] [INFO] FAILED - RETRYING: [10.168.22.112 -> localhost]: wait for system finished reboot (118 retries left).
2023-04-17 17:27:32,802 cmdline.py 169 [2697] [INFO] FAILED - RETRYING: [10.168.22.112 -> localhost]: wait for system finished reboot (117 retries left).
2023-04-17 17:27:32,802 cmdline.py 169 [2697] [INFO] FAILED - RETRYING: [10.168.22.112 -> localhost]: wait for system finished reboot (116 retries left).
2023-04-17 17:27:32,802 cmdline.py 169 [2697] [INFO] ok: [10.168.22.112 -> localhost]
2023-04-17 17:27:32,802 cmdline.py 169 [2697] [INFO]
2023-04-17 17:27:32,803 cmdline.py 169 [2697] [INFO] TASK [shutdown system by reboot on ansible host] *******************************
2023-04-17 17:27:32,858 cmdline.py 169 [2697] [INFO] skipping: [10.168.22.112]
2023-04-17 17:27:32,858 cmdline.py 169 [2697] [INFO]
2023-04-17 17:27:32,858 cmdline.py 169 [2697] [INFO] PLAY RECAP *********************************************************************
2023-04-17 17:27:32,859 cmdline.py 169 [2697] [INFO] 10.168.22.112              : ok=12   changed=5    unreachable=0    failed=0    skipped=4    rescued=0    ignored=2
2023-04-17 17:27:32,859 cmdline.py 169 [2697] [INFO]
2023-04-17 17:27:34,063 cluster.py 479 [2697] [INFO] Reboot 10.168.22.112 hosts successfully

Output note

The progress and results of node shutdown or restart are displayed in the command output.