Adjusting the cluster system time

When users modify the NTP server configuration of a cluster, such as switching between using an external NTP server and using an internal host as the NTP server, it might lead to inaccuracies in the system time of certain nodes in the cluster. Therefore, relevant commands should be used to correct this, ensuring all nodes in the cluster have synchronized the system time.

Applicable scenario

The following procedure applies to scenarios where the system time of certain nodes in an ACOS (AVE) cluster or ACOS (VMware ESXi) cluster is inaccurate.

If the cluster uses an internal host as an NTP server and the system time of all nodes remains consistent with the actual time, but users still wish to adjust the cluster time, the adjustment can be made using the zbs-cluster sync_internal_leader_to_normal --time <time_string> command. The format of its parameters can be referenced from the description in Inaccurate system time of the NTP Leader node.

Precaution

Prometheus service cannot save data earlier than the current system time. Therefore, if the adjusted system time is earlier than the currently displayed system time, Prometheus will not be able to update the data between these two time points.

Preparation

Log in to AOC, go to the cluster management interface, click Settings, select Cluster time, and determine whether the cluster uses an internal or external host as its NTP server.

Using the external NTP server

Prerequisite

Ensure that the system time of the external NTP server used by the cluster is accurate.

Procedure

Execute the following command on any node in the cluster to synchronize the clocks of all nodes with the external NTP server.

zbs-cluster sync_time

Note:

For an ACOS (AVE) cluster, the system temporarily halts the elf-vm-monitor service during clock synchronization, triggering an alert from the monitoring system for this service. However, this alert can be ignored because the system will automatically resume the elf-vm-monitor service once the synchronization completes. You still need to confirm that the elf-vm-monitor service has restarted successfully after the synchronization process is finished.

Using an internal host as the NTP server

Principle for adjusting the cluster system time

Since other nodes in the cluster will automatically adjust their system time to stay in sync with the NTP Leader node as long as the time difference does not exceed 10 minutes, the principle for adjusting system time in the cluster is to ensure that the time difference across all nodes does not exceed 10 minutes.

Accurate system time of the NTP Leader node

If the NTP Leader node in the cluster is synchronized with the actual time, but certain nodes in the cluster have a time difference greater than 10 minutes and fail to synchronize automatically, you must manually adjust the system time on each of these nodes by executing the following command to align them with the NTP Leader.

zbs-node sync_time

Inaccurate system time of the NTP Leader node

If the system time of the NTP Leader node in the cluster is not synchronized with the actual time, you can synchronize the system time of all nodes in the cluster with the correct time by running the following command on any node in the cluster.

zbs-cluster sync_internal_leader_to_normal --time <time_string>

Information:

• The format of the parameter time_string aligns with the format of time_str in the command date --set="time_str".
• For an ACOS (AVE) cluster, the system temporarily halts the elf-vm-monitor service during clock synchronization, triggering an alert from the monitoring system for this service. However, this alert can be ignored because the system will automatically resume the elf-vm-monitor service once the synchronization completes. You still need to confirm that the elf-vm-monitor service has restarted successfully after the synchronization process is finished.

For example, run the following command on any node in the cluster:

zbs-cluster sync_internal_leader_to_normal --time "+600 second"

zbs-cluster sync_internal_leader_to_normal --time "-600 second"

In the example above, +600 seconds indicates adjusting the system time of all nodes forward by 600 seconds, while -600 seconds means adjusting the system time of all nodes backward by 600 seconds.

If you want to adjust the system time of the node to a specific time, such as exactly 19:51, run the following command:

zbs-cluster sync_internal_leader_to_normal --time "19:51:00"