Abnormal packet sending or receiving on host network ports

Description

The system periodically collects and tracks the number of dropped and error packets on physical network ports, and triggers the alert when the count exceeds the threshold.

Alert message

• The physical network port { .labels._device } of the host { .labels.hostname } has more than { .threshold } dropped packets in 2 minutes while receiving packets.

• The physical network port { .labels._device } of the host { .labels.hostname } dropped more than { .threshold } packets in 2 minutes while sending packets.

• The physical network port { .labels._device } of the host { .labels.hostname } has more than { .threshold } error packets in 2 minutes while receiving packets.

• The physical network port { .labels._device } of the host { .labels.hostname } has more than { .threshold } error packets in 2 minutes while sending packets.

• The physical network port { .labels._device } of the SCVM { .labels.hostname } has more than { .threshold } dropped packets in 2 minutes while receiving packets.

• The physical network port { .labels._device } of the SCVM { .labels.hostname } dropped more than { .threshold } packets in 2 minutes while sending packets.

• The physical network port { .labels._device } of the SCVM { .labels.hostname } has more than { .threshold } error packets in 2 minutes while receiving packets.

• The physical network port { .labels._device } of the SCVM { .labels.hostname } has more than { .threshold } error packets in 2 minutes while sending packets.

Impact

This issue typically does not affect services, but the cause still needs to be investigated.

Cause

Increases in dropped and error packet counts on physical NICs may be caused by the following:

• Burst traffic: The NIC cannot process traffic fast enough during peak loads, causing packet loss. This is common during large-scale throughput testing using traffic generation tools.

• Small queue size: The default transmit or receive queue depth of the physical NIC is too small, which can cause queue overflow and packet loss during traffic bursts.

• MTU mismatch between the switch and NIC: Inconsistent MTU settings between the switch and NIC cause packets exceeding the NIC MTU to be dropped, or to be counted as error packets due to checksum failures during reassembly.

• Malformed packets: Issues such as CRC check failures or physical layer interference also increase error packet counts.

Troubleshooting approach

1. Check MTU settings: confirm whether the MTU values configured on the switch and the host's physical NIC are consistent.

2. Check queue configuration: log in to the host and run ethtool -g <NIC name> to view the queue settings. For example:

   [root@node31 11:11:10 ~]$ ethtool -g enp94s0f0np0
   Ring parameters for enp94s0f0np0:
   Pre-set maximums:
   RX:             8192
   RX Mini:        0
   RX Jumbo:       0
   TX:             8192
   Current hardware settings:
   RX:             1024
   RX Mini:        0
   RX Jumbo:       0
   TX:             1024

   The output contains two sections:

   • Pre-set maximums: the maximum queue depth supported by the physical NIC.

   • Current hardware settings: the current actual queue configuration.

   If the current queue depth is significantly smaller than the maximum, you can adjust the queue depth configuration as appropriate.

Solution

Take the action based on the investigation results:

• MTU configuration mismatch

  If the MTU values of the switch and the physical NIC are inconsistent, adjust the MTU on the switch or the physical NIC to the same value. Ensure that all elements (switches, physical NICs, system networks, VM networks, etc.) on the same network path have consistent MTU values; otherwise, packets may be dropped, or data abnormal fragmentation may occur.

• Small NIC queue size

  If the transmit and receive queue sizes of the physical NIC are too small, run ethtool -G <NIC name> tx <queue size> and ethtool -G <NIC name> rx <queue size> to adjust the transmit and receive queue depths, respectively.

  For example:

  [root@node31 11:11:14 ~]$ ethtool -G enp94s0f0np0 rx 8192
  [root@node31 11:40:01 ~]$
  [root@node31 11:40:02 ~]$ ethtool -G enp94s0f0np0 tx 8192
  [root@node31 11:40:06 ~]$
  [root@node31 11:40:07 ~]$ ethtool -g enp94s0f0np0
  Ring parameters for enp94s0f0np0:
  Pre-set maximums:
  RX:             8192
  RX Mini:        0
  RX Jumbo:       0
  TX:             8192
  Current hardware settings:
  RX:             8192
  RX Mini:        0
  RX Jumbo:       0
  TX:             8192