Network performance for applications in a virtual environment depends on many factors including, network protocols, congestion, and buffering along the network route. When network issues reach critical levels and network buffers become filled, transmitted and received packets may begin to be dropped. VMware host dropped packets report shows the total number of transmitted or received packets dropped by all vNICs on the host across all VMs.
Definition
Dropped Received Packets
Network switches, physical NICs, device drivers, and network stacks may all contain queues where data is buffered before being passed along. These queues are all finite in size. When they fill up, no more packets can be received, and additional arriving packets must be dropped.
For received packets, the virtual NIC buffers packet data coming from a virtual switch (vSwitch) until it is retrieved by the device driver running in the guest OS. The vSwitch, in turn, contains queues for packets sent to the virtual NIC. If the Guest OS does not retrieve packets from the virtual NIC rapidly enough, the queues in the virtual NIC device can fill up. This may cause the queues within the corresponding vSwitch port to fill. A vSwitch port with a full packet queue that receives a packet sent to a VM must drop the packet.
The following problems can cause the VMware guest OS to fail to retrieve packets quickly enough from the vNIC:
- High CPU utilization
While the applications and guest OS are driving the VM to high CPU utilization, there can be extended delays from the time the guest OS is notified that receive packets are available until those packets are retrieved from the virtual NIC. Sometimes the high CPU utilization is caused by high network traffic since network packet processing can place a significant demand on CPU resources. - Guest OS driver configuration
Device drivers for networking devices are usually capable of tuning parameters within the guest OS. These may control behavior such as the use of interrupts versus polling, the number of packets retrieved on each interrupt, and the number of packets a device should buffer before interrupting the OS. Failure to configure them properly can cause poor network performance and dropped packets.
Definition
Dropped Transmitted Packets
Transmitted packets on a vNIC are buffered in associated vSwitch ports until they can be transmitted onto their perspective physical uplink devices. When traffic from VMs that share the vSwitch exceeds the capabilities of the uplink NICs, the vSwitch buffers can fill. Any additional transmitted packets sent will be dropped.
Solutions
Dropped Received Packets
The solutions to the problem of dropped received packets are all related to ways of improving the ability of the guest OS to quickly retrieve packets from the virtual NIC. To solve the issue of dropped received packets you must find ways to improve the ability of the guest OS to quickly receive packets from the vNIC. Some of them include:
- Increasing CPU Resources to the VM
It may be necessary to add vCPUs to provide sufficient CPU resources if the VM is dropping receive packets. Refer to your OS documentation if high CPU utilization is the result of network processing. This may interfere with the guest OS's ability to take advantage of multiple CPUs. Increasing CPU resources should help to alleviate the effects of network processing congestion. - Increasing Efficiency of CPU Resource Usage for the
VM
Proper tuning of application with high CPU utilization may be able to improve the efficiency of their CPU resource usage. See the tips for increasing VM efficiency in the Host CPU Saturation discussion. - Tuning the Guest OS Network Stack
Tuning the network stack within the guest OS may also result in improved performance speed and efficiency of packet handling. Refer to the documentation for the OS. - Spread the Network Load Across Additional Virtual
NICs
Interrupts in some guest OSes are sent to a single processor core. This single processor can easily become a bottleneck that results in dropped packets. Adding additional virtual NICs will spread the load across multiple processor cores and decrease the stress of network interrupts on any single vNICs
Solutions
Dropped Transmitted Packets
Here are several ways to avoid dropped transmitted packets sent through Oracle VMware and SQL Server VMware applications and databases:
- Adding Uplink Capacity to the vSwitch
To mitigate some conditions that cause transmit packets to be dropped, you can add physical uplink NICs to the vSwitch. You must monitor the traffic, however, to ensure that NIC teaming policies result in the proper load distribution among available uplinks. - Moving High Network Demand VMs to a Different
vSwitch
When vSwitch uplinks become overloaded, you can rebalance demand load by moving high-demand VMs to additional vSwitches - Increasing Network Capacity
Ocaisionally, you will find that bottlenecks are created by network infrastructure. You may need to add additional network capacity to handle VM demands. - Reducing Network Traffic
There are a number of ways to reduce network traffic generated by VMs, including caching network data or tuning the network stack to use larger packets. What you can do will depend on the applications and guest OS you are using.
Confio IgniteVM
Confio IgniteVM helps identify the impact of host dropped packets for sites running Oracle on VMware, SQL Server on VMware, and other virtual databases. IgniteVM helps DBAs maintain performance and availability on virtual servers. IgniteVM is the only virtualization-aware database monitoring solution.
Learn more about IgniteVM solutions for:
Follow Us