Overloaded Storage

Storage infrastructure can have a major impact on the performance of Oracle VMware and SQL Server VMware applications and databases. One of the most significant issues is overloaded storage. When memory is overloaded operation time-outs can cause issued commands to be aborted causing noticeable application-level performance problems.

Definition

There are two main causes of overloaded storage within VMware environments:

  1. Excessive demand
    The storage load being placed on the device exceeds the device's ability to respond.
  2. Misconfigured storage
    Storage devices have a large number of configuration parameters, including:
    • Number of disks per LUN
    • RAID level of a LUN
    • Assignment of array cache to a LUN

Changing any of these variables can affect the ability of the device to handle the I/Oload. Guidelines are provided from storage vendors on proper configuration and expected performance for different levels and types of I/O load. Consult them to determine what may result from changing these parameters.

Overloaded storage configurations within a virtual environment are capable of reducing overall performance of any Oracle VMware or SQL Server VMware applications operating within it.

Solutions

Due to the complexity and variety of storage infrastructures and configurations, it is difficult to define specific solutions for slow or overloaded storage. vSphere is capable of high storage performance using any supported storage technology, but little can be donebeyond re-balancing load to solve problems related to slow or overloaded storage. Consult OEM storage documentation to monitor the demand being placed on the storage device.

Vendor-specific configuration recommendations should be followed to configure the device for the demand. If the device is not capable of satisfying the I/O demand, the load should be distributed among multiple devices, or faster storage should be obtained.

Storage Information

The following is a guide to help detect overloaded storage performance problems related caused by storage infrastructure issues. It also contains some information on storage and the differences between virtual and physical environments that should be considered when trying to problem solve storage performance problems in vSphere:

  • Tune storage intensive databases and applications
    Review Ignite trend charts for SQL statements that spend the most time waiting on storage and tune them to read less data. In the Ignite GUI on the Waits tab, look for large amounts of response time associated with storage related waits and tune the associated SQL statements if possible:
    • Oracle - db file sequential read, db file scattered read, etc
    • SQL Server - PAGEIOLATCH_xx, IO_COMPLETION, WRITELOG, etc
    • Sybase - waiting for regular buffer read to complete, etc
    • DB2/LUW - SQLM_FETCH, etc
  • Size storage for performance and capacity
    As physical disks continue to grow in storage capacity, it becomes possible to satisfy the storage capacity requirements of applications with a very small number of disks. The I/O rate that each individual disk can handle, however, is limited. This means it is possible to fit the storage for an application on a disk device that cannot physically handle the I/O rates it may generate. The ability of a storage device to handle the bandwidth and IOPS demands of an application is as important a selection criterion as is the capacity of the device. 

Sizing storage for performance becomes more important in a virtualized environment. When applications run on a server in a physical environment, the storage for that application is often placed on a dedicated LUN with specific performance characteristics. When moved to a vSphere environment, the storage for an application may be taken from a pre-existing VMFS volume, which may be shared by multiple VMs running different applications. If the LUN on which the VMs are placed was not sized and configured to meet the needs of all VMs, the storage performance available to an application may suffer. This makes it critical to consider not only the performance capabilities of VMFS volumes, and their backing LUNs, but their available capacity as well.
  • Moving to a virtualized environment changes storage workloads
    
In a physical environment, the configuration of LUNs is often based on the I/O characteristics of single applications. When multiple applications are consolidated onto a single VMFS volume, the workload presented to that volume will be different than the storage workload of the individual applications. This will require an increase the number of physical disk devices needed to meet the storage performance demands of the applications.
  • Understand the load being placed on storage devices
    
Many storage arrays have tools that enablethe capture of workload statistics. In addition to providing information on IOPS and I/O sizes, these tools may provide information on queuing in the array and statistics that measure the effectiveness of the caches in the array.
  • Simple benchmarks can help isolate storage performance problems
    Once storage performance problem have bee identified, troubleshooting the problem using application-level loads may be difficult. When using live applications, problems may occur intermittently or only during peak periods. Driving applications with synthetic loads can be time-consuming and require complex tools. Disk benchmarking tools, such as IOmeter, can be used to generate loads similar to application loads in a controllable manner.
  • Consider the tradeoff between memory capacity and storage demand
    Some applications can take advantage of additional memory capacity to cache frequently used data to reduce storage loads. This may also provide the added benefit of improving application performance. Because running applications in VMware ESX makes it easy to add additional memory capacity to a VM, this tradeoff should be considered when storage is unable to meet existing demands. Some applications require changes to storage configuration parameters to make use of additional memory.

Confio IgniteVM

Confio IgniteVM helps identify the impact of overloaded storage for sites running Oracle on VMware, SQL Server on VMware, and other virtual databases. IgniteVM helps DBAs maintain performance and availability on virtual servers. IgniteVM is the only virtualization-aware database monitoring solution.

Learn more about IgniteVM solutions for: