Performance Troubleshooting Clustered ONTAP

A simple rule of thumb: If a particular system or application is running slower than you expect, or slower than it has historically, it might be a performance issue. However, if a particular system or application is not working at all, it is likely not a performance related issue.

  • The recommended method for monitoring performance in a Cluster Data ONTAP system is to use volume workloads. You can use the commands below to help determine where your issue might reside.
The following is a review of the commands above, and a look at how to use/interpret them:

Use the preceding command to view overall latency on volume workloads and get a sense of work performed on the cluster.

  • Workload – Concatenation of volume name and internal workload identifier.
    Output can be filtered:

    • Autovol (qos statistics volume latency show) -vserver -volume 
    • User Defined (QOS statistics workload latency show) -workload -workload-id ·
  • ID – Internal workload identifier.
  • IOPS – Average number of operations processed every second.
  • Throughput – Average amount of payload data moved into and out of the workload every second.
  • Latency – Average operation residency time into and out of the workload. · -total-. For throughput metrics, the sum of averages of all workloads on the cluster. For latency, the average of average latencies on the cluster
More can be learned about the offered loads presented to the volume using the QoS volume characteristics command displayed earlier. In particular, the concurrency calculation shows the level of application offered load in relation to cluster consumption of that load. Both these factors individually are highly complex and do not provide enough information to draw any major conclusions. However, concurrency does provide information about application behavior, such as the request arrival rate:
  • Workload – Concatenation of volume name and internal workload identifier
  • ID – Internal workload identifier
  • IOPS – Average number of operations processed every second
  • Request size – Calculation of throughput divided by IOPS. Given that all the metrics available are averages, this is the best that can be done. For more detailed request size information, a histogram will be required.
  • Read – Percentage of workload that covers read operations. The remaining percentage covers write operations for SAN protocols and is the sum of writes and metaoperations (or other) for NAS protocols.
  • Concurrency – Product of latency and IOPS; see “Little’s Law: A Relationship of Throughput, Latency, and Concurrency.” This is the number of operations resident in the cluster being serviced at a given point in time.
  • -total- – For throughput and concurrency metrics, the sum of averages of all workloads on the cluster. For remaining metrics, the average of averages on the cluster.
The volume latency command breaks down the latency contribution of the individual clustered Data ONTAP components. Among the monitoring commands presented here, this is possibly the most useful, in that it provides visibility into workload internals across the cluster. It is worth repeating that latency is the equivalent of operation residency time and is the sum of operation wait time and execution time for a given component. In the preceding example, it can be seen that workload bobvol5 is an indirect workload averaging a 2.17ms latency. The largest portion of that latency is attributed to the 1.82ms disk contribution. The remaining component contributions are most likely CPU execution time and network and cluster interconnect transmission delay, and account for very little of the total latency. This command does not provide enough information to know every detail. However, when considering a singular workload under normal operating conditions, execution time will almost always be far less significant in relation to wait time (in this example disk wait time):
  • Workload – Concatenation of volume name and internal workload identifier
  • ID – Internal workload identifier
  • Latency – Overall average operation residency time into and out of the cluster
  • Network – Average latency contribution of the server (or client) facing transport component. This includes the delay introduced by front-end SAN or NAS transport stacks.
  • Cluster – Average latency contribution of the cluster interconnect transport component. This includes the delay introduced by all cluster interconnect transport protocols.
  • Data – Average latency contribution of the clustered Data ONTAP proprietary file system, known as WAFL. After the operation has been transported by the underlying protocols, it will be processed by WAFL and the response is returned to the protocols for delivery as rapidly as possible.
  • Disk – Average latency contribution of the physical disks
  • QoS – Applicable only when QoS policy limits are in place. When actively limiting, this is the average latency penalty incurred to enforce the user-defined policy limit
  • NVRAM – Average latency incurred to replicate write operations in NVRAM to the high-availability (HA) and/or cluster partner.
  • -total- – The average of average latencies on the cluster.
Some workloads are more expensive than others in terms of CPU utilization due to application-specific behavior. The preceding volume resource cpu command displays the specific CPU utilization for a given volume workload. Note that this in no way represents the total physical node-level CPU utilization. That is a different topic of discussion, because there are many internal Data ONTAP processes that can be running, not accounted for here. It should also be noted that indirect workloads are present here to account for the transport protocol CPU overhead (see bobvol5-wid9864 earlier):
  • Workload – Concatenation of volume name and internal workload identifier
  • ID – Internal workload identifier
  • CPU – Processor resource utilization attributed to the workload
  • -total- – Sum of all the workload CPU utilization metrics.

In a similar fashion to CPU, physical disks are a limited shared resource where some workloads are more expensive than others. That is where the similarities end, though. In the preceding command context, disk utilization represents the amount of time a disk is busy servicing requests on behalf of the volume workload. Disk utilization (and thus disk latency) can widely vary among workloads due to many factors previously discussed, such as data access patterns, working set size, free space, and cache resources. Unlike the volume resource cpu command, the volume resource disk command only includes workloads that are local to the node, because it is fundamentally impossible for disk (or aggregate) utilization to be split across multiple nodes:

  • Workload – Concatenation of volume name and internal workload identifier
  • ID – Internal workload identifier
  • Disk – Average disk utilization attributed to the workload
  • -total- – Average disk utilization of all disks on node.

Some more generic commands that can provide useful data:
Enter privileged mode:

  • Testcluster::> set –privilege advanced
  • Testcluster::*> statistic show-periodic
    What values do you see? Is CPU high? Are total Ops where you expect them to be? Does anything look unusual?
    Note: When reviewing CPU information in Data ONTAP, CPU AVG is a better indicator of overall CPU utilization compared to CPU BUSY.

What can be done with this data? Are any volumes showing more IOPs than what is expected? Are any volumes showing more latency than what is expected? Do any volumes show higher disk latency than what is expected?

Using QoS, set up policy groups to throttle offending or ‘bully’ workloads. Here is how:

If you have identified a rogue workload, the next step would be to create a new QoS policy group and set the I/O throughput limit to 1,000 IOPS [an arbitrary value, used just for this example].
Once this is complete, view the new group by using the qos policy-group show command.
Next, the offending volume needs to be assigned to the QoS policy group to begin throttling the rogue workload. The modify option of storage objects is used to assign an existing storage object to a QoS policy group.
Finally, verify that the volume has been assigned to the QoS policy group using the volume showcommand.
This policy successfully limits the amount of system resources this workload can consume. Users on this volume might begin to experience latency they are not used to seeing, with this limit in place. However, this ensures that users of other volumes will not be impacted by this workload any more.
Note: It is important to keep in mind that all systems have a set performance ‘budget’. Any workload and any options enabled will all have a performance cost, subtracted from the system’s performance budget. You can use a combination of the above commands and the baseline understanding of what your system should be doing, from a workload perspective, to determine where an issue might reside.
twitterlinkedinmailtwitterlinkedinmail
Arco

About

View all posts by