Performance assessment and tuning experiences with RHEL
Performance management is the process of making sure that adequate computing resources (i.e. CPU, Memory, Disk and Networks) are available to accomplish the business needs of all users. Before performing any performance assessment we hope the system has design properly and have conducted below exercise:
Workload optimization: The target workload may be hampered by these settings.
Capacity Planning: To estimate the resources that will be necessary to support a system's workload for a specific period of time.
Throughput and Latency: Throughput is the measure of how much data can be transferred or processed by a resource in a given time. Latency is the delay that a resource must wait to start data transfer or processing as opposed to Throughput.
For the performance assessment and tuning, we have to follow some approaches. The Utilization Saturation and Errors (USE) Method that is highly regarded among performance tuning experts.
With Utilization Saturation and Errors (USE) Method, we can create our metric table. Below an example of very high level view of listing of resources, consider the metric types: utilization, saturation and errors and so on.
resource | type | metric |
CPU | utilization | CPU utilization (either per-CPU or a system-wide average) |
CPU | saturation | run-queue length or scheduler latency |
Memory capacity | utilization | available free memory (system-wide) |
Memory capacity | saturation | anonymous paging or thread swapping (maybe "page scanning" too) |
Network interface | utilization | RX/TX throughput / max bandwidth |
Storage device I/O | utilization | device busy percent |
Storage device I/O | saturation | wait queue length |
Storage device I/O | errors | device errors ("soft", "hard") |
Utilization: 100% utilization is usually a sign of a bottleneck (check saturation and its effect to confirm the bottleneck). Greater than 70% utilization for an extended time (many seconds or minutes) can hide short bursts of 100% utilization.
Saturation: Any degree of saturation can be a problem (non-zero) and is measured as the length of a wait queue, or the time spent waiting in the queue before being processed.
Errors: Non-zero error counters are worth investigating, especially if they are still increasing while facing degradation in performance.
Note: More details refere to: https://queue.acm.org/detail.cfm?id=2413037
Profiling:
Profiling is the process of gathering information in different ways of retrieving performance data for a system. For example, Application Profiling is gathering information about a program's behavior as it executes. And will determine which areas of a program can be optimized to increase the program's overall speed, reduce its memory usage, etc. Application profiling tools help to simplify this process.
In a nutshell, we need to prepare the set of tools to gathering information performance data for a system.
System monitoring:
There are a bunch of monitoring tools that commonly view information, and can be used by way of the command line or a graphical user interface, as determined by the system administrator. And system monitoring is a helpful approach to provide the user with data regarding the actual timing behavior of the system to perform further analysis using the data that these monitors provide.
It’s always advisable to use more than one monitoring tool or alternative tool to verify the data regarding the actual timing behavior of the system.
Process Management in Linux:
Whatever program that we execute in our linux system will consider process and we need to understand the type of process (Foreground or Background), states of the process (Running, Sleeping, etc.), resource utilization (CPU,Memory, etc) and so on.