Here's a report on the webserver on a typical day:
This is an analysis of the data contained in the file 20060613. The data was collected on 06/13/2006, from 00:00:00 to 23:00:00, from system 'linux11444.dn.net'. Operating system was 2.4.23. 4 processors are present. The ht (hyperthreading) flag was seen in /proc/cpuinfo. We will analyze the data as reported by the /proc filesystem but we can not be sure if hyperthreading has been enabled because of BIOS settings and motherboard support. It is possible that the number of CPUs are not being correctly reported. 3032 megabytes of memory are present.
Data collected by the ps -elf command on 06/13/2006 from 00:00:00 to 23:00:00, and stored in the file /opt/sarcheck/ps/20060613, will also be analyzed.
Table of Contents
SUMMARY
When the data was collected, a severe CPU bottleneck may have existed. No memory bottleneck was seen and the system has sufficient memory. No significant I/O bottleneck was seen. A change has been recommended to at least one tunable parameter. Recommendations can be found in the Recommendations Section of this report.
RECOMMENDATIONS SECTION
All recommendations contained in this report are based solely on the conditions which were present when the performance data was collected. It is possible that conditions which were not present at that time may cause some of these recommendations to result in worse performance. To minimize this risk, analyze data from several different days and implement only regularly occurring recommendations.
Adjusting process priorities with the nice(C) command, optimizing applications, or a CPU upgrade may help performance. Heavy CPU utilization was seen and a CPU upgrade may be needed to improve performance.
Change the bdflush parameter 'nfract' from 54 to 58. This is the percentage of dirty buffers allowed in the buffer cache before the kernel flushes some of them.
Change the bdflush parameter 'nfract_sync' from 84 to 86. This is the percentage of dirty buffers in the buffer cache before the kernel agressively flushes them synchronously.
Change the bdflush parameter 'nfract_stop_bdflush' from 44 to 48. This is the percentage of dirty buffers in the buffer cache used to decide when to stop bdflush.
To change the value of the bdflush parameters immediately as described in the above recommendations, use the following command:
sysctl -w vm.bdflush="58 500 512 0 500 3000 86 48 0"
If you are able to make this change and it improves performance, you can make the change permanent by adding the command to the /etc/sysctl.conf file.
RESOURCE ANALYSIS SECTION
A reboot was detected between 11:30:04 and 11:40:02. Data collected during reboot periods are ignored by this tool.
The CPU was busy an average of 74.0 percent of the time. This indicates that spare capacity is limited and a CPU bottleneck may exist. Any significant increase in this workload is likely to result in performance degradation. CPU utilization peaked at 100.00 percent from 10:40:03 to 10:50:02. This peak is higher than expected, possibly because of minor differences in the elapsed time between timestamps and /proc samples.
Application optimization, use of the nice(C) command, or a CPU upgrade may help performance.
CPU number 0 was busy for an average of 74.28 percent of the time. During the peak interval from 20:30:04 to 20:40:02, this CPU was 100.01 percent busy. The CPU was busy with user work 64.56 percent of the time and was busy with system work 9.73 percent of the time. The sys/usr ratio on this CPU was 0.15:1. This is below the threshold of 2.50:1.
CPU number 1 was busy for an average of 73.93 percent of the time. During the peak interval from 20:30:04 to 20:40:02, this CPU was 100.01 percent busy. The CPU was busy with user work 67.91 percent of the time and was busy with system work 6.02 percent of the time. The sys/usr ratio on this CPU was 0.09:1.
CPU number 2 was busy for an average of 74.28 percent of the time. During the peak interval from 10:40:03 to 10:50:02, this CPU was 100.01 percent busy. The CPU was busy with user work 67.98 percent of the time and was busy with system work 6.30 percent of the time. The sys/usr ratio on this CPU was 0.09:1.
CPU number 3 was busy for an average of 73.71 percent of the time. During the peak interval from 10:40:03 to 10:50:02, this CPU was 100.01 percent busy. The CPU was busy with user work 67.58 percent of the time and was busy with system work 6.12 percent of the time. The sys/usr ratio on this CPU was 0.09:1.
Individual CPU Statistics CPU# Average %Busy Average %User Average %Sys Average %Nice Peak %Busy 0 74.28 64.56 9.73 0.00 100.01 1 73.93 67.91 6.02 0.00 100.01 2 74.28 67.98 6.30 0.00 100.01 3 73.71 67.58 6.12 0.00 100.01
The average amount of free memory was 404219.0 pages or 1579.0 megabytes. The minimum amount of free memory was 261658 pages or 1022.10 megabytes at 13:30:05. The /proc/sys/vm/freepages data was not found and this is normal on newer kernels.
The average amount of free memory including cached memory and memory used for buffers was 561903.8 pages or 2194.9 megabytes. This is 72.4 percent of all the memory seen on the system. The minimum amount of free memory including cached memory and memory used for buffers was 426561 pages or 1666.25 megabytes at 13:30:05.
No memory bottleneck was seen and the system has sufficient memory.
The value of nfract, the dirty buffer threshold used to wake up bdflush, was set to 54 percent. The goal of tuning nfract is to keep it low enough that the number of dirty buffers in the cache is not enough to degrade performance, but high enough to allow as many dirty buffers in the cache as possible. In this case a recommendation was made to increase the value to 58 percent. A recommendation has been made to change the value of nfract_sync to 86 percent in order to optimize its value relative to nfract. A recommendation has been made to change the value of nfract_stop_bdflush to 48 percent in order to optimize its value relative to nfract.
The value of ndirty was set to allow bdflush to write 500 buffers to the disk at one time.
The nrefill parameter was set to 512 buffers. This parameter controls the number of buffers to be added to the free list whenever bdflush calls refill_freelist().
The interval parameter in /proc/sys/vm/bdflush controls how frequently the kernel update daemon runs, and it was set to 500 jiffies. A jiffie is a clock tick and on x86 systems, there are 100 jiffies per second.
The value of the page_cluster parameter was 3. This means that 8 pages are read at once. Values of 2 or 3 are typically better for systems will small memory sizes, systems where response time is important, and systems where most I/O is not sequential. There may be an advantage in raising this value if you want to speed up the performance of programs which do mostly sequential I/O.
The kswapd parameter tries_base was set to 512. This controls the number of pages that kswapd will try to free each time it runs. The kswapd parameter tries_min was set to 32. This controls the number of times that kswapd tries to free a page of memory when it's called. The kswapd parameter swap_cluster was set to 8. This controls the number of pages that kswapd will try to write when it is called.
The average page in rate was 0.306 per second. Page ins peaked at 30.39 per second from 00:50:00 to 01:00:00. The average page out rate was 309.884 per second. Page outs peaked at 610.90 per second from 12:30:02 to 12:40:05. An unusually high page out rate was detected. This may be normal for your environment, but it is still worth noting.
The average swap in rate and the average swap out rate were both zero.
The amount of swap space in use was always 0.00 megabytes. The size of swap space was 2013.11 megabytes. The peak amount of swap space in use was 0.000 percent of the total. This is a very small amount of the total available swap space. The amount of swap space in use did not change during the monitoring period.
There was one swap file seen in /proc/swaps.
The system-wide disk I/O rate averaged 19.79 per second and peaked at 37.84 per second from 12:30:02 to 12:40:05 on 06/13/2006. The read rate averaged 0.03 per second and peaked at 1.64 per second from 00:50:00 to 01:00:00 on 06/13/2006. The write rate averaged 19.75 per second and peaked at 37.84 per second from 12:30:02 to 12:40:05 on 06/13/2006.
Partitions data was not available. This data is required to calculate the "percent busy" statistics for disks.
The I/O rate on disk device (8,0) averaged 19.37 per second and peaked at 37.19 per second from 12:30:02 to 12:40:05. The read rate averaged 0.03 per second. The write rate averaged 19.34 per second. The (x,x) notation refers to the major and minor device numbers and is the way that devices are identified in /proc.
The I/O rate on disk device (8,1) averaged 0.42 per second and peaked at 0.67 per second from 14:00:03 to 14:10:03. The read rate averaged 0.0001 per second. The write rate averaged 0.42 per second.
The noatime option was not specified on any of the mounted filesystems. Data from /proc/partitions is not available and so the likelihood of performance gains with this option is unknown.
The value of the ctrl-alt-del parameter was 0. The value of 0 is better in almost all cases because it prevents an immediate reboot if the ctrl, alt, and delete keys are pressed simultaneously.
There were an average of 3078.80 interrupts per second and the peak interrupt rate seen was 4665.90 per second from 12:50:02 to 13:00:02.
At 14:00:01 ps -elf data indicated that there were a peak of 278 processes present. This was the largest number of processes seen with ps -elf but it is not likely to be the absolute peak because the operating system does not store the true "high-water mark" for this statistic. There were an average of 182.3 processes present.
No runaway processes, memory leaks, or suspiciously large processes were detected in the data contained in file /opt/sarcheck/ps/20060613.
CAPACITY PLANNING SECTION
The section is designed to provide the user with a rudimentary linear capacity planning model and should be used for rough approximations only. These estimates assume that an increase in workload will affect the usage of all resources equally. These estimates should be used on days when the load is heaviest to determine approximately how much spare capacity remains at peak times.
Based on the limited data available in this single day of data, the system cannot support an increase in workload at peak times without some loss of performance or reliability, and the bottleneck is likely to be CPU utilization. Implementation of some of the suggestions in the recommendations section may help to increase the system's capacity.
The CPU can support an increase in workload of approximately 0 percent at peak times. For more information on peak CPU and disk utilization, refer to the Resource Analysis section of this report. There was nothing seen to suggest that the amount of memory present would be unable to support a significantly greater load. The busiest disk can support a workload increase of at least 100 percent at peak times. For more information on peak CPU and disk utilization, refer to the Resource Analysis section of this report.
Please note: In no event can Aptitune Corporation be held responsible for any damages, including incidental or consequent damages, in connection with or arising out of the use or inability to use this software. All trademarks belong to their respective owners. This software is provided for the exclusive use of: Werner Technologies. This software expires on 06/15/2006 (mm/dd/yyyy). Code version: SarCheck for Linux 6.01.05. Serial number: 00067559.
Thank you for trying this evaluation copy of SarCheck. To order a licensed version of this software, just type 'analyze -o' at the prompt to produce the order form and follow the instructions.
(c) Copyright 2003-2006 by Aptitune Corporation, Plaistow NH 03865, USA, All Rights Reserved.
http://www.sarcheck.com/
Statistics for system: linux11444.dn.net Start of peak interval End of peak interval Date of peak interval Statistics collected on: 06/13/2006 MAC Address: 00:07:E9:24:96:FD Average combined CPU utilization: 74.05% Average user CPU utilization: 67.01% Average sys CPU utilization: 7.04% Average 'nice' CPU utilization: 0.00% Peak combined CPU utilization: 100.00% 10:40:03 10:50:02 06/13/2006 Average page out rate: 309.88/sec Peak page out rate: 610.90/sec 12:30:02 12:40:05 06/13/2006 Average swap out rate: 0.00/sec Peak swap out rate: 0.00/sec 06/13/2006 Average swap space in use: 0.00 megabytes Peak swap space in use: 0.00 megabytes Average amount of free memory: 404219 pages or
1579.0 megabytes Minimum amount of free memory: 261658 pages or
1022.10 megabytes 13:30:05 06/13/2006 Average system-wide I/O rate: 19.79/sec Peak system-wide I/O rate: 37.84/sec 12:30:02 12:40:05 06/13/2006 Average read rate: 0.03/sec Peak read rate: 1.64/sec 00:50:00 01:00:00 06/13/2006 Average write rate: 19.75/sec Peak write rate: 37.84/sec 12:30:02 12:40:05 06/13/2006 Average Interrupt rate: 3078.80/sec Peak Interrupt rate: 4665.90/sec 12:50:02 13:00:02 06/13/2006 Avg number of processes seen by ps: 182.3 Max number of processes seen by ps: 278 14:00:01 06/13/2006 Approx CPU capacity remaining: 0.0% Approx I/O bandwidth remaining: 100%+ Can memory support add'l load: Yes