Most of our pages include graphs for one given proxy. What if you want to track performance patterns for a group of caches? This page is an index to graph bags. A graph bag is a page with graphs of a given type for many proxies. Participating proxies are listed in the page header.

Click on a graph title to get the corresponding graph bag.

If you are lost, the start page is here.

The graph shows the distribution of network transfer sizes for hits and misses and disk transfer sizes for swap requests based on 24 hour data. To show the real traffic rather than file size distribution, we count every access to a document. Thus, a document that had 3 accesses would be counted once as a miss (if the document was not cached before) and twice as a hit (if the document was still in the cache).

The graph shows the distribution of file sizes for hits, misses, and swap requests based on 24 hour data. To show file size distribution rather that traffic, we count only the first access to a document. This graph is helpful in estimating the number of cached objects given the cache size as well as per file memory requirements.

The graph shows the change in proxy server load with time. Load is measured as number of requests per second processed by the proxy. Hits and misses are determined using Squid's action field in access.log.

Proxy response time is the total time it takes to serve a client request. The graph shows median response time in milliseconds during the day.

The graph shows proxy response time versus document size.

The graph shows the number of concurrent requests present in the proxy server. We count the number of requests in the system using 10 msec intervals and calculate the median based on 20 minute grouping. Small 10 msec intervals assure that we count the number of concurrent requests rather than total number of requests per [large] interval. Note that this graph is not a "request per second" graph.

This graph is helpful in estimation of per request resources needed to support studied traffic.

The graph shows the variation of the Document Hit Ratio and Byte Hit Ratio during the day.

This graph analyzes relative impact of request processing stages. Such analysis is essential for performance optimizations since it helps in identifying performance bottlenecks.

For misses, we distinguish four major stages: client connect, proxy connect, server reply, and proxy reply. The graph shows relative contribution of each stage towards total delay. Total delay is calculated as a sum of delays of all stages.

Note that median total delay may differ from median response time because of pipelining and such non-accounted activities as DNS lookups. We are working on a more precise model that accounts for these side effects.

We cannot account for pipelining affects that may change relative contributions for large requests. However, usually more than 80% of all requests cannot be pipelined due to small document sizes. We believe that our estimations are very close to actual performance.

See also "Request Response Time Components (200 Hits)".

See "Request Response Time Components (Misses)" experiment for the graph description and important caveats.

For hits, we distinguish three major stages: client connect, swap-in, and proxy reply. The graph shows relative contribution of each stage towards total delay. Total delay is calculated as a sum of delays of all stages.

The graph shows the speed of processing swap-in and swap-out requests during the day (in requests per second).

The graph shows the number of concurrent swap requests present in the proxy server. We count the number of requests in the system using 2 msec intervals and calculate the median based on 20 minute grouping. Small 2 msec intervals assure that we count the number of concurrent requests rather than total number of requests per [large] interval. Note that this graph is not a "disk request per second" graph.

We plot the 50th and 75th percentiles. The 50th percentile is the same as median.

This graph is useful in determining the increase in the length of disk queues during peak hours (if any).

Disk utilization is measured in the percent of time there was at least one active swap request. The measurements are done using 2 msec intervals. Note that only swap requests are taken into consideration. Disk utilization is represented by the "all" curve. Curves for swap-in and swap-out requests are given to compare the contribution of each class towards disk utilization.

The patch does not measure per disk utilization. The graph represents the utilization of the disk I/O subsystem as a whole. In other words, if there is always one disk I/O in the system, then utilization is 100% regardless of the number of physical disks installed.

Disk response time is the total time it takes to load/store (swap in/out) a document from/into the disk cache. The graph shows median disk response time in milliseconds during the day.

The patch allows for quantifying varios I/O delays. Let's consider a swap request. Swapping a document is done in several steps. First, the corresponding file should be opened for reading or created for writing. This requires an open(2) system call which may incur significant OS overhead: An open(2) call may result in extra I/Os if OS has to write/read i-nodes to/from disk. Then the content in swapped to/from disk using blocks of fixed size. Disk cache and various delays in-between these I/Os affect the total response time.

To estimate the OS overhead on swapping a file we plot the median disk response time of a request versus file size. Response times for files smaller than 16 KB were grouped using 1 KB granularity. Larger files used 1 KB granularity to get enough entries per group. The graph is based on the 24 hour data.

Squid attempts to swap files using blocks of fixed length (e.g. 8 KB). For each I/O direction, we plot the total request response time and the time it takes to swap the first block. The "total" and "1st delay" curves for files smaller than 8 KB are the same.

Since various per I/O delays dominate disk transfer time, the size of an I/O is not very important (the number of I/Os is). This explains step-like shape of the "total" curves: If I/O block size is 8 KB, the times to read 5 KB and 8 KB are the same!

The first disk delay always includes OS overhead on opening a file. Consecutive I/Os for the same file (if any) do not have this overhead. Thus, for file sizes equal to two I/O blocks (16 KB), the difference between the first delay and second I/O approximates OS overhead on opening a file. The patch does not measure the duration of the second I/O. However, we can compute it, assuming that overhead does not depend on the file size for small files:

    1st_Delay   = Overhead + I/O
    Total( 8KB) = 1s_Delay
    Total(16KB) = 1s_Delay + I/O
 =>
    I/O      = Total(16KB) - Total( 8KB)
    Overhead = Total( 8KB) - I/O

It is tempting to improve the overall response time by pipelining disk and network transfers. The graph studies the percentage of atomic swap requests. An atomic requests is served using a single disk I/O. Thus, there is no opportunity for a cache server to pipeline the processing of an atomic request.

Hot memory buffer has two major functions. First, it caches incoming (new) documents. Second it caches documents swapped in from disk as a result of a disk hit. It is not obvious what documents should be cached in memory if any. To further understand how Squid memory buffer works, we track down the memory hits that correspond to documents that were never swapped in:

When a document is retrieved from its source, it is placed into memory buffer. Later, a document is swapped out to free space for incoming requests. However, before the document content is freed from memory buffer, it may be requested by other clients. Such requests result in no swap-in memory hits that we are interested in. These hits are interesting because they show how effective memory buffer is in caching new (previously uncached) documents.

There are hits and hits. We consider four hit categories or classes:

• Disk hits: Hits that were resolved by reading content from disk. Only 200 replies are considered.
• Memory hits: Hits that were resolved by sending content from memory. The requested document was in the hot memory buffer and, thus, no disk activity was necessary. Only 200 replies are considered.
• IMS hits: The client sent an If-Modified-Since request. The object was in the cache and still fresh. This corresponds to the TCP_IMS_HIT action in Squid. Note: The cache may have more recent data than the client, so Squid may send the full body with the new content. It is still a hit from Squid point of view.
• 304 hits: IMS hits (see above) with 304 reply code. Note that the 304 code guarantees that the object was not read from disk and its content was not sent to a client.
• Negative hits: Hits with reply code other than 200 or 304.

To illustrate relative importance of each class, we plot the percentage of all hits a class represents.

The graph shows the number of concurrent outgoing connections present in the proxy server.

The majority of the connections are due to misses. A hit may require an outgoing connection to verify the freshness of an object.

Proxy connect time is the time it takes to send an HTTP request to a server. A proxy may request a document from the original server or another proxy. In case of a local hit for an "old" object, a proxy may send an If-Modified-Since request. Note that IP lookup activity is not included in proxy connect time. For this graph, we ignore requests that did not contact other servers.

The graph plots the distribution of proxy connect time during peak load.

This experiment measures the time it takes to receive a reply from the original server or another proxy after a request has been sent.

Server reply time depends on the size of the reply. Here we plot this dependency. Variations in reply time for large files are due to insufficient number of such files that leads to less stable median.

Server response time is the total time it takes to send a request and receive a reply from a primary server. In other words, it is the sum of connect and reply times.

Client connect time is the delay from accept() system call till receiving a parse-able HTTP request.

Proxy reply time is the time it takes to send a reply to a client after the document was retrieved from the cache or another server. Note that due to pipelining, a reply process may start prior to receiving the last byte from the server.

To understand what affects proxy reply time it is important to distinguish several subclasses of replies. On this graph we isolate 200 hits from 304 hits. 200 hits require transmission of the entire document as in case of a miss. 304 hits require transmittion of minimal amount of information and are usually done in one system call.

Proxy reply time depends on the amount of information to be sent. This graph depicts the dependency based on 24 hour data.

Most documents served by proxies are smaller than 8 KB. Here we show reply time for small files. We plot 200 and 304 hits separately to compare reply time of a 200 hit with a miss in a fair setup.

There are hits and hits. We consider four hit categories or classes:

• Disk hits: Hits that were resolved by reading content from disk. Only 200 replies are considered.
• Memory hits: Hits that were resolved by sending content from memory. The requested document was in the hot memory buffer and, thus, no disk activity was necessary. Only 200 replies are considered.
• IMS hits: The client sent an If-Modified-Since request. The object was in the cache and still fresh. This corresponds to the TCP_IMS_HIT action in Squid. Note: The cache may have more recent data than the client, so Squid may send the full body with the new content. It is still a hit from Squid point of view.
• 304 hits: IMS hits (see above) with 304 reply code. Note that the 304 code guarantees that the object was not read from disk and its content was not sent to a client.
• Negative hits: Hits with reply code other than 200 or 304.

To illustrate relative importance of each class, we plot the percentage of all hits a class represents.

Hot memory buffer has two major functions. First, it caches incoming (new) documents. Second it caches documents swapped in from disk as a result of a disk hit. It is not obvious what documents should be cached in memory if any. To further understand how Squid memory buffer works, we track down the memory hits that correspond to documents that were never swapped in:

When a document is retrieved from its source, it is placed into memory buffer. Later, a document is swapped out to free space for incoming requests. However, before the document content is freed from memory buffer, it may be requested by other clients. Such requests result in no swap-in memory hits that we are interested in. These hits are interesting because they show how effective memory buffer is in caching new (previously uncached) documents.

http://www.cs.ndsu.nodak.edu

This page is maintained by Alex Rousskov (rousskov@plains.NoDak.edu).