Run Performance Tests

When a performance test is run, Web Performance Trainer^TM will simulate a number of users using their browser to access your web site. You can configure the test to automatically increase the number of users, up to the limit of your license. The product can simulate up to 200 simultaneous users per machine. This may seem like a small number compared to the large number of hits reported by major sites, but it is a typical maximum number for a single server to handle while giving an acceptable response time.

Preparing the Test Machine
Running a performance test is a CPU and memory intensive operation. In order to get the most out of your test machine, and insure the most accurate statistics you should make sure there are no background processes running on your computer. While the performance test runs Web Performance Trainer^TM 2.2 will monitor your CPU usage and make sure you don't overload your machine. When the CPU Usage/Load Average gets too high Web Performance Trainer^TM will stop adding new virtual users. Note that if your machine's CPU load is too high at the beginning of the test, no users will be added at all! At this time check your machine's CPU load average:

Configuring The Performance Test
The following screen shot shows the playback controls configured to run a test on a single business case, "Purchase Transaction", for a duration of 10 minutes, starting with 25 simulated users. The number of users will increase by 10 every 1 minutes, so that the potential number of users at the end of the test is 115. It is best to start with a low number of users and verify the correct working of your web-back end before performing more complicated tests. Also note that the number of virtual users you can simulate is limited by the speed and memory of the playback machine, so that the actual number of virtual users generated can be lower than the value in the potential field.

Tests can be run on either single business cases, or on "load profiles", which are groups of business cases. Select the test type by clicking on the appropriate radio button, and "Select Test" pull down menu will be populated with either business cases or load profiles you previously created. When business cases are run directly they are automatically configured for the most common settings, using the random start and recorded playback pacing features. In order to override these settings you'll need to configure a load profile.

Duration can be in units of hours, minutes, or days. The duration of the test should change depending on your testing goals. If you are just trying to get an idea of the speed of certain operations on your site, useful performance information can be gained for tests that are a few minutes long. You can then tweak parameters in scripts or machine configuration and see if it has an effect on performance. If, however, you are trying to stress your web site to see if anything breaks, you'll want to run the test over a longer period of time. While the program can be configured to run for long periods of time, this will generate a lot of data, and may cause the program to run out of memory. In this case, you should configure the sample period, or the time period between saving statistical data, to be longer. Typically, if you're going to run for multiple hours, you should have sample periods that are on the order of minutes, while during short tests you can have sample periods as small as 5 seconds.

The modem speed parameter describes the speed of each virtual user in the simulation. This means if you recorded a business case over a local LAN, playing that business case back at modem speeds will take much longer. The default is to give each virtual user the same modem speed. If you wish for each business case to be played back at different modem speeds, you'll need to configure a load profile.

The goal of a performance test is to determine a relationship between the number of virtual users and performance. In order to do that, you'll want to describe a ramping number of virtual users, to see how performance changes in relationship to the number of users.

This section of the GUI allows the user to describe the performance test in terms of the starting number of virtual users, and how fast to add new virtual users. You will want to pick a starting number of virtual users that's at least as big as the number of business cases, while keeping the starting number low enough to leave room for the number of users to increase. A typical value is between 1 and 50 virtual users. Note that you need to start with at least as many virtual users as you have business cases.

The "increase by" value is how many virtual users to add in a period, usually between 1 and 5 minutes. Typically this value ranges from 1 to 50. Again, you want to generate a ramping value, in order to determine the relationship between load and performance. Note that be default the maximum number of users to increase is 50.

When running larger number of virtual users generated by multiple computers these values may be low. In that case, edit the configuration file trainer.cfg in the installation directory, and modify the parameters MaximumStartUsers and MaximumIncrementUsers.

The sample period is the length of time over which statistics will be sampled before saving the values. For example, if the sample period is 15 seconds, the statistics views showing the results of a test will have values every 15 seconds. This value should be shorter for short tests, and longer for long tests. For example, if your test only lasts an hour, then having samples every 10 seconds makes sense. If, though, your test is intended to run overnight, then the sample period should be much longer, in the area of 5 minutes. This helps make the data easier to interpret.

Potential:	The number of virtual users the current settings will create if the test is allowed to run to completion.
Limit To:	A value you choose to limit the number of virtual users. If you know your computer can only handle 200 virtual users with good performance, then you can set the limit to 200.
License:	The number of virtual users you are licensed to simulate.

Running The Performance Test
Once the performance test is configured it can be started by clicking on the Start button. The test can be ended at any time by clicking on the Stop button.

Field:	Definition:
Start Time	The time the test was started.
Duration	The length of time the test has been running
Hits	The total number of HTTP commands that have been generated.
Hits/s	The current number of HTTP commands per second that are being generated.
KB/s	The total number of kilobytes per second that are being transferred to and from the web site. This includes all data, including HTTP Request Headers, HTTP Reply Headers, HTML content, and images.
Cases/s	The number of business cases that are being generated per second by the test.
Errors	The number of HTTP errors that were generated. This will indicate if the simulated users are able to access the web site.
Users	The current number of users being simulated.

Inspecting Errors
Errors are displayed on the Error Tab on the same page, shown in a list giving the time the error occurred, the URL that generated the error, and a description of the error. This table will update dynamically during playback so you can view any errors instantly during the test. Note that it may take up to a minute between the time an error is reported in the instantaneous statistics, updated every two seconds, and the Error Tab, which is updated when all test statistics are collected each minute. The section on validation has a list of the possible causes of errors and their meanings.

More details about each error (if available) can be viewed by double-clicking a row in the Error Tab. The following dialog will show the headers and content received from the server (if available). The "View in Browser" button allows the page to be inspected in a browser (if configured).

Monitoring the Test Machines
When the performance test is running Web Performance Trainer^TM will monitor the CPU Load and increase the number of virtual users if that option has been selected. Playback can occur from either a local playback engine, or from one or more other computers, each running a remote playback engine.

While the performance test is running you can also use the Statistics and Graphs tabs to view the data collected during a test. Note that while you can view the stats during a test, they will not update dynamically, meaning you'll have to click on the selected business case, web page, or url to redisplay the statistics associated with that element. This is done to keep performance demands of the GUI at a minimum, so that the most CPU power is available for generating virtual users.

Field: Definition:

Computer Starting in version 2.2, multiple computers can be controlled so that large numbers of virtual users can be generated. This field contains the names of the computers that are running remote engines. If the value is "localhost", this means the playback engine embedded in the current computer is being used. This is not recommended if remote engines are also being used.

Status Indicates whether playback is occurring or not.

Active Users The number of virtual users actually running.

Est User Capacity The estimated number of virtual users that could potentially be generated by Web Performance Trainer^TM on each computer. This is not the current number of virtual users being generated, nor does it have anything to do with the web server's capabilities. This number is displayed to give the user an idea of the power of the computer running Web Performance Trainer in terms of how many virtual users could potentially be simulated, and is used in the load balancing algorithm to distribute the load generating task among multiple computers.
Note that the estimated number usually is inaccurate at lower load averages, so that your computer very well may be able to generate a larger number of virtual users. This is because at low load averages the estimation is not as accurate as at a high load average. Also, the response of many computers is nonlinear, so that the load average could hover at 20%, for example, and stay there while the number of virtual users climbs.

% Memory Used This measures how much of the memory allocated for internal buffers is actually in use. This number has no relation to any operating system specific information you might view using the NT Task Manager.

This value will go up and down during the performance test, but will slowly creep up towards the value of the Total Memory below when using large numbers of virtual users, or when running the performance test for a long period of time. When the program absolutely has run out of memory you will see this value quickly climb into the 90% range every 30 seconds or so. When this happens the playback may stop to prevent the program from running out of all memory and ceasing to operate.

When the figure hits about 90% of the Total Memory, you can conserve memory by deleting old performance statistics and graphs, or by starting a new file and importing the business cases to be used in further tests. See the Memory Management page for more information.

Memory Status This indicates if the program is low on internal memory buffers. It is normal for the status to be "overloaded" during operation. When this happens the internal memory buffers will be reclaimed and the status should go back to normal. If the status is continually in the "overloaded" state then the program may stop playback in order to prevent a total loss of memory.

Percentage Load The CPU utilization of the playback computer, where 100% has all of the machine cycles being used. Note that on UNIX this value is greatly affected by background processes that are blocked, so even though a process isn't taking up any CPU time, if it is stuck in a disk wait or otherwise hung your load average will be higher. Use "ps", "top" or other programs to find and stop background processes that may be increasing the system load so that the full power of the computer is available for the performance test.

Note that there is lag in getting the information from the NT operating system, so the value will not be exactly the same as the value displayed in the NT Task Manager.

Load Status This indicates how much available load capacity your computer has available. Any value over 80% is considered to be "overloaded", and your computer is working at maximum capacity.