Introducing Microsoft Web Application Stress Tool
The Microsoft WAS web stress tool is designed to realistically simulate multiple browsers requesting pages from a web site. You can use this tool to gather performance and stability information about your web application. This tool simulates a large number of requests with a relatively small number of client machines. The goal is to create an environment that is as close to production as possible so that you can find and eliminate problems in the web application prior to deployment.
Benefits of Microsoft Web Application Stress
Ø Create scripts in several different ways: manually, by recording browser activity, by pointing to an IIS log file, by pointing to the content tree, or importing a script.
Ø Multiple client interfaces. Run scripts from the standard Web Application Stress C++ client, use the Active Server Page client, or create your own client using the Web Application Stress object model.
Ø Use multiple users and passwords to access sites that use the most common forms of authentication and encryption
Ø Dynamic cookies that maintain a relationship with the Web Application Stress users, enabling realistic personalized test scenarios and session support.
Ø Runs as an NT service on the client machines, allowing you to log off of a client without interrupting a running test.
Ø Summary reports with extensive performance data including percentiles that remove outliers.
Ø Domain name (DNS) support allowing you to stress entire clusters.
Ø Page groups, which allow you to logically group files and control script flow execution.
Ø Delay between script item requests, producing exact time sequences for testing race conditions.
Web stress testing overview
The main objective of a web stress tool is to realistically simulate a large number of users with a relatively small number of client machines. The following discussion provides some general guidelines for web stress testing.
Web stress testing is divided into two main categories:
Performance testing
Stability testing
The first task in performance testing is to use Microsoft Web Application Stress to apply stress to the web site and measure the maximum requests per second that the web server can handle. This is a quantitative measurement. The second task is to determine which resource prevents the requests per second from going higher, such as CPU, memory, or backend dependencies. This is known as finding the bottleneck and is more of an art than a measurement.
In many situations, the web server processor is the bottleneck. Increase the stress to the point where the requests per second start to decrease, then back the stress off slightly. This is the maximum performance that the web site can achieve. Increasing stress is accomplished by increasing the stress level (threads) and/or sockets in Web Application Stress, and increasing the number of Web Application Stress client machines.
On a single processor web server, Internet Information Server (IIS) can handle 10 ASP requests at one time when the web server is configured with the default settings. If requests are incoming at a higher rate than they are being processed, then all 10 threads dedicated to handling ASP requests will be busy. After that, additional requests are placed in the ASP queue, which can hold up to 490 pending requests. If the queue fills up to 490, then the web server returns the error message "HTTP/1.0 Server Too Busy".
One performance measurement that can be used to demonstrate a web site's performance is MHz/Request/Second. This number then provides a function into which you can simply plug in the sites expected load and determine if the web pages are too expensive. The formula for this is:
(Number of Processors) * (Speed of Processors) * (average of "Processor: % Processor Time _Total" perfmon counter) / (average of "ASP: Requests per second" perfmon counter)
For example:
4 processors * 200 MHz => 800 MHz total
70% processor utilization => (800) * (0.70) => 560 MHz used
80 ASP requests per second
560/80 => 56/8 => "7 MHz per asp request per second"
Stability testing insures that the site has no threading or memory issues. Many of the short-term stability issues can be discovered during the performance tests, such as bad memory leaks and thread safety among SSO's. To determine the long-term stability of a web site, run the stress test for more than 24 hours at peak performance. Log the test run using Performance Monitor.
Using the WAS sample script
After installing and opening WAS for the first time, a script is created called Sample Script. Use this script to get a feel for some of the features in the WAS web stress tool. The actual files used in the Sample Script are stored in the folder where you installed WAS. Copy the 'Samples' folder to the root directory of your web site before continuing with this tutorial.
Script view, script items, and script item detail view
The left hand window of WAS is known as the script view. This view displays all of the scripts stored in the current WAS database. If this is a new database then the only items you see in the script view are Defaults and Sample Script.
There are seven script items in the Sample Script. Each script item utilizes a special feature of WAS. For example, notice that one of the script items is a POST. Using your mouse, highlight the row header to the left of the POST script item, then double-click. This opens the script item details view. From this view you can edit the query string name-value pairs, change POST data, modify the header, and enable secure socket layer.
Take a look at the Querystring tab view. Notice that the names User and Password are being passed in the querystring. Also notice that %Username% and %Password% are the values being passed. These are not literal values, they are special variables that tell WAS to pass the next available user and next available password from the list. WAS automatically cycles through the user names and passwords, passing the next set with each POST. See below for more information on WAS users.
Close the script item details view by clicking the OK button.
Page groups
The last two script items in the Sample Script contain the text 'adGrp' under the Group column. This is a page group. A page group is shown as 'Default' unless you change it. Page groups are used to reorganize the order in which the script items are invoked. It is also used to change the number of times that each script item is invoked while a script is running.
You can see a list of all the page groups by selecting the Page Groups node in the script tree view. You may also change the distribution percentages from this view. Notice that keep-alives are enabled for entire page groups at a time. For more in-depth discussion of page groups in Microsoft WAS, see the following knowledge base article: Using page groups.
Performance counters
Select the Perf Counters node from the script tree and click on the Add counter button. It may take a moment or so to load the Add Counters dialog when this button is clicked for the first time. Add the following counters:
· Web Service: Get Requests/sec
· Web Service: Post Requests/sec
· System: % Total processor time
· Active Server Pages: Requests/sec
Change the Collection interval to every 5 seconds. Capturing the correct performance monitor counters is critical to obtaining the correct data to analyze when the test completes. There are several important performance monitor counters to choose from, based on the type of application you are testing. Click here to view a list of the most common web related counters and an explanation of each.
Settings
Select the Settings node and change the Test Run Time to 1 minute, down from 15. Leave the other settings as they are for the time being but look over the other options in this view to get an idea of what can be configured for a script. You may also set the default settings for all new scripts by selecting the Defaults node and changing those options. Keep in mind that the settings in the Defaults node will not affect existing scripts, such as the Sample Script.
The Settings dialog lets you configure how the requests will be run against the server. You can specify the number of simulated clients by setting the number of threads and number of sockets on each thread – in this case 10 threads and 10 sockets yields 100 simulated clients.
This dialog is your most important tool in tailoring the test for the appropriate load factor on your test Web site. The first items are Stress Level and Stress Multiplier which determine the simulated number of users:
This property determines the number of threads that will be run by WAS to hit the client application.
This property determines the number of sockets that are created on each of the above threads.
The end result is that the Stress Level times the Stress Multiplier equal the number of clients you are simulating. Threads * Sockets = Total clients.
Test Run Time
The request delay allows you to provide more realistic user simulation, since users don't continuously click on links as soon as a page loads. Typically users look around a page, find a link and then click it. Even a familiar user may take 5 seconds between requests – new users will take much longer. I'll talk more about when you want to add a delay and when you don't later in the article.
This option causes WAS to monitor the traffic being generated both on the outgoing on incoming links and optionally allows limiting the bandwidth available. This is useful if you're generating large amounts of traffic and you may have to contend with the possibility of testing for overloading your incoming Internet pipe. In typical application it takes a huge amount of volume to overflow even a T1 connection, but this may vary depending on your site's content. Even though throttling is available, there's no mechanism in WAS that tells you when you overrun the bandwidth – you have to look at the resulting report for hints of bandwidth usage.
Follow-Redirects
If the
tested site includes redirect headers which cause pages to stop executing the
current page and instead go to another page, this option allows WAS to follow
those redirects. Redirects are common in ASP applications to route code logic
from one page to another. Many applications that login or otherwise manually
authenticate users tend to often use Redirects. The Max value determines how
many successive redirects are followed – one would hope to never see these more
than 1 level deep, but the folks of Microsoft were thinking ahead for sick and
twisted minds.
The remaining options should be left alone except in special situations – you can review those items in the WAS help file.
Select
the Users node and
double-click on the Default user population, indicated by the following icon: 
.
This opens the Users view where you can add and delete users from the default
population and create new populations. Each WAS user stores cookie information
and authentication data. Notice that there are 20 users in the default
population.
WAS users are not the same as Stress level (threads), a setting located in the Concurrent Connections section of the Settings node. The two concepts should be kept separate.
Clients
Select
Scripts from the
View menu to return
to the Script view. Select the
Clients node under the Sample Script and double-click on the
Default client group, indicated by the following icon: 
.
This opens the Clients view where you can add and delete client machines from
the current group or add new groups of client machines. Notice that 'localhost'
is the only client and that it has a check box next to it. This means that the
current machine is acting as a WAS client. Leave the Client view as is and
select Scripts from
the View menu to
return to the Script view.
For more in-depth discussion of users and clients in Microsoft WAS, see the following knowledge base article: Understanding Threads, Users, and Clients.
Running a test
Once you have created a script and configured all the settings, users, and clients, it is time to start the test. Select the Sample Script and choose Run form the Scripts menu. Allow the test to complete.
Reporting
Select Reports from the View menu to open the Reports view. Expand the Sample Script report to display all of the report nodes. There should be at least one node who's title is the date and time in which your latest test was started. Expand and select the top level of this report node to view a summary of this test.
You can select a specific node of the report tree to view more in-depth information. For example, select the Result Codes node to display a sum of the HTTP result codes for every request in the test. Expanding the Perf Counters node displays the counters that have been collected during the test.
Expand the Page data node and select the first script item. The right hand pane displays detailed information regarding this script item. Reports provide the response time for specific pages by determining when the script has finished downloading on the client. This is a good source of performance data.
The Time To First Byte (TTFB) calculates the time from the request for the page until WAS receives the first byte of data, in milliseconds. The Time To Last Byte (TTLB) calculates the total time from the request until the last byte of data has been received on the client, in milliseconds. This number includes the TTFB time and any additional time needed to receive the last byte of data. All of the requests are sorted, then the data is divided into percentiles. For a more in-depth discussion of the p-squared algorithm and percentiles, see the following WAS knowledge base article: Percentiles in WAS.
While in report view, you can select Export to CSV from the File menu. Exporting the report values to a format that Microsoft Excel can read allows you to create charts that show where most of the requests fall. For example, the following chart illustrates Time To Last Byte:
Description of Scripts
Virtual root
The virtual root field of the content tree window is used to establish a place holder for the paths in a script. If all of your paths are located in a consistent vroot, you can specify this in the Virtual Root box so that all script items are preceded with the correct vroot name. For example, if all of the path's in a script are located in the /production/content virtual root of your web server, type this in the Virtual Root box.
Setting
Concurrent Connection
Stress level vs. stress multiplier
In the settings window under concurrent connections, Web Application Stress allows you to specify the Stress level (threads) and Stress multiplier (sockets per thread) for greater control of a test run. Stress level is the total number of Windows NT threads that are created across all of the clients. Each thread can create multiple sockets and each socket is a concurrent request. The following formula explains this relationship:
Total Concurrent Requests = Stress level (threads) x Stress multiplier (sockets per thread) = Total Number Sockets
From the web server's perspective, it just sees lots of incoming socket connections. The server doesn't care what the thread architecture is behind the incoming requests, so your stress is equal to the multiple of threads and sockets, not simply threads alone.
In general there isn't much reason to increase the Stress multiplier (sockets per thread) value above 1. However, there is one scenario where increasing the sockets per thread is useful. Windows NT has a limit of 2,000 threads per server, and could start to bog-down after 100 threads are created. Increasing the sockets per thread allows a single client machine to more efficiently generate current connections greater than 100, thereby increasing the stress. This is most effective when testing large static (HTML only) files with bandwidth throttling enabled.
Request delay
The request delay imposes a limit to the size of data Web Application Stress receives from each request. To set this option, click Settings in the Script Tree and check Use random delay under the Request Delay section.
One way to use this feature is for fine tuning modem throughput simulation. For example, Web Application Stress receives data in 1.2K chunks. A 28.8 modem's throughput is typically 3 kilobytes per second. Setting the random delay between 300 and 400 milliseconds will prevent Microsoft Web Application Stress from receiving more then 3 kilobytes per second per thread.
Warmup Time
The length of time used in the warm up setting, in the format: hh:mm:ss.
Cooldown Time
The length of time used in the cool down setting, in the format: hh:mm:ss.
Bandwidth throttling
Microsoft Web Application Stress simulates modem throughput. This allows you to simulate any bandwidth from a 14.4 modem to a T1 line during your stress tests. The largest benefit of this feature is the ability to increase the number of concurrent connections on a web server, thereby simulating lots of modem users. The limitation of this feature is that it will not work with files that are less than 1.2 kilobytes in size.
To use this feature:
1. Select Settings under a script in the Scripts view
2. Click Throttle bandwidth.
3. Select the bandwidth of interest.
The following throughput settings are available:
14.4
28.8
56 k
T1
T3
HTTP redirect support
Many web applications use HTTP redirects to forward user requests to another page. This is commonly used to maintain consistency as pages change or move. Web Application Stress will follow a configurable number of redirects so that you don’t have to track these changes when creating your test script.
Redirection is on by default but can be disabled from the Settings for each script. The Max edit box in the Redirects section allows you to specify a maximum number of redirects allowable from one request to prevent an endless loop condition from occurring in your test script.
Performance Counter
Microsoft Web Application Stress has an integrated Windows NT performance monitor interface to simplify test data collection. You can store your favorite Performance Monitor counters with each script.
To add performance monitor counters, do the following:
1. Select Perf Counters, located under a script in the script view window.
2. Set a collection interval. This is the time, in seconds, that the samples will occur.
3. Click Add Counter. This may take a moment to load the first time you invoke it.
4. Select the machines, objects, and counters you are interested in collecting by clicking Add after each selection.
5. Select OK when you are finished.
Creating users
Users in Microsoft Web Application Stress allow you to simulate the feature of browsers that provide personalized data storage and authentication. To use cookies or authentication you must add at least one user. To create new users, do the following:
1. Click the View menu and choose Users to open the users view.
2. In Number of new users, type the number of users you would like to create.
3. In User name prefix, type a value that you would like to precede each user’s number. If you would like to simply use numbers for the user names, leave this field blank.
4. In Password, type a password. This same password will be assigned to all users.
Client status
When a client is added to the list of stress clients, Microsoft Web Application Stress checks the connection to the remote client and displays one of the messages listed below. Web Application Stress does not prevent you from starting a test with clients that display "FAILED" or "Wrong Version". These clients will not be used; however there must be at least one working client machine in order for the Web Application Stress master to collect data at the end of the test.
Using Cookies
Intelligent use of cookies in your stress tests will make the results more accurate. There are two types of cookies in Microsoft Web Application Stress, static and dynamic.
Static cookies do not change with each request the way dynamic cookies can. These cookies are stored in the header of each script item as static text. Static cookies are created in two ways:
1. Recording a script against a site that hands out cookies.
2. Reading an IIS log file that contains cookies in one of the imported fields.
Dynamic cookies require that you first create at least one virtual user. Each of these users can store one dynamic cookie for each web server that they are directed against. This type of cookie is created in two ways:
1. The first, and easiest way, is to direct users against a web page on the site that real life visitors use to customize. Typically the page can be hit with a querystring that sets personalized data. An advantage to this method is that both the pages that set personalized cookies and the pages that use them are tested. A disadvantage is that typically all users will have the same personalized settings, determined by the query string used to hit the page.
2. The second way is to create a custom web page which sets cookies specifically for Web Application Stress users. Since you have complete freedom over this page, you can set cookies that span a range of random values. Each user will then have a unique cookie. This method is more advanced and requires some knowledge of writing web pages. It also requires a familiarity with the cookies used on the site and their expected range of values.
Reporting
Reports provide you with considerable data from which to build your final test results. The overview section displays general information about the test such as how long it was run. The script settings section displays the configuration used during the test run, such as the number of concurrent connections. The test clients section displays which clients were used. The result codes section summarizes the number of successful and unsuccessful hits. The page summary shows you the performance of each page at a glance. Page groups expand on the distribution of hits across page groups. Page data displays in-depth information associated with each script item in the test and perf counters show you the performance monitor counter values captured over the test run.
Reporting - overview section
Report name: The name of the current report. You can change the name,
otherwise it defaults to the date and time that the current script ended.
Run on : The date and time that the current script started.
Run Length: The length of time that the current script ran, in the format
Dd hh:mm:ss.
Stress Version: The current version of Web Application Stress used to create
this report.
Number of clients: The number of clients specified in the clients view. Note that
this does not reflect the number of clients used in a test since some of them may have failed.
Number of hits: The total number of requests made by Microsoft Web
Application Stress during this test run.
Requests per second:The average requests per second throughout the test.
Socket errors: Connect – The number of times any client failed to connect
To the web server.
Send: The number of times any client failed to send data to the
Web server.
Recv: The number of times any client failed to receive data from
The Web server.
Timeouts: The number of threads that timed-out, and subsequently
Were Closed.
Reporting - script settings section
Server: The name of the current report. You can change the
name, otherwise it defaults to the date and time that the
current script ended.
Number of threads: The number of threads that the test used. These are
Concurrent connections.
Sockets per thread: The number of sockets that used in this test.
Test length: The total test time from start to finish, in the format: dd
hh:mm:ss.
Warm up: The length of time used in the warm up setting, in the
format: hh:mm:ss.
Cool down: The length of time used in the cool down setting, in the
format: hh:mm:ss.
Use random delay: A value of either Yes or No stating whether this test used a
Request delay or not.
Throttle Bandwidth: Displays whether bandwidth throttling was used in this test
Or not.
Follow Redirects: Displays whether the use of redirects was supported for the
current test.
Max Redirect Depth Displays the maximum number of redirects that were
allowed in the test run.
Reporting - test clients section
Clients used: A list of machines that Web Application Stress was
Successfully able to invoke during the test.
Clients not used: A list of machines that Web Application Stress could not use
during the test. See Socket Errors in the Overview section for more information.
Reporting - result codes section
Result codes: The count of each type of HTTP result code that occurred
during the test.
Reporting - page summary section
Page: The script items used in the test.
# Hits: The number of times a request for this page occurred. Note:
this value may also include errors.
TTFB Avg: Time To First Byte Average. The total time, in milliseconds,
from the initial request until the first byte of data is received on the client. This is an average over the test.
TTLB Avg: Time To Last Byte Average. The total time, in milliseconds,
from the initial request until the last byte of data is received on the client. This is an average over the test.
Reporting - page group section
The following items are displayed for each page group.
Distribution: The total distribution for this particular page group.
% Total Distribution The percentage that this page group
was utilized as compared
to all other page groups in the test.
Hit Count: The total number of times a page in this page group was
requested.
% Total Hits: The percentage, of the total hits against all page groups,
that this page group was utilized.
Result Codes: The count of each type of HTTP header result codes
experienced during the test.
Reporting - page data section
Path: The name of the path that the following fields apply to.
Hit count: The total number of times a page in this page group was
requested.
Socket errors: The total number of times a page in this page group was
requested.
Result codes: The percentage, of the total hits against all page groups,
That this page group was utilized.
Result Codes: The count of each type of HTTP header result codes
Experienced during the test.
Downloaded Content
Length (in bytes) Min-The minimum size of data received by the client, in
bytes.
Max –The minimum size of data received by the client, in
bytes.
Time to first byte
(in milliseconds) Min – The minimum time that it took to receive the first
byte of data on the client.
Max –The maximum time that it took to receive the first byte of data on the client.
Time to last byte
(in milliseconds) Min – The minimum time that it took to receive the last
byte of data on the client.
Max – The maximum time that it took to receive the last
Byte of data on the client.
Limitations of WAS
Advantages aside, WAS does have a few limitations. The current known bugs and issues are listed on the WAS Web site. Here are some features that are not currently supported in WAS:
The ability to modify the parameters of a URL, based on results returned from previous requests.
The ability to run or emulate client-side logic.
The ability to specify a fixed number of test cycles for the duration of the test.
The ability to run tests simultaneously against multiple Web servers with different IP addresses or domain names.
The ability to support redirection of pages to another server with a different IP address or domain name .
The ability to record SSL (Secure Socket Layer) pages directly from a Web browser