Thoughts about LoadComplete - Trial User

Mike, I appreciate the time you spent to review all my observations.  Some of them were my mistake for not reading all the documentation (such as on complex scenarios).  Otherwise, my replies follow.



Q:

Expected
HTTP Responses:  I would like to be able to globally configure HTTP
Response Status Codes to mark them as OK, rather than having to go
through the Scenarios, request by request marking 302s acceptable as 200
OK.



A:

What
do you mean by "globally"? Do you mean the scenario or project level?
IAC, if LoadComplete was configured to treat 302 responses as OK, it
would fail to point out real mistakes in the log. Can you share a use
case for such a setting?



Comment: After recording relatively brief scenarios and then running them, I typically end up with 5 or 6 warnings for 302.  The recording acts as expected based on viewing the user traversal via the request/response pairs, so I'm fine with treating the 302s as OK.  However, as far as I know, I have to go click on each warning after it was run the first time and manually choose to treat 302 as OK.  I would like to have a way to dismiss all 302 warnings as OK in one step.



Q:

When
viewing a scenario, with NO grouping of Requests as Pages, I would
like to sort Requests by extension type.  The available columns are
only Request and ThinkTime.  I'd like an Extension column that is
sortable to I can group and then delete undesired extensions in one
swoop.



I would like to have a GLOBAL option to exclude
recording or
testing of list of file extensions.  In other words, I'd like an
option to exclude all images, all javascript, and/or all style sheets
from the test scenario, as well as by specific file extension or
pattern.



A:

Could you please give us a use case for excluding such requests from scenarios?



Comment: I only care about testing the web application server (i.e. ColdFusion), not the web server (IIS).  I don't want to load test requests for JavaScript, Images, or CSS sheets.  I want to easily eliminate from my recorded scenario all request types that I don't care about.  To facilitate that, I would like to be able to sort requests from a recorded scenario so that I can group the extensions, the select by groups, and delete those I don't care about.  Presently, I go to Page view, then expand the page, expand the "Other Files", visually inspect the extensions listed there, then delete them.  Then I go to the next Page view, Page by Page.  In Request view I can see all requests for all pages at once, so it makes sense to sort/group extensions there for easier removal.



To avoid having to edit a recording to remove unwanted extensions from scenarios, a global setting to exclude by extension type (.gif, .jpg, .css, .js) and another to exclude by category (images, css, javascript). That would let me record without those file types without having to go back and remove them later.



Q:

After
adding new parameters, if I accidentally click
"Edit one more parameter" again unintentionally then there is no
option to just Finish or Go Back.  I must either Cancel or create a new
Variable.  I'm afraid that clicking Cancel will cancel all the variables
I previously added.



A:

Do you think that renaming the button to "Finish and edit one more parameter" will be OK? Or, do you have some other vision?



Comment: After clicking Edit One More Parameter, I think a Go Back button would be best.  When using it, I realized I had made a mistake and I wanted to go back then click Finish instead, but I couldn't go back.



Q:

When renaming a Scenario, the Run button drop down list does
not update to reflect the new name, and cannot play the old named Scenario



A:

I'm
not sure how to reproduce the issue. Can you give me more details along
with images that will demonstrate the steps to reproduce it?



Comment: The issue exists, but I didn't describe it accurately.  When multiple "Tests" exist, the green button for Run will have a drop down option list showing the test names.  If I rename a Scenario then choose a Test from that Scenario in the Run button drop down list, then I get a popup alert for "Unable to find the "XXXXX" scenario" where XXXXX is the original name of the scenario before it was renamed.



Q:

For a while, the Play button showed as disabled (gray'd out) even though I have a
scenario that passed Verify Scenario.  I had to relaunch LoadComplete to get the play button to work again.



A:

Does this problem persist on Beta 2? If it does, can you share the steps to reproduce it along with your test project?



Comment: I only encountered this problem once and haven't seen it again.  If I do, I'll try to document the behavior better.



Q:

I
want to be able to complete several performance tests repeatedly, then
have LoadComplete average the results of all trial runs as a group.  Then I want to
modify my application, and run another group of performance tests,
average those.  Finally, I want to compare the different test groups and see JUST the
delta of changes between the two test groups. 



A:

Again, could you please share a use case?



Comment: I want to know the effect on performance of one change to my web application.  I would like to have a statistical analysis to tell me if the change to the web app had a statistically significant impact on performance.  In order to perform valid statistical analysis, I need to have a reasonable sample size of test results before the change AND test results after the change.  Then I can apply a test like Chi Squared (as explained here) to tell me definitively if the change that was made really impacted performance or not.



To do this properly, it would be great to have a load test tool that can repeat a load test N number of times, then let me make a change to my web app and reset the test environment, then have the load test tool run the same load test the same N number of times.  Ideally the load test tool would automatically apply the chi squared formula to the test results and show me in simple terms if the change had an impact on performance (in either direction, better or worse).  You could hide the complex math explanation from the user, and just provide an easy to use wizard with a simple graphical metaphor to indicate to the end user if the change was significant or not.



Thanks again for all your replies.