Community Day 2021 - Regression Testing of Performance

[sonya_m]

Let's move on to the next session of the day. Not to miss any sessions, subscribe to the event tag - #CommunityDay2021. We will be posting great content for the entire day today!

Regression Testing of Performance

by Alexandr Gubarev, TestComplete Senior Test Lead

The session takes a really deep dive into the subject. This talk is based on the experience of the TestComplete and ReadyAPI QA teams when they had decided to rapidly extend test suites to protect users from performance regressions.

This video will be useful for those who want to add performance checks to their functional tests, regardless of the type of application: Desktop, Web, Mobile, or even a web service. We will define a common approach when and for which scenarios you need to do regression performance testing, tell you about the useful features of TestComplete, learn advanced Google Sheets practices, and share recipes for quickly implementing performance checks in your tests.

Become a Community Day Winner!🏆

Watch the video sessions, post your questions and give Kudos to get the event points!

Read more about participation rules.

Watch the session:

The VideoRecorder extension developed by Alexandr mentioned in the interview.

Timestamps:

00:00 Speaker introduction

03:22 About regression and performance testing

04:45 Example of the case

06:04 Defining scenarios

09:28 Steps of tests

10:58 Using the Google Sheets for POC

11:53 TestComplete: aqPerfomance object

13:22 TestComplete: Time with Children in Log

15:15 TestComplete: Performance Counters monitoring

16:26 Demo: defining of the scenario

17:15 Demo: high-level steps

18:12 Demo: measuring

20:13 Demo: Google Sheets as storage

21:04 Demo: report table

22:43 Demo: creating web service on Google Sheets

24:51 Demo: how to publish your report for manager

25:22 Demo: work with web service from TestComplete

26:38 Demo: tests run

29:37 Some final words

Any questions on regression testing of performance? Ask away in the comments below!

[AGubarev]

@AlexKaras ,

yes, we have a lot of internal things which we will be glad to share.

Thanks @sonya_m and @TanyaYatskovska we've had a chance to "say" something to community 🙂

For now, we have no ready video/tutorial for that. But I found that we already have something in Community: https://community.smartbear.com/t5/TestComplete-Questions/Allure-Reporting-using-TestComplete-14-1-J...

[AlexKaras]

Alexandr,

> I found that we already have something in Community:

Thank you for the link! I seem to miss this thread as I don't remember it. Will read it for details.

> For now, we have no ready video/tutorial for that.

This was a kind or request for the future 🙂

[cauline]

There is also another way to present trends with your tests in Jenkin's.  We use an extension called 'Plots', that in our case, we use to graph page load times daily/weekly.

[AlexKaras]

Off-topic:

> after that can investigate this deeper with some additional tools like profilers: https://smartbear.com/product/aqtime-pro/features/performance-profiler/

Irony of life: AQTime (MemProof) was the initial product by SmartBear (AutomatedQA and TotalQA (if I remember it right)) and TestComplete was created to automate its testing (as it was said here somewhere in time).

Now AQTime is used to performance-tune TestComplete... 🙂

I would like to believe that performance, resources usage and memory (leaks) control and usage optimization are still of significant importance for  at least some companies / projects and that AQTime is not an internal product of low importance for SmartBear.

Might it be an option to have an event like this Community Day but dedicated to AQTime sometime in the future? Maybe with some detailed and technical demos of how to use AQTime with modern .Net applications?

[AlexKaras]

> Thanks @sonya_m and @TanyaYatskovska we've had a chance to "say" something to community

Our sincere kudos go to Tanya, Sonya, management who approved this event and all guys who participated in the event and assisted in its preparation.

[Marsha_R]

Thanks for the video!  Do you have a preference when to start performance testing for a baseline of a new project.  Would you start at the beginning and see how performance is affected by each new feature or wait till the end or close to the end and then try to fix it?  I can see a case for both.

[AGubarev]

Hi, @Marsha_R 

Thanks for the question.

From my experience, when we start test projects from scratch, first of all Devs need to "understand" and implement basic functionality. And this functionality is near to POC. So, it can be changed in any moment. So, I'd concentrate all QA resources on functional tests. Maybe the only exception, if we talk about performance, is load/stress testing. I remember the case, when we as QA perform load / stress of the the raw version of client-service app, and as result developers totally change the infrastructure and solution vendors (the previous had limited scale capabilities). So, if you test project which potentially will be scaled (it has database, or server on background) and it's not typical project for your company. It's reasonable to investigate limitations on the earlier stages.

Regarding to performance regression testing, I still think that QA resources are "luxury", so, product teems should use it on most critical scenarios. And it's hard to predict such scenarios if the app is not used by real customers. Of cause, there are obvious issues with performance. But from my experience even ordinary functionality tests can found them. So, to sum up, I cannot see the cases, when it reasonable to develop such kind of tests at least before the core functionality is covered with functionality tests

[AlexKaras]

Hi Marsha,

> Do you have a preference [...]

If I may jump in, I would suggest intermediate suggestion - start when the foundation of solution has been established and it was decided that it is good enough and may and will be used from now on (i.e. that generic architecture of solution has been tried and it was decided that architecture (core engine) is approved). This seems to be a good moment to create initial baseline and then move on and monitor how each new feature affects performance.

Does this sounds reasonable?

P.S. Update: It appeared that I wrote practically the same that @AGubarev replied three minutes before 🙂

[sonya_m]

This has been one of the most active threads, thanks everyone for being so involved!

The best question bonus points will go to @hannecroonen! 

[Lee_M]

@AGubarev 

arguments.callee.caller

I will further investigate this approach, but last time I ran it, it got one of the subfunctions names (eg Close dialogue, which is called multiple times and is not the parent script run)