Forum Discussion
william_roe wrote:
I'm curious why you don't use keyword tests and name mapping. We use a mix of keyword and script and find only the keyword referebces refactor / rename when objects change (i.e. move). The same would be true for not mapping. I can't image managing references such as:
Sys.Browser("chrome").Page("http://localhost/CMWebTest/Go/?p=/CMWebTest/Crib/Properties/1").Panel("primaryWrapper").Section("primaryContent").Frame("t_cmwebtest_crib_properties_1_content").Section("actionWrapper").Section("actionContent").Panel(0).Form("mainForm").Panel("partialCribProperties").Panel("partialCribProperties_CC").Panel("partialCribProperties_C0").Panel(0).Article(0).Panel(0).Panel(6).Panel(1).Table("AlertEMailAddress_ET").Cell(0, 1).Table("AlertEMailAddress").Cell(0, 0).Textbox("AlertEMailAddress_I")
we create a function call for each gui object to find / refer to the object.
so we'd have something like SetText( alert_email_textbox(), ' some text); and then in an objects file we'd have
function alert_email_textbox() {
return alert_page().FindChild( new Array("ObjectType", "idStr"), new Array("Textbox", "AlertEMail"), FIND_DEPTH);
}
chicks wrote:
william_roe wrote:
I'm curious why you don't use keyword tests and name mapping. We use a mix of keyword and script and find only the keyword referebces refactor / rename when objects change (i.e. move). The same would be true for not mapping. I can't image managing references such as:
Sys.Browser("chrome").Page("http://localhost/CMWebTest/Go/?p=/CMWebTest/Crib/Properties/1").Panel("primaryWrapper").Section("primaryContent").Frame("t_cmwebtest_crib_properties_1_content").Section("actionWrapper").Section("actionContent").Panel(0).Form("mainForm").Panel("partialCribProperties").Panel("partialCribProperties_CC").Panel("partialCribProperties_C0").Panel(0).Article(0).Panel(0).Panel(6).Panel(1).Table("AlertEMailAddress_ET").Cell(0, 1).Table("AlertEMailAddress").Cell(0, 0).Textbox("AlertEMailAddress_I")
we create a function call for each gui object to find / refer to the object.
so we'd have something like SetText( alert_email_textbox(), ' some text); and then in an objects file we'd have
function alert_email_textbox() {
return alert_page().FindChild( new Array("ObjectType", "idStr"), new Array("Textbox", "AlertEMail"), FIND_DEPTH);
}
So in lieu of mapping objects the FindChild method is used. I could see how this would reduce the sensitivity to objects getting moved..How's the speed? I'm assuming you use 'eval' on consumer side of this method. Can you show what that code looks like? Is the idStr attribute always used? Our templates use ObjectIdentifier
- jsc9 years agoRegular Contributor
Team
Our general setup in our scrum team was this:
- 4 developers
- 1 tester (spending ~70% of his time for this team for general testing purposes)
- 1 product manager (spending ~60% of his time for this team)
Now we are
- 7 developers
- 1 tester (spending ~70% of his time for this team for general testing purposes)
- 1 product manager (spending ~60% of his time for this team)
- 1 student working ~30% for testing purposes
I am the tester above, having studied computer science, but I do not have any experience working as a developer.
Process
In our team I was the only tester, responsible for testing in general. Before I started there was no automated testing at all and developers had the attitude that developing software is just writing code that is writting as quick as possible and did not take responsibility for any testing purpose.
My believe is, that creating good code could not be done without various testing-activities including automated regression testing, code reviews,... Nobody wants to (or could do) do regression testing all the time so it is a must to automate this.
So I started step by step introducing more and more of these activities. I introduced them in this order:
- automated GUI regression tests (TestComplete and TestExecute)
- code reviews
- integration tests
- unit tests
It would have been better to do it in a different order but it was not possible. It have been hard fights with the head of development but the product manager was on my side and we could convince our CEOs as well. (some major bugs helped :-) )
We were able to make the developers take responsibility for manual testing, code reviews, unit tests and integration tests.
Now the last step is coming that the developers also should do the automated regression testing.
Right now I am doing a bit of automation but mainly focus on improving the process (teaching one of the developers checking daily the logs of our tests) and improving the process and testenvironment of the regression test (testing architect). We have a computer science student working ~1.5 days per week automating issues with TC. (we had another computer science student working for us the last year. Most of the time she worked 1.5 days per week and half a year she worked full time)
Regression testsuite (TC, TE)
Our testing suite in TestComplete is huge.
We track coverage of what to automate and what not in all of our JIRA-issues. Everything that can somehow be seen in the UI has to be covered in this testing suite.
If the suite would be run completely I think it would take ~11-12 hours. As this is too slow to get results in time we split the tests and run them in parallel. Unfortunately this could not be done properly in TestComplete itself. So we wrote some scripts doing this for us. Now the tests are automatically started each night (or manually during the day) by a job in jenkings. Afterwards the test runs on 7 virtual machines in parallel providing the result within ~2 hours. So after 2 hours our complete software is regression tested.
The test is under version control (git) and before the start of the test the test is checkout out.
All of our testitems can be run on their own. No testitem depends on another. The testitems should take less than 4-5 minutes (there are some legacy tests, that run stable but take way too long).
We are mapping the needed items for test before writing tests using them.We only map manually! The namemapping is the core of the test. Without a good and clean namemapping the test is not possible.
We extensively use extended find (if possible for all elements).
We do not map elements by name or stuff but by IDs. The developer are responsible for providing those IDs and it is a bug if they do not.
We remove all elements (panel, cell, table,...) from the namemapping that have no ID.
This way the test could be read by someone not familar with TC but knowing the software under test.
We use keyword tests but for all regularly needed stuff we have scripts implemented, like restoring databases, logging in, checking logs, comparing excel files, working with SVG-elements, event handlers, ...
We also make scripts outside TestComplete relocate the log folder dynamically on some central huge harddrive so that logs are grouped by software version tested, so we do not have to collect them from the 7 machines. Other scripts also rename the logname and logfolder so that we could directly recognize which version tested, which part of the test and how many bugs.
The test also makes an entry in a database when it is run (majorVersion, branch, revision, timestamp, testpart, number of errors,...).
A website gathering data from this database gives an overview for all builds if they are tested or not, if the test is OK or NOT OK and if the software was deployed on any of our systems (the OK, NOK naturally has to be set manually).
Our next task is make the test run in various browsers (mainly Edge). Right now the test just runs in Chrome.
But we already have scripts written that switch the region-folder for each project. (as there is no possibility to do this in TestComplete itself)
The first testrun showed that ~80% of the test runs properly in Edge, thanks to the good namemapping. Just some elements have to be changed slightly.
- baxatob9 years agoCommunity Hero
We were able to make the developers take responsibility for manual testing, code reviews, unit tests and integration tests. Now the last step is coming that the developers also should do the automated regression testing.
No, the last step should be - to make the developers take responsibility for making the coffee for testers :)
(I'm just kidding. Thank you for your detailed feedback)
- jsc9 years agoRegular Contributor
we do it like this (Jscript)
function startTask(taskname, waitTime)
{
Log.AppendFolder("startTask: " + taskname);
Log.Message("starting task '" + taskname + "'");
var oShell = new ActiveXObject("WSCript.shell");
oShell.run('NET START "' + taskname + '"', "1", true);
Log.Message("trying to start task '" + taskname + "'");
Delay(waitTime);
Log.Message("waited for " + waitTime + " ms");Log.PopLogFolder();
}Stopping a task is the same. But we combine it with checking afterwards if the corresponding process does still exist. If the process still exists it is just killed. (script as well)
- william_roe9 years agoSuper Contributor
jsc wrote:Team
...
All of our testitems can be run on their own. No testitem depends on another. The testitems should take less than 4-5 minutes (there are some legacy tests, that run stable but take way too long).
Excellent stuff. Thanks for all your feedback. I'll probably ask several questions from the one post.
How are all testitems designed to run on their own and not depend on another? If a record is added in a testitem and not removed how can the test be run again without creating a duplicate? If there are dependencies (i.e. supplier / product / purchase / receive / customer / sale / receivable...) how can individual testitems be descrete and reusable and run on their own?
- jsc9 years agoRegular Contributor
How are all testitems designed to run on their own and not depend on another?
- they all start with a defined state
- there are 2 defined starting-states in our testitems: either a new and empty database is created at the start of the testitem or an existing database is restored- often we import specified data (this is possible in our software, e.g. from an excel file that is also under version control)
- afterwards they all start the browser and log in
If a record is added in a testitem and not removed how can the test be run again without creating a duplicate?
- see above
- additionally we do some preparing work at the start of the project suite (e.g. installing the program, deleting all existing databases, terminating some processes...) (setup: ~9 minutes)
- additionally we have a onStartTest routine running at the start of each testitem (terminating browser, terminating processes,...) (onStartTest: ~2-5 seconds)- we also have a onStopTest-Routine (~1-3 seconds) and a teardown project that clears all stuff at the end of the project suite (~20-30 seconds)
- no testitem is run twice. there is no need for that in our case. and if so, we would take care about a defined state at the start of the testitem (see above)
If there are dependencies (i.e. supplier / product / purchase / receive / customer / sale / receivable...)
how can individual testitems be descrete and reusable and run on their own?
- I do not get what dependencies this could be, but I try to guess.
- e.g. we have a database that we restore where a webservice-connector or other connectors are already defined- we have surrounding testenvironment that provides data (some vm sontaining ppm-systems, excel files, webservices,...) that is either under version control or must not be changed at all.
- is it this what you meant?Additionally we make sure that a failed testitem does not at all affect following testitems.
We therefore wrote our own Runner.nextTestitem(...) routine that skips the current testitem and starts the next. This routine is called when some inproper state is detected (restoring database failed, timeout is over,...). This routine combined with the onStartTest and onStopTest routines guarantee us that this is working perfectly well.
- william_roe9 years agoSuper Contributor
jsc wrote:If there are dependencies (i.e. supplier / product / purchase / receive / customer / sale / receivable...)
how can individual testitems be descrete and reusable and run on their own?
- I do not get what dependencies this could be, but I try to guess.
- e.g. we have a database that we restore where a webservice-connector or other connectors are already defined- we have surrounding testenvironment that provides data (some vm sontaining ppm-systems, excel files, webservices,...) that is either under version control or must not be changed at all.
- is it this what you meant?l.
Examples of dependencies our system:
- Product(s) cannot be created with Supplier(s)
- Purchase Order(s) cannot be created without Product(s)
- Receipts cannot be created without Purchase Order(s)
- Sale(s) cannot be created without Inventory (and Customers)
- Receivable(s) cannot be created without Sale(s)
I understand your approach but have a question. How are you restoring your database while the application pool (IIS) is running? I regularly restore the database to certain pre-defined states (i.e. pre-sales) when there is a failure but this requires stopping the web server application pool and killing an application process before a restore can be performed.
Are you a Microsoft shop? We use IIS and SQL Server (donning fire suit).
- william_roe9 years agoSuper Contributor
baxatob wrote:We were able to make the developers take responsibility for manual testing, code reviews, unit tests and integration tests. Now the last step is coming that the developers also should do the automated regression testing.
No, the last step should be - to make the developers take responsibility for making the coffee for testers :)
(I'm just kidding. Thank you for your detailed feedback)
The better the automated testing the lazier developers become :smileytongue:
- jsc9 years agoRegular Contributor
william_roe wrote:- Product(s) cannot be created with Supplier(s)
- Purchase Order(s) cannot be created without Product(s)
- Receipts cannot be created without Purchase Order(s)
- Sale(s) cannot be created without Inventory (and Customers)
- Receivable(s) cannot be created without Sale(s)
I understand your approach but have a question. How are you restoring your database while the application pool (IIS) is running? I regularly restore the database to certain pre-defined states (i.e. pre-sales) when there is a failure but this requires stopping the web server application pool and killing an application process before a restore can be performed.
Are you a Microsoft shop? We use IIS and SQL Server (donning fire suit).
I think we do not have such dependencies, but I would (or at least try to) integrate them in a stable state in our testenvironment.
We use a postgreSQL databases and apacache Tomcat in our web application.
The restoring databases is easy. The service is stopped and started via script as well as the Tomcat service. So we make sure that all services are started properly when login starts.
(in other departments we do similar stuff with MS SQL Server (no problem at all) and Oracle DMBS (bigger and therefore slower to restore)).
No, we are no webshop, we create some sort of ppm-software. (I am not sure if it is allowed to post links to our software here, so I do not...)
- william_roe9 years agoSuper Contributor
jsc wrote:
william_roe wrote:- Product(s) cannot be created with Supplier(s)
- Purchase Order(s) cannot be created without Product(s)
- Receipts cannot be created without Purchase Order(s)
- Sale(s) cannot be created without Inventory (and Customers)
- Receivable(s) cannot be created without Sale(s)
I understand your approach but have a question. How are you restoring your database while the application pool (IIS) is running? I regularly restore the database to certain pre-defined states (i.e. pre-sales) when there is a failure but this requires stopping the web server application pool and killing an application process before a restore can be performed.
Are you a Microsoft shop? We use IIS and SQL Server (donning fire suit).
.... So we make sure that all services are started properly when login starts.
(in other departments we do similar stuff with MS SQL Server (no problem at all) and Oracle DMBS (bigger and therefore slower to restore)).
Sounds as though our next step is to figure out how to start / stop IIS (via script) and how to terminate / launch applications via script.
GREAT DISCUSSIONS!!!
- chicks9 years agoRegular Contributor
william_roe I'm assuming you use 'eval' on consumer side of this method. Can you show what that code looks like? Is the idStr attribute always used? Our templates use ObjectIdentifier
William, I don't know what you mean by " eval on the consumer side". idStr is not always used, we use whatever is most specific. sometimes it's only object identifier. Speed may be a concern, but not to the point where we have spent any time investigating alternattives.
Regards, Curt
Related Content
Recent Discussions
- 10 hours agoSvetaG