cancel
Showing results for 
Search instead for 
Did you mean: 

TestLeft support for UI-testing of "VisualStudio Extensions" - Ready to use? HowTo?

SOLVED
New Contributor

TestLeft support for UI-testing of "VisualStudio Extensions" - Ready to use? HowTo?

Hello. 

I installed TestLeft and I use the trial license.

I like to use the test suite for automated ui tests for VisualStudio Extensions.

My simple validation scenario is:

 

[1] start VisualStudio 2019 Preview experimental instance

[2] wait for started instance of [1]

[3] open "Create new project"-dialog   (i.e. File | New | Project...)

[4] select any project template

[5] click "Next"

[6] click "Create"

[7] wait for finished project creation

 

Do you have any manuals, howtos, hints to cover this scenario?

I played a little bit around and I still stuck at step [2] and [3].

1 ACCEPTED SOLUTION

Accepted Solutions
New Contributor

Re: TestLeft support for UI-testing of "VisualStudio Extensions" - Ready to use? HowTo?

Thanks for your reply, but that does not help.

I used TestLeft's UI Spy, the code above is generated by that tool.

But during execution of the unit tests the menus are not recognized and do not open.

 

Sometimes the following rule is valid: Code reading != Code execution   (I guess "expert" should know this.)

 

"[..] believe" and "[..] should work [..]" comments to get a better community rank is absolutley not very helpful. Try to execute my example code, when it works on your machine, just say "[..] works on my machine [..]", in that case the software by SmartBear has a bug or our environment is not supported.

 

Anyway, our company canceled the evaluation.
In case of not getting any working easy-case version for tests the software will not fit our needs. 

5 REPLIES 5
Community Hero

Re: TestLeft support for UI-testing of "VisualStudio Extensions" - Ready to use? HowTo?

Hi,

 

According to https://support.smartbear.com/testleft/docs/general-info/supported-technologies/supported-applicatio..., you should be able to get what you need.

Note, that as Visual Studio does not provide debug information for itself, TestLeft will not be able to access its internals (except of .Net-based modules if any) and will get access to system-exposed windows only.

 

Also, I did not see any relation to Visual Studio Extension in your scenario. If your extension is .Net-based or provides debug info you should be able to access its windows and public methods/properties.

 

> I still stuck at step [2] and [3].

What exact problems do you have?

 

Regards,
Alex
[Community Expert Group]
____
[Community Expert Group] members are not employed by SmartBear Software but
are just volunteers who have some experience with the tools by SmartBear Software
and a desire to help others. Postings made by [Community Expert Group] members
may differ from the official policies of SmartBear Software and should be treated
as the own private opinion of their authors and under no circumstances as an
official answer from SmartBear Software.
[Community Expert Group] signature is used with permission by SmartBear Software.
http://smartbear.com/forums/f83/t86934/community-experts/
================================
Highlighted
New Contributor

Re: TestLeft support for UI-testing of "VisualStudio Extensions" - Ready to use? HowTo?

Thanks for your reply.

 

I just mentioned that I like to test VisualStudio Extensions because that is my main domain.

I can not test an VisualStudio Extension when the test tooling (i.e. TestLeft) does not support the application barebone. You have to know that most menus and toolbar are weaved dynamically during runtime (i.e. base on a xml-dialect called VSCT [Visual Studio Command Table, see https://docs.microsoft.com/en-us/visualstudio/extensibility/internals/visual-studio-command-table-do...]. We provide several commands with our VisualStudio Extension as well some individual toolwindows and several editor implementations, etc.pp.

 

Anyway, we like to have UI-testing for integration-tests (i.e. blackbox tests) with a "release" setup of our VisualStudio Extension. In this case we have a minimum set of debug information. We do some reflection and we use "Automation Interface"-approaches (e.g. EnvDTE.*) but 50-75% can be visual checks (e.g. screenshot compares, color checks, etc.) We do not need any VisualStudio-insides.

 

After said this, the easiest way to check if any test tooling allow UI-testing for VisualStudio is, to just clarify if the basic menu-walks work, e.g. go to Menu | File | New… to create a new project. When this does not work any effort to handle other calls will not work as well and the test tooling is not usable for our scenarios.

Currently I do not know how to traverse any dynamically weaved (vsct-based) UI-elements in VisualStudio, here for example, to create a new project (see attached screenshot).

TestLeftA.png

 

Currently I tried three ways to query the menu item "Project...":

 

using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using SmartBear.TestLeft.TestObjects;
using SmartBear.TestLeft.TestObjects.WPF;

namespace TestLeftProject1
{
    [TestClass]
    public class TestLeftTest : UnitTestClassBase
    {
        #region Class initializers
        [ClassInitialize]
        public static void ClassInitialize(TestContext context)
        {
            UnitTestClassBase.InitializeClass(context);
        }

        [ClassCleanup]
        public static void ClassCleanUp()
        {
            UnitTestClassBase.FinalizeClass();
        }
        #endregion

        private IProcess _process = null;

        private IProcess RunVisualStudio()
        {
            if (_process == null)
            {
                _process = Driver.Applications.Run(
                @"C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\Common7\IDE\devenv.exe",
                @"/rootSuffix Exp");
            }

            var proc = System.Diagnostics.Process.GetProcessById(_process.ProcessId);
            if(proc.HasExited)
            {
                _process = null;
                _process = RunVisualStudio();
            }

            return _process;
        }
        
        [TestMethod]
        public void CreateProject_Test01()
        {
            IProcess process = RunVisualStudio();
            
            for (int i = 0; i < 300; ++i)
            {
                try
                {
                    IControl mainWindow = process.Find<IControl>(new WPFPattern()
                    {
                        ClrFullClassName = "Microsoft.VisualStudio.PlatformUI.MainWindow"
                    }, 2);

                    if (mainWindow == null)
                    {
                        System.Threading.Thread.Sleep(500);

                        continue;
                    }

                    mainWindow.Keys("ctrl+shift+n");
                }
                catch(Exception ex)
                {
                    Console.WriteLine(ex.Message);
                }
            }
        }

        [TestMethod]
        public void CreateProject_Test02()
        {
            IProcess process = RunVisualStudio();

            IControl ctrl = process.Find<IControl>(new WPFPattern()
            {
                ClrFullClassName = "Microsoft.VisualStudio.PlatformUI.MainWindow"
            }, 2).Find<IWPFMenu>(new WPFPattern()
            {
                WPFControlAutomationId = "MenuBar"
            }, 2).Find<IControl>(new WPFPattern()
            {
                ClrFullClassName = "Microsoft.VisualStudio.PlatformUI.VsMenuItem",
                WPFControlText = "_File"
            });

            ctrl.Click();

            var ctrlNew = ctrl?.Find<IControl>(new WPFPattern()
            {
                ClrFullClassName = "Microsoft.VisualStudio.PlatformUI.VsMenuItem",
                WPFControlText = "_New"
            });

            ctrlNew?.Click();

            var ctrlNewProject = ctrlNew?.Find<IControl>(new WPFPattern()
            {
                ClrFullClassName = "Microsoft.VisualStudio.PlatformUI.VsMenuItem",
                WPFControlText = "_Project..."
            });

            ctrlNewProject?.Click();
        }

        [TestMethod]
        public void CreateProject_Test03()
        {
            IProcess process = RunVisualStudio();

            IControl vsMenuItem = process.Find<IControl>(new WPFPattern()
            {
                ClrFullClassName = "System.Windows.Controls.Primitives.PopupRoot"
            }, 2).Find<IControl>(new WPFPattern()
            {
                ClrFullClassName = "Microsoft.VisualStudio.PlatformUI.VsMenuItem",
                WPFControlText = "_Project..."
            }, 3);

            vsMenuItem.Click();
        }
    }
}

 

 

 

Community Hero

Re: TestLeft support for UI-testing of "VisualStudio Extensions" - Ready to use? HowTo?

Hi,

 

I tried to use TestLeft's UI Spy in the Point and Fix mode and all menu items (File, New and Project) were highlighted. This makes me think that they must be accessible from code and must work.

I believe that your _Test01 and _Test02 methods should work. Don't they? What problems do you have?

 

Regards,
Alex
[Community Expert Group]
____
[Community Expert Group] members are not employed by SmartBear Software but
are just volunteers who have some experience with the tools by SmartBear Software
and a desire to help others. Postings made by [Community Expert Group] members
may differ from the official policies of SmartBear Software and should be treated
as the own private opinion of their authors and under no circumstances as an
official answer from SmartBear Software.
[Community Expert Group] signature is used with permission by SmartBear Software.
http://smartbear.com/forums/f83/t86934/community-experts/
================================
New Contributor

Re: TestLeft support for UI-testing of "VisualStudio Extensions" - Ready to use? HowTo?

Thanks for your reply, but that does not help.

I used TestLeft's UI Spy, the code above is generated by that tool.

But during execution of the unit tests the menus are not recognized and do not open.

 

Sometimes the following rule is valid: Code reading != Code execution   (I guess "expert" should know this.)

 

"[..] believe" and "[..] should work [..]" comments to get a better community rank is absolutley not very helpful. Try to execute my example code, when it works on your machine, just say "[..] works on my machine [..]", in that case the software by SmartBear has a bug or our environment is not supported.

 

Anyway, our company canceled the evaluation.
In case of not getting any working easy-case version for tests the software will not fit our needs. 

Community Hero

Re: TestLeft support for UI-testing of "VisualStudio Extensions" - Ready to use? HowTo?

Hi,

 

"[..] believe" and "[..] should work [..]" comments to get a better community rank is absolutley not very helpful.

One friend of mine somewhere in time had fantastic phrase in his avatar:

"All extreme judgements are ultimately wrong. Even this one seems to be too extreme."

 

Try to execute my example code, when it works on your machine, just say "[..] works on my machine [..]"

a) I am not employed by SmartBear;

b) This is public Community forum where you are welcome to ask questions but nobody owes you anything and well may decide not to spend his time to execute some piece of code without clear description of what goes wrong;

c) One of the best ways during evaluation is to contact Support for guaranteed and official reply. Support can be contacted via the https://support.smartbear.com/message/?prod=TestLeft form;

d) Been an end-user and a tester, I do not much care about something that works on someone's machine. The fact that something works on my machine means nothing to you. That is why I wondered about your exact problems in order to try to help;

e) It's a pity that your company cancelled evaluation as it looks like that the one who evaluated TestLeft just did not read documentation and did not went through provided code samples;

f) My experience with C# is really basic but the below two test methods created with the help of TestLeft's UI Spy "worked on my machine" :

using Microsoft.VisualStudio.TestTools.UnitTesting;

using SmartBear.TestLeft;
using SmartBear.TestLeft.TestObjects;
using SmartBear.TestLeft.TestObjects.WPF;

namespace VSMenuTest
{
    [TestClass]
    public class TestLeftTest : UnitTestClassBase
    {
        #region Class initializers
        [ClassInitialize]
        public static void ClassInitialize(TestContext context)
        {
            UnitTestClassBase.InitializeClass(context);
        }

        [ClassCleanup]
        public static void ClassCleanUp()
        {
            UnitTestClassBase.FinalizeClass();
        }
        #endregion

        [TestMethod]
        public void VSMenuTest()
        {
            IControl vsMenuItem = Driver.Find<IProcess>(new ProcessPattern()
            {
                ProcessName = "devenv"
            }).Find<IControl>(new WPFPattern()
            {
                ClrFullClassName = "Microsoft.VisualStudio.PlatformUI.MainWindow"
            }, 2).Find<IWPFMenu>(new WPFPattern()
            {
                WPFControlAutomationId = "MenuBar"
            }).Find<IControl>(new WPFPattern()
            {
                ClrFullClassName = "Microsoft.VisualStudio.PlatformUI.VsMenuItem",
                WPFControlText = "_File"
            });
            vsMenuItem.Click();

            IControl vsMenuItem2 = Driver.Find<IProcess>(new ProcessPattern()
            {
                ProcessName = "devenv"
            }).Find<IControl>(new WPFPattern()
            {
                ClrFullClassName = "System.Windows.Controls.Primitives.PopupRoot"
            }, 2).Find<IControl>(new WPFPattern()
            {
                ClrFullClassName = "Microsoft.VisualStudio.PlatformUI.VsMenuItem",
                WPFControlText = "_New"
            }, 3);
            vsMenuItem2.Click();

            IControl vsMenuItem3 = Driver.Find<IProcess>(new ProcessPattern()
            {
                ProcessName = "devenv"
            }).Find<IControl>(new WPFPattern()
            {
                ClrFullClassName = "Microsoft.VisualStudio.PlatformUI.VsMenuItem",
                WPFControlText = "_Project..."
            }, 6);
            vsMenuItem3.Click();

            ITopLevelWindow hwndSource = Driver.Find<IProcess>(new ProcessPattern()
            {
                ProcessName = "devenv"
            }).Find<ITopLevelWindow>(new WindowPattern()
            {
                WndCaption = "New Project"
            });

            hwndSource.Close();
        }

        [TestMethod]
        public void VSMenuTest2()
        {
            ITopLevelWindow hwndSource = Driver.Find<IProcess>(new ProcessPattern()
            {
                ProcessName = "devenv"
            }).Find<ITopLevelWindow>(new WindowPattern()
            {
                WndCaption = "*Microsoft Visual Studio"
            });

            hwndSource.Keys("^!N");

            ITopLevelWindow hwndSource2 = Driver.Find<IProcess>(new ProcessPattern()
            {
                ProcessName = "devenv"
            }).Find<ITopLevelWindow>(new WindowPattern()
            {
                WndCaption = "New Project"
            });

            hwndSource2.Close();
        }
    }
}

 

P.S.: BTW, thank you for your message - it made me to start VisualStudio and refresh my memory as for how C# looks like. Something that I was going to do for some time but did not have a good enough reason. Smiley Happy

 

Regards,
Alex
[Community Expert Group]
____
[Community Expert Group] members are not employed by SmartBear Software but
are just volunteers who have some experience with the tools by SmartBear Software
and a desire to help others. Postings made by [Community Expert Group] members
may differ from the official policies of SmartBear Software and should be treated
as the own private opinion of their authors and under no circumstances as an
official answer from SmartBear Software.
[Community Expert Group] signature is used with permission by SmartBear Software.
http://smartbear.com/forums/f83/t86934/community-experts/
================================
New Here?
Join us and watch the welcome video:
Update to TestLeft 2.3
Labels