Forum Discussion

root's avatar
root
New Contributor
5 years ago
Solved

TestLeft support for UI-testing of "VisualStudio Extensions" - Ready to use? HowTo?

Hello. 

I installed TestLeft and I use the trial license.

I like to use the test suite for automated ui tests for VisualStudio Extensions.

My simple validation scenario is:

 

[1] start VisualStudio 2019 Preview experimental instance

[2] wait for started instance of [1]

[3] open "Create new project"-dialog   (i.e. File | New | Project...)

[4] select any project template

[5] click "Next"

[6] click "Create"

[7] wait for finished project creation

 

Do you have any manuals, howtos, hints to cover this scenario?

I played a little bit around and I still stuck at step [2] and [3].

  • root's avatar
    root
    5 years ago

    Thanks for your reply, but that does not help.

    I used TestLeft's UI Spy, the code above is generated by that tool.

    But during execution of the unit tests the menus are not recognized and do not open.

     

    Sometimes the following rule is valid: Code reading != Code execution   (I guess "expert" should know this.)

     

    "[..] believe" and "[..] should work [..]" comments to get a better community rank is absolutley not very helpful. Try to execute my example code, when it works on your machine, just say "[..] works on my machine [..]", in that case the software by SmartBear has a bug or our environment is not supported.

     

    Anyway, our company canceled the evaluation.
    In case of not getting any working easy-case version for tests the software will not fit our needs. 

5 Replies

  • AlexKaras's avatar
    AlexKaras
    Champion Level 3

    Hi,

     

    According to https://support.smartbear.com/testleft/docs/general-info/supported-technologies/supported-applications.html, you should be able to get what you need.

    Note, that as Visual Studio does not provide debug information for itself, TestLeft will not be able to access its internals (except of .Net-based modules if any) and will get access to system-exposed windows only.

     

    Also, I did not see any relation to Visual Studio Extension in your scenario. If your extension is .Net-based or provides debug info you should be able to access its windows and public methods/properties.

     

    > I still stuck at step [2] and [3].

    What exact problems do you have?

     

    • root's avatar
      root
      New Contributor

      Thanks for your reply.

       

      I just mentioned that I like to test VisualStudio Extensions because that is my main domain.

      I can not test an VisualStudio Extension when the test tooling (i.e. TestLeft) does not support the application barebone. You have to know that most menus and toolbar are weaved dynamically during runtime (i.e. base on a xml-dialect called VSCT [Visual Studio Command Table, see https://docs.microsoft.com/en-us/visualstudio/extensibility/internals/visual-studio-command-table-dot-vsct-files?view=vs-2019]. We provide several commands with our VisualStudio Extension as well some individual toolwindows and several editor implementations, etc.pp.

       

      Anyway, we like to have UI-testing for integration-tests (i.e. blackbox tests) with a "release" setup of our VisualStudio Extension. In this case we have a minimum set of debug information. We do some reflection and we use "Automation Interface"-approaches (e.g. EnvDTE.*) but 50-75% can be visual checks (e.g. screenshot compares, color checks, etc.) We do not need any VisualStudio-insides.

       

      After said this, the easiest way to check if any test tooling allow UI-testing for VisualStudio is, to just clarify if the basic menu-walks work, e.g. go to Menu | File | New… to create a new project. When this does not work any effort to handle other calls will not work as well and the test tooling is not usable for our scenarios.

      Currently I do not know how to traverse any dynamically weaved (vsct-based) UI-elements in VisualStudio, here for example, to create a new project (see attached screenshot).

       

      Currently I tried three ways to query the menu item "Project...":

       

      using System;
      using Microsoft.VisualStudio.TestTools.UnitTesting;
      using SmartBear.TestLeft.TestObjects;
      using SmartBear.TestLeft.TestObjects.WPF;
      
      namespace TestLeftProject1
      {
          [TestClass]
          public class TestLeftTest : UnitTestClassBase
          {
              #region Class initializers
              [ClassInitialize]
              public static void ClassInitialize(TestContext context)
              {
                  UnitTestClassBase.InitializeClass(context);
              }
      
              [ClassCleanup]
              public static void ClassCleanUp()
              {
                  UnitTestClassBase.FinalizeClass();
              }
              #endregion
      
              private IProcess _process = null;
      
              private IProcess RunVisualStudio()
              {
                  if (_process == null)
                  {
                      _process = Driver.Applications.Run(
                      @"C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\Common7\IDE\devenv.exe",
                      @"/rootSuffix Exp");
                  }
      
                  var proc = System.Diagnostics.Process.GetProcessById(_process.ProcessId);
                  if(proc.HasExited)
                  {
                      _process = null;
                      _process = RunVisualStudio();
                  }
      
                  return _process;
              }
              
              [TestMethod]
              public void CreateProject_Test01()
              {
                  IProcess process = RunVisualStudio();
                  
                  for (int i = 0; i < 300; ++i)
                  {
                      try
                      {
                          IControl mainWindow = process.Find<IControl>(new WPFPattern()
                          {
                              ClrFullClassName = "Microsoft.VisualStudio.PlatformUI.MainWindow"
                          }, 2);
      
                          if (mainWindow == null)
                          {
                              System.Threading.Thread.Sleep(500);
      
                              continue;
                          }
      
                          mainWindow.Keys("ctrl+shift+n");
                      }
                      catch(Exception ex)
                      {
                          Console.WriteLine(ex.Message);
                      }
                  }
              }
      
              [TestMethod]
              public void CreateProject_Test02()
              {
                  IProcess process = RunVisualStudio();
      
                  IControl ctrl = process.Find<IControl>(new WPFPattern()
                  {
                      ClrFullClassName = "Microsoft.VisualStudio.PlatformUI.MainWindow"
                  }, 2).Find<IWPFMenu>(new WPFPattern()
                  {
                      WPFControlAutomationId = "MenuBar"
                  }, 2).Find<IControl>(new WPFPattern()
                  {
                      ClrFullClassName = "Microsoft.VisualStudio.PlatformUI.VsMenuItem",
                      WPFControlText = "_File"
                  });
      
                  ctrl.Click();
      
                  var ctrlNew = ctrl?.Find<IControl>(new WPFPattern()
                  {
                      ClrFullClassName = "Microsoft.VisualStudio.PlatformUI.VsMenuItem",
                      WPFControlText = "_New"
                  });
      
                  ctrlNew?.Click();
      
                  var ctrlNewProject = ctrlNew?.Find<IControl>(new WPFPattern()
                  {
                      ClrFullClassName = "Microsoft.VisualStudio.PlatformUI.VsMenuItem",
                      WPFControlText = "_Project..."
                  });
      
                  ctrlNewProject?.Click();
              }
      
              [TestMethod]
              public void CreateProject_Test03()
              {
                  IProcess process = RunVisualStudio();
      
                  IControl vsMenuItem = process.Find<IControl>(new WPFPattern()
                  {
                      ClrFullClassName = "System.Windows.Controls.Primitives.PopupRoot"
                  }, 2).Find<IControl>(new WPFPattern()
                  {
                      ClrFullClassName = "Microsoft.VisualStudio.PlatformUI.VsMenuItem",
                      WPFControlText = "_Project..."
                  }, 3);
      
                  vsMenuItem.Click();
              }
          }
      }
      

       

       

       

      • AlexKaras's avatar
        AlexKaras
        Champion Level 3

        Hi,

         

        I tried to use TestLeft's UI Spy in the Point and Fix mode and all menu items (File, New and Project) were highlighted. This makes me think that they must be accessible from code and must work.

        I believe that your _Test01 and _Test02 methods should work. Don't they? What problems do you have?