[FAQ] iOS Test Automation Frameworks
We supportXCTest,XCUITest,Appium,Jasmine,KIFandCalabashforiOS test automation with Bitbar. Is iOS Test based on UI Automation, which means it can only test UIKit components or? As of XCode8 release UIAutomation is no longer supported in Bitbar testing public cloud, as Apple has dropped support for it. How are you handling the provisioning profiles to be able to test on devices? We replace your provisioning profile with our provisioning profile. No need to root or jailbreak devices to get this done. Can I test Cocos2d or C++ OpenGL based games? Yes, you can. To get this done effectively you have to think what test automation framework fits to best for your needs and will the framework work with your app/game. Is there support to automatically handle iOS dialogues? Currently the handling of iOS dialogues is handled through used test framework. We have internal tools using which we are able to avoid and dismiss OS dialogues outside of test runs. How can I start an iOS Appcrawler run using Bitbar RiC? iOS Appcrawler isn’t available at the moment in Run in Cloud -plugin How can I start an iOS Appcrawler using API? You can do it on iOS UI Automation project -type. API Calls: Get current project config: GET /api/v2/projects/{projectId}/config Set Project Config: POST /api/v2/projects/{projectId}/config projectId, {projectId} mode, "IOS_CRAWLER" usedDeviceGroupId, {deviceGroupId} (mandatory, or test run start fails) Start new Test Run: POST /api/v2/runs1.3KViews0likes0Comments[FAQ] General Bitbar questions
Do we need to share the source code of the app in the Cloud? No you don’t, unless you choose to use a test framework that specifically requires this. How does Bitbar manage the cleaning of devices? Each phone is cleaned of apps that are not expected to be there. Android browser: we do remove any browser app local data implying there will not be any navigation data on next session. We also uninstall all apps that is possible to uninstall and remove files we know are created during tests. iOS browser: we rely on iOS’s own browser history cleaning services to revert Safari to an initial state. Each device is put through a clean up phase after each test run. Regardless of all these efforts some files or data may stay hidden somewhere on the device in the Public Cloud. For a truly private environment we recommend a Private or Enterprise Cloud installation. What are the requirements for signing my iOS applications to use on Bitbar’s iOS devices? The app needs to be an ad-hoc distribution developer debug build. You can find more informationhere. How do you support Bitbar devices communicating with servers behind our firewall? For Public cloud the only way is for the customer servers to whitelist our public IPs to allow connections from our devices. For private clouds or dedicated devices we can implement a VPN or proxy connection. In on-premise setups the customer has complete freedom on implementing the networking as needed. Do you support turning off and on Wifi on Bitbar devices? This is not supported in Public Cloud. There is support for this in Private and Enterprise environments. My tests require a Bluetooth (BLE) connection between the device and my external HW. What should I do? For Public Cloud you could try to mock the BLE connection data as required by the tests. Otherwise we can pair devices to required Bluetooth devices in Private or Enterprise Cloud environments. Do you use emulators or real devices? Real devices only. A lot of our content will not work outside country X. How can this be solved in the Cloud? For Public cloud the only way is for the customer servers to whitelist our public IPs allowing our devices to connect. For private cloud we can implement a VPN or proxy connections. In Enterprise Cloud it is up to the customer to define as the installation is in their own premise. Do you support in app purchases? In Public Cloud it is complicated to manage Google account cash. If a test account can be used to log in and do the purchase completely inside the tested app, then we support it. For Private and Enterprise environments we can set accounts controlled by the customer to enable such purchases. Do you support testing of push notifications? If the used test framework supports testing push notifications, then we can support them too. Triggering of push notifications is up to the customer to handle. Can we change device settings on Bitbar? In Public Cloud it is not allowed to access device settings at all. For Private and Enterprise environments settings can be changed through remote access to the device. What types of device performance data does Bitbar provide for test runs? We provide CPU and Memory usage data. Can we do remote manual testing with Bitbar devices? Yes, for most Android and iOS devices. Some models are incompatible with the used technology. Manual testing on iOS devices is supported on public cloud through dedicated devices. Enterprise and Private cloud users can decide which users get access to manual testing. Does Bitbar support parallel or concurrent test runs? Yes.Here is more information. Can I use my own Google accounts on Bitbar devices? In Public Cloud setting a Google account on the device isn’t allowed. Using a Google API/Service to log in to an app with your own account can be done. For Private and Enterprise environments a customer’s own Google account can be set on the devices. Which Google Play account is used on your devices? In Bitbar public cloud a number of Google accounts are in use. These Google accounts may appear when running tests in Bitbar public cloud. play@bitbar.com play-x@bitbar.comwhere x is in 1…18 play-USx@bitbar.comwhere x is either 1 or 2 play.us1.public@gmail.com patchcordX@gmail.comwhere X > 18 Where can I see a list of your devices? How often are you updating it? And how long does it take you to have new devices? The complete list of devices - with all details – can be found underBitbar Cloud. We are adding new devices to increase device availability on existing models and add new models as they come out. Can Bitbar provide dedicated private devices for my testing? Dedicated devices are available through Private Cloud installations or as part of the public cloud. In a private cloud installation the customer is able to freely select the number and type of devices and these are managed by Bitbar. In Public cloud users can reserve one or multiple devices to their use only to assure device availability and security. For more information please contact sales. Can I change or choose the OS on the devices? If not, how do you choose what OS should be on there? No, devices have certain version of OS and cannot be changed. We do the changing when time goes on and if OEM releases relevant OS update. The purpose is to keep up with the actual use of certain OS versions globally and provide as versatile setup of different devices and OS as possible. Can the handsets under test receive an email via wifi? Yes, but your application needs to be configured to receive email. The regular email application in device cannot be currently configured for sending/receiving emails. Where can I find the free test trial and how can I get started with Bitbar Cloud? You can create yourself a free account atbitbar.com. After leaving your email address, you’ll get an activation email with a link guiding you through the registration process. You’re now able to login to Bitbar Cloud using your new credentials and access our trial device group. My test run failed on most devices. Why was that? There are several reasons why test runs can fail on Bitbar Cloud. First and the most typical case is that there is something wrong with the application, and instrumentation makes it crash. A good rule of thumb is that if AppCrawler run crashes with your app, then the problem is in app itself. Device problems is typically seen as pending test runs. Are the devices jail broken/rooted? No. None of our devices are jail broken or rooted. Our app requests Device Administrator privileges from the user. After the user grants the app Device Admin privileges these privileges cannot be removed without first entering a password. It is conceivable that some tests may leave an app on the device. Does any of this cause a problem for your test environment? No, this does not cause problems to our devices or environment. Our system automatically cleans, reboots and hard resets devices before any new test run. The app connects to our internal server. Our IT department will only allow connections from known IP address ranges. Is it possible for you to tell us what IP ranges are used to originate traffic from your test devices? We have two public cloud data centers with IP ranges 185.75.2.0/28 and 205.234.8.112/28. Public Cloud users can also implement a test app to find the current IP of used device/connection and communicate it to external service that can open that IP for connections. For Private and Enterprise Cloud installations most special network configurations are possible. Which port ranges are used for when running Appium client side tests? When you are running an Appium client side test, your host will be making a connection to a dynamic IP address and a port range 10010 to 10090. Where are your devices located? Bitbar devices are located in our data centers in San Jose, CA and Wroclaw, Poland. How safe are you? All our user data, applications and results are stored in Amazon Cloud and are very safe. We also use SSL and encrypt the data as seen appropriate. Only users with valid accounts can access their own data and results. We also track all IPs connecting to our Bitbar. How long is usually the queue? In each priority group? Do I have to wait for 2 hour or day? The time between a start of test run and delivering results depends entirely on test case length, current queue, time of the day and a desired device set. Typically we’re talking about 2 to 15 minutes. Sometimes it can be longer if you start multiple test runs on same devices. We have multiple copies of devices at Bitbar so some devices can ensure results in few minutes (depending on test run length). The customers with higher priority plans (platinum > gold > silver) can get their app running first in case of queue. How long will our projects be online? We periodically clean unused projects and project files from our public cloud. For unused projects (where tests have not been run) after three months the project files are removed. After this the whole project is removed after next three months. Can I take pictures, sound recording (mic) with the phones? Yes, you can. Our devices are not positioned for any specific photo target and recording of audio can give you arbitrary recording. However, our devices are fully functioning Android and iOS devices, and both mentioned functions are enabled on the devices.2.2KViews0likes2Comments[FAQ] Android Test Automation Frameworks
When people consider what framework works the best, these supplementary documents, slidedecks and webinars can be recommended: Appium, Calabash, Espresso, Robotium, uiautomator:https://www.slideshare.net/bitbar/different-android-test-automation-framework-what-works-you-the-best Robotium vs. uiautomator:http://bitbar.com/the-pros-and-cons-of-different-android-testing-methods Appium:https://www.slideshare.net/bitbar/testdroid-webinar201401071.2KViews0likes0Comments
