Tallan's Technology Blog

Tallan's Top Technologists Share Their Thoughts on Today's Technology Challenges

Top 5 Free Mobile Application Testing Platforms

Chris Fedge

The testing phase of mobile development is critical to the success of  any application, and with the multitude of variations in both hardware and operating systems, this stage is arguably the most difficult to manage and execute properly.  Fortunately there are a handful of useful tools that assist with the distribution and testing of mobile applications with the majority of these being free or free try.  Each platform offers a unique set of functionality, so the best way to choose is experimenting with each one.  Here are some of the best mobile testing platforms available today:

Crashlytics

Summary: Crashlytics provides a powerful lightweight solution for capturing and analyzing crashes in mobile applications and delivering beta distributions to testers.  Crashlytics succeeds in delivering a robust platform which supplies essential beta testing features without making things overly complicated.

Getting Started: Plugins for Android Studio, Eclipse, and IntelliJ are available for Android or XCode for iOS.  The plugin allows for easy one-click integration of Crashlytics into an application which actually makes the necessary changes to the project automatically.  Support is also provided for Ant, Maven and Gradle build tools.

Distribution: Beta builds are distributed by uploading or simply dragging in the application package file to the plugin within the IDE.  Crashlytics also provides a mobile application which is installed on each testing device and allows for easy distribution of updates and new versions.

Analytics: Crashlytics includes the following information at the time of a crash:

  • Device and Operating System
  • If the Device is Rooted
  • Free Storage Space
  • Free RAM
  • If the Application is in Focus
  • Proximity Sensor On or Off
  • Orientation
  • Battery Level

In addition to these metrics, a full stack trace of each thread running in the application at the time of the crash is displayed.  There is also an option to add custom keys and custom logging.

Pricing: Everything Crashlytics has to offer is completely free.

Support: iOS and Android

TestFairy

Summary: TestFairy is a beta testing platform exclusively for Android.  It is by far the easiest testing platform to integrate and use.  TestFairy also delivers a plethora of information surrounding crashes and the devices that are being used for testing.

Getting Started: Integration with TestFairy is achieved simply by uploading an application package file to the TestFairy website.  Before uploading the file, options are available for capturing specific metrics, limiting session duration, and enabling video recording.

Distribution: The beta distribution is achieved by emailing a groups of testers which can be managed through the website.  There is also an option to release your application as a public beta which can be accessed through a public URL.

Analytics: TestFairy includes the following information over the course of each testing session:

  • Memory Performance
  • CPU Performance
  • Network Throughput
  • Phone Signal
  • WIFI Signal
  • Battery Performance
  • Logcat Entries
  • Location
  • IP Address
  • Device and Operating System
  • Screen Resolution
  • Stack Trace

A video recording of the session is also available, however this can be disabled if the device is not on Wi-Fi in order to save mobile data.

Pricing: The basic version is free, but a paid enterprise version is available.

Support: Android

Ubertesters

Summary: Ubertesters is a beta testing platform that attempts to cover all parts of the beta testing process including management of team roles, distribution, crash analytics, and detailed bug reporting.  Ubertesters also provides a process for hiring testers.

Getting Started: Integration with Ubertesters is accomplished by downloading the library and including the library in the project along with a Product ID which identifies the application.  The application code must be modified to initialize Ubertesters.

Distribution: Ubertesters provides a mobile application which is installed on each testing device and allows for easy distribution of updates and new versions.  New revisions are uploaded to the Ubertesters website where they can be marked as current or closed.

Bug Reporting: From within the application being tested, the tester can access a panel for reporting the issue.   The panel can be activated by a slider overlaid on the screen, shaking the device, or a manual trigger programmed into the application.  Testers can send reports which include the following:

  • Summary
  • Type (Bug, Feature, Task, Improvement, or Crash)
  • Severity (Low, Normal, High, Critical, or Show Stopper)
  • Description
  • Attached Screenshots which can be marked up with lines, highlighting, text or arrows

Analytics: The following information is available for each testing session:

  • Device Model and Platform
  • Hardware Version
  • Operating System and Kernel Version
  • Storage Space
  • Free RAM
  • Location
  • IP Address
  • Screen Resolution
  • Enabled Sensors (Cameras, Light, Compass, Barometer, Gyroscope, Proximity, Temperature, Accelerometer)
  • Battery Level
  • External Power
  • Session Events and Crashes with Stack Trace

Pricing: Ubertesters is free with paid Indie, Startup, Pro and Enterprise versions available.

Support: iOS, Android, and Windows Phone

Crittercism

Summary: Crittercism is bug reporting platform which delivers a numerous array of available features for tracking down and resolving bugs.  Fully integrated, it is hard to imagine any issue that could not be tracked down quickly and easily.

Getting Started: In order to integrate Crittercism with your beta the library can be downloaded and included in the project along with a Crittercism App ID.  Typically the application code must be modified to initialize Crittercism, however an alternate method called App Wrapping allows for iOS applications to be integrated with Crittercism without modifying the source code.

Analytics: A valuable feature of Crittercism is the grouping of identical issues under one list item.  It is then possible to see graphs and statistics regarding the state of the devices that experienced those problems.  The following information is also available for each issue:

  • Disk space
  • SD Card Space
  • Battery Level
  • Mobile Network or Wi-Fi Status
  • App Version
  • Model
  • Carrier
  • System Version
  • Location
  • Orientation
  • Memory Usage

For android applications, Crittercism will track during which activity the issue occurred.  Additionally, if Crittercism is configured correctly it can also track performance metrics such as Latency, Error Rate, Request Volume and Data In/Out.

Pricing: The basic version is free, with paid Professional, Pro Plus and Enterprise versions.

Support: Android, iOS, HTML5 and Windows Phone 8

BugSense

Summary: BugSense is bug reporting platform which supplies useful information about crashes and exceptions, and displays them in easy to understand reports.   One of the most interesting features of BugSense is the Insights view which displays trends in issues most severely affecting the application in an easy to read summary.

Integration: To integrate BugSense into your application, the SDK must be downloaded and included in the project.  The API key must also be added to the application code for BugSense to identify the application.

Analytics: Bugsense displays errors in an application in real time, as a summary, or as a list of distinct errors which can be filtered by version, tags, number of occurrences, etc.  The following information is available related to each error:

  • Class and File Where the Error Occurred
  • Time of First and Most Recent Occurrence
  • Wi-Fi and Mobile Network
  • GPS
  • Memory Usage
  • App Version
  • Device Name
  • Operating System Version
  • Stack Trace when Error Occurred

Pricing: BugSense is free with paid Indie, Plus, and Enterprise versions available.

Support: Android, iOS, Windows 8, HTML5 and Windows Phone 8

With all of the options out there, selecting beta testing tools can be difficult and unfortunately not every feature of each platform is covered here, but hopefully this article will give some direction.  The platforms listed here were tested using Android 4.2.2 and Android Studio, so anything without support for Android, such as TestFlight, was not included.  If none of these work for you feel free to check out these other options:

3 Comments. Leave new

Highly descriptive post, I liked that a lot.
Will there be a part 2?

Thanks, I’m glad to hear it! But sorry, no plans for a part 2 at the moment.

With Bugclipper we are trying to make app testing simple and more efficient. Would love to share more :)

\\\