Running XCTests radically faster using SnapshotQuery and getting better insights

Published: 19th September 2021; Updated: 10th August 2022

Running automated UI tests at scale poses various challenges, such as:

Reproducibility and tracing of rare test failures
Tooling and infrastructure reliability
Performance / throughput

In this article, we will primarily focus on testing performance. Of all types of automated tests: unit tests, integration tests, and snapshot tests, the UI tests are the ones that take the longest time to run.

For inspiration, here are a few numbers on how certain companies approached UI testing in 2021 based on Building Mobile Apps at Scale by Gergely Orosz¹:

Spotify: ~500 UI tests
Robinhood: ~15 UI tests
Shopify: ~20 UI tests
Uber: a handful of UI tests
Lyft: a few UI tests

Tip

If you are running only a few UI tests, this post might still be interesting for you, but the real benefits come if you are running automated UI tests at scale.

XCTests at a high level

Let's use a very simple XCTest as an example:

// 1. Verify the welcome screen
let app = XCUIApplication()

// 1. Verify the welcome screen
XCTAssertTrue(app.staticTexts["Welcome"].exists)
XCTAssertTrue(app.buttons["Login"].exists)
XCTAssertTrue(app.buttons["Register"].exists)

// 2. Start the registration flow
app.buttons["Register"].tap()

// 3. Verify the registration screen
XCTAssertTrue(app.staticTexts["Create an account"].exists)
XCTAssertTrue(app.textFields["firstName"].exists)
XCTAssertTrue(app.textFields["lastName"].exists)
XCTAssertTrue(app.textFields["email"].exists)
XCTAssertTrue(app.buttons["Submit"].exists)
XCTAssertFalse(app.buttons["Submit"].isEnabled)

Conceptually (and even formally) we can think of any UI automated test as a series of action and verification steps:

UI Tests flow

Test actions typically involve invoking tap(), typeText(_:), and other XCUIElement functions, after which, the app under test would result in a new state which is then verified using various XCTAssert* functions.

If all the assertions pass, the next action is performed and the cycle of actions and verification continues.

The problem(s) with state verification in XCTests

1. Performance

The more complex the screen, the more XCTAssert* calls it will require in order for its content to be fully verified. Ideally, every element should get verified, for example:

Static text labels
Button labels and whether the buttons are enabled, disabled, hittable or selected
Text field placeholders and their initial text values
Cells have labels, images etc.

For UITest-Runner, each assertion means:

Sending a query to the app under test
Traversing the view hierarchy of the app and evaluating it against the query
Sending the results back to the UITests-Runner

On average (and if the queried element already exists on screen and the UITest-Runner doesn't need to retry the query after a delay) one such round takes 50-100ms² depending on the screen complexity and hardware. Multiplied by the number of assertions required to verify a screen, that can very quickly reach rather significant amounts of time and cause UI tests to become slow.

2. Reliability

The second, and perhaps even bigger/conceptual problem is that native XCTest APIs don't allow for the whole screen content to be verified atomically, but rather their sub-parts are verified sequentially, a single element at a time.

As a consequence, the tests might be less reliable and flakier, especially with larger test suites.

Addressing both problems at the same time

XCTests allow taking snapshots of various elements using snapshot APIs:

let snapshot = try element.snapshot()

The snapshot is of XCUIElementSnapshot type, which is a tree structure and contains children XCUIElementSnapshot instances.

Measuring the performance of taking a snapshot of various screen elements, we found out that the time it took for making a snapshot of any individual element on the screen was almost the same as taking a snapshot of the entire screen - still in range of those 50-100ms.

Query	Average duration
`app.buttons["Login"].snapshot()`	70ms
`app.secureTextFields["Enter PIN"].snapshot()`	67ms
`app.snapshot()`	78ms

Taking a snapshot of the whole app and performing the verification on that snapshot (instead of querying and verifying elements one by one) would give us the possibility of executing all the verification steps atomically plus it will make the tests run radically faster.

The only problem is that XCUIElementSnapshot doesn't provide a nice way of querying its tree structure like XCUIElementQuery does, with its familiar chaining and subscripts syntax.

SnapshotQuery

To solve this, we developed a custom XCUIElementSnapshot querying algorithm and exposed over 100 APIs to allow working with the snapshot in the same way as you are used to with standard XCUIElementQuery while bringing down the time it takes to query a tree from an average of 50-100ms to an average of 60μs effectively making it almost 1000x faster.

Let's take a look at how this reflects on our code example from before:

// 1. Verify the welcome screen
var app = try XCUIApplication().snapshot()

XCTAssertTrue(app.staticTexts["Welcome"].exists)
XCTAssertTrue(app.buttons["Login"].exists)
XCTAssertTrue(app.buttons["Register"].exists)

// 2. Start the registration flow
app.buttons["Register"].tap()

// 3. Verify the registration screen
app = try app.snapshot()

XCTAssertTrue(app.staticTexts["Create an account"].exists)
XCTAssertTrue(app.textFields["firstName"].exists)
XCTAssertTrue(app.textFields["lastName"].exists)
XCTAssertTrue(app.textFields["email"].exists)
XCTAssertTrue(app.buttons["Submit"].exists)
XCTAssertFalse(app.buttons["Submit"].isEnabled)

Notice that the only change is that instead of querying the XCUIApplication we are querying the snapshot of the application, but the remaining queries remained the same. This was one of the goals we set when developing the SnapshotQuery - keeping source code/syntax compatibility with the XCUIElementQuery.

Even more importantly, due to its efficient design, the querying performance will remain practically constant regardless of how many queries/assertions are performed to verify a screen because the query time on a snapshot is almost negligible in comparison to taking the initial snapshot.

Understandably, given that obtaining a snapshot still takes those 50-100ms, the benefits of using a snapshot and the SnapshotQuery APIs only outweigh the cost when making > 1 verification/assertion query per screen.

Applications with the biggest gains

We found the SnapshotQuery particularly useful in these use cases:

Verifying the state of any screen after an action is performed
Verifying any screen content that involves a collection (UITableView, UICollectionView or SwiftUI List)
Verifying the state of very dynamic screens (making verification atomic)

Screen Flow Pattern in XCUITests

Check out this article if you are interested in how to further improve XCUITests using Screen Flow Pattern.

AppeliumTests SDK

The SnapshotQuery APIs are part of AppeliumTests SDK for iOS. The availability of SnapshotQuery is not tied to any Appelium license, but we believe that you can get the biggest benefits when combining these APIs with the powerful UI tests recording and analysis functionality of the Appelium platform. With just a moment that takes to integrate the SDK, you will get:

Complete test execution reporting in an online dashboard (with test trends and stats)
Intelligent test failures grouping and annotations (like you are used to treating crash reports)
Network and console logs of the app under test (unavailable with plain XCTest)
Screen recordings of test failures
Queries, assertions, actions, network logs, console logs and stack traces of the UITest-Runner
Test performance improvement suggestions for each test execution

Book Building Mobile Apps at Scale by Gergely Orosz ↩
Average results in our testing app. Actual performance may vary based on the hardware and query complexity. ↩