Build a Repeatable iOS Simulator Test Matrix on a Cloud Mac

The same UI test suite may pass locally yet time out unpredictably on a remote node. The usual cause is not application code, but stale simulator state, shifting shard boundaries, or excessive concurrency. A reliable test matrix should answer three questions for every run: which runtime was used, which tests ran together, and where the failure artifacts were saved. The following workflow uses xcodebuild and simctl to make each run repeatable.

Define the Matrix Boundaries First

Do not start by testing every device and OS version. Begin with one primary runtime and one representative device to establish a stable baseline, then add compatibility combinations. Before each run, record the Xcode version, macOS version, runtime identifier, commit, and test manifest. Device names are not suitable as unique keys because multiple simulators can share the same name. Save the UDID instead.

The goal of a test matrix is not to maximize the number of combinations. It is to produce explainable results for the same commit under the same inputs.

Create a Disposable Simulator Baseline

First query the runtimes and device types actually installed on the node. Do not hard-code version numbers in the script.

xcrun simctl list runtimes
xcrun simctl list devicetypes
xcrun simctl list devices available

Select the appropriate identifiers from the output, then create a separate device for each shard. The name is only for readability; all subsequent commands should use the UDID.

RUNTIME_ID="com.apple.CoreSimulator.SimRuntime.iOS-XX-X"
DEVICE_TYPE_ID="com.apple.CoreSimulator.SimDeviceType.iPhone-XX"

UDID=$(xcrun simctl create "ui-shard-01" "$DEVICE_TYPE_ID" "$RUNTIME_ID")
xcrun simctl boot "$UDID"
xcrun simctl bootstatus "$UDID" -b

Do not reuse simulators that developers use for day-to-day debugging. Tests may change permissions, locale settings, keyboards, application data, and background tasks. Disposable devices keep those changes confined to a single run. If tests depend on camera, notification, or location permissions, configure them explicitly during test setup rather than assuming authorization from a previous run still exists.

Build Once, Then Shard by Manifest

A matrix should not compile the same project again for every shard. Run build-for-testing once to create a shared test artifact, then have each shard invoke test-without-building.

DERIVED_PATH="$PWD/.derived-test"

xcodebuild build-for-testing \
  -workspace App.xcworkspace \
  -scheme AppUITests \
  -destination "generic/platform=iOS Simulator" \
  -derivedDataPath "$DERIVED_PATH"

XCTESTRUN=$(find "$DERIVED_PATH" -name "*.xctestrun" -print -quit)

Commit shard manifests to the repository. Avoid ad hoc splits based on alphabetical order or execution time, which can move the boundaries from run to run. For example, keep sign-in and sign-out tests in one group, and offline and synchronization tests in another. Disable Xcode's built-in secondary parallelization during execution so that concurrency is controlled only by the outer scheduler.

xcodebuild test-without-building \
  -xctestrun "$XCTESTRUN" \
  -destination "platform=iOS Simulator,id=$UDID" \
  -only-testing:"AppUITests/AuthenticationTests" \
  -parallel-testing-enabled NO \
  -resultBundlePath "$PWD/results/shard-01.xcresult"

If shard durations differ substantially, adjust the manifests using the median duration from several recent runs. After making changes, version the updated manifests and record what changed. Do not let the scheduling script reshuffle tests automatically before every run, or failures cannot be compared directly with the previous run.

Control Concurrency and Clean Up Reliably

More simulator concurrency does not always mean faster execution. Every additional shard introduces another set of startup processes, graphics services, test processes, and application instances. Start with two shards and monitor memory pressure, CPU saturation, boot time, and total duration. Reduce concurrency immediately if swapping occurs, bootstatus times out, or test durations fluctuate significantly.

The script must clean up after successful runs, failures, and interruptions. Register an exit handler for every created UDID, shutting down the device before deleting it so that the next run cannot accidentally reuse stale simulators.

cleanup() {
  xcrun simctl shutdown "$UDID" >/dev/null 2>&1 || true
  xcrun simctl delete "$UDID" >/dev/null 2>&1 || true
}

trap cleanup EXIT INT TERM

Do not use a global deletion command to remove every simulator on the node, because other jobs may be using the same node. Only operate on UDIDs created and recorded for the current run. If failure state must be preserved, archive the logs and result bundles before deleting the devices.

Use Result Bundles to Separate Real Failures from Flakes

Every shard must have its own .xcresult path. Also save its standard output, start time, end time, exit code, and UDID. Reusing a path can overwrite the most valuable artifact: the initial failure state. After a failure, rerun only the affected shard once instead of immediately rerunning the entire matrix.

A test that fails first and then passes usually points to unstable wait conditions, animations, network dependencies, or state races. A test that fails at the same step twice is more likely to be a deterministic regression. If the two failures occur at different points, investigate node resources and shared state between tests first.

Before deployment, verify the following in order:

1. The runtime and device type both come from query results on the current node.
2. Every shard uses a unique UDID and a separate result directory.
3. Test artifacts are built only once, with no rebuilds in individual shards.
4. Test manifests are versioned and are not randomly reshuffled before a run.
5. Reruns do not overwrite the initial failure results.
6. After an interruption, only simulators created for the current run are deleted.

Frequently asked questions