5 metrics for monitoring mobile CI/CD performance using Bitrise Insights

Posted on by

To succeed with mobile CI/CD, monitoring key performance metrics is essential to evaluate the efficiency and overall performance of your mobile app. Using the right metrics in the right way can help produce a higher-quality app, and Bitrise Insights can help you accomplish that.

In this article, we’ll cover 5 key metrics your team can monitor to ensure that your app is meeting growing user expectations, and you’ll learn how to use Bitrise Insights to optimize your mobile app’s performance.

1. Build time

Build time is the measure of time it takes to run a mobile build and is a key metric for measuring the efficiency of the overall mobile app development process. The faster the build time, the faster developers can deliver new features to users.

Why it matters

On the other hand, slow builds take up more resources — both in terms of credits and developer time. Slow builds can not only lead to wasted developer time and a poor developer experience, but also can be more expensive to run.

Shaving off a few seconds every now and again might not seem worthwhile initially. However, those saved seconds will accumulate over time and might lead to a dramatic change in how you work. You can reduce context switching, enter a flow state, and make the best of your limited amount of time each day.

How to speed up build times

  • Use customizable workflows in your CI pipeline that automate manual processes like code signing, releasing to app stores, and more.
  • Separate workflows and run builds in parallel to reduce overall build time.
  • Run builds on Apple silicon machines to speed up build times by up to 50%.
  • Reuse cached dependencies from previous builds to speed up build times by using key-based caching
  • Reuse unchanged build outputs from previous builds to speed up build times by using build caching tools. 

💡Bitrise tip: Bitrise Insights helps you identify negative trends in build times and drill down to identify the root cause to optimize your processes. This is especially useful when you are running multiple builds and need to filter your data to view a single build or app.

‍

2. Build failure rate

Build failure rate measures the percentage of builds that failed. A high build failure rate means that your builds are failing most of the time and you will most likely have to spend time debugging, trying the build again, and then waiting for the results. This is especially time-consuming for long builds. Tracking build failure rate can help you reduce the wait time throughout the app development process and can increase your team’s productivity.

On the other hand, a low build failure rate means less time spent on context switching, less waiting for builds to complete, and less credit consumption.

How to lower build failure rate

  • Identify negative build failure trends to know when your builds started failing more than you expected.
  • Using filters and breakdown, dig deeper into the failure trends to identify the root cause of the failure. No guesswork, just debug, and fix the issues.
  • Set up alerts so that you don’t have to open Bitrise to know when your failure rates increase.
  • Use build and test caching to reduce manual workload and errors for new builds.

💡Bitrise tip: Check out your top 5 apps that fail most frequently to optimize for a lower build failure rate. With Bitrise Insights, you can identify bottlenecks, which calculate the time & credit impact of the failures based on how many times the build failed and how long the builds are. Why is this important? Because the failure percentage in itself can be misleading as one failed build out of two builds in a week is 50% failure rate, but that single failed build might not be the most impactful to fix if you have an issue which caused 10% of your PR builds (of which you do tens or hundreds a day) to fail.

3. Test duration

Test duration is the measure of time it takes to run a test. For mobile apps, monitoring testing duration is key as long testing times can stall releases. Short testing times with enough testing coverage lead to a faster time-to-market and better user experience.

How to shorten test duration

  • Automate testing to run more tests and increase test coverage, while reducing manual workloads.
  • Use a testing pyramid framework to outline the type, sequence, and frequency of tests you plan to run.

💡Bitrise tip: Use Bitrise Insights to monitor mobile testing duration and identify tests that take up the most amount of time to optimize processes. Filter down to a ‘per workflow’ or ‘per app’ view to view test duration for a single app or build.

4. Test failure rate

Tracking test failure rate helps to see where people frequently have to wait during the development process. A low test failure rate ensures that the app is free from errors and bugs — leading to a better user experience.

Conversely, a high test failure rate means that the app might be buggy and engineers are having to spend time debugging, re-running tests, and waiting for builds. There’s more time spent on hotfixes or code changes, which usually leads to a poor user experience. A high test failure rate also typically means that the build has also failed. 

How to lower test failure rate

  • Use testing automation to reduce any errors associated with manual testing.
  • Prioritise test failures based on the frequency of test failures or devices/environments in which they are occurring to fix the high-frequency, more prevalent failures.
  • Improve your local testing environment and strategy. Make it simple for engineers to run the relevant tests themselves before committing the code change. 

💡Bitrise tip: View test failure rate per app or workflow with Bitrise Insights. You can drill down  to a ‘failure rate per test suites’ view or a ‘failure rate per branches’ view to determine which test case is failing most frequently. Then, you can filter down further into the most frequently failing test case to show the previous runs of a specific test case, how long the runs were, and whether they were successful or not.

5. Flaky tests

Flaky tests produce different results even when the code hasn’t changed. In other words, flaky tests are those that are sometimes successful and sometimes fail — without any code changes. Flaky tests are the most crucial to fix because they can cause confusion and set backs, even when code is functioning as intended.

Eliminating flaky tests prevents wasted developer time, and is crucial to improving developer confidence. Successful tests ensure your processes are working for you — not against you. And, if tests frequently produce flaky results, developers may start to ignore the results over time — leading to poor developer confidence.

How to reduce the number of flaky tests

  • Isolate flaky tests to determine the root cause of the problem.
  • Test in a stable, consistent environment to avoid inconsistencies due to hardware or software configurations.
  • Monitor test results for inconsistencies.

💡Bitrise tip: Bitrise Insights can detect flaky tests that produce inconsistent results for the same code state. It can detect across builds when builds have  code commit hash information specified when the build is triggered. However, it can also detect flaky tests in a given build if the same test was performed multiple times in that given build, even if there was no code specified.

Using Bitrise Insights to improve mobile CI/CD performance

Mobile app development requires continuous monitoring and improvement to achieve success and stay competitive. Mobile CI/CD is used to automate the mobile app development process and ensure faster and more efficient development. 

And to achieve success with mobile CI/CD, you need Bitrise Insights to monitor critical performance metrics and optimize mobile app performance. Bitrise Insights helps mobile teams track the top five key metrics to monitor including build time, build failure rate, test duration, test failure rate, and flaky tests. Mobile teams use Bitrise Insights to identify negative trends in these metrics, investigate root causes, and optimize processes. 

By utilizing Bitrise Insights, app developers can efficiently measure their mobile app development process’s performance and produce a higher-quality app that meets user expectations. Get started with Bitrise Insights today to improve your mobile CI/CD performance.