Building Visual Screenshot Change Detection Tests for Android with Roborazzi

09 Mar, 2025

When developing Android apps, verifying that UI changes have been correctly applied is essential. This is especially important in the Android ecosystem, which must support a wide variety of screen sizes and resolutions. However, code review alone makes it difficult to accurately identify UI changes, and as team size grows, it becomes increasingly easy to miss unintended UI modifications.

Recently, I built a screenshot testing system using Roborazzi for a project at my company. In this post, I will share my experience from adopting Roborazzi through building a CI pipeline with GitHub Actions.

Introduction to Roborazzi

Roborazzi is an Android screenshot testing library based on Robolectric. It leverages Robolectric’s Native Graphics Mode (RNG) to visualize Android integration tests running in a JVM environment. Although still in the experimental stage, it also provides experimental support for Compose Multiplatform iOS and Compose Desktop.

The greatest advantage of Roborazzi is that it can generate and compare screenshots in a JVM environment without a physical device, enabling fast test execution in CI environments. Additionally, it is used in Google’s official sample project Now in Android, providing production-level implementation examples as reference.

Paparazzi vs. Roborazzi

Paparazzi is an excellent tool for visualizing displays in a JVM environment, but it is not compatible with Robolectric, which mocks the Android framework. Roborazzi bridges this gap. By integrating with Robolectric, it enables testing with dependency injection tools like Hilt and allows interaction with actual components. Essentially, Roborazzi extends Paparazzi’s capabilities by capturing screenshots through Robolectric, providing a more efficient and reliable testing process.

Key Gradle Tasks

Roborazzi provides the following Gradle tasks:

Task Name	Description
recordRoborazziDebug	Capture and save screenshots
compareRoborazziDebug	Compare current images with saved images
verifyRoborazziDebug	Verify differences between current and saved images
verifyAndRecordRoborazziDebug	Verify images and record new baseline images if differences are found
clearRoborazziDebug	Delete saved screenshots (experimental)

Project Setup

Gradle Configuration

First, add the Roborazzi plugin to your project-level build.gradle.kts.

// root build.gradle.kts
plugins {
    // ...
    id("io.github.takahirom.roborazzi") version "[version]" apply false
}

If you are using a buildScript block, configure it as follows:

// root build.gradle.kts
buildscript {
  dependencies {
    // ...
    classpath("io.github.takahirom.roborazzi:roborazzi-gradle-plugin:[version]")
  }
}

Then configure your module-level build.gradle.kts as follows:

// module build.gradle.kts
plugins {
    // ...
    id("io.github.takahirom.roborazzi")
}

android {
    // Existing configuration

    testOptions {
        unitTests {
            isIncludeAndroidResources = true
            // Use Robolectric's hardware rendering mode
            all {
                it.systemProperties["robolectric.pixelCopyRenderMode"] = "hardware"
            }
        }
    }
}

dependencies {
    // Core functions
    testImplementation("io.github.takahirom.roborazzi:roborazzi:[version]")

    // Jetpack Compose
    testImplementation("io.github.takahirom.roborazzi:roborazzi-compose:[version]")

    // JUnit rules
    testImplementation("io.github.takahirom.roborazzi:roborazzi-junit-rule:[version]")
}

// Roborazzi configuration
roborazzi {
    // Set reference image output directory
    outputDir.set(file("src/test/screenshots"))

    // Set comparison image output directory (Experimental option)
    compare {
        outputDir.set(file("build/outputs/screenshots_comparison"))
    }
}

Additional gradle.properties Configuration

Roborazzi supports various options that can be configured in the gradle.properties file.

# Screenshot test activation options
roborazzi.test.record=true
# roborazzi.test.compare=true
# roborazzi.test.verify=true

# Image resize scale setting
roborazzi.record.resizeScale=0.5

# File path strategy setting
# Default is relativePathFromCurrentDirectory
roborazzi.record.filePathStrategy=relativePathFromRoborazziContextOutputDirectory

# Image format setting (WebP support, Experimental)
# roborazzi.record.image.extension=webp

# Automatic cleanup of old screenshots (use with caution)
# roborazzi.cleanupOldScreenshots=true

Writing Screenshot Tests

Now let’s write actual screenshot tests. Roborazzi supports various screenshot capture methods, with the key APIs as follows:

Capture Target	Code Example
Jetpack Compose	`composeTestRule.onNodeWithTag("AddBoxButton").captureRoboImage()`
Espresso View	`onView(ViewMatchers.isRoot()).captureRoboImage()`
Plain View	`view.captureRoboImage()`
Compose Lambda	`captureRoboImage { Text("Hello Compose!") }`
Full Screen (Experimental)	`captureScreenRoboImage()`
Bitmap	`bitmap.captureRoboImage()`

Here is an example of a screenshot test for a Jetpack Compose login screen:

@RunWith(RobolectricTestRunner::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE) // Enable Robolectric Native Graphics mode (required)
class LoginScreenshotTest {

    @get:Rule
    val composeTestRule = createComposeRule()

    @Test
    fun captureLoginScreen_defaultState() {
        // Set up Compose UI
        composeTestRule.setContent {
            MyAppTheme {
                LoginScreen(
                    uiState = LoginUiState.Default,
                    onLoginClick = {},
                    onEmailChanged = {},
                    onPasswordChanged = {}
                )
            }
        }

        // Capture screenshot
        composeTestRule.onRoot()
            .captureRoboImage(
                // Specify filename (saved to default output directory)
                "loginScreen_defaultState",
                // Additional option settings
                roborazziOptions = RoborazziOptions(
                    // Image resizing
                    recordOptions = RoborazziOptions.RecordOptions(
                        resizeScale = 0.75f
                    ),
                    // Comparison options
                    compareOptions = RoborazziOptions.CompareOptions(
                        changeThreshold = 0.01f
                    )
                )
            )
    }

    @Test
    fun captureLoginScreen_errorState() {
        composeTestRule.setContent {
            MyAppTheme {
                LoginScreen(
                    uiState = LoginUiState.Error("Invalid credentials"),
                    onLoginClick = {},
                    onEmailChanged = {},
                    onPasswordChanged = {}
                )
            }
        }

        // You can also capture specific components only
        composeTestRule.onNodeWithTag("errorMessage")
            .captureRoboImage(
                "loginScreen_errorMessage"
            )

        // Full screen capture
        composeTestRule.onRoot()
            .captureRoboImage(
                "loginScreen_errorState"
            )
    }
}

Using RoborazziRule

Roborazzi supports more convenient test writing through JUnit rules. RoborazziRule is optional, but provides the following benefits:

Provides context such as RoborazziOptions and outputDirectoryPath
Supports various capture types (LastImage, AllImage, Gif, etc.)

@RunWith(RobolectricTestRunner::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class RuleTest {
    @get:Rule
    val composeTestRule = createComposeRule()

    @get:Rule
    val roborazziRule = RoborazziRule(
        composeRule = composeTestRule,
        captureRoot = composeTestRule.onRoot(),
        options = RoborazziRule.Options(
            // Capture images only on test failure
            captureType = RoborazziRule.CaptureType.Gif(onlyFail = true),
            // Image comparison option settings
            roborazziOptions = RoborazziOptions(
                compareOptions = RoborazziOptions.CompareOptions(
                    // Settings to address anti-aliasing issues
                    imageComparator = SimpleImageComparator(
                        maxDistance = 0.007f,
                        vShift = 2,    // Allow vertical shift
                        hShift = 2     // Allow horizontal shift
                    )
                )
            )
        )
    )

    @Test
    // Annotation to ignore RoborazziRule
    // @RoborazziRule.Ignore
    fun testWithRule() {
        composeTestRule.setContent {
            MyAppTheme {
                // ...
            }
        }

        // RoborazziRule automatically captures images during test execution
    }
}

GIF Image Capture

Roborazzi also supports generating GIF images to show animations or user interactions.

@Test
fun captureRoboGifSample() {
    onView(ViewMatchers.isRoot())
        .captureRoboGif("build/test.gif") {
            // Launch the app
            ActivityScenario.launch(MainActivity::class.java)
            // Navigate to next page
            onView(withId(R.id.button_first))
                .perform(click())
            // Go back
            pressBack()
            // Navigate to next page again
            onView(withId(R.id.button_first))
                .perform(click())
        }
}

Testing with Various Device Configurations

To better reflect real-world usage environments, it is recommended to capture screenshots across various device configurations (screen size, orientation, dark mode, etc.). Roborazzi provides predefined device configurations through RobolectricDeviceQualifiers.

// Set default device configuration at class level
@RunWith(RobolectricTestRunner::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
@Config(qualifiers = RobolectricDeviceQualifiers.Pixel5)
class DeviceSpecificScreenshotTest {
    // ...
}

// Apply different configuration to specific test methods only
@Test
@Config(qualifiers = RobolectricDeviceQualifiers.MediumTablet + "+land")
fun captureLoginScreen_landscapeTablet() {
    // ...
}

// Dark mode configuration
@Test
@Config(qualifiers = "+night")
fun captureLoginScreen_darkMode() {
    // ...
}

// Locale configuration
@Test
@Config(qualifiers = "+ja")  // Japanese locale
fun captureLoginScreen_japaneseLocale() {
    // ...
}

Rendering Differences Across Operating Systems

There are a few caveats to be aware of when using Roborazzi.

1. Cross-OS Rendering Inconsistency

Roborazzi uses the Skia library to render UI, and rendering results may differ slightly across operating systems. This is a known issue also mentioned in the FAQ - Why do my screenshot tests fail inconsistently across different operating systems like MacOS, Ubuntu, and Windows?.

This issue was also discussed in the Now in Android project, confirming that identical rendering cannot be guaranteed across all environments. For reference, this is also why screenshot verification tasks failed on my local PC (Mac). The main causes are:

Differences in graphics driver implementations across OSes
Differences in font rendering approaches
Differences in anti-aliasing handling

While you could address this by adjusting options in RoborazziOptions.CompareOptions, I felt that gradually relaxing the verification conditions would defeat the purpose of comparing screenshots. Therefore, following the advice of Roborazzi’s author takahirom, I configured both baseline image generation and verification to run in the same CI environment for consistency.

In fact, you can confirm that nowinandroid also does not compromise on image comparison options.

2. Resolving Memory Issues

Running many screenshot tests with complex UIs can cause OutOfMemoryError. This is a known issue also discussed in Issue #272, and can be resolved by adjusting maxHeapSize as follows:

// build.gradle.kts
android {
    testOptions {
        unitTests.all {
            maxHeapSize = "4096m"
        }
    }
}

3. Resolving Corrupted Image Issues

In some cases, images captured by Roborazzi may appear broken. This is typically caused by an API level that is too low or Robolectric rendering mode configuration issues. As discussed in Issue #255, the following configuration can resolve this:

// Apply to test class
@Config(sdk = [35])  // Use latest API level
@GraphicsMode(GraphicsMode.Mode.NATIVE)  // Native Graphics mode is required

// build.gradle.kts
android {
    testOptions {
        unitTests.all {
            it.systemProperties["robolectric.pixelCopyRenderMode"] = "hardware"
        }
    }
}

These settings resolved most rendering issues.

For reference, Robolectric 4.14 and later supports Android V (SDK 35).

Building the CI Pipeline

After writing tests, integrating them into a CI pipeline is crucial. I built a system using GitHub Actions that automatically runs screenshot tests whenever a PR is created and visually displays any changes.

GitHub Actions Workflow Configuration

Here is an example workflow configuration using GitHub Actions:

name: Android Pull Request CI

on:
  pull_request:
    branches: [ main, develop ]

jobs:
  verify-screenshots:
    runs-on: ubuntu-latest

    permissions:
      checks: write
      pull-requests: write
      contents: write

    steps:
      - name: Checkout the code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Set up JDK
        uses: actions/setup-java@v4
        with:
          distribution: 'temurin'
          java-version: '17'
          cache: 'gradle'

      - name: Set up Android SDK
        uses: android-actions/setup-android@v3

      - name: Grant execute permission for gradlew
        run: chmod +x gradlew

      - name: Run ktlint
        run: ./gradlew ktlintCheck

      # Screenshot verification step
      - name: Verify Roborazzi Screenshots
        id: screenshotsverify
        continue-on-error: true
        run: ./gradlew verifyRoborazziDebug

      # Generate new screenshots if verification fails
      - name: Generate new screenshots if verification failed
        id: screenshotsrecord
        if: steps.screenshotsverify.outcome == 'failure'
        run: ./gradlew recordRoborazziDebug

      # Extract JIRA ticket from branch name (adjust to your team's workflow)
      - name: Extract JIRA ticket
        id: extract_ticket
        if: steps.screenshotsrecord.outcome == 'success'
        run: |
          BRANCH_NAME=$(echo ${{ github.head_ref }})
          echo "branch_name=$BRANCH_NAME" >> $GITHUB_OUTPUT

          if [[ $BRANCH_NAME =~ (TICKET-[0-9]+) ]]; then
            TICKET="${BASH_REMATCH[1]}"
            echo "ticket=$TICKET" >> $GITHUB_OUTPUT
            echo "Found JIRA ticket: $TICKET"
          else
            echo "ticket=NO-TICKET" >> $GITHUB_OUTPUT
            echo "No JIRA ticket found in branch name"
          fi

      # Commit new screenshots
      - name: Push new screenshots if available
        uses: stefanzweifel/git-auto-commit-action@v5
        if: steps.screenshotsrecord.outcome == 'success'
        with:
          file_pattern: '**/*.png'
          disable_globbing: false
          commit_message: "${{ steps.extract_ticket.outputs.ticket != 'NO-TICKET' && format('{0} feat: update screenshots', steps.extract_ticket.outputs.ticket) || 'NO-TICKET feat: update screenshots' }}"

      # Collect comparison images
      - name: Copy screenshot comparison files
        id: copy_screenshots
        if: steps.screenshotsverify.outcome == 'failure'
        run: |
          mkdir -p /tmp/screenshot-diff

          cd ${{ github.workspace }}
          find . -name "*_compare.png" -exec cp {} /tmp/screenshot-diff/ \;

          if [ -z "$(find /tmp/screenshot-diff -type f -name '*.png')" ]; then
            echo "No screenshot files found"
            echo "has_images=false" >> $GITHUB_OUTPUT
          else
            echo "Found screenshot comparison images:"
            ls -la /tmp/screenshot-diff
            echo "has_images=true" >> $GITHUB_OUTPUT
          fi

      # Create Companion Branch
      - name: Create companion branch for screenshot comparison
        id: companion_branch
        if: steps.copy_screenshots.outputs.has_images == 'true'
        run: |
          BRANCH_NAME="screenshot-compare-${{ github.event.pull_request.number }}"
          echo "branch_name=$BRANCH_NAME" >> $GITHUB_OUTPUT

          git config --global user.name "GitHub Actions"
          git config --global user.email "actions@github.com"

          git stash -u || true

          git branch -D "$BRANCH_NAME" || true
          git checkout --orphan "$BRANCH_NAME"
          git rm -rf .

          mkdir -p screenshot-diff

          cp /tmp/screenshot-diff/*.png screenshot-diff/

          git add screenshot-diff
          git commit -m "Add screenshot comparison results"
          git push origin "$BRANCH_NAME" -f

          echo "companion_push_success=true" >> $GITHUB_OUTPUT

          git checkout -f ${{ github.head_ref }} || git checkout -f HEAD

      # Generate markdown report
      - name: Generate markdown report for screenshot diffs
        id: generate_report
        if: steps.companion_branch.outputs.companion_push_success == 'true'
        run: |
          REPORT="## Screenshot Comparison Results\n\n"
          REPORT+="| Screen Name | Comparison Image |\n"
          REPORT+="|:--------:|:----------:|\n"

          BRANCH_NAME="${{ steps.companion_branch.outputs.branch_name }}"

          for FILE in /tmp/screenshot-diff/*.png; do
            FILENAME=$(basename "$FILE")
            SCREENNAME=${FILENAME%_compare.png}

            IMG_URL="https://github.com/${{ github.repository }}/blob/$BRANCH_NAME/screenshot-diff/$FILENAME?raw=true"

            REPORT+="| \`$SCREENNAME\` | ![Screenshot]($IMG_URL) |\n"
          done

          EOF=$(dd if=/dev/urandom bs=15 count=1 status=none | base64)
          echo "report<<$EOF" >> $GITHUB_OUTPUT
          echo -e "$REPORT" >> $GITHUB_OUTPUT
          echo "$EOF" >> $GITHUB_OUTPUT

          echo "found_images=true" >> $GITHUB_OUTPUT

      # Find existing PR comment
      - name: Find existing comment
        uses: peter-evans/find-comment@v3
        id: find_comment
        if: steps.generate_report.outputs.found_images == 'true'
        with:
          issue-number: ${{ github.event.pull_request.number }}
          comment-author: 'github-actions[bot]'
          body-includes: '## Screenshot Comparison Results'

      # Create or update PR comment
      - name: Create or update comment with comparison results
        uses: peter-evans/create-or-update-comment@v4
        if: steps.generate_report.outputs.found_images == 'true'
        with:
          comment-id: ${{ steps.find_comment.outputs.comment-id }}
          issue-number: ${{ github.event.pull_request.number }}
          body: ${{ steps.generate_report.outputs.report }}
          edit-mode: replace

      # Run unit tests
      - name: Run unit tests
        run: ./gradlew testDebugUnitTest --stacktrace

When the process completes successfully, you can verify that the comment is properly created as shown below: (Screen names are redacted with white boxes for security reasons.)

This workflow operates in the following sequence:

When a PR is created, the screenshot verification task (verifyRoborazziDebug) is executed.
If verification fails (meaning there are UI changes), new screenshots are generated (recordRoborazziDebug).
The JIRA ticket number is extracted from the branch name and included in the commit message. This part can be adjusted to match your company or team’s workflow. (For example, you could use GitLab issue numbers or other work ID formats instead of JIRA.)
Screenshot comparison images are stored in a separate temporary branch (companion branch).
A markdown report containing the comparison images is generated and added as a PR comment.

The Companion Branch Approach

You might naturally wonder: if the number of screenshot images grows excessively, won’t the repository size keep increasing?

That is why the Companion Branch Approach was proposed in DroidKaigi/conference-app-2022.

In the workflow I wrote, comparison images are stored in a temporary branch to prevent the main codebase repository from growing in size, but reference images are still committed to the repository. As shown in the attached diagram, you could further improve this by uploading reference images to GitHub Actions Artifacts and performing the verification task against comparison images created in the companion branch. Also, while the default retention period for GitHub Actions Artifacts is 90 days, if you estimate that the time from PR creation to review and merge completion is typically within one month, you could set the retention period to 30 days as follows:

- name: Store reference images as artifacts
  uses: actions/upload-artifact@v4
  with:
    name: reference-screenshots
    path: src/test/screenshots
    retention-days: 30

In fact, Part 3 of The Software Engineer’s Guidebook also covers this topic in the Snapshot Testing chapter:

The biggest downside of using a snapshot test suite is that the reference images used for comparison can become too large to keep in the same repository as the test code. At companies with a large number of snapshot tests, it is common to store reference images outside the code repository.

Conclusion

Screenshot testing with Roborazzi can be a great help in improving the UI quality of Android apps. In particular, integrating it with a CI pipeline to build an automated UI verification system was valuable, as it enables early detection and visual confirmation of unintended UI changes.

Note that this post does not cover all of Roborazzi’s features. It is important to continuously monitor new features and changes with each release.

References

#robolectric #roborazzi #screenshot-test