Recently, I worked on automating some internal processes for building and releasing Flutter applications. Part of this work has involved integrating integration test runs and screenshots as part of the release and deployment process.

I wanted to be able to use the same set of tests to validate our code on Android and iOS devices without having to write large amounts of platform-specific code or configuration. These tests needed to be able to integrate with Fastlane, so they can be utilized by CI/CD. Specifically, this restricted setup runs using command line tools with no manual steps in Xcode or Android Studio, aside project level configuration.

My objective was to write an entry point to launch tests that can…

1. Validate the end-to-end navigation through the app.
2. Verify that the application visually renders as expected on a wide variety of devices.
3. Automatically generate screenshots for the app store rather than relying on manual collection.

This work took me on an exploration of different strategies for testing in Flutter as well as the console tools available for interacting with both Android and iOS emulators.

This blog is Part 2 of a three-part series exploring automating Flutter CI/CD on CircleCI. Part 1 covered setting up Fastlane to build and deploy applications locally, this post outlines automating screenshot capture and test runs, and part 3 discusses configuring CircleCI to automate these processes.

The Application

My example application is a combination of the Flutter counter app and the Random Word generator from the Flutter Getting Started Codelab. It includes a button to increment a counter, save a favorite number, and navigation to a favorites list.

While this isn’t functionally useful, it is complex enough to have application state and basic navigation without introducing extraneous dependencies.

Basic Integration Tests

Flutter currently recommends utilizing the integration_test package to run integration tests. This package uses the same API as the widget tests, but since it launches the full application, it requires a target device such as a physical device or emulator to run the tests.

The official instructions include setup steps for running integration tests on Android and iOS, but these steps don’t meet my goal of using the same code to set up and run the tests for both device types – they are more oriented toward running on Firebase.

The snapshot documentation seemed to be more device agnostic, so I went ahead with the steps to create a basic test by adding the dependencies and creating a basic test in the recommended directory. The test just opens the home page, then navigates to the favorites screen.

integration_test/main_test.dart

import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';
import 'package:ex_flutter_fastlane/main.dart' as app;


void main() {
 IntegrationTestWidgetsFlutterBinding.ensureInitialized();


 group('end-to-end test', () {
   testWidgets('render home and navigate', (tester) async {
     app.main();
     await tester.pumpAndSettle();
     expect(find.text('0'), findsOneWidget);


     final Finder favorites = find.text('Favorites');
     await tester.tap(favorites);
     await tester.pumpAndSettle();
     expect(find.text('No favorites yet.'), findsOneWidget);
   });
 });
}

To ensure my project configuration was correct, I started an iOS emulator and an Android emulator then ran the tests on each using the command: flutter test integration_test.

Integration Test Snapshots

Now that I had the basic tests running, I went ahead and updated the code to add screenshots following the screenshot guidelines.

I modified integration_tests/main_test.dart to assign the binding to a variable, turn off the debug banner, and add some Android-specific configuration.

void main() { 
  final binding = IntegrationTestWidgetsFlutterBinding.ensureInitialized();
 setUpAll(() {
   return Future(() async {
     WidgetsApp.debugAllowBannerOverride = false; // Hide the debug banner
     if (Platform.isAndroid) { // from dart:io
       await binding.convertFlutterSurfaceToImage();
     }
   });
 });

Then I added two snapshots to the test:

     expect(find.text('0'), findsOneWidget);
     await binding.takeScreenshot('0-home');

…and…

     expect(find.text('No favorites yet.'), findsOneWidget);
     await binding.takeScreenshot('1-favorites');

Before moving on, I re-ran flutter test integration_test on both devices. For Android, everything was fine, but iOS was not happy.

The console output showed an exception.

The following MissingPluginException was thrown running a test:
MissingPluginException(No implementation found for method captureScreenshot on channel plugins.flutter.io/integration_test)

I looked up the issue and found that this is a known issue: [integration_test] Plugin not registered unless running via XCTest. This is a problem since applying a Flutter patch to CI would be tricky. There is another workaround in the comments that seems like it applies, so I updated AppDelegate.swift with the following code.

let controller : FlutterViewController = window?.rootViewController as! FlutterViewController
IntegrationTestPlugin.instance().setupChannels(controller.binaryMessenger)

This is not ideal and will need to be removed as soon as the issue is corrected, but for now, the tests will run again with flutter test integration_test.

Even though the tests run, the output isn’t being written anywhere. To take care of that, I needed to invoke them through a driver and designate an output location. The driver acts as an intermediary between the tests running on the target device and the host device file system. I created a driver file based on the example driver in the snapshot documentation.

test_driver/integration_test.dart

import 'dart:io';
import 'package:integration_test/integration_test_driver_extended.dart';


Future<void> main() async {
 await integrationDriver(
   onScreenshot: (String screenshotName, List<int> screenshotBytes) async {
     final File image =
         await File('screenshots/$screenshotName.png').create(recursive: true);
     image.writeAsBytesSync(screenshotBytes);
     return true;
   },
 );
}

This differs slightly from the code in the integration_test documentation by placing the screenshots in a directory rather than at the project root.

I was then able to run the tests using…

flutter drive --driver=test_driver/integration_test.dart --target=integration_test/main_test.dart 

…and the screenshots were saved to the screenshots directory!

Multiple Devices

For each of the prior commands, after execution, I had to specify the device I wanted to use. This was avoidable by adding -d $deviceName to the commands, but it didn’t solve the issue of overwriting the screenshots.

To correct the overwriting, I decided to pass the device name as an environment variable to the driver. Then, it can be prefixed to the snapshot name. To do so, I updated test_driver/integration_test.dart by adding a few lines at the top to read the device name:

 String deviceName = Platform.environment['DEVICE_NAME'] ?? '';
 String devicePrefix = deviceName.isEmpty ? '' : '${deviceName}_';

Then, use it when the file is created.

final File image = await File('screenshots/$devicePrefix$screenshotName.png')
             .create(recursive: true);

Now, when the DEVICE_NAME environment variable is set, it will be prefixed to the screenshots.

Before implementing more code, I needed to decide which devices to use. I wanted the far ends of the size spectrum supported by the application, but I also needed to meet the requirements for app store upload.

I consulted the Apple App Screenshot Specifications and Play Console Screenshot Documentation. I decided, for this first implementation, I would just use one phone and one tablet for each type, and then I would add more after the proof of concept is complete.

For iOS, I decided to use the iPhone 8 Plus and the iPad Pro 6th generation, but I needed to make sure they were available on my system. I executed xcrun simctl list to check which devices were present. iPhone 8 Plus was missing, so I installed it through the Simulator UI. I also could have used xcrun simctl create.

For Android, I decided to use the Pixel 5 and Galaxy Nexus. I used emulator -list-avds to check if they existed. They did not, so I created them with Android Studio Device Manager. I also could have used avdmanager create avd.

Now that I had my list of devices, I created a dart file to list the devices and issue the command to run the tests. I put it with the other driver code.

test_driver/emulator_runner.dart

Future<void> main() async {
  List<String> emulatorIds = [
   'Pixel_5_API_33',
   'Galaxy_Nexus_API_33',
   'iPhone 8 Plus',
   'iPad Pro (12.9-inch) (6th generation)'
 ];
}


Future<void> runFlutterIntegrationTests(
   String deviceId, String deviceName) async {
 const String integrationTestDriver = 'test_driver/integration_test.dart';
 const String integrationTestTarget = 'integration_test/main_test.dart';


 final result = await Process.run('flutter', [
   'drive',
   '-d',
   deviceId,
   '--driver=$integrationTestDriver',
   '--target=$integrationTestTarget',
 ], environment: {
   'DEVICE_NAME': deviceName
 });
}

This was a good start, but there is an obvious gap – how will it map between the device name and the device ID that the flutter drive command needs?

For manual testing, I started the emulators, then used flutter devices to output the device IDs. I then tested that it worked, but for automation, I’d like to instead start the emulator by name, run the tests, then shut down the device. This code needs to vary by operating system and utilize a lot of command line tools, xcrun, emulator, and adb at a minimum, so instead of writing all of that, I went with a package instead.

The emulators package wraps that tooling to manage setup and cleanup for each emulator. I just had to provide it with the list of names that I already had. It also provides some snapshot functionality that would prove useful later. I followed the example in the package documentation and added a new function to the emulator runner.

test_driver/emulator_runner

Future<void> runFlutterScreenshotTests(List<String> emulatorIds) async {
 final emulators = await Emulators.build();
 final configs = [
   {'locale': 'en-US'},
 ];
 await emulators.forEach(emulatorIds)((device) async {
   for (final c in configs) {
     DeviceState state = device.state;
     await runFlutterIntegrationTests(state.id, state.name);
   }
 });
}

Then, I just had to call await runFlutterScreenshotTests(emulatorIds) from the main function. Now that everything was wired up, I closed all of my open emulators, then re-ran the tests with dart test_driver/emulator_runner.dart.

Excellent. I now had images for all of my devices in a single directory. There was just one thing bothering me. There was no status bar, and there was a glaring placeholder where it should be.

This makes sense. The integration_test package takes screenshots within the context of the Flutter application; it doesn’t know about OS features like status bars. These screenshots are perfectly fine for some use cases, like checking for overflow or golden tests.

It just doesn’t look right to me for uploading to the app store. I did some research into the available options, and it seemed like the two main options were to use an image manipulation library and apply overlays or take our screenshots at the emulator level instead of at the application level.

Luckily, the package that I am using for starting and stopping the devices also contains some functionality that assists with taking screenshots. Unluckily, it is not compatible with integration_test; it uses the prior flutter_driver tests. I would have to write a separate set of tests for it.

I was working through this problem and realized I needed to be careful about how many images I planned to upload for the app store. Google Play has a limit of eight for each device type. However, when I run our full suite of integration tests, I may want a lot of screenshots to compare to each other across runs.

I decided to go ahead and decouple the app store images from the rest of the integration test suite and write a separate, smaller test for the app store using flutter_driver. This was a tradeoff, and I spent some time looking into other options, but each one I identified was significantly more code and introduced even more dependencies into the stack.

Emulator Screenshots

First, I needed to write the new test implementation using flutter_driver. I wanted this separate from the other tests, so I put it with the drivers rather than the rest of the integration tests.

This will set up the tests with a clean status bar if possible (I noticed that it does not always work on iOS devices), run the tests, and then output them to an Android or an iOS directory. These directories are arbitrary in this case. Some automation tools, such as Fastlane, require specific paths to handle uploading screenshots to the app stores automatically.

test_driver/emulators/screenshot_test.dart

void main() async {
 const androidScreenshotPath = 'screenshots/store/android';
 const iosScreenshotPath = 'screenshots/store/ios';


 final driver = await FlutterDriver.connect();
 final emulators = await Emulators.build();
 final screenshot = emulators.screenshotHelper(
   androidPath: androidScreenshotPath,
   iosPath: iosScreenshotPath,
 );


 setUpAll(() async {
   await screenshot.cleanStatusBar();
   await driver.waitUntilFirstFrameRasterized();
 });


 tearDownAll(() async {
   await driver.close();
 });


 takeScreenshot(identifier) async {
   await driver.waitUntilNoTransientCallbacks();
   await screenshot.capture(identifier);
 }


 group('end-to-end test', () {
   test('Navigate and Screenshot', () async {
     find.text('0'); // this will wait for the render to complete
     await takeScreenshot('0_home');


     final favoritesButton = find.text('Favorites');
     await driver.tap(favoritesButton);
     await takeScreenshot('1_favorites');
   });
 });
}

This also needs a driver:

test_driver/emulators/screenshot.dart

import 'package:flutter/material.dart';
import 'package:flutter_driver/driver_extension.dart';
import 'package:ex_flutter_fastlane/main.dart' as app;


void main() {
 enableFlutterDriverExtension();
 WidgetsApp.debugAllowBannerOverride = false;
 app.main();
}

The naming convention of the driver and the test is significant. The driver will look for a test file that shares its name with _test appended.

This was all the code that I needed to write to run the tests. I just needed to hook it into the existing emulator code by adding a new call in the existing runFlutterScreenshotTests function execution loop in emulator_runner.

test_driver/emulator_runner.dart

 await emulators.forEach(emulatorIds)((device) async {
   for (final c in configs) {
     DeviceState state = device.state;
     await runFlutterIntegrationTests(state, c);
     await emulators.drive(device, 'test_driver/emulators/screenshot.dart',
         config: c);
   }
 });
}

Now, when I run the tests, they will integrate with the existing tests and generate new screenshots that include the status bar.

Conclusion

At this point, the testing code meets all of the goals outlined at the start of the project and has introduced all of the foundational tooling and concepts I used for my task.

Because it is launched with a single command after the emulators have been created, it is compatible with use on Fastlane. Prior to using it on CI devices, I recommend doing some preparation.

First, think through the specific use cases of your application that you want to test and configure the test runs accordingly. Generally, I would expect full integration tests only need to be run once per device family on CI. They are likely to be more resource intensive than a smaller test to just create screenshots.

Second, keep in mind that iOS tests will require a box that can run Xcode. If this is not already available, it will need to be a consideration.

Third, running Android emulators on VMs can be tricky with regard to resource consumption and GPU emulation. Expect to spend some time tuning the creation of the emulators so that they perform as expected.

The code in this post is a proof of concept and an outline rather than production-ready source material. I did use something similar for my implementation, but I added logging, error handling, and some additional environment configuration that fit my specific use case.

Overall, I was pleased that I was able to automate the screenshot generation. I learned a lot about the flutter test infrastructure and running device emulators, and I look forward to continued work in the integration_test package around screenshot functionality.

In our next installment, we will discuss configuring CircleCI to automate these processes. Stay tuned by subscribing to the Keyhole Dev Blog!