Last week I wrote about my initial learnings with Maestro. Today I continue by sharing what I have learned while developing Maestro tests for an mvp of an cross-platform app¹.

This post will cover:

1. The Basics of Maestro Tests
2. Leveling Up Maestro Tests with JavaScript
3. Can LLMs Write Maestro Tests?
4. Using Maestro’s Cloud
5. Integrating Maestro with a continous integration (CI) workflow

The Basics of Maestro Tests

The unit of a Maestro test is a called a flow and is contained in a yaml file. Flow files have two main sections:

1. The config section: This is where you define the appId, name, and any flows that should run before and after the test.
2. The test section: This is where you write your tests.

Honestly, the documentation is great. So rather than re-writing it here, I will highlight a few of my favorite commands and features.

• Selectors with heirachical lookup. This means you can use a number of different properties to find an element: ids, text, regex, etc. This feature allows you to write more readable and more specific tests.
• Nested flows. With the runFlow command you can run flows within other flows. This is useful for larger apps with similar component on multiple pages.
• Multiple ways of moving the screen: Maestro has three unique commands that give options when manipulating the screen: the simple scroll, more precise scrollUntilVisible, and the most flexible swipe – which has many options.
• Simple access of variables and the ability to extend Maestro with JavaScript.

Here is a full annontated test flow file.

Maestro + JavaScript

Maestro’s built in commands handle a lot of tests cases, but if you have a case that is not covered then you can run some JavaScript! For simple uses, you can write javascript right in your yaml file with the evalScript command and for more complex scripts you can use the runScript command. Below is an example that validates an elements values as a number in a currency format:

# flow.yaml
# ...
- copyTextFrom: {id: "MarketCap"}  # Copy the marketcap text/value for use in the script
- runScript: script.js  # Run JS to validate the value
- assertTrue: ${output.marketCap}  # Check that the value passed the validation

Here is the JavaScript code that validates the marketcap value.

This is a simple example. However, it illustrates how Maestro’s JavaScript availablity can greatly enhance writing unique, custom, and creative tests.

An additional benefit of Maestro’s use of JavaScript is that you can utilize Page Object Models. These are js files that house objects with properties corresponding to specific pages. Having models in a central location and format allows writing tests that are more readable and easier to maintain over time.

// login.js
output.login = {
    email: 'email_text',
    password: 'password_text',
    loginBtn: 'loginButton',
    registerBtn: 'registerButton'
}

# flow.yaml
- runScript: login.js
- assertVisible: ${output.login.loginBtn}

Can LLMs Write Maestro Tests

In short yes.

I fed Claude 3.5 a really simple prompt² and the relevant components. It generated a pretty good test with only two noticable mistakes³ that could be easily corrected with a more comprehensive prompt.

If I were writing many tests from scratch I would spend a little time to refine the prompt, lean on LLMs to generate the tests, and then manually verify and tweak them. On a philosophical note: I find it very cool that LLMs can be used both in the generation of tests and in executing the tests themselves!

Maestro Cloud

You can run a flow, or suites of flows, on Maestro’s Cloud to avoid the need to set up simulators! All you need to do is build the app and run maestro cloud <BUILT_APP_FILE> <TEST_FILE>. The outcome, logs, and a recording of the test is available in the the cloud console. A great tool and really handly for CI workflows.

Basics of Continuous Integration with Maestro

In just five basic steps I had a CI workflow:

1. Create a new hello world project in Android Studio
2. Add a .maestro directory with a simple test named flow.yaml
3. Add a .github/workflow directory with the examlpe from Maestro Cloud’s docs⁴
4. Create a github repo and added my Maestro Cloud API key as a repository secret.
5. Push the project to github where the github action builds the app and then runs the tests using maestro cloud.

While, this CI workflow is super simple it shows the potential and usefulness of CI in realworld scenarios – you can validate your app on multiple devices and frameworks without the lengthy local build processes or tedium of configuring simulators!

In my next post I will document my learning of CI basics, Github Actions, and of the Jenkins framework.

Thanks for reading.