shinytest2 vs Cypress: End-to-End (E2E) Testing in Shiny

Appsilon values robustness and understands the importance of End-to-End (E2E) testing for building reliable Shiny applications. Even if you’re unfamiliar with testing, it is likely that you have already done so. You likely manually tested different functions and checked the app ran as planned. In this article, we will compare two popular testing frameworks for R/Shiny: Cypress and shinytest2. 

Are you creating multiple versions of an R/Shiny app? Discover how to measure performance improvements using {shiny.benchmark}.

While Cypress has long been a reliable choice, shinytest2 is a newer option that has recently gained popularity. We will discuss the features and benefits of each framework to help you decide which one is the best fit for your needs. 

TOC:

• What is Automated Testing?
• shinytest2
• Cypress
• Comparing shinytest2 vs Cypress
• E2E Testing in Rhino: shinytest2 OR Cypress

What is Automated Testing?

Automated testing is a test method without much user involvement during the process. The keyword here is “during.” And yes, writing automated tests requires a significant amount of development time.

Automated testing ensures reliability, but don’t forget the end user. Project Leader Maria Grycuk shares her experience on the importance of user tests.

But automated tests offer several benefits, including increased robustness and reliability. As the application grows and the development team changes, these tests help ensure that the correct product is being built. 

In the long run, automated tests pay off by helping to maintain the integrity of the application. There are different types of automated tests, including unit tests and end-to-end tests. This article will focus on end-to-end tests, which test the entire application and how its components work together.

shinytest2 – for E2E Testing Shiny Apps

Note: Please note that the shinytest2 version described in this article is version 0.2.0. It is a fairly recent package and is expected to undergo significant changes. 

Shinytest2 is a new package for testing created by the team at Posit and announced at the rstudio::conf(2022). In particular, shinytest2 tests Shiny apps. This specialized testing framework is something Shiny has lacked in the past. It works by tapping deep into Shiny and the way components interact with the server and testthat machinery.

While complex tests are possible with shinytest2, an important feature is that it  conducts snapshot testing

Snapshot Testing & Shiny

Snapshot testing is a method of comparing the current behavior of an application to a pre-defined snapshot. It is useful for apps that do not change unexpectedly, such as traditional Shiny applications.

Snapshot tests are useful when the UI does not change often. In other words, for apps that look pretty much the same and only have their outputs change with a change in input. This makes it a quality fit with traditional Shiny applications which, once set, rarely change. In this lies the power of shinytest2 and also, its drawbacks.

To test an application through shinytest2, you first launch the application with shinytest2::record_test(). Then, you interact with it and “expect values”. This records your clicks and inputs and these values are saved to the snapshot files. The next time you run the test, these values serve as the single source of truth. The snapshot serves as a comparison to what the application displays later.

Advantages of shinytest2

• Incredibly easy to set up: you just install an R package and you’re done.
• You can get started as soon as you install the package; you do not need to know complex JS.
• The creation of tests is simple and fast.
• This is a good approach to Shiny tests as the app stays static in terms of UI, elements, and components.

Disadvantages of shinytest2

• Real interaction is not emulated very closely – shinytest2 relies on Shiny mechanics.
  • For example, typing is not emulated – the input value is set directly instead. In some situations, this can bypass code that should be tested (e.g. React-based components or custom JS) and limit the test value.
• There is a limited range of options for finding and interacting with elements with only CSS Selectors being available. Interaction is also, only limited to Clicks which makes a complex app with a varied range of inputs (text, scroll) difficult to test.

R/Shiny & Snapshot Testing

The disadvantages of Snapshot tests being ill-suited for dynamic Shiny applications that go beyond the basics also contribute here.

• Snapshot tests might be somewhat costly to maintain.
  • A change of the output ID or a slight rewording of a rendered message can break the test even though it is irrelevant.
  • Care must be taken to limit the scope of the test (only include relevant inputs/outputs) and review the changes to the snapshots whenever they happen.
• You cannot create a snapshot test for a feature/bugfix before you implement it.
  • It can be useful to first write a test to verify a feature / reproduce a bug and only then implement the necessary changes for it to pass (TDD, test-driven development).
  • Snapshot tests can only verify that the application continues to work as it did when the test was created.

Cypress – for E2E Testing Shiny Apps

If you’ve been around web app development, you may have heard of Cypress before. This is because Cypress is a Node.js package that has been around since 2015 when it was first released. Built with the Open Source (MIT License) philosophy and with commercial backing (Cypress.io – a paid platform for uploading and viewing test results) Cypress has become a staple of the developer’s toolbox.

Cypress can test everything from basic websites to complex web applications, but it is not specifically designed to test R/Shiny apps. It is technology agnostic so it does not necessarily care what you used to code the web application. In other words, Cypress is a general test kit.

Programmatic Testing

Shinytest2 works mainly on Snapshot testing, whereas Cypress supports programmatically writing the test cases you want to test against and then having a suite compare those values. If you’ve heard of assertion, you could say that is what Cypress does.

Advantages of Cypress

• Cypress is modern but mature. It allows for almost any kind of element selection with its rich set of functions and is more stable than Selenium which was released in 2004.
• It emulates human interaction and does not rely on R/Shiny, which means the features and the problems of Shiny applications and testing them are not a caveat here.
• You can easily write robust tests which verify specific functionality.
• You can also easily integrate the test execution and do time travel, which simply means checking the before-after states of elements.

Disadvantages of Cypress

• Cypress is dependent on Node.js which can be difficult to use in R/Shiny projects.
• All tests must be written in JavaScript and therefore, you need to learn another language.

Comparing shinytest2 vs Cypress

Below, you can find a comparative summary of the two frameworks, for easy reference and selection.

E2E Testing in Rhino: shinytest2 OR Cypress

If you think Cypress works for you, you can also try Rhino by Appsilon, which is a simpler way to build production-grade Shiny applications. All you need is Node.js on your system, and it will work without any hiccups. 

Additionally, with Rhino v1.3.0, we’ve added shinytest2. This means with Rhino, you can choose the testing framework that best fits your preference, team structure, and specific application requirements.

Ready to make the switch to Rhino? Learn how to migrate an exisiting app to Rhino.

Conclusion: shinytest2 vs Cypress for E2E testing in Shiny

As is with most things, there is no universal right or wrong answer. We recommend going with the method that makes the most sense for your application. Regardless, our recommendation is to conduct E2E Testing. 

Even though it takes time, it makes things that much easier in the future. This long-term focus can make your application more stable over time and stand out as a sustained MVP. We’re seeing Shiny applications move beyond the quick POC and turn into business-critical tooling. So we encourage testing and development of best practices whenever possible.

Looking to upgrade from pagination in R Shiny? Try adding infinite scroll to your Shiny app with our guide.