add to E2E docs (#14149)
This commit is contained in:
Suzanne Aitchison
GitHub
parent
2686eecb64
commit
dcab661070
1 changed files with 280 additions and 107 deletions
|
|
@ -4,156 +4,329 @@ title: End to End Tests
|
|||
|
||||
# End to End Tests
|
||||
|
||||
[Cypress](https://www.cypress.io), a JavaScript based framework, is used for end
|
||||
to end (E2E) testing. Tests and associated utilities are within the `cypress`
|
||||
directory.
|
||||
|
||||
```shell
|
||||
$ tree cypress
|
||||
We use [Cypress](https://www.cypress.io), a JavaScript based framework, for end
|
||||
to end (E2E) testing. You can find tests and associated utilities in the
|
||||
`cypress` directory:
|
||||
|
||||
```
|
||||
cypress
|
||||
├── fixtures
|
||||
│ ├── images
|
||||
│ │ └── admin-image.png
|
||||
│ └── users
|
||||
│ ├── adminUser.json
|
||||
│ └── changePasswordUser.json
|
||||
├── integration
|
||||
│ ├── adminFlows
|
||||
│ │ └── config
|
||||
│ │ └── authenticationSection.spec.js
|
||||
│ └── loginFlows
|
||||
│ ├── userChangePassword.spec.js
|
||||
│ └── userLogin.spec.js
|
||||
├── plugins
|
||||
│ └── index.js
|
||||
└── support
|
||||
├── commands.js
|
||||
└── index.js
|
||||
├── fixtures (Any hard-coded data, e.g. test users, images)
|
||||
├── integration (The actual tests, grouped by user flow)
|
||||
├── plugins (Cypress plugins)
|
||||
└── support (Where custom commands are found and added)
|
||||
```
|
||||
|
||||
In addition to Cypress, we use
|
||||
[cypress-testing-library](https://github.com/testing-library/cypress-testing-library).
|
||||
This provides custom Cypress commands and utilities to improve how E2E tests are
|
||||
written. dom-testing-library and preact-testing-library are already used for
|
||||
[front-end tests](/tests/frontend-tests), which both offer a similar API in the
|
||||
context of testing a Preact or Stimulus component. All these tools are part of
|
||||
the [Testing Library](https://testing-library.com) family.
|
||||
## Additional tools used
|
||||
|
||||
For the test web server, the
|
||||
[cypress-rails](https://github.com/testdouble/cypress-rails) gem is used to
|
||||
We enhance our use of Cypress with a couple of additional packages.
|
||||
|
||||
### cypress-testing-library
|
||||
|
||||
We use
|
||||
[cypress-testing-library](https://github.com/testing-library/cypress-testing-library)
|
||||
for custom Cypress commands and utilities to improve how we write our tests.
|
||||
This package is part of the [Testing Library](https://testing-library.com)
|
||||
family that we also use in [front-end tests](/tests/frontend-tests), offering a
|
||||
similar API.
|
||||
|
||||
### cypress-rails
|
||||
|
||||
We use the [cypress-rails](https://github.com/testdouble/cypress-rails) gem to
|
||||
start a test web server that runs a rails test environment (`RAILS_ENV=test`).
|
||||
|
||||
It also resets the database between test runs by starting a database transaction
|
||||
at the beginning of a test and performs a rollback at the end of a test being
|
||||
run. The cypress-rails gem also provides a rake task that allows us to
|
||||
coordinate all this work.
|
||||
|
||||
## Running E2E Tests Locally
|
||||
## How to run E2E tests locally
|
||||
|
||||
From the command line, run `yarn e2e` to start E2E testing.
|
||||
### 1. From the command line, run `yarn e2e`
|
||||
|
||||
1. `bundle check` will run to ensure all the gems for the project are up to
|
||||
date. If they are not, you will be prompted to run `bundle install`.
|
||||
Some initial setup and checks will automatically run as part of this command:
|
||||
|
||||
```bash
|
||||
Doing a quick bundle check to make sure gems are all up to date.
|
||||
- `bundle check`: You will be prompted to run `bundle install` if gems for the
|
||||
project are not up to date.
|
||||
- `yarn install`: will ensure front-end packages are up to date.
|
||||
|
||||
* zonebie (0.6.1)
|
||||
Install missing gems with `bundle install`
|
||||
Unable to launch end to end tests. Ensure that all your gems are installed and up to date.
|
||||
### 2. You will then be prompted to set up the end to end (E2E) test database
|
||||
|
||||
```
|
||||
Do you need to set up your end to end (E2E) testing database?
|
||||
|
||||
Answer yes if this is your first time running E2E tests on your local machine or you need to recreate your E2E test database. (y/n)
|
||||
```
|
||||
|
||||
2. `yarn install` will run to ensure front-end packages are up to date.
|
||||
Type `y` if you need to install the E2E test database. Typically you only need
|
||||
to do this the first time you run e2e tests, but you can also run it if:
|
||||
|
||||
```bash
|
||||
[1/5] 🔍 Validating package.json...
|
||||
[2/5] 🔍 Resolving packages...
|
||||
success Already up-to-date.
|
||||
```
|
||||
- you have added additional seed data to `seeds_e2e.rb`
|
||||
- your data based might be corrupted and you want to reset it back to its
|
||||
original state
|
||||
|
||||
3. You will be prompted to set up the end to end (E2E) test database. Type `y`
|
||||
or `Y` to install the E2E test database. Typically you only need to select
|
||||
`y` the first time you run e2e tests, but it can also be run if ever you
|
||||
corrupt your database and need to reset it back to its original state.
|
||||
### 3. Ready to run tests!
|
||||
|
||||
```bash
|
||||
Do you need to set up your end to end (E2E) testing database? Answer yes
|
||||
if this is your first time running E2E tests on your local machine or you
|
||||
need to recreate your E2E test database. (y/n)
|
||||
```
|
||||
The
|
||||
[Cypress test runner](https://docs.cypress.io/guides/core-concepts/test-runner.html#Overview)
|
||||
will open and you are now ready to run end to end tests.
|
||||
|
||||
4. If `y` or `Y` is pressed, the E2E test database will install.
|
||||
|
||||
```bash
|
||||
Setting up the E2E database before running E2E tests...
|
||||
|
||||
|
||||
Dropped database 'PracticalDeveloper_test'
|
||||
Created database 'PracticalDeveloper_test'
|
||||
|
||||
...
|
||||
```
|
||||
|
||||
5. The
|
||||
[Cypress test runner](https://docs.cypress.io/guides/core-concepts/test-runner.html#Overview)
|
||||
will open and you are now ready to run end to end tests.
|
||||
While the test runner is open, any new or updated tests will be dynamically
|
||||
reflected in the UI.
|
||||
|
||||
|
||||
|
||||
## E2E Tests on CI/CD
|
||||
|
||||
E2E tests run on a dedicated build node on Travis. It runs headless via the
|
||||
`bin/e2e-ci` command. These tests currently do not run in parallel with Knapsack
|
||||
Pro as there were issues integrating the cypress-rails gem with the
|
||||
E2E tests automatically run on a dedicated build node on Travis. It runs
|
||||
headless via the `bin/e2e-ci` command. These tests currently do not run in
|
||||
parallel with Knapsack Pro as there were issues integrating the cypress-rails
|
||||
gem with the
|
||||
[knapsack-pro-cypress](https://github.com/KnapsackPro/knapsack-pro-cypress) npm
|
||||
package.
|
||||
|
||||
## Cypress Custom commands
|
||||
## Writing Cypress tests
|
||||
|
||||
[Cypress custom commands](https://docs.cypress.io/api/cypress-api/custom-commands.html)
|
||||
allow you to extend the functionality of the E2E testing framework. In the case
|
||||
of Forem, we need custom commands to create an article, for example.
|
||||
### Seed data
|
||||
|
||||
A custom command is prefixed like any Cypress command by `cy.` All custom
|
||||
commands can be found in the
|
||||
[commands.js](https://github.com/forem/forem/blob/master/cypress/support/commands.js)
|
||||
file.
|
||||
When the E2E test database is set up (via the `yarn e2e` command), it is seeded
|
||||
with the data in
|
||||
[`spec/support/seeds/seeds_e2e.rb`.](https://github.com/forem/forem/blob/816855ce3b1c49eaa38f65c63878b33228b1ee9f/spec/support/seeds/seeds_e2e.rb)
|
||||
|
||||
### Creating a Custom Article Command
|
||||
This seeds file can be added to as needed, and the new data will be reflected
|
||||
when you next run `yarn e2e` and select `y` to the question "Do you need to set
|
||||
up your end to end (E2E) testing database?".
|
||||
|
||||
To create an article as part of your test's setup, use the `cy.createArticle`
|
||||
custom command. It can be called like so:
|
||||
### Test setup
|
||||
|
||||
```javascript
|
||||
cy.createArticle({
|
||||
title: 'Test Article',
|
||||
tags: ['beginner', 'discuss'], // tags are optional
|
||||
content: 'This is a test article',
|
||||
Any individual test setup steps can be included in the `beforeEach` hook. You
|
||||
will almost always want to call the
|
||||
[custom Cypress command](https://docs.cypress.io/api/cypress-api/custom-commands.html):
|
||||
|
||||
```js
|
||||
cy.testSetup();
|
||||
```
|
||||
|
||||
This makes sure that previous cookies are cleared, and the Rails database state
|
||||
is reset (e.g. clearing any articles or changes made in previous tests).
|
||||
|
||||
Some other useful custom commands for setting up tests include:
|
||||
|
||||
- **`cy.loginAndVisit(user, url)`**: Logs in the given user and navigates to the
|
||||
URL, waiting for any user login side effects to complete.
|
||||
- **`cy.loginUser(user)`**: Logs in the given user, without routing to any page.
|
||||
This is handy if you need to complete any other setup steps before visiting
|
||||
the page under test.
|
||||
- **`cy.createArticle(articleData)`**: Creates an article for the currently
|
||||
logged in user. The response returns the URL path to the new article, e.g:
|
||||
`response.body.current_state_path`
|
||||
- **`cy.visitAndWaitForUserSideEffects(url)`**: Visits the given page and waits
|
||||
for any user related network requests to complete to make sure the UI is in a
|
||||
'ready' state for testing. Particularly useful if you couldn't use
|
||||
`cy.loginAndVisit`.
|
||||
- **`cy.signOutUser()`**: Logs out the current user and returns to the home
|
||||
page, waiting on any side effects completing.
|
||||
|
||||
You can
|
||||
[see all custom commands in the Cypress support.commands.js file](https://github.com/forem/forem/blob/d21a5d8ed4501d19c4f499013081b212cc8dd97f/cypress/support/commands.js)
|
||||
|
||||
### Finding elements
|
||||
|
||||
In almost all cases, we use
|
||||
[cypress-testing-library](https://github.com/testing-library/cypress-testing-library)
|
||||
commands to find elements in tests.
|
||||
|
||||
The most robust way to do this is to find by role and accessible name, e.g.:
|
||||
|
||||
```js
|
||||
cy.findByRole('button', { name: 'Log in' });
|
||||
```
|
||||
|
||||
We favor `findByRole` queries where possible because:
|
||||
|
||||
- It is more specific and reliable than e.g. `cy.findByText('Log in')` -
|
||||
narrowing our selector to only buttons helps us make sure we match with the
|
||||
correct element.
|
||||
- It will only return accessible elements. For example, any element with
|
||||
`display: none` or similar property that would stop a user from perceiving or
|
||||
interacting with the element will not be returned.
|
||||
- It draws attention to problematic HTML that could impact users of assistive
|
||||
technology. For example, if the "Log in" button was actually a `div`, it won't
|
||||
be returned by the above selector.
|
||||
- It can help highlight issues with accessible names. For example, if
|
||||
`cy.findByRole('link', { name: 'Profile' })` returns 10 links for different
|
||||
profiles, we can readily identify an accessiblity issue where screen reader
|
||||
users would not be able to differentiate between the links.
|
||||
|
||||
#### Scoping selectors
|
||||
|
||||
Cypress allows for a few methods to scope our element selectors to specific
|
||||
sections of the page. Scoping to a smaller area of the page can allow us to
|
||||
select elements more easily, and focus in on the area of the app under test.
|
||||
|
||||
##### Chaining
|
||||
|
||||
One way to scope your selector is to chain it off of a previous selector. For
|
||||
example, the below code will find the article link contained within the `<main>`
|
||||
element, ignoring any similar links the header, etc:
|
||||
|
||||
```js
|
||||
cy.findByRole('main').findByRole('link', { name: 'My article' });
|
||||
```
|
||||
|
||||
This is particularly useful if you only need to find a single element in the
|
||||
given section of the page.
|
||||
|
||||
##### Using the `within` callback
|
||||
|
||||
Another way to scope your selectors is by using the `within` method. This scopes
|
||||
any selectors in the callback to the given element, and can be particularly
|
||||
useful if you want to conduct all of your test steps within the same container.
|
||||
|
||||
For example, the below code will find the "profile preview" card, and then scope
|
||||
all queries to that card alone, ignoring any other content on the page:
|
||||
|
||||
```js
|
||||
cy.findByTestId('profile-preview-card').within(() => {
|
||||
cy.findByRole('link', { name: 'User profile' });
|
||||
cy.findByRole('button', { name: 'Follow' });
|
||||
});
|
||||
```
|
||||
|
||||
If you want to do something with the article creation's response, you can call
|
||||
it like so.
|
||||
#### Best practice in selecting elements
|
||||
|
||||
```javascript
|
||||
cy.createArticle({
|
||||
title: 'Test Article',
|
||||
tags: ['beginner', 'discuss'], // tags are optional
|
||||
content: 'This is a test article',
|
||||
}).then((response) => {
|
||||
cy.visit(response.body.current_state_path); // path to article
|
||||
We tend to follow the
|
||||
[testing-library guiding principles](https://testing-library.com/docs/guiding-principles)
|
||||
for selecting elements:
|
||||
|
||||
> The more your tests resemble the way your software is used, the more
|
||||
> confidence they can give you.
|
||||
|
||||
You can
|
||||
[find the suggested priority order of testing-library queries on their website](https://testing-library.com/docs/queries/about#priority),
|
||||
but as a general rule, we try to favor queries which are accessible to
|
||||
everyone - i.e. how would a user find a "Log in" button? They'd look for a
|
||||
button with the name "Log in", so `cy.findByRole('button', { name: 'Log in' })`
|
||||
seems like a good fit
|
||||
|
||||
Avoid selecting elements by classname or any other property that our users
|
||||
wouldn't be aware of. If you need to find an element that doesn't have an
|
||||
obvious semantic HTML or accessible name query, then give your element a
|
||||
`data-testid` and use `cy.findByTestId('my-element')` to find it in your test.
|
||||
This should be a last resort, and will likely be most useful for scoping
|
||||
selectors.
|
||||
|
||||
For further reading, check out
|
||||
[Kent C. Dodd's article on making UI tests resilient to change](https://kentcdodds.com/blog/making-your-ui-tests-resilient-to-change).
|
||||
|
||||
## Common gotchas
|
||||
|
||||
We've noticed some common "gotchas" that can cause flakiness in our Cypress
|
||||
tests.
|
||||
|
||||
### 1. Route changes should be followed by a unique selector
|
||||
|
||||
If the page changes, for example if your test steps click on a link to an
|
||||
article, then it's important to make your next selector unique to the new page.
|
||||
This makes sure that Cypress doesn't find a matching element on the page you
|
||||
just left.
|
||||
|
||||
See the examples below of how to make these route changes more robust.
|
||||
|
||||
#### 🚫 Before: Route changes, but we find the `main` element on the previous page
|
||||
|
||||
```js
|
||||
cy.findByRole('main').findByRole('link', { name: 'Test article' }).click();
|
||||
// After clicking the link we can _sometimes_ accidentally get a reference to the 'main' element on the page we just left
|
||||
cy.findByRole('main').findByRole('button', { name: 'Share post' });
|
||||
// Cypress fails to find the 'Share post' button inside the previous page's `main` element, and the test fails
|
||||
```
|
||||
|
||||
#### ✅ After: Route changes, and we find a unique element before proceeding
|
||||
|
||||
```js
|
||||
cy.findByRole('main').findByRole('link', { name: 'Test article' }).click();
|
||||
// The 'Share post' button doesn't exist on the page we just left, so Cypress will wait for it to be shown on the new page
|
||||
cy.findByRole('button', { name: 'Share post' });
|
||||
```
|
||||
|
||||
### 2. Interactive elements must be initialized before clicking
|
||||
|
||||
In a lot of places we present views in Rails-generated HTML and asynchronously
|
||||
attach JavaScript event listeners after the page has loaded. This means that in
|
||||
an automated test environment it is possible to click a button before its click
|
||||
handler has been attached.
|
||||
|
||||
For this reason, it's important to double check how a feature's click handlers
|
||||
are initialized and, if necessary, make sure Cypress waits for the button to be
|
||||
ready to click.
|
||||
|
||||
See the examples below of how to make this kind of button interaction more
|
||||
robust.
|
||||
|
||||
#### 🚫 Before: We click a button without waiting for initialization
|
||||
|
||||
```js
|
||||
cy.findByRole('main').findByRole('link', { name: 'Test User Profile' }).click();
|
||||
// We immediately try to click a button that's initialized asynchronously in JS. Sometimes the test will fail as the click handler is not yet attached.
|
||||
cy.findByRole('button', { name: 'Follow' }).click();
|
||||
```
|
||||
|
||||
#### ✅ After: We wait for a data attribute that indicates the initialization has completed
|
||||
|
||||
```js
|
||||
cy.findByRole('main').findByRole('link', { name: 'Test User Profile' }).click();
|
||||
// A data attribute is added to initialized follow buttons, and Cypress waits until this is present on the page
|
||||
cy.get('[data-click-initialized]');
|
||||
cy.findByRole('button', { name: 'Follow' }).click();
|
||||
```
|
||||
|
||||
### 3. Lingering network requests interfere with test setup or new user login
|
||||
|
||||
Before each test we usually call `cy.testSetup()` to ensure cookies are cleared
|
||||
and a user may be logged in fresh. However, if a previous test triggered
|
||||
user-related network requests, and didn't wait until their completion, then
|
||||
occasionally responses to these requests interfere with test setup and cause the
|
||||
previous user to be persisted.
|
||||
|
||||
This is particularly prevalent in very short tests, but can also happen if you
|
||||
try in the middle of a test to sign out as one user, and immediately log back in
|
||||
as another.
|
||||
|
||||
This issue is best avoided by:
|
||||
|
||||
- Utilising custom Cypress commands (e.g. `cy.loginAndVisit(user, url)`,
|
||||
`cy.visitAndWaitForUserSideEffects(url)`, `cy.signOutUser()`) that help ensure
|
||||
side effects from network requests are accounted for
|
||||
- Splitting tests into 'single user' tests (i.e. avoid logging in as multiple
|
||||
users in the same test)
|
||||
|
||||
#### 🚫 Before: Using the `cy.visit(url)` command directly without awaiting side effects
|
||||
|
||||
```js
|
||||
beforeEach(() => {
|
||||
cy.testSetup();
|
||||
cy.fixture('users/articleEditorV2User.json').as('user');
|
||||
|
||||
cy.get('@user').then(() => {
|
||||
cy.loginUser(user).then(() => {
|
||||
// The `visit` command does not take user-related network requests into account. If a test runs quickly, the responses may bleed into the next test setup
|
||||
cy.visit('/dashboard');
|
||||
});
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Creating a Response Template Command
|
||||
#### ✅ After: Using the custom `loginAndVisit` command
|
||||
|
||||
To create a response template as part of your test's setup, use the
|
||||
`cy.createResponseTemplate` custom command. It can be called like so:
|
||||
```js
|
||||
beforeEach(() => {
|
||||
cy.testSetup();
|
||||
cy.fixture('users/articleEditorV2User.json').as('user');
|
||||
|
||||
```javascript
|
||||
cy.createResponseTemplate({
|
||||
title: 'Test Canned Response',
|
||||
content: 'This is a test canned response',
|
||||
cy.get('@user').then(() => {
|
||||
// The custom command logs in the user and visits the page, ensuring that user-related network requests are awaited
|
||||
cy.loginAndVisit(user, '/dashboard');
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue