Creating Test Suites
Features described in this document are being deprecated as part of the transition to Test Workflows - Read More.
A large IT department has a frontend team and a backend team, everything is deployed on Kubernetes clusters, and each team is responsible for its part of the work. The frontend engineers test their code using the Cypress testing framework, but the backend engineers prefer simpler tools like Postman. They have many Postman collections defined and want to run them against a Kubernetes cluster but some of their services are not exposed externally.
A QA leader is responsible for release trains and wants to be sure that before the release all tests are completed successfully. The QA leader will need to create pipelines that orchestrate each teams' tests into a common platform.
This is easily done with Testkube. Each team can run their tests against clusters on their own, and the QA manager can create test resources and add tests written by all teams.
Test Suites
stands for the orchestration of different test steps, which can run sequentially and/or in parallel.
On each batch step you can define either one or multiple steps such as test execution, delay, or other (future) steps.
By default the concurrency level for parallel tests is set to 10, you can redefine it using --concurrency
option for CLI command.
Passing Test Suite Artifacts between Steps
In some scenarios you need to access artifacts generated on previous steps of the test suite. Testkube provides two options to define which artifacts to download in the init container: all previous step artifacts or artifacts for selected steps (step number is started from 1) or artifacts for latest executions of previously executed tests (identified by names). All downloaded artifacts are stored in /data/downloaded-artifacts/:executionId folder. See a few examples below.
Test Suite Creation
Creating tests is really simple - create the test definition in a JSON file and pass it to the testkube
kubectl
plugin.
An example test file could look like this:
echo '
{
"name": "testkube-suite",
"description": "Testkube test suite, api, dashboard and performance",
"steps": [
{"execute": [{"test": "testkube-api"}, {""test": "testkube-dashboard"}]},
{"execute": [{"delay": "1s"}]},
{"downloadArtifacts": {"previousTestNames": ["testkube-api"]}, "execute": [{"test": "testkube-dashboard"}, {"delay": "1s"}, {""test": "testkube-homepage"}]},
{"execute": [{"delay": "1s"}]},
{"downloadArtifacts": {"previousStepNumbers": [1, 3]}, "execute": [{"test": "testkube-api-performance"}]},
{"execute": [{"delay": "1s"}]},
{"downloadArtifacts": {"allPreviousSteps": true}, "execute": [{"test": "testkube-homepage-performance"}]}
]
}' | kubectl testkube create testsuite
To check if the test was created correctly, you can look at TestSuite
Custom Resource in your Kubernetes cluster:
kubectl get testsuites -ntestkube
NAME AGE
testkube-suite 1m
testsuite-example-2 2d21h
To get the details of a test:
kubectl get testsuites -ntestkube testkube-suite -oyaml
apiVersion: tests.testkube.io/v3
kind: TestSuite
metadata:
creationTimestamp: "2022-01-11T07:46:12Z"
generation: 4
name: testkube-suite
namespace: testkube
resourceVersion: "57695094"
uid: ea90a79e-bb46-49ee-a3ef-a5d99cee0a2c
spec:
description: "Testkube test suite, api, dashboard and performance"
steps:
- stopOnFailure: false
execute:
- test: testkube-api
- test: testkube-dashboard
- stopOnFailure: false
execute:
- delay: 1s
- stopOnFailure: false
downloadArtifacts:
allPreviousSteps: false
previousTestNames:
- testkube-api
execute:
- test: testkube-dashboard
- delay: 1s
- test: testkube-homepage
- stopOnFailure: false
execute:
- delay: 1s
- stopOnFailure: false
downloadArtifacts:
allPreviousSteps: false
previousStepNumbers:
- 1
- 3
execute:
- test: testkube-api-performance
- stopOnFailure: false
execute:
- delay: 1s
- stopOnFailure: false
downloadArtifacts:
allPreviousSteps: true
execute:
- test: testkube-homepage-performance
Your Test Suite
is defined and you can start running testing workflows.
Test Suite Steps
Test Suite Steps are the individual components or actions that make up a Test Suite. They are typically a sequence of tests that are run in a specific order. There are two types of Test Suite Steps:
Tests: These are the actual tests to be run. They could be unit tests, integration tests, functional tests, etc., depending on the context.
Delays: These are time delays inserted between tests. They are used to wait for a certain period of time before proceeding to the next test. This can be useful in situations where you need to wait for some process to complete or some condition to be met before proceeding.
Similar to running a Test, running a Test Suite Step based on a test allows for specific execution request parameters to be overwritten. Step level parameters overwrite Test Suite level parameters, which in turn overwrite Test level parameters. The Step level parameters are configurable only via CRDs at the moment.
For details on which parameters are available in the CRDs, please consult the table below:
Parameter | Test | Test Suite | Test Step |
---|---|---|---|
name | ✓ | ✓ | |
testSuiteName | ✓ | ||
number | ✓ | ||
executionLabels | ✓ | ✓ |