Analyzing Results
Reporting
Once your tests results are in, you’ll need to analyze those to take corresponding actions:
- Tactical reporting of test results is geared towards fixing immediate problems with your tests: test failures, flaky tests, long execution times or high resource consumption.
- Strategic reporting of test results is geared towards planning out your test activities for your teams: Which areas of your applications should be tested more? Are certain components showing an increased failure rate over time? Are certain tests taking longer and longer?
CI/CD tooling will help with the tactical analysis to a certain extent, but rarely has any support for strategic analysis of test results, especially if you are running multiple testing tools and/or are triggering tests outside the traditional CI/CD workflow. External tooling can help in this regard, provided that it can ingest any output provided by your testing tools, which of course has to be configured/scripted accordingly.
Analysis of tests is important to both DevOps for test execution infrastructure analysis/optimization and QA for planning both tactical and strategic testing activities.
Analyzing Test Results with Testkube
Testkube includes a powerful Test Insights module that collects individual result metrics from each executed test and allows for powerful analysis for both operational and strategic decision making, for example:
- Which test(s) are consuming the most time/resources in our testing pipelines?
- How are our overall testing efforts trending in regard to pass/fail ratios, number-of-tests, flakiness, etc.
- Which components or types of tests should we be focusing more (or less) on?