Updating manual tests with scraper changes
Status: Draft
Assignee: @e12231202
Problem
Manual updating of the manual test cases according the scraper data seems to be very error-prone, the most frequent errors being heatmap related (wrong color expected, non-existing plant relations, inconsistent time periods in temporal relations, etc.).
It also implicates work for any scraper update/fix, i.e. anything that changes finalDataset.csv.
Constraints
Assumptions
- Scraper data is correct
Solutions
Alternative A
Test cases don't have test data included and during testing the tester (including automated tests) directly look at CSVs
Alternative B
We generate the test cases from the data
Alternative C
Creating a bash script that must be run before tests and that inserts special test fixtures (and then another script for removal) that the relevant tests are executed on
Currently implemented in MR !2140
Alternative D
Simply maintain manual test cases & guided tour by hand, maybe with a warning in scraper if these plant values change
Tests have been updated accordingly in MR !2231
Alternative E
Build a consistency checker between scraper data and test data
Alternative F
We could (should) add a few test cases that checks the resulting CSV of the scraper, i.e. finalDataset.csv, e.g. if it includes hierarchy data etc. to avoid something like issue #2625 where the inheritance of certain attributes gets broken.
See also #1949.
In these test cases, we can link to the manual test cases that use the same test data.
When one of these test cases breaks, we update the manual test cases as well.
If a test checks for data correctness it should belong as an automated test on the scraper output.
These tests should run against fixed input fixtures rather than live scraped data; if we use live data we would experience the same issues we are currently facing with the manual test cases.
Alternative G
Fuzzy some manual tests/make manual test assertions independent from data.
A data-independent version would assert the behavior instead of the concrete value, e.g. set up a known previous planting that has some relation, and check that the heatmap is colored at all.