Testing Strategy

Testing Approach

The software testing strategy aims to ensure the delivery of high-quality software by employing a comprehensive testing approach throughout the development lifecycle. This strategy combines various testing types and levels to validate the functionality, performance, and security of the software.

The testing approach for PermaplanT will follow a mix of unit testing, integration testing, and system testing.

• Unit testing will be conducted for individual components and functions to ensure that they are working as expected.
• Integration testing will be conducted for API endpoints and interactions between components to ensure that they are functioning correctly together.
• System testing will be conducted as manual tests and automated e2e tests to ensure the proper functionality of the system as a whole.

The effort necessary to write and maintain the tests should match their impact. Therefore, it is important to identify and use the appropriate types of tests for the current use case. Non-trivial single-component functionality should be covered by unit tests. Non-trivial multi-component functionality should additionally be covered by integration tests. Wherever possible and reasonable system tests should be automated to avoid the variance in scope and granularity that comes with manual testing.

Goals

• Prefer tests that are easy to maintain, e.g., prefer unit tests for detailed logic tests.
• Ensure that the software meets functional requirements and operates as intended.
• Identify and mitigate risks associated with software defects and failures.
• Enhance user experience by maintaining low bug rate and responsiveness.
• Improve overall software quality and reliability.
• Ensure that the 'mr', 'dev', and 'master' environments are tested with full End-to-End (E2E) tests.
• Ensure that the 'www' environment is tested with smoke tests.

Objectives

• Maintain an 80% backend branch coverage.
• Run tests with all environments.
• TBD frontend objectives
• TBD additional objectives

In Scope:

• E2E testing of use cases
• E2E testing of UI for different sizes/devices
• Integration testing of various modules and components
• System tests
• Performance tests
• Security tests
• Regression testing
• User acceptance testing
• Compatibility testing on different platforms, browsers, and devices

Out of Scope:

• Database tests
• Stress tests

Current State of CI/CD Tests

For further information about the environments see doc/ci.md

Production Environment (www)

• Testing: No Tests are currently run on the production environment.
• Since the production environment deploys the master branch, which has already been tested in the CI/CD pipeline, it is not necessary to run additional non-E2E tests.

Master Branch

• Testing: The master branch is currently missing E2E tests.

Test Coverage Across Environments

Goals

The goal is to ensure that all environments pass all tests. Specifically:

1. Production:

   • Must pass at least E2E and smoke tests.
   • If these tests fail, administrators should be alerted immediately.

2. Master:

   • E2E tests must be added and pass consistently.
   • If this test fail, administrators should be alerted immediately.

Testing Types and Levels

1. Unit Testing

Unit tests are used to test individual units of code in isolation from the rest of the system. This is important because it allows us to validate that each unit is working as intended, without interference from other parts of the system.

• Unit tests must be created by every developer during the development process, following the Arrange-Act-Assert (AAA) pattern.

Short example:

#![allow(unused)]
fn main() {
struct Plants;

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_abs_for_a_negative_number() {
    // Arrange
    let negative = -5;

    // Act
    let answer = abs(negative);

    // Assert
    assert_eq!(answer, 5);
    }
}
}

Frontend

For the frontend unit tests will be used to test the following areas:

• State management logic: Since the frontend relies heavily on state management, it is important to ensure that the state is being properly managed and updated.
• Functions that handle data manipulation, such as sorting or filtering
• Components that render UI elements
• Correct event handling
• Form validation and submission

Backend

For the backend, unit tests will be used to test the following areas:

• Database queries and operations
• Business logic and data manipulation, e.g. code that performs calculations, manipulates data, or makes decisions based on input
• Authentication and authorization
• Error handling

In the backend unit tests can be found in the src/test directory.

2. Integration Testing

Integration tests are used to test the integration between different parts of the system. This is important because it allows us to validate that the different parts of the system are working together correctly.

In the application the integration tests will be used to test the following areas:

• Continuous Integration (CI) pipelines must run integration tests for different modules and components.

Frontend

• API calls using a mock API

Backend

• API endpoints
• Database queries

In the backend integration tests can be found in the src/test/ directory. The whole module is annotated with #[cfg(test)] and will therefore only be compiled for tests.

3. System Testing

System testing will include end-to-end testing of the application to ensure overall functionality and user experience. This will include testing of all features, navigation, and error handling.

• System test scenarios must be derived from the use cases and the requirements they imply.
• Manual system tests are documented in doc/tests/manual/protocol.md.
• Automated system tests (e2e tests) can be found in the top-level e2e folder.
• Both are specified in Gherkin syntax.
• Exploratory testing techniques must be utilized to uncover potential issues and edge cases.
• A subset of system tests (and e2e tests), called smoke tests, must be run on the production environment to ensure that the most important features are working as expected and that the application is healthy.

4. Performance Testing

Load testing will be used to measure system reliability, performance.

5. Security Testing

TBD by @e12022492?

Test Environment and Infrastructure

• Test environments must mirror production environments as closely as possible.
• Virtualization and containerization technologies, such as Docker must be utilized for test environment management.
• Test environments must be version controlled and easily reproducible.

Testing Tools

• E2E: Python Playwright & Pytest + plugins
• Backend: Rust built-in tests by cargo
• Frontend: Vitest

Test Data Management

• Test data must be synthetic.
• Test data must cover a variety of scenarios, including at least valid and invalid cases.
• Equivalence partitioning and boundary value analysis tests should be done when it makes sense.

Test Execution and Reporting

• Test cases must be executed manually or automated, as appropriate for each testing type.
• All automated tests must be executed by CI.
• Integration tests must be executed by the CI pipeline on every push to PR and every merge to master.
• Test metrics, such as test coverage and test execution time, must be tracked in the pipeline.
• System tests and User acceptance tests must be performed before every release.
• Manual tests will be documented under doc/tests/manual/reports
• An Email is sent whenever master fails, with a small report containing information since last build.

Test Automation

• Unit and Integration tests must be fully automated.
• Security and Performance tests may be manual.
• System tests are automated (if possible) according to the e2e guidelines.

Other Considerations

• Tests should be run before committing via pre-commit hooks
• Tests should be run on different configurations (e.g. production, staging)