CI/CD Pipeline

Hotline

Our Team

Emergency service Jenkins+Infrastructure:

• Mondays: Hannes Zettl
• Tuesdays Morning: Mo Moritz Adelsberger
• Tuesdays Evening: Lukas Oberhauser
• Wednesdays: Patrick Graßl
• Thursdays Morning: Lukas Oberhauser
• Thursdays Evening: Patrick Graßl
• Fridays: Hannes Zettl
• Weekends: Mo Moritz Adelsberger

"Wartungsfenster" fixed times where upgrades are done, may cause downtime

• Mondays or Fridays as early as possible (e.g. 7:00) until about 10:00 or longer up to 10:30 on troubles: when finished it will be written in Nextcloud

GitLab

GitLab has their "Wartungsfenster" always on the first Wednesday of the month. There may be "Internal Error", "Service Unavailable", etc. Otherwise, in case of errors, email to:

• To: help@it.tuwien.ac.at, gitlab-admin@it.tuwien.ac.at
• CC: markus.raab@tuwien.ac.at

Triggers

The pipeline will be executed only for pushes to Pull Requests and for pushes to master+dev.

If the pipeline fails and you think it is not the fault of your code, you can re-execute the pipeline by typing jenkins build please as a comment. In any case (for person in charge see "emergency dienst" above):

• If there is already an open issue about it, ping the person in charge daily in this issue by posting the link to the failed build with @mention.
• Otherwise, please create a issue with the failing build log and assign the person in charge.

For users with login credentials for Jenkins, you can manually execute the pipeline for a branch or merge request via the Jenkins UI. Please ask if you want/need login data.

Succeeding Pushes

If you push to branches:

• Only the newest commit will be built. It does not matter if commits in between would pass.
• Builds of older pushes are aborted.
• Also jenkins build please will abort previous builds and start the build again.
• Jenkins writes comments if MR pipelines were successful / failed.
• The latest MR build job will stay deployed on (https://mr.permaplant.net).
• The deployed branch and commit version can be seen on (https://mr.permaplant.net/version).
• The master and dev branch build when new commits get pushed to them.

Stages

Sanity stage

This stage performs rapid checks like pre-commit, migrations and schema building.

Local pre-commit currently does not perform the codespell hook, but it is performed in the pipeline. Cargo fmt, eslint and groovy linting is not performed through pre-commit.

Tests and Build

This is a parallel stage which fails fast (exits if one stage fails) or times out after 2 hours.

It can be subdivided in following categories:

Multiple Parallel Cargo Stages

Standard cargo build, clippy, doc and test is performed here.

One Sequential Frontend Stage

Frontend is built, tested, linted, format checked and additionally package.json version is verified to be up to date. Storybook and typedocs are also generated here.

One Sequential Mdbook Stage

Mdbook is built, tested and links are checked.

Deploy MR

Every merge request will be deployed on a publicly available instance on mr.permaplant.net. Jenkins will acquires a lock before deploying to mr.permaplant.net and release it after finishing the E2E stage. This lock prevents other jobs to overwrite the deployment while we are performing E2E tests.

Since there is only one agent for MRs available, the last built MR wins.

E2E Tests

E2E tests are run on mr.permaplant.net. When a test times out, it is retried up to two times with a five second delay between retries.

Test reports and results can be found in the jobs artifacts.

Failed tests will generate videos and/or screenshots inside e2e/test-results/ in the jobs artifacts, depending on what failed. The videos have to be downloaded to be viewed.

There is always a html report inside e2e/test-reports/ in the jobs artifacts and a cucumber report on a separate page.

Deploy Dev

The dev branch will be automatically deployed to dev.permaplant.net.

Deploy master

The master branch will be automatically deployed to master.permaplant.net.

Deploy Prod

There is a separate Jenkinsfile (/ci/Jenkinsfile.release) for this pipeline, which is only manually executable through Jenkins, it:

• uses the master branch (our release branch),
• runs only build and deploying stages (currently no testing), and
• deploys to www.permaplant.net.

See /.gitlab/issue_templates/Release.md for additional manual steps during the release.

Disk Space Management

Jenkins stores different types of data in different places.

A workspace is a directory on a build agent where Jenkins checks out source code and executes pipeline steps. Workspaces are temporary execution directories and can usually be cleaned after a build has finished.

A build lives on the Jenkins controller and contains the build record, including metadata, console output, test reports and archived artifacts such as e2e test videos. The workspace and build concepts are relevant for all Jenkins jobs and pipelines.

The Orphaned Item Strategy is, in our case, only relevant for Multibranch Pipelines. It defines how long old branch and MR jobs should be kept after they are no longer discovered by Jenkins.

Workspace Cleanup

The Jenkins Workspace Cleanup Plugin is available to help manage disk space by automatically cleaning the active build workspace.

This prevents workspace accumulation on build agents. However, workspace cleanup only affects workspace directories. It does not delete build history, console logs, archived artifacts or old Multibranch Pipeline jobs.

Build discarder

The build discarder works on a per-job basis and defines build and artifact retention.

Artifacts are part of a build. Therefore, build retention takes precedence over artifact retention: once a build is discarded, its artifacts are discarded as well.

properties([
    buildDiscarder(logRotator(
        numToKeepStr: '',           // max number of builds to keep
        daysToKeepStr: '14',        // max age in days builds are kept
        artifactNumToKeepStr: '',   // max number of artifacts to keep
        artifactDaysToKeepStr: '14' // max age in days artifacts are kept
    ))
])

Orphaned Item Strategy

The Orphaned Item Strategy is a configuration available for computed folders, such as Multibranch Pipelines.

In a Multibranch Pipeline, Jenkins scans the source control system and automatically creates jobs for discovered branches and merge requests. When a branch or merge request is no longer discovered e.g. because an MR was merged or closed, the corresponding Jenkins job becomes orphaned.

The Orphaned Item Strategy defines how long these orphaned branch or MR jobs should be kept before Jenkins deletes them. It does not control how many builds are kept inside an active branch or MR job. That is handled by the build discarder.

Summary

Node Labels

Nodes are labeled for pipeline node selection.

Labels which are not in use anymore

Pipeline Development Aids

Jenkins provides built-in documentation that is very helpful when reading, writing or modifying pipeline code.

https://build.perma.health/pipeline-syntax/

This page contains the Snippet Generator, which can generate Pipeline syntax for available Jenkins steps based on the plugins installed on the controller. This is especially useful because the available steps and parameters depend on the actual Jenkins installation.

Additional helpful documentation is available from the same area, for example:

• the Steps Reference, which documents available pipeline steps
• the Global Variable Reference, which documents available global variables such as env, params and currentBuild
• the Declarative Directive Generator, which helps generate Declarative Pipeline blocks such as agent, stage and post

These pages are useful both for understanding existing Jenkinsfiles and for writing new pipeline code, because they reflect the features and plugins available on the Jenkins controller.