Infrastructure/BuildBot/Changelog

BuildBot Changelog

Features of the new BuildBot (2021)

BuildBot Changelog

  • Blender build pipeline CLI (command line interface) can fully run locally with no external CI/CD system.
  • MacOS - arm64 (M1/Apple Silicon) builds.
  • Documentation delivery
    • Blender Manual
    • Blender Code API
  • Delivery to digital store’s test channels/branches
    • Steam
    • Snap
    • Epic (backlog)
  • Various automated testing strategies such as:
    • Unit and smoke tests
    • Acceptance (wip), performance (wip), security (backlog), benchmark (backlog)
  • GPU kernel compile split into own step.
    • Saves time on incremental builds as this steps is skipped.
    • Tracking step times on just GPU compilation.
  • Allows for debug build configuration.
  • Package file names follow SemVer 2.0 specification
  • Simplify configuration and management of BuildBot workers.
  • Configurable build notification strategies. (backlog)
  • Coordinated triggerable builders.
  • Specific use cases that require authorization.
  • Mouse over build status.
  • Builds are purged from archive folder after 100 days.
  • Generate patch notes for LTS releases

CI Scripts Moved

All CI related scripts have been moved out of blender.git into their own repository.

Workers

  • Workers are now managed by Proxmox open-source platform for virtualization.
  • The supporting hardware uses HDD based drives that have slowed down the build times.
    • However, we have added many more virtual machines to compensate.
    • We will add new higher performance workers and dedicate them to important pipelines soon after Buildbot deployment. (planned)
  • Having updated to Cuda 11.4, we noticed a considerable amount of time building (2x to 3x) especially on Windows.
    • We are able to run the GPU compile in parallel on Linux reducing the time by 3x.
    • This is not possible on Windows. However, we can improve total build time by switching the pipeline to Ninja instead of MSBuild. (done)

The Buildbot production (PROD) environment has the following workers.

Worker Count Notes
Ubuntu 20.04 3 Manual, API build and store deliveries
Centos 7 3 Code builds only
MacOS Catalina - Intel 3 Code builds only
MacOS Big Sur - Arm 1 Code builds only
Windows Pro 10 4 Code builds only
Local coordinators 4 Coordinates builders
NOTE
Other workers are on standby and can be added to this environment in short amount of time.


Updates

Currently in UATEST

  • None

September 06, 2021

  • Added studio-sprite-fright branch to experimental nightly builds.
  • Added studio-sprite-fright and cycles-x to incremental builds.
  • Refactored Buildbot master configuration file into modules.
  • Added machine provisioning and build support for Ubuntu and PopOS 21.04 systems.
  • Rename Experimental to Branch menu item
  • Allow to list builds by branch or patch name

August 23, 2021

  • Upgrade all machines to Cuda 11.4.1
  • Check installed Cuda version and validate against configured value.
    • The configure-machine step will now fail the build if the worker does not have the correct version installed.
    • On Linux, the patch value is a date instead of the expected number and it installs in the same folder replacing the previous version.
  • Check installed Optix version and validate against configured value.
    • The configure-machine step will now fail the build if the worker does not have the correct version installed.

August 18, 2021

  • On MacOS, validate app bundle code sign.
  • On Windows, fix CUDA 11 path setting and log determined paths.

August 16, 2021

  • Use Hashicorp Vault for service credentials.

August 10, 2021

  • Windows: Workaround code sign issues with long paths.
  • Add filter check for Exception and Abort on unit tests.
  • Ensure cleanup of svn paths in case of prior interruption.

August 06, 2021

  • Fix missing packages for official release deployment.
  • Monitoring deployed build files now report missing files from mirrored site.
  • Add 1TB space for archived builds.
    • Archived builds will now be purged after 100 days, up from 25 days.
  • Add performance and acceptance test builders
    • The builders will appear once workers have been assigned to them.

July 30, 2021

  • Add option to skip tests for Experimental and Patch builders only.
  • Rename Steam branches as per feedback.
  • Switch build pipeline configuration file to yaml format.
    • The pipeline_config.json file will be removed shortly after this deployment - done
  • CentOS builds now uses the configured gcc version.
    • It was previously hardcoded to gcc 9 for all branches.
    • Added support for gcc 10 on Ubuntu/PopOS 20.04 LTS and Debian Testing.
  • Split smoke tests into separate GUI and CLI steps.
    • Only CLI smoke tests run for code builds.
  • Any Experimental and Patch builds are now archived after 7 days.
    • The builds are still available for another 18 days in the Archived section then purged from the system.
  • Step generate-code-patch-notes no longer fails when commit hash not found.
    • An error is logged instead.
  • Pull artifacts specific to platform and purpose.
    • Steam and Windows store builders only needs the zip file.
  • Upgrade software
    • Pipeline python version upgraded from 3.8 to 3.9
    • Install CUDA 11.4 on all machines.
    • Buildbot master/worker upgraded to 3.2.0

July 14, 2021

  • Optix SDK 7.3.0 available on all machines.
    • On Linux, the configure-machine step will install the requested package.
    • On Windows, they are pre-installed.

July 12, 2021

  • Clean ~/Download/Blender folder on clean-code step
    • Ran out of space on low sized VM's on Store build steps.
    • Freeing up to ~30 GB space
  • Fix parsing issues with LTS generate-code-patch-notes step
    • Will display as WARNING
    • No longer halts job on errors

July 01, 2021

  • On MacOS builders, skip compile-code-gpu step
    • Buildbot still runs the step, however the pipeline will internally not run any commands.
    • This saves ~1-2 minutes total build time on slow machines.
  • Halt on test-code-unit and test-code-smoke steps for tracked LTS versions
  • Added CLI smoke test
  • Standardized pipeline step summary details
  • Added all workers to monitoring system

June 24, 2021

  • On Windows builders, the package-code-binaries step skips msi packaging for vdev and vexp tracks.
    • This saves ~7-9 minutes total build time.
  • Added step generate-code-patch-notes for release tracks.
    • Step reads Phabricator:Maniphest release branch tracking tasks to generate notes for steam, wiki and html formats.
  • Archive Experimental and Patch builds older than 7 days
    • Builds are still accessible via the All Archived Builds link
    • The archived builds will be purged after 25 days.

June 18, 2021

  • Buildbot can now deploy Blender packages to final release locations.
    • Monitors all mirrors until all files have been replicated on all sites.
  • Moved package-code-store-windows step from code build pipelines to store related pipeline.
    • This saves ~10 min per build delivery for Windows builds.
    • It was not useful for Experimental or Patch builds.
  • Added nightly store delivery builds.
  • Removed blender- and -release words from package file names built from the release branches.

June 14, 2021

  • Switched to Ninja build system
    • Windows GPU kernel compile now runs in parallel
    • Mostly beneficial for VM's using HDD
    • 3x-4x faster GPU kernel builds on Windows
    • Ninja will force a rebuild for first time running on machine
  • More robust code signing
    • Added configurable timeservers to round-robin when needed
    • Added retry on command errors
  • Archiving will now consider previous patch versions.
  • Build retention now down to 25 days from 30 days.
  • Fixed packaging issue when using patch-code pipeline step on Windows.
    • Pipeline was removing the arc branch on the code-install step
    • We no longer remove the arc branch so that the package-code-binaries step using cpack on Windows can get the correct files.

May 27, 2021

  • Web interface changes
    • Normalized build type navigation and Buildbot links
    • Added sha256 hash file download link.
    • Added link to view all archived builds.
    • NOTE: Clearing browser cache might be required for new UI update.
  • Added branch based Buildbot overrides.
    • Hard coded values have been moved from pipeline code and into a json based configuration file.
  • Example overrides:
    • Svn artifacts and Git submodule branch/commit overrides
    • Tools and SDK Versions: gcc, cuda, optix
    • CMake Options
      • Some options are restricted.
      • Still needs further testing.
  • Added support for Experimental branch builds based on any release branch.
    • Allows rebuild of older branches
  • Added test fail error patterns.
NOTE
The use of new branch based configuration file can be phased in anytime.


May 20, 2021

  • Use vexp track for Experimental branch builds.
    • The pipeline will use a separate folder on the workers.
    • The vdev track is now dedicated to master branch
    • This should reduce the times the master branch fully rebuilds GPU kernels.
    • Added asset-browser-poselib to nightly builds
  • Added ability to override git submodule branches for experimental builds - T88432
  • Renamed the coordinator trigger buttons.
NOTE
Experimental coordinators can now be found in the vexp track.
Retrieved from "https://wiki.blender.org/w/index.php?title=Infrastructure/BuildBot/Changelog&oldid=27744"