Kaitai Struct project sports a relatively complex CI (Continuous Integration) pipeline. This document describes how it’s working.

High-level overview

Kaitai Struct CI strives to be highly modular and, by design, tries not to rely on a single CI engine provider. Currently, we use:

With so many different jobs/flows, it’s hard to rely on CI providers internal tools (such as job browsers, test browsers, log explorers, etc), so we’re using our own CI dashboard to unify all the data coming from different test runs/sources and monitor current status of the compiler.

The following (clickable) chart gives an overview of CI pipeline, tracing the path for one particular target language (Ruby):

Kaitai Struct CI high-level overview
Figure 1: Kaitai Struct CI high-level overview

On a high level, it can be summarized as:

  • We build the compiler: sources are in compiler repo, all dependencies are fetched automatically by build tooling (sbt).

  • This results in several packages, which get published as artifacts to several artifact stores (our "unstable builds"). Some of these unstable builds trigger other products' pipelines (such as Web IDE).

  • We run internal compiler unit tests (test_compiler).

  • We fetch test .ksy files from tests repo (~150 tests), and run compiler on them, asking for every possible target language.

  • This produces ~150 tests × ~15 targets ~ 1500 compiled format files in various target languages. We push them all into ci_targets repo at GitHub.

  • Once ci_targets is updated, this triggers lots of CI jobs in various environments in parallel. Every such job tests one particular target language in one particular environment. To identify them, we give every such job a name in form of language/environment. For example:

    • ruby/1.9 tests "ruby" target language in default (Linux) environment, using Ruby 1.9 (at Travis).

    • cpp_stl_11/msvc141_windows_x64 tests "cpp_stl_11" (which is actually a subvariant of "cpp_stl" target with compiler options set to generate exactly C11 code using STL library), building and running it in Windows environment with Microsoft Visual C compiler, toolkit version 141, x64 architecture. This job runs in AppVeyor CI.

  • Every test job eventually produces test results, which get pushed into ci_artifacts repo. Every job gets individual branch: for example, ruby/1.9 branch keeps results of ruby/1.9 test run.

  • Finally, these test results can be viewed online at our CI dashboard, which is a simple JavaScript app which fetches and aggregates all test results from all runs on the fly.

Main CI pipeline in details

Compiler

Compiler is the heart of everything in Kaitai Struct, so it’s only natural that we focus on making compiler maintenance as automated as possible.

  • Everything starts with compiler repo.

  • A new commit (or build request) for compiler repo triggers building of new compiler binaries. There are two builds running:

  • These build results are already useful (as unstable builds for other projects / people who would like to test bleeding edge features) and they get published as artifacts:

Note
 Direct links / instructions how to reach these downloads are available as "unstable" at Kaitai homepage / Download

  • After compiler builds are finished, we run the internal compiler unit tests (test_compiler). These are included inside compiler repo and are supposed to test individual functions & methods inside the compiler.

Building tests formats

After we’ve got the compiler, next steps is to take "test formats" (a large bunch of different input .ksy files) and run compiler on them. build_formats process does that. To do that, we’ll need:

This results in many files in target languages generated in compiled/$LANG directories. That directory gets pushed into ci_targets repo. One can use version control history in that repo to track which formats code generation has changed over development iterations.

Running test formats

The exact mechanism of "building and running tests" largely depends on target language and environment, but there are a few common things:

Other CI pipelines

Web IDE

TODO

Formats gallery

Formats gallery is a static website, which provides user-friendly rendition of contents of our formats repo.

Its pipeline is very simple and consists of only one job, running on GitHub Actions:

Documentation

TODO