KS project consists of several subprojects, each of which has its own repository:

  • compiler — everything starts here, this is the core of the project that everything uses

  • tests — used to prove that compiler behaves as expected and there are no regressions introduced by the changes

  • runtimes, one per each target language — used by the code generated by a compiler

  • documentation — keeps users informed about the language and all new features

  • console visualizer and tools: ksv, ksdump

Technically, one can download and work with each of these subprojects individually, but most people just download everything by checking out our main umbrella project that includes (almost) everything:

git clone --recursive https://github.com/kaitai-io/kaitai_struct.git

Note the --recursive option, which would download all the submodules. If you forgot it and cloned without it, you can still do it manually after cloning:

git submodule update --init --recursive

Compiler

KS compiler is written in Scala language and thus uses SBT for building. It can be compiled in one of 2 ways, each of which can be packaged in one of several ways:

  • generating Java .class files, to be run in JVM

    • not packaged, ready to be ran from a build dir (so called "stage build")

    • packaged as universal (.zip) package

    • packaged as Debian (.deb) package

    • packaged as Windows (.msi) installer package

  • generating JavaScript .js file, to be run either inside a browser or in node.js environment

    • not packaged, used as a source JS file on a website

    • packaged as npm package

Prerequisites

To build the compiler, one generally needs just these two things:

  • Java Runtime Environment (JRE)

  • Scala Build Tool (sbt)

For Debian/Ubuntu, this should be enough to download every prerequisite (JRE is pulled automatically by sbt):

echo "deb https://repo.scala-sbt.org/scalasbt/debian all main" | sudo tee /etc/apt/sources.list.d/sbt.list
echo "deb https://repo.scala-sbt.org/scalasbt/debian /" | sudo tee /etc/apt/sources.list.d/sbt_old.list
curl -sL "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823" | sudo apt-key add
sudo apt-get update
sudo apt-get install sbt

For RedHat-based systems, use the following:

curl -L https://www.scala-sbt.org/sbt-rpm.repo > sbt-rpm.repo
sudo mv sbt-rpm.repo /etc/yum.repos.d/
sudo yum install sbt

Manual installation:

Note
 You don’t need whole Java Development Kit (JDK), unless you want to test & develop for Java target — just JRE is enough for KS compiler.

All other dependencies (including Scala runtimes, compilers, etc) required for the compiler are downloaded and installed by sbt automatically. It shouldn’t really matter even which version of sbt you use for bootstrap, as sbt will pull relevant sbt update packages automatically as well.

Configuring proxy

Note, that if you behind proxy, you need to run sbt with flags

  -Dhttp.proxyHost=<your proxy server>
  -Dhttp.proxyPort=<your proxy server port>
  -Dhttps.proxyHost=<your proxy server>
  -Dhttps.proxyPort=<your proxy server port>

For example

  sbt -Dhttp.proxyHost=proxy.com -Dhttp.proxyPort=3128 -Dhttps.proxyHost=proxy.com -Dhttps.proxyPort=3128

Unfortunately, sbt doesn’t understand http(s)_proxy environment variable properly, if it contains address of proxy server in host:port format, so flags is necessary.

This flags needed only on first run or when you want to check and upgrade dependencies. After first sbt run all dependencies will be downloaded and access to the internet not required anymore.

Building for JVM

We use sbt-native-packager to build deployable formats. Note that in order to perform the following builds, one must be in the kaitai_struct/compiler directory.

Building an universal (.zip) package

SBT native packager: universal package docs

  1. sbt compilerJVM/universal:packageBin

  2. Get result in jvm/target/universal/kaitai-struct-compiler-*.zip

Building Debian package

  1. Install prerequisites:

    • On Debian/Ubuntu host: sudo -i apt-get install dpkg dpkg-sig dpkg-dev lintian fakeroot

    • On RedHat host: sudo dnf install dpkg fakeroot

  2. sbt compilerJVM/debian:packageBin

  3. Get result in jvm/target/kaitai-struct-compiler_*_all.deb

Building RedHat package

  1. Install prerequisites:

    • On RedHat host: sudo dnf install rpm rpm-build

  2. sbt compilerJVM/rpm:packageBin

  3. Get result in jvm/target/rpm/RPMS/noarch/kaitai-struct-compiler-*.noarch.rpm

Building Windows package

SBT native packager: Windows package docs

  1. Install WIX

  2. sbt compilerJVM/windows:packageBin

  3. Get result in jvm/target/windows/kaitai-struct-compiler.msi

  4. Rename to add version to kaitai-struct-compiler-$VERSION.msi

Building for JavaScript platform

Building to JavaScript platform is done using a Scala.js project. Note that it uses a somewhat different set of dependencies, as they must actually be JavaScript libraries, not Java jars.

  1. Run sbt fastOptJS

  2. Get result in js/target/scala-2.11/kaitai-struct-compiler-fastopt.js

  3. Use this JavaScript file on a website

Publishing a new version

  1. Choose a new version number (WIX imposes harsh requirements for version to look like x.x.x.x) and update it in build.sbt, version := …​, commit

  2. Prepare an entry in RELEASE_NOTES.md, commit

  3. Create version tag:

    • git tag $VERSION

    • git push --tags

  4. Update main repository

  5. Create new version at:

  6. Upload:

  7. Publish them all

Publishing new version to oss.sonatype.org

  1. Verify that one has OSS Sonatype login/password for iokaitai org.

  2. Preliminary setup (needs to be done once per machine — verified for sbt 1.1)

    • Enable the sbt-pgp plugin globally by adding this line to ~/.sbt/1.0/plugins/gpg.sbt:

      addSbtPlugin("com.github.sbt" % "sbt-pgp" % "2.1.2")

    • Set up credentials: create $HOME/.sbt/.credentials with the following contents (replacing XXX with username and password):

      realm=Sonatype Nexus Repository Manager
host=oss.sonatype.org
user=XXX
password=XXX

    • Set up $HOME/.sbt/1.0/plugins/credentials.sbt with the following contents:

      credentials += Credentials(Path.userHome / ".sbt" / ".credentials")

    • For publishing the Java runtime library: set up ~/.m2/settings.xml with the following contents:

      <settings>
  <servers>
    <server>
      <id>ossrh</id>
      <username>XXX</username>
      <password>XXX</password>
    </server>
  </servers>
</settings>

    • Make sure GPG keys are present

  3. sbt publishSigned

  4. Go to https://oss.sonatype.org/#stagingRepositories

  5. Continue to follow Java runtime publishing instructions

Tests

TODO

Publishing runtime libraries

Java

JavaScript

  • Pump version in package.json and package-lock.json (search for "version": ) to X.Y.Z format (e.g. 0.9.0 for 0.9 release)

  • Make a commit with the changes and tag it $VERSION

  • npm login - fill npmjs.org login credentials

  • npm publish --tag latest

Python

  • Install/update the needed Python packaging tools: python3 -m pip install --upgrade setuptools twine wheel

    • If this gives you permission errors, don’t use sudo to run pip. Instead use your system package manager (e. g. apt) to install/update the tools, or add the --user option to pip install.

  • Pump version in kaitaistruct.py, seek __version__ =

  • Delete the dist directory (if it exists), so that built files from previous or development versions don’t get uploaded by accident.

  • python3 setup.py sdist bdist_wheel

  • twine check dist/*

  • twine upload dist/*

    • This will prompt for your PyPI credentials. See the Twine docs for info on how to preset/permanently configure the credentials.

    • To upload to TestPyPI instead: twine upload --repository=testpypi dist/*

  • Check that new version appears under https://pypi.org/project/kaitaistruct/#history

  • git tag $VERSION

  • git push origin $VERSION

Ruby

  • Pump version in lib/kaitai/struct/struct.rb, seek `VERSION = `

  • gem build kaitai-struct.gemspec

  • Test gem (i.e. by installing it to a live system)

  • gem push kaitai-struct-$VERSION.gem

  • git tag $VERSION

  • git push --tags

C#

  • Pump versions in kaitai_struct_runtime_csharp.csproj, commit and put $VERSION tag on it

  • dotnet pack -o . → get resulting .nupkg and .snupkg files in the current directory

  • Create an API key on https://www.nuget.org/account/apikeys

  • dotnet nuget push *.nupkg -k <API key> -s https://api.nuget.org/v3/index.json

Adding new language

Overall routine for adding new language is described in Adding support for new target language.

After addition, don’t forget to update lists of languages: