Kaitai Struct User Guide

If you’re going to try Kaitai Struct for the first time, the Web IDE is probably the easiest way to get started. Just open Kaitai Struct Web IDE and you’re ready to go:

A list of Web IDE features is available on the kaitai_struct_webide GitHub wiki.

Note that there are two different versions of the Web IDE:

If you want to use the latest features, use the devel Web IDE.

If you don’t fancy using a hex dump in a browser, or want to integrate Kaitai Struct into your project build process automation, you’d want a desktop / console solution. Of course, Kaitai Struct offers that as well.

Probably the simplest thing Kaitai Struct can do is reading fixed-size structures. You might know them as C struct definitions — consider something like this fictional database entry that keeps track of dog show participants:

And here is how it would look in .ksy:

It’s a YAML-based format, plain and simple. Every .ksy file is a type description. Everything starts with a meta section: this is where we specify top-level info on the whole structure we describe. There are two important things here:

With that out of the way, we use a seq element with an array (ordered sequence of elements) in it to describe which attributes this structure consists of. Every attribute includes several keys, namely:

The YAML-based syntax might look a little more verbose than C-like structs, but there are a few good reasons to use it. It is consistent, it is easily extendable, and it’s easy to parse, so it’s easy to make your own programs/scripts that work with .ksy specs.

A very simple example is that we can add docstrings to every attribute, using syntax like that:

These docstrings are not just the comments in the .ksy file, they’ll actually get exported into the target language as well (for example, in Java they’ll become JavaDoc, in Ruby they’ll become RDoc/YARD, etc). This, in turn, is super helpful when editing the code in various IDEs that will generate reminder popups for intelligent completion, when you browse through class attributes:

Many file formats use some sort of safeguard measure against using a completely different file type in place of the required file type. The simple way to do so is to include some "magic" bytes (AKA "file signature"): for example, checking that the first bytes of the file are equal to their intended values provides at least some degree of protection against such blunders.

To specify "magic" bytes (i.e. fixed content) in structures, Kaitai Struct includes a special contents key. For example, this is the beginning of a seq for Java .class files:

This reads the first 4 bytes and compares them to the 4 bytes CA FE BA BE. If there is any mismatch (or less than 4 bytes are read), it throws an exception and stops parsing at an early stage, before any damage (pointless allocation of huge structures, waste of CPU cycles) is done.

Note that contents is very flexible and you can specify:

In the case of using an array, all elements' byte representations would be concatenated and expected in sequence. Some examples:

More extreme examples to illustrate the idea (i.e. possible, but definitely not recommended in real-life specs):

To ensure attributes in your data structures adhere to expected formats and ranges, Kaitai Struct provides a mechanism for validating attribute values using the valid key. This key allows you to define constraints for values, enhancing the robustness of your specifications. Here’s how you can enforce these constraints:

For most cases, the direct valid: value shortcut is preferred for its simplicity, effectively functioning as valid/eq.

When a value does not meet the specified criteria, Kaitai Struct raises a validation error, halting further parsing. This preemptive measure ensures the data being parsed is within the expected domain, providing a first layer of error handling.

Many protocols and file formats tend to conserve bytes, especially for strings. Sure, it’s stupid to have a fixed 512-byte buffer for a string that typically is 3-5 bytes long and only rarely can be up to 512 bytes.

One of the most common methods used to mitigate this problem is to use some integer to designate length of the string, and store only designated number of bytes in the stream. Unfortunately, this yields a variable-length structure, and it’s impossible to describe such a thing using C-style structs. However, it’s not a problem for Kaitai Struct:

Note the size field: we use not a constant, but a reference to a field that we’ve just parsed from a stream. Actually, you can do much more than that — you can use a full-blown expression language in size field. For example, what if we’re dealing with UTF-16 strings and my_len value designates not a number of bytes, but number of byte pairs?

One can just multiply my_len by 2 — and voila — here’s our UTF-16 string. Expression language is very powerful, we’ll be talking more about it later.

Last, but not least, we can specify a size that spans automatically to the end of the stream. For that one, we’ll use a slightly different syntax:

Another popular way to avoid allocating huge fixed-size buffers is to use some sort of trailing delimiter. The most well-known example of this is probably the null-terminated string which became a standard string representation in C:

These 4 bytes actually represent the 3-character string "abc", plus one extra trailing byte "0" (AKA null) which serves as a delimiter or terminator. By agreement, C strings cannot include a zero byte: every time a function in C sees that either in stream or in memory, it considers that as a special mark to stop processing.

In Kaitai Struct, you can define all sorts of delimited structures. For example, this is how you define a null-terminated string:

As this is a very common thing, there’s a shortcut for type: str and terminator: 0. One can write this as:

Of course, you can use any other byte (for example, 0xa, AKA newline) as a terminator. This gives Kaitai Struct some limited capabilities to parse certain text formats as well.

Reading "until the terminator byte is encountered" could be dangerous. What if we never encounter that byte?

Another very widespread model is actually having both a fixed-sized buffer for a string and a terminator. This is typically an artifact of serializing structures like this from C. For example, take this structure:

and do the following in C:

After the first strcpy operation, the buffer will look like:

And after the second strcpy, the following will remain in the memory:

Effectively, the buffer is still 16 bytes, but the only meaningful contents it has is up to first null terminator. Everything beyond that is garbage left over from either the buffer not being initialized at all (these ?? bytes could contain anything), or it will contain parts of strings previously occupying this buffer.

It’s easy to model that kind of behavior in Kaitai Struct as well, just by combining size and terminator:

This works in 2 steps:

The nature of binary format encoding dictates that in many cases we’ll be using some kind of integer constants to encode certain entities. For example, an IP packet uses a 1-byte integer to encode the protocol type for the payload: 6 would mean "TCP" (which gives us TCP/IP), 17 would mean "UDP" (which yields UDP/IP), and 1 means "ICMP".

It is possible to live with just raw integers, but most programming languages actually provide a way to program using meaningful string names instead. This approach is usually dubbed "enums" and it’s totally possible to generate an enum in Kaitai Struct:

There are two things that should be done to declare a enum:

What do we do if we need to use many of the strings in such a format? Writing so many repetitive my_len- / my_str-style pairs would be so bothersome and error-prone. Fear not, we can define another type, defining it in the same file, and use it as a custom type in a stream:

Here we define another type named str_with_len, which we reference just by doing type: str_with_len. The type itself is defined using the types: key at the top level. That’s a map, and inside it we can define as many subtypes as we want. We define just one, and inside it we nest the exact same syntax as we use for the type description on the top level — i.e. the same seq designation.

Of course, one can actually have more levels of subtypes:

TODO

Expression language (used, for example, in a size key) allows you to refer not only to attributes in the current type, but also in other types. Consider this example:

If the body_len attribute was in the same type as body, we could just use size: body_len. However, in this case we’ve decided to split the main header into a separate subtype, so we’ll have to access it using the . operator — i.e. size: header.body_len.

One can chain attributes with . to dig deeper into type hierarchy — e.g. size: header.subheader_1.subsubheader_1_2.field_4. But sometimes we need just the opposite: how do we access upper-level elements from lower-level types? Kaitai Struct provides two options here:

Some protocols and file formats have optional fields, which only exist in some conditions. For example, one can have some byte first that designates if some other field exists (1) or not (0). In Kaitai Struct, you can do that using the if key:

In this example, we again use expression language to specify a boolean expression in the if key. If that expression is true, the field is parsed and we’ll get a result. If that expression is false, the field will be skipped and we’ll get a null (or its closest equivalent in our target programming language) if we try to get it.

At this point, you might wonder how that plays together with enums. After you mark some integer as "enum", it’s no longer just an integer, so you can’t compare it directly with the number. Instead you’re expected to compare it to other enum values:

There are other enum operations available, which we’ll cover in the expression language guide later.

Most real-life file formats do not contain only one copy of some element, but might contain several copies, i.e. they repeat the same pattern over and over. Repetition might be:

Kaitai Struct supports all these types of repetitions. In all cases, it will create a resizable array (or nearest equivalent available in the target language) and populate it with elements.

"TLV" stands for "type-length-value", and it’s a very common staple in many formats. The basic idea is that we use a modular and reverse-compatible format. On the top level, it’s very simple: we know that the whole format is just an array of records (repeat: eos or repeat: expr). Each record starts the same: there is some marker that specifies the type of the record and an integer that specifies the record’s length. After that, the record’s body follows, and the body format depends on the type marker. One can easily specify that basic record outline in Kaitai Struct like that:

However, how do we specify the format for body that depends on rec_type? One of the approaches is using conditionals, as we’ve seen before:

However, it’s easy to see why it’s not a very good solution:

That is why Kaitai Struct offers an alternative solution. We can use a switch type operation:

This is much more concise and easier to maintain, isn’t it? And note that size is specified on the attribute level, thus it applies to all possible type values, setting us a good hard limit. What’s even better — even if you’re missing the match, as long as you have size specified, you would still parse body of a given size, but instead of interpreting it with some user type, it will be treated as having no type, thus yielding a raw byte array. This is super useful, as it allows you to work on TLV-like formats step-by-step, starting by supporting only 1 or 2 types of records, and gradually adding more and more types.

You can use "_" for the default (else) case which will match every other value which was not listed explicitly.

Switching types can be a very useful technique. For more advanced usage examples, see Advanced switching.

So far we’ve done all the data specifications in seq — thus they’ll get parsed immediately from the beginning of the stream, one-by-one, in strict sequence. But what if the data you want is located at some other position in the file, or comes not in sequence?

"Instances" are Kaitai Struct’s answer for that. They’re specified in a key instances on the same level as seq. Consider this example:

Inside instances we need to create a map: keys in that map are attribute names, and values specify attribute in the very same manner as we would have done it in seq, but there is one important additional feature: using pos: …​ one can specify a position to start parsing that attribute from (in bytes from the beginning of the stream). Just as in size, one may use expression language and reference other attributes in pos. This is used very often to allow accessing a file body inside a container file when we have some file index data: file position in container and length:

Another very important difference between the seq attribute and the instances attribute is that instances are lazy by default. What does that mean? Unless someone would call that body getter method programmatically, no actual parsing of body would be done. This is super useful for parsing larger files, such as images of filesystems. It is impractical for a filesystem user to load all the filesystem data into memory at once: one usually finds a file by its name (traversing a file index somehow), and then can access file’s body right away. If that’s the first time this file is being accessed, body will be loaded (and parsed) into RAM. Second and all subsequent times will just return a cached copy from the RAM, avoiding any unnecessary re-loading / re-parsing, thus conserving both RAM and CPU time.

Note that from the programming point of view (from the target programming languages and from internal Kaitai Struct’s expression language), seq attributes and instances are exactly the same.

Sometimes, it is useful to transform the data (using expression language) and store it as a named value. There’s another sort of instances for that — value instances. They’re very simple to use, there’s only one key in it — value — that specifies an expression to calculate:

Value instances do no actual parsing, and thus do not require a pos key or a type key (the type will be derived automatically). If you need to enforce the type of the expression, see typecasting.

Quite a few protocols and file formats, especially those which aim to conserve space, pack multiple integers into one byte, using integer sizes less than 8 bits. For example, an IPv4 packet starts with a byte that packs both a version number and header length:

It’s possible to unpack bit-packed integers using old-school methods with bitwise operations in value instances:

However, Kaitai Struct offers a better way to do it — using bit-sized integers.

We introduced the doc key early in this user guide as a simple way to add docstrings to the attributes. However, it’s not only attributes that can be documented. The same doc key can be used in several different contexts:

The meta key is used to define a section which stores meta-information about a given type, i.e. various complimentary stuff, such as titles, descriptions, pointers to external linked resources, etc:

Quite often binary formats use the following technique:

Consider this example:

Of course, one can derive this manually:

Thus:

But this is very inconvenient and potentially error prone. What will happen if at some time in future the record contents are updated and we forget to update this formula?

It turns out that substreams offer a much cleaner solution here. Let’s separate our "header" and "body" into two distinct user types, and then we can just specify size on this body:

For comment, we just made its size to be up until the end of stream. Given that we’ve limited it to the substream in the first place, this is exactly what we wanted.

The same technique might be useful for repetitions as well. If you have an array of same-type entries, and a format declares the total size of all entries combined, again, you can try to do this:

And do some derivations to calculate number of entries, i.e. "total_len / sizeof(entry)". But, again, this is bad because:

Solving it using substreams is much more elegant. You just create a substream limited to total_len bytes, and then use repeat: eos to repeat until the end of that stream.

Another useful feature that’s possible with substreams is the fact that while you’re in a substream, the pos key works in the context of that substream as well. That means it addresses data relative to the start of that substream:

In this example, body allocates a substream spanning from 20th byte (inclusive) till 100th byte (exclusive). Then, in that stream:

This comes super handy if your format’s internal structures somehow specify offsets relative to some other structures of the format. For example, a typical filesystem/database often uses a concept of blocks, and offsets that address stuff inside the current block. Note how KSY with substreams is easier to read, more concise and less error-prone:

The more levels of structure offset nesting there are, the more complicated these pos expressions would get without substreams.

If you ever need to "escape" the limitations of a substream by wishing to use a pos key of a parse instance to address something absolutely (i.e. in the main stream), it’s easy to do so by adding an io key to choose the root’s stream:

That’s the typical situation encountered in many file container formats. Here we have a list of files, and each of its entries has been limited to exactly 80 bytes. Inside each 80-byte chunk, there’s a file_name, and, more importantly, a pointer to the absolute location of the file’s body inside the stream. The body instance allows us to get that file’s body contents quickly and easily. Note that if there was no io: _root._io key there, that body would have been parsed inside a 80-byte substream (and most likely that would result in an exception trying to read outside of the 80 byte limit), and that’s not what we want here.

The technique above is not limited to just the root object’s stream. You can address any other object’s stream as well, for example:

Some formats obscure the data fully or partially with techniques like compression, obfuscation or encryption. In these cases, incoming data should be pre-processed before actual parsing takes place, or we’ll just end up with garbage getting parsed. All such pre-processing algorithms have one thing in common: they’re done by some function that takes a stream of bytes and returns another stream of bytes (note that the number of incoming and resulting bytes might be different, especially in the case of decompression). While it might be possible to do such transformation in a declarative manner, it is usually impractical to do so.

Kaitai Struct allows you to plug-in some predefined "processing" algorithms to do decompression, de-obfuscation and decryption to get a clear stream, ready to be parsed. Consider parsing a file, in which the main body is obfuscated by applying XOR with 0xaa for every byte:

Note that:

Expression language operates on the following primitive data types:

Integers come from uX, sX, bX type specifications in sequence or instance attributes (i.e. u1, u4le, s8, b3, etc), or can be specified literally. One can use:

It’s possible to use _ as a visual separator in literals — it will be completely ignored by the parser. This could be useful, for example, to:

Floating point numbers also follow the normal notation used in the vast majority of languages: 123.456 will work, as well as exponential notation: 123.456e-55. Use 123.0 to enforce floating point type to an otherwise integer literal.

Booleans can be specified as literal true and false values as in most languages, but also can be derived by using type: b1. This method parses a single bit from a stream and represents it as a boolean value: 0 becomes false, 1 becomes true. This is very useful to parse flag bitfields, as you can omit flag_foo != 0 syntax and just use something more concise, such as is_foo.

Byte arrays are defined in the attribute syntax when you don’t specify anything as type. The size of a byte array is thus determined using size, size-eos or terminator, one of which is mandatory in this case. Byte array literals use typical array syntax like the one used in Python, Ruby and JavaScript: i.e. [1, 2, 3]. There is a little catch here: the same syntax is used for "true" arrays of objects (see below), so if you try to do stuff like [1, 1000, 5] (1000 obviously won’t fit in a byte), you won’t get a byte array, you’ll get an array of integers instead.

Strings normally come from using type: str (or type: strz, which is a shortcut that implicitly adds terminator: 0). Literal strings can be specified using double quotes or single quotes. The meaning of single and double quotes is similar to those of Ruby, PHP and Shell script:

TODO: Enums

Streams are internal objects that track the byte stream that we parse and the state of parsing (i.e. where the pointer is). There is no way to declare a stream-type attribute directly by parsing instructions or specify it as a literal. The typical way to get stream objects is to query the _io attribute from a user-defined object: that will give us a stream associated with this particular object.

There are two composite data types in the expression language (i.e. data types which include other types as components).

Literals can be connected using operators to make meaningful expressions. Operators are type-dependent: for example, the + operator applied to two integers would mean arithmetic addition, while the same operator applied to two strings would mean string concatenation.

Just about every value in expression language is an object (including literals), and it’s possible to call methods on it. The common syntax to use is obj.method(param1, param2, …​), which can be abbreviated to obj.method if no parameters are required.

Note that when the obj in question is a user-defined type, you can access all its attributes (both sequence and instances) using the same obj.attr_name syntax. One can chain that to traverse a chain of substructures: obj.foo.bar.baz (given that obj is a user-defined type that has a foo field, which points to a user-defined type that has a bar field, and so on).

There are a few pre-defined methods that form a kind of "standard library" for expression language.

In some rare cases, you need a type that actually does absolutely nothing. For example, you purposely want to ignore parsing a certain switch case and avoid running it through the default type, e.g. a situation like this:

This is very easy to achieve. Here are a few examples of type definitions which do nothing when invoked:

Delimited structures, having terminator specified to define a structure of arbitrary size, are pretty common and useful. However, sometimes you’re dealing with more advanced versions of these which require you to fine-tune certain aspects of delimiting.

As your project grows in complexity, you might want to have multiple .ksy files: for example, for different file formats, structures, substructures, or to reuse the same subformat in several places. As in most programming languages, Kaitai Struct allows you to have multiple source files and has imports functionality for that.

Using multiple files is very easy. For example, if you have a date.ksy file that describes the date structure:

and you want to use it in a file listing specification filelist.ksy, here’s how to do that:

Generally, you just add an array in meta/imports and list all you want to import there. There are 2 ways to address the files:

Sometimes you want Kaitai Struct-generated code to call code in your application to do the parsing, for example, to parse some text- or state-based format. For that, you can instruct ksc to generate code with so-called "opaque" types.

Normally, if the compiler encounters a type which is not declared either in the current file or in one of the imported files, for example:

... it will output an error:

If we want to provide our own implementation of a custom_encrypted_object type, first we need to compile our .ksy file with the --opaque-types=true option. This will avoid the error, and the compiler will consider all unknown types to be "opaque", i.e. it will treat them as existing in some external space.

Alternatively, instead of specifying the command line argument --opaque-types=true to the compiler, as of Kaitai Struct version 0.7, it is now possible to specify meta field ks-opaque-types as follows:

This will generate the following code (for example, in Java):

As you see, CustomEncryptedObject is instantiated here with a single argument: an IO stream. All that’s left is to create a class with a compatible constructor that will allow a call with a single argument. For statically typed languages, note that the constructor’s argument is of type KaitaiStream.

An example of what can be done (in Java):

As discussed in Processing: dealing with compressed, obfuscated and encrypted data, Kaitai Struct utilizes the process key to invoke processing of the data for the purposes of "bytes in - bytes out" transformations. It is meant to be used to implement compression & decompression, encryption & decryption, obfuscation & de-obfuscation, those kind of transformations.

Kaitai Struct runtime libraries come bundled with a "standard" set of such transformations, but quite often one encounters the need to implement some custom data transformation algorithm. There are many thousands of encryption and compression algorithms. It’s impractical to both try to implement them in declarative form using standard Kaitai Struct types (because as an end-user, you’re most likely interested in the decoded result, not internal structures of the algorithm/cipher), and it’s next to impossible to bundle all the data processing algorithms in the world into Kaitai Struct runtime (not only it would become very bloated, but also quite a few such algorithms are encumbered by software patents and licensing restrictions).

To alleviate this problem, Kaitai Struct allows one to invoke custom processing algorithms, implemented in imperative code in target languages. This works very similar to opaque external types, but for process invocations, not for type invocations.

Calling a custom process type is easy:

This would generate something like this (example is for Java, other target languages a use similar technique):

A typical implementation of a custom processor would look like this (again, example in Java; refer to language-specific notes for documentation on other languages):

This example is mostly self-explanatory. Strongly typed languages, such as Java, usually provide some sort of interface that such a custom processor class should implement. For Java, it’s named CustomDecoder. And, as outlined there, we implement:

By default, specifying a plain name invokes a custom processing class in the same namespace/package where the code is generated. If you want to keep your generated code in a separate namespace/package than your custom hand-made code, you can specify it like this:

For Java, this would result in an invocation of com.example.MyRle class. Other languages use similar rules of translation; see language-specific notes for details.

A special namespace prefix "kaitai." is reserved for extended libraries provided by the Kaitai project. As of 0.8, none of these have been published, but in future you can expect implementations like "kaitai.crypto.aes" or "kaitai.compress.lzma" to be provided by libraries implemented in multiple languages that would be released along with the minimal core Kaitai Struct runtime.

Every object (except for the top-level object) in a .ksy file has a parent, and that parent has a type, which is some sort of user-defined type. What happens if two or more objects use the same type?

In this example, both opcodes use same type arg. Given that these are different types, Kaitai Struct infers that the only thing they have in common is that they are objects generated by Kaitai Struct, and thus they usually implement KaitaiStruct API, so the best common type that will be ok for both parents is KaitaiStruct. Here’s how it looks in any statically-typed language, e.g. in Java: