Serialization Guide

Don’t worry, it should be straightforward:

The most important option is --read-write, which enables read-write mode. This adds the methods needed for serialization to the generated classes.

--no-auto-read is explicitly specified here for demonstrative purposes. If you omit it, the compiler will still behave as if it were specified, because it’s implied by --read-write. Normally in read-only mode, if you don’t specify --no-auto-read, you can just use the fromFile / from_file static method to parse a file and get the object with the extracted data immediately:

Or you can instantiate the Gif class directly by invoking the class constructor and passing the stream to read from:

This is because the _read method (responsible for parsing the data) is automatically called from constructors of the generated classes, and is also private (in Java) because you never need to call it explicitly.

However, in read-write mode, it’s no longer clear to Kaitai Struct why you’re creating a particular object. The purpose may just be to create an empty object to be filled with data and later written, in which case you don’t want to read from any stream. For this reason, _read is never called automatically in read-write mode - you need to call it explicitly if you want to read from a stream:

As with the compiler, the latest released 0.10 KS runtime libraries don’t have serialization capabilities yet.

Let’s focus on what the _check method does. We know that foo is expected to be a list of exactly 2 integers (because of repeat-expr: 2 in the source .ksy). Every parsing of the hello_world type tries to read 2 integers, and in any successfully parsed HelloWorld object, foo will be always 2 elements long. However, the setFoo setter allows us to set any integer list - even if its length is 0, 1 or greater than 2.

Nevertheless, if we set foo to a list of length other than 2 and write the hw object to bytes, we won’t be able to get the same state of the HelloWorld object by parsing these bytes again: either the parsing fails with an EOF exception if the stream was shorter than 8 bytes, or we get garbage values in foo (if we attempted to write foo with less than 2 elements) because we interpret some bytes outside foo as if they were foo values, or we may read 2 correct values, but the object we serialized had actually more. In such cases, it’s very much possible that not only the parsed foo wouldn’t match the foo we wrote, but also the offsets of all fields after foo would be shifted, so their values would be incorrect too.

This is because by setting foo to anything other than a 2-integer list, we violate the property of consistency - the data is not consistent with the constraints directly following from how the format is specified in the source .ksy file. Kaitai Struct knows these constraints, and generates assertions for them in the _check method whenever possible. If _check detects a consistency issue, it throws a ConsistencyError, telling you to fix the problem and try again. This protects you from proceeding to the writing phase with inconsistent values, which would inevitably result into corrupt data that cannot be faithfully decoded back to the original values.

To see it in action, let’s try what happens if we set foo to a list of length 3 and ask the HelloWorld class what it thinks about the consistency of this object:

As expected, the _check method caught the problem and threw an exception - the expected length of field foo was 2, but it was 3, which doesn’t match the format definition.

Real-world .ksy specifications often define custom types in the types section. For example:

A typical way to serialize such format would be as follows:

First, we instantiate the root class UserTypes as usual. Then we need the instance of the user-defined chunk type, translated as UserTypes.Chunk in Java or Python. We use the usual way to create an instance of a class, but this time using all 3 arguments of the constructor:

The reason is that we must provide values for the _parent and _root parameters (see their description in the User Guide). These built-in references should be valid in all KS types so that it’s possible to rely on them in expressions inside the .ksy spec when needed. When you instantiate inner objects (any object instances of user-defined types other than the root object) manually, you have to set these properties correctly.

Note the generally-applicable rule of what should go there (let’s call the parent object as p):

If you don’t set the correct values to both _parent and _root, it’s a consistency issue that will be reported in _check of the parent object (ut in this case):

After we create an instance of the UserTypes.Chunk subtype, we set its properties, and then we call _check. This is important: _check always works only for the one object on which you call it, it doesn’t recursively descend into substructures (unlike _read and _write which do that, so you call them just on the top-level object). So it’s not enough to call _check just on the top-level object - you have do it for every KS object on which you use setters.

After creating a new KS object, you must also set fields with contents or valid on them, even if there’s only one valid value they can have. Kaitai Struct doesn’t set them automatically at the moment. For example, the following magic field

needs to be set as follows:

The _check method validates such fields, so you get notified if the values are not valid.

They don’t have setters. If you need to make value instances change, you have to set their inputs (fields they depend on). For example:

We set a 5-byte array to data, so for the object to be consistent, we need len_data to be 5. Since it’s defined as len_data_raw - 3, we set len_data_raw to 8, which makes len_data to be 8 - 3 = 5.

What happens if you want to change the length of data in this existing object? Instances in KS are cached, so even if you change len_data_raw, len_data will keep returning the old cached value (5):

To fix this, you need to call a special method _invalidate{Inst} (_invalidate_{inst} in Python) associated with the value instance after changing len_data_raw:

The Java’s _invalidate{Inst} / Python’s _invalidate_{inst} method invalidates the cached value of the instance so that it’s recalculated on the next access.

They have setters and their own _check{Inst} (_check_{inst}) method which you should call. Additionally, you can also use a special boolean set{Inst}_ToWrite setter (in Python you’d assign a boolean to a property {inst}__to_write), allowing you to disable writing of a specific instance (as r.set{Inst}_ToWrite(false) in Java, or r.{inst}__to_write = False in Python) in a particular KS object. This may be useful for C-style union members (several overlapping fields with different types, but only one applies in any object), lookaheads or other positional instances you don’t want to write.

You can give them to the constructor when instantiating the KS type and you can later change them via setters. Again, KS doesn’t set almost anything automatically, so you’re usually in charge of setting all parameters, even though you need to set the parameters to same values that the parent type would pass to them. The _check method of the parent type contains checks whether this holds.

Current serialization support relies on fixed-length streams, meaning that once you create a stream, it’s not possible to resize it later. Therefore, you’ll often need to calculate sizes "manually" in your application along with setting the object properties (at least for the root stream, which you have to provide to the _write method). The recommended way to do that is outlined in this GitHub comment.

Enum values not present in the enum definition are not supported in Java or Python right now. An attempt to write them causes NullPointerException in Java, AttributeError in Python.

Unlike the existing parser implementation of bit types which relied on explicit alignToByte() calls (and this resulted in many problems, because in many cases the compiler failed in where to insert them and where not), all byte-aligned operations in Java and Python runtime libraries with serialization support now perform the byte alignment automatically, and the explicit alignToByte() calls shouldn’t be needed anymore.

When you write a structure with X-bit type: bX fields, only full bytes are written once they’re known. This means that if your format ends at an unaligned bit position, the bits of the final partial byte remain in the internal "bit buffer", but they will not be written to the underlying stream until you do some operation which aligns the position to a byte boundary (e.g. writeBytes(0), seek(…​), or explicit writeAlignToByte()). However, if you don’t have anything else to write and don’t need to work with that stream anymore, it’s recommended to close() the stream, which automatically writes the remaining bits (if any) before closing the stream.

Sometimes a consistency check cannot be performed in _check because the user expressions from the .ksy specification that the check needs to use do not allow it. A typical example is when the expression makes use of the built-in _io variable, for example:

Since it’s a fixed-length byte array with the size expression denoting its length, it’s necessary to check whether the length of the rest byte array (that might have been changed via a setter) and the value of the size expression _io.size - _io.pos match. But this expression uses _io, so it cannot be performed in _check: _check is meant to check pure data consistency and the _io may not be available at this point. So this consistency check will be moved to _write just before the rest field would be written.