On-disk serialization

Write to a serial, peristed queue in OutputManager. OutputManager will also be responsible for truncating this queue to asssert that any size limitations are respected.

This would allow for a temporary loss of network, without losing data.


The entire queue is serialized into a set of files (segments) which are limited in size according to maxSegmentSize, with the default being 100 MB.

Each segment is named according to a hex-encoded, zero-padded base offset of that segment (example: 00000000ff).

Incoming data is written to the tail segment linearly, until it would be forced to grow larger than maxSegmentSize. When this happens a new tail segment is allocated and the blob will be written to the newly allocated tail segment.

A consumer maintains its position in the queue in the index file. At a regular interval, a process will scan the current offset of all consumers and trim the head of the queue. Trimming involves unlinking all whole segments prior to a given position, partial segments where the trim position is in the middle of the segment will be kept.


Each segment is a binary file, with the following structure.

magic   | 4 | 4 byte magic, making up "FFLG" (0x46 0x46 0x4c 0x47) in ASCII.
version | 2 | Unsigned 2-byte short, indicating the current version of the
              segment format.
offset  | 8 | Unsigned offset in number of messages that is the start of this
size    | 4 | An unsigned integer indicating the size of the next entry.
blob    | n | A byte blob with the above size.
... other entries until EOF.

The index is a binary file, with the following structure.

idlength | 4 | A 4 byte unsigned integer, indicating the length of the next id.
offset   | 8 | An 8 byte unsigned long, indicating the offset of the next id.
id       | n | A UTF-8 encoded string, which is the id of the given consumer.
               The encoded length of this string is given by `idlength`.
... other entries until EOF.