.acomm File Format Specification

This document describes the binary layout of .acomm files -- the on-disk format for AgenticComm communication data. The format is designed for fast sequential reads, compact storage, and forward compatibility.

Design Goals

• Compact: Messages are serialized with bincode and compressed with flate2 (gzip). Typical agent conversations compress at 3:1 or better.
• Atomic writes: The checksum footer is written last. A crash during write leaves the previous valid state intact. Readers verify the checksum and fall back to the last known-good state on mismatch.
• Forward compatible: Unknown sections are skipped using the section table offsets. Older readers can safely open files written by newer versions.
• Indexed: Channel-to-message and timestamp indexes are stored inline for fast lookup without full deserialization.
• Bounded: Maximum file size is 2 GB. Maximum message count is 10 million. These limits prevent unbounded growth and ensure predictable performance.

File Layout

An .acomm file consists of seven contiguous sections:

+-------------------------+
|   Magic + Header        |  96 bytes
+-------------------------+
|   Section Table         |  variable (6 entries * 24 bytes = 144 bytes)
+-------------------------+
|   Channel Section       |  variable (bincode serialized)
+-------------------------+
|   Message Section       |  variable (bincode + flate2 compressed)
+-------------------------+
|   Subscription Section  |  variable (bincode serialized)
+-------------------------+
|   Index Section         |  variable (bincode serialized)
+-------------------------+
|   Footer                |  40 bytes (checksum + metadata)
+-------------------------+

Section 1: Header (96 bytes)

The header occupies the first 96 bytes of the file.

Flags Bitfield

Validation Rules

• magic must be exactly ACOMM001 (8 bytes).
• version must be <= 1 for this specification. Readers encountering a higher version should either upgrade or refuse to open the file with a clear error message.
• section_count must match the number of entries in the section table.
• channel_count and message_count must be consistent with their respective sections.
• total_size must match the actual file size.
• Reserved bytes must be zero. Readers must ignore reserved bytes (do not reject files with non-zero reserved bytes, for forward compatibility).

Section 2: Section Table

Immediately following the header. Each entry is 24 bytes. The table has section_count entries (currently 6).

Section Types

Forward Compatibility

When a reader encounters an unknown section type:

1. Log a warning (not an error).
2. Skip the section using the offset and length fields.
3. Continue processing remaining sections.

This allows older readers to open files written by newer versions that introduce additional sections.

Section 3: Channel Section

Contains all channel records, serialized sequentially with bincode.

Channel Record Layout

Each channel is serialized as a bincode struct. The section is a length-prefixed sequence:

[channel_count: u64]
[channel_0: bincode bytes]
[channel_1: bincode bytes]
...
[channel_N: bincode bytes]

Each channel record includes:

Participant Sub-record

Section 4: Message Section

Contains all active messages, serialized with bincode and optionally compressed with flate2.

Compression Envelope

When the COMPRESSED flag is set in the header:

[uncompressed_size: u64]  -- 8 bytes, original size before compression
[compressed_data: bytes]  -- flate2 gzip compressed bincode data

When the COMPRESSED flag is not set:

[raw_data: bytes]  -- uncompressed bincode data

Message Record Sequence

After decompression (or directly if uncompressed), the message section contains:

[message_count: u64]
[message_0: bincode bytes]
[message_1: bincode bytes]
...
[message_N: bincode bytes]

Messages are stored in creation order (ascending by created_at). This enables efficient temporal range queries on the raw data.

Message Record Layout

Each message is a bincode-serialized struct with the fields defined in Data Structures. Key considerations:

• content is stored as a length-prefixed byte array (UTF-8). The content is part of the compressed block, so individual message content is not independently accessible without decompressing the entire section.
• signature, if present, is stored as a length-prefixed byte array.
• metadata, if present, is serialized as a bincode map.
• Optional fields use bincode's Option encoding (1 byte tag: 0=None, 1=Some followed by value).

Section 5: Subscription Section

Contains all pub/sub subscriptions.

[subscription_count: u64]
[subscription_0: bincode bytes]
[subscription_1: bincode bytes]
...
[subscription_N: bincode bytes]

Each subscription record follows the Subscription struct layout from Data Structures.

Section 6: Index Section

Contains lookup indexes for fast queries without full message scanning.

Index Types

The index section contains multiple sub-indexes, each prefixed with a type tag and length:

[index_count: u32]
[index_type_0: u32] [index_length_0: u64] [index_data_0: bytes]
[index_type_1: u32] [index_length_1: u64] [index_data_1: bytes]
...

Channel-Message Index (type = 1)

Maps channel IDs to the message IDs they contain.

[entry_count: u64]
For each entry:
  [channel_id: u64]
  [message_id_count: u64]
  [message_ids: [u64; count]]   -- sorted ascending

Timestamp Index (type = 2)

Sorted array of (timestamp, message_id) pairs for binary search.

[entry_count: u64]
For each entry:
  [timestamp: u64]
  [message_id: u64]

Entries are sorted by timestamp ascending. Ties are broken by message_id ascending.

Topic Index (type = 3)

Maps exact topic strings to message IDs.

[entry_count: u64]
For each entry:
  [topic_len: u32]
  [topic: [u8; topic_len]]
  [message_id_count: u64]
  [message_ids: [u64; count]]   -- sorted ascending

Sender Index (type = 4)

Maps sender participant IDs to their message IDs.

[entry_count: u64]
For each entry:
  [sender_len: u32]
  [sender: [u8; sender_len]]
  [message_id_count: u64]
  [message_ids: [u64; count]]   -- sorted ascending

Correlation Index (type = 5)

Maps correlation IDs to all messages in the thread.

[entry_count: u64]
For each entry:
  [correlation_id_len: u32]
  [correlation_id: [u8; len]]
  [message_id_count: u64]
  [message_ids: [u64; count]]   -- sorted ascending

Section 7: Footer (40 bytes)

The footer is the last 40 bytes of the file. It is written last to ensure atomic updates.

Checksum Verification

On file load:

1. Read the last 40 bytes to extract checksum and footer magic.
2. Verify footer magic is ACEND001.
3. Compute SHA-256 of bytes 0 through (file_size - 40).
4. Compare computed hash with stored checksum.
5. If mismatch: reject the file with an integrity error. Do not silently load corrupt data.

Atomic Write Protocol

When writing a .acomm file:

1. Write to a temporary file (<path>.acomm.tmp).
2. Write header, section table, all sections.
3. Compute SHA-256 of everything written so far.
4. Write footer (checksum + magic).
5. Flush and sync the temporary file.
6. Atomically rename the temporary file to the target path.

This ensures that readers never see a partially-written file. If the process crashes during write, the temporary file is left behind and can be cleaned up on the next successful write.

Size Limits

When a limit is reached, the store returns an error with a descriptive message. The store does not silently truncate or drop data.

Versioning

The version field in the header enables format evolution:

Version Upgrade Protocol

When a reader encounters version > 1:

1. If the reader supports the version, proceed normally.
2. If the reader does not support the version, check if the section table contains only known section types.
3. If all section types are known, attempt to read the file (the new version may only add new sections that can be skipped).
4. If unknown section types are present, skip them and read known sections.
5. If the header structure itself has changed (different header size), reject with a version error.

Version Downgrade

Files written by version N can be read by readers supporting version N or higher. Downgrade (opening a version-2 file with a version-1 reader) is supported only if the version-2 file uses no features beyond version-1 and all unknown sections can be safely skipped.

File Locking

When multiple processes access the same .acomm file:

1. Writers acquire an exclusive lock on <path>.acomm.lock (advisory file lock).
2. Readers do not require a lock (they read the last atomically-written version).
3. The lock file contains the PID of the holding process and a timestamp.
4. Stale locks (process PID no longer running) are automatically recovered after 60 seconds.

Lock File Format

PID: <process_id>
STARTED: <unix_timestamp>
HOSTNAME: <hostname>

Plain text, human-readable. This allows manual inspection and recovery.

Example File Sizes