.amem File Format Specification

This document describes the binary layout of .amem files -- the on-disk format for AgenticMemory brain data. The format is designed for fast random access, memory-mapped I/O, and compact storage.

Design Goals

• Zero-copy reads: Node and edge records are fixed-size, enabling direct memory access without deserialization.
• Memory-mapped friendly: The layout is aligned for mmap() access. The OS handles paging, so brain files larger than available RAM work efficiently.
• Compact: Content is LZ4-compressed. Feature vectors are stored as contiguous float arrays. No JSON overhead, no field names repeated per record.
• Atomic writes: Writes update the header last. A crash during a write leaves the previous valid state intact.
• Forward compatible: The version field and reserved header bytes allow format evolution without breaking existing readers.

Overview

An .amem file consists of six contiguous sections:

+------------------+
|   Magic + Header |  64 bytes
+------------------+
|   Node Records   |  node_count * 64 bytes
+------------------+
|   Edge Records   |  edge_count * 13 bytes
+------------------+
|   Content Block  |  variable (LZ4 compressed)
+------------------+
|  Feature Vectors |  node_count * dimension * 4 bytes
+------------------+
|     Indexes      |  variable
+------------------+

Section 1: Header (64 bytes)

The header occupies the first 64 bytes of the file.

Validation rules:

• magic must be 0x414D454D.
• version must be <= 1 for this specification.
• node_count and edge_count must be consistent with file size.
• dimension must be a positive integer (typically 128).

Section 2: Node Records

Immediately following the header. Each node record is a fixed 64 bytes.

Total: 64 bytes per node.

Notes:

• Node IDs are implicit -- node N is at offset 64 + (N * 64) from the start of the file.
• content_offset and metadata_offset are offsets into the decompressed content block, not the raw file.
• vector_offset is a byte offset into the vector block section.

Section 3: Edge Records

Immediately following the node records. Each edge record is a fixed 13 bytes.

Total: 13 bytes per edge.

Notes:

• Edges are sorted by source node ID for efficient adjacency lookups.
• Both source and target must be valid node IDs (less than node_count).

Section 4: Content Block

A single LZ4-compressed block containing all node content and metadata strings, concatenated end-to-end.

Structure (after decompression):

[node_0_content][node_1_content]...[node_N_content][node_0_metadata][node_1_metadata]...

All content is UTF-8 encoded. Metadata is JSON-encoded UTF-8 (a flat object with string keys and string values).

Compression:

• Algorithm: LZ4 (frame format).
• The header field content_length stores the compressed size.
• The header field content_uncompressed stores the decompressed size.
• If the flags bit 2 is not set, the content block is stored uncompressed (for very small brains where compression overhead is not worthwhile).

Rationale for LZ4: LZ4 decompression runs at memory bandwidth speeds (typically 3--5 GB/s), making it effectively free compared to I/O. The compression ratio for natural language text is typically 2--3x.

Section 5: Feature Vectors

A contiguous block of IEEE 754 single-precision floating-point values. Each node's vector is dimension floats (default: 128 floats = 512 bytes per vector).

Layout:

[node_0_vector: f32 * dimension][node_1_vector: f32 * dimension]...[node_N_vector]

Notes:

• Vectors are stored in node ID order.
• If a node has no vector (vector_offset is u64::MAX), the corresponding slot contains all zeros.
• The contiguous layout is critical for SIMD-accelerated similarity search -- the CPU can scan vectors without pointer chasing.
• Total size: node_count * dimension * 4 bytes.

Section 6: Indexes

The index section contains auxiliary data structures for accelerating queries. It is present only if the flags bit 1 is set.

Type Index (Bitmap)

A bitmap index that maps event types to node IDs. Each event type has a bitset of node_count bits, where bit N is set if node N has that event type.

Session Index

Maps session IDs to their constituent node ID ranges.

Time Index

A sorted array of (timestamp, node_id) pairs for efficient time-range queries.

Cluster Map

Pre-computed cluster assignments for approximate nearest-neighbor search. Nodes are grouped into clusters based on their feature vectors.

Version Compatibility

Version 1 (Current)

The initial release format as described in this document.

Readers must:

• Reject files where magic != 0x414D454D.
• Reject files where version > 1.
• Handle the absence of vectors (flags bit 0 unset) gracefully.
• Handle the absence of indexes (flags bit 1 unset) by falling back to linear scans.

Forward Compatibility

Future versions may:

• Add new event types (values >= 6 in the event_type field). Old readers should treat unknown types as opaque.
• Add new edge types (values >= 7 in the edge_type field). Old readers should treat unknown types as opaque.
• Add new index types. Old readers should skip unknown index types.
• Increase the header size. Old readers should use content_offset to find the content block rather than hardcoding offsets.
• Utilize the reserved header bytes for new fields.

Byte Order

All multi-byte integers and floats are stored in little-endian byte order.

Size Estimates

For a brain with N nodes, M edges, and 128-dimensional vectors:

Example: A brain with 100,000 nodes, 500,000 edges, and average content of 200 bytes per node: