Audit Format

Last updated 15 days ago

This is the normative specification of j17's audit artifacts — hash chains, Merkle trees, proofs, and signed anchors — as produced by format version 2. It is written to be implementable without reading j17's source: a verifier built from this page alone, in any language with SHA-256 and Ed25519, must accept every artifact the platform produces and reject every forgery this format defends against.

Format 2 differs from format 1 in exactly two ways: Merkle hashing is domain-separated (leaves and internal nodes use distinct prefixes, per RFC 6962), and anchor signatures cover a canonical serialization of the anchor's claims rather than the bare root hash. Event chain hashes are unchanged — anything verifying chains against GET /{type}/{id}/events?raw=true is unaffected by the version bump.

Version signaling

Every artifact that carries hashes or signatures carries a format field:

  • "format": 2 — this specification.
  • Field absent — format 1 (produced before platform 0.16.3). Format 1 trees use no domain separation (leaves are used raw; internal nodes are SHA-256(left || right)), and format 1 anchor signatures cover the lowercase hex merkle_root string only.

A verifier must key on this field. New artifacts are always format 2; format 1 artifacts remain valid as format 1 and are never re-signed.

Definitions

  • SHA256(x) — SHA-256 over the bytes x, yielding 32 bytes.
  • hex(x) — lowercase hexadecimal encoding of bytes x.
  • unhex(s) — the bytes of a hex string (case-insensitive on input; j17 always emits lowercase).
  • || — byte concatenation.
  • All hashes on the wire are 64-character lowercase hex strings.

1. Event chain hashes (unchanged from format 1)

Events in a stream are hash-chained. For a stream's events e_1 .. e_n, with bytes_i the event's JSON exactly as stored (retrievable via GET /{type}/{id}/events?raw=true as the raw field):

chain_1 = hex( SHA256( bytes_1 ) )
chain_i = hex( SHA256( chain_(i-1) || bytes_i ) )      for i > 1

Note the preimage concatenates the previous hash's hex string (64 ASCII bytes) with the raw event bytes. The chain hash of an event is returned as hash by /events, and is the input leaf value for its stream's Merkle tree.

2. Merkle trees

All j17 Merkle trees — per-stream (over chain hashes), per-instance (over stream roots), and platform (over instance roots) — use the same construction. The input is an ordered list of 64-hex strings v_1 .. v_n (n ≥ 1; an empty input produces no tree and no anchor).

2.1 Leaf hashing

leaf_i = SHA256( 0x00 || unhex(v_i) )

A single 0x00 byte, then the 32 raw bytes of the input value. Leaves are hashed even when n = 1: a single-leaf tree's root is hex(leaf_1), not v_1.

2.2 Internal nodes

Levels are built bottom-up. If a level has an odd number of nodes, its last node is duplicated (so the level being paired always has even length). Adjacent pairs combine as:

parent = SHA256( 0x01 || left || right )

where left and right are the 32-byte child hashes. The root of the last level is the tree's root, emitted as lowercase hex.

The 0x00/0x01 prefixes are RFC 6962's domain separation: no internal node can be reinterpreted as a leaf or vice versa, so second-preimage arguments do not depend on what leaf values look like.

On duplicate-last padding. This construction is known to give [A, B, C] and [A, B, C, C] the same root (the CVE-2012-2459 class). It is harmless here because no j17 tree can legitimately contain equal adjacent inputs: stream-tree inputs are chain hashes, and chain_i == chain_(i-1) requires SHA256(chain_(i-1) || bytes_i) == chain_(i-1) — a SHA-256 break; instance- and platform-tree inputs are roots over byte streams that embed their own stream key or instance id, so equal inputs across entries likewise require a collision. Duplication has one further consequence a verifier MUST handle: the duplicated node exists at a real index, so an inclusion proof for a phantom index — index 3 in a padded [A, B, C, C] tree — folds to the signed root and verifies. A verifier MUST therefore reject any proof whose claimed index is not below the signed count for its tree (event.index < stream.event_count; a stream's index below the anchor's stream_count): the format-2 anchor signs those counts precisely so this bound is available offline. The j17 endpoints enforce it server-side; an independent verifier consuming exported bundles must enforce it itself or it will accept a proof of a nonexistent event. A verifier that wants no argument at all may additionally reject any proof or tree whose adjacent inputs are equal. (RFC 6962's unbalanced construction avoids the property entirely; j17 keeps duplication for construction continuity with format 1, on the argument above. The 0x00/0x01 prefixes are the part of RFC 6962 this format adopts; the tree shape is not RFC 6962's.)

(Format 1 differs here only in having no prefixes and no leaf hashing: leaves were used raw and parents were SHA256(left || right).)

2.3 Which values feed which tree

Tree Input values v_i Order
Stream the stream's event chain hashes event order
Instance each non-empty stream's root stream key, lexicographic ascending
Platform each instance's anchored root instance id, lexicographic ascending

3. Inclusion proofs

A proof for input value v at index i (0-based) consists of siblings (hex strings) and directions ("left" / "right"), one entry per tree level below the root. To verify against a claimed root:

current = SHA256( 0x00 || unhex(v) )            # the leaf hash
for (sibling, direction) in zip(siblings, directions):
    if direction == "right":                     # sibling is on the right
        current = SHA256( 0x01 || current || unhex(sibling) )
    else:                                        # sibling is on the left
        current = SHA256( 0x01 || unhex(sibling) || current )
accept iff hex(current) == root

A single-leaf tree's proof has empty siblings and directions — this is legal and expected: verification is then just the leaf hash itself, so accept iff hex(SHA256(0x00 || unhex(v))) == root. A verifier must not reject empty proof arrays.

Before the fold, a verifier must check len(siblings) == len(directions) and reject on mismatch — a zip that silently truncates the longer list would turn a malformed proof into a shorter fold that can still "verify".

Proof responses carry "format": 2. A format 1 proof verifies the same way minus both prefixes and minus the leaf-hash step (start with current = unhex(v); a format 1 single-leaf proof accepts iff v == root).

4. Anchors and signatures

An anchor is a signed commitment to a tree root at a point in time. Format 2 signatures cover a canonical payload — a JSON object serialized with the exact field order given below, no whitespace, integers in plain decimal — so the anchor's claims (whose data, which environment, when) are bound by the signature and cannot be relabelled.

All values in the payload are platform-generated (UUIDs, environment names, hex digests, ISO 8601 UTC timestamps, integers) and contain no characters requiring JSON escaping; a verifier can reconstruct the payload by direct string interpolation.

4.0 Snapshot cut

An instance anchor describes one coherent cut across all of the instance's event streams. The platform atomically captures the chain-tip hash of every non-empty public stream, then reads each stream only through its captured tip. Events appended after that cut are excluded even if their stream is read later in the Merkle scan. An atomic write spanning several streams is therefore included in full or excluded in full; an anchor never includes only some of its events.

The full scan does not hold a write lock. If a destructive stream rewrite makes a captured tip unavailable before it can be read, anchor construction abandons that attempt and starts again from a fresh cut rather than signing a mixed view.

created_at is the signed time at which the completed anchor is constructed. It is not an event timestamp or a claim that every captured stream tip was written at that instant.

4.1 Instance anchor payload

{"format":2,"scope":"instance","instance_id":"<uuid>","environment":"<env>","merkle_root":"<hex>","event_count":<int>,"stream_count":<int>,"created_at":"<iso8601>"}

4.2 Platform anchor payload

{"format":2,"scope":"platform","merkle_root":"<hex>","instance_count":<int>,"cutoff_at":"<iso8601>","created_at":"<iso8601>"}

4.3 Signature

signature = hex( Ed25519_sign( private_key, payload_bytes ) )

The anchor record stores each payload field individually, plus signature and public_key (32-byte Ed25519 public key, hex). To verify: reconstruct the payload from the record's fields exactly as specified, and check the signature over those bytes with the record's public key. Any field substitution — a different environment, instance, timestamp, or root — must fail.

The created_at in the payload is generated by the builder before signing and stored verbatim; it is the anchor's one authoritative timestamp.

(Format 1 signatures are Ed25519_sign(private_key, merkle_root_hex_string) — they bind the root and nothing else.)

4.4 Key identity

Instance anchors are signed with the instance's key; platform anchors with the platform key. GET /_audit/signing-key returns the instance's current public key; the platform public key is published in the platform anchor records themselves and pinned by external timestamps (OpenTimestamps / RFC 3161) — see the audit API guide for the trust chain.

5. Verification recipes

A page of events is untampered relative to itself: fetch /events?raw=true, fold §1 over the entries, compare each computed hash to the entry's hash where served. (The fold is self-sufficient — the served hash is a cross-check, not an input, so verification works even where hash is not served.)

An event is in its stream's anchored root: recompute its chain hash (§1), verify its stream proof (§3) against the stream root.

A stream is in a signed instance anchor: verify the stream root's inclusion proof against the anchor's merkle_root, then verify the anchor signature (§4.3) with the anchor's public key.

An instance root is in a signed platform anchor: same shape one level up, using the platform payload (§4.2). The platform anchor's external timestamps prove the root existed at a point in time independent of j17.

6. Attestation bundles

GET /{type}/{id}/audit/attestation/{index}?anchor={anchor_id} returns, in one response, everything §5's recipes need for one event — logarithmic in size and disclosing nothing about other streams:

{
  "ok": true,
  "format": 2,
  "event":          { "index": 2, "id": "...", "raw": "{...}", "hash": "...", "prev_hash": "..." },
  "stream":         { "key": "order:...", "merkle_root": "...", "event_count": 4 },
  "stream_proof":   { "siblings": ["..."], "directions": ["right", "left"] },
  "instance_proof": { "index": 0, "stream_count": 3, "siblings": ["..."], "directions": ["right"] },
  "anchor":         { "format": 2, "id": "...", "instance_id": "...", "environment": "...",
                      "merkle_root": "...", "signature": "...", "public_key": "...",
                      "event_count": 9, "stream_count": 3, "created_at": "..." },
  "anchored_at": "..."
}

To verify a bundle offline:

  1. Bind the bundle to its claimed stream: decode event.raw and check its "key" field equals the stream you expect — and, belt-and-suspenders, equals stream.key in the response. This check is not redundant and must not be deleted as such: a sibling-only instance proof shows stream.merkle_root is in the anchor's tree somewhere, but nothing positional binds that root to a stream key the way the (redacted) streams map once did. The bytes are the binding — the chain hash covers event.raw, and event.raw embeds the key — but only if the verifier reads them. Without this step, a valid bundle for document A passes every other check when presented as a bundle for document B.
  2. Check the coordinates: event.index equals the index you requested, and instance_proof.stream_count equals anchor.stream_count (a signed total — a free check that the proof was built against the signed tree, not a differently-sized one). Proof arrays obey §3's length rule.
  3. Recompute the chain hash from raw and prev_hash (§1); compare to event.hash.
  4. Verify stream_proof from event.hash to stream.merkle_root (§3).
  5. Verify instance_proof from stream.merkle_root to anchor.merkle_root (§3).
  6. Reconstruct the instance anchor payload from anchor's fields (§4.1) and verify signature with public_key.

The anchor is redacted: it never includes the anchor's streams map — an inclusion proof needs only siblings.

anchored_at is a convenience echo of anchor.created_at and event.id is the storage-level stream id; both are advisory. The authoritative timestamp is anchor.created_at — it is inside the signed payload; nothing else in the envelope is, so nothing else should be treated as load-bearing.

Semantics:

  • anchor (query parameter) pins the anchor by id; omitted means the latest anchor for the key's environment. Batch workflows should pin: resolve one anchor, then request every bundle against it.
  • The bundle attests the stream as of the anchor: proofs are built over the first stream.event_count events, and events written after the anchor are not covered by it.
  • Only format-2 anchors can back bundles.

Error contract (machine-readable error_type, always):

error_type Status Meaning Caller's move
beyond_anchor_coverage 409 The anchor predates this event ("reason": "index") or this stream ("reason": "stream"). Carries anchor_id, anchor_created_at, covered_event_count. Wait for, or mint, the next anchor. Not a fault.
invalid_index 422 Malformed or negative index. Fix the request.
anchor_format_unsupported 409 A format-1 anchor cannot back a format-2 bundle. Mint a fresh anchor.
anchor_stream_mismatch 409 The stream no longer reproduces the root the anchor signed (e.g. rewritten by a GDPR tombstone). Mint a fresh anchor; the old one cannot attest this stream.

Change log

  • v2 (platform 0.16.3): domain-separated Merkle hashing (§2.1, §2.2); canonical signed anchor payload (§4). Event chain hashing unchanged. Attestation bundles (§6) added in 0.16.4.
  • v1 (through platform 0.16.2): raw leaves, unprefixed internal nodes, signature over bare root hex. Identified by the absence of format.
Next
FAQ
Can't find what you need? support@j17.app