Raft Clustering

LoomCache is fundamentally a distributed system: in production deployments with persistence enabled and a healthy Raft quorum, a single node failure should not lose committed data. To coordinate this replicated state, LoomCache implements the Raft Consensus Algorithm, a widely adopted consensus protocol. This page explains the leader role, log replication, leader election, the write-ahead log, and dynamic membership.

Raft Consensus Flow

Waiting for client request...

Client

Leader Node

WAL Term

Follower 1

Follower 2

The Leader Role

At any given time, one node in the LoomCache cluster is the leader. The leader receives all write requests (for example, PUT and DELETE). It writes each request to its local log and immediately broadcasts an AppendEntries RPC to the followers.

Clients discover the leader through the client-side leader cache. If a write reaches a follower, the client receives the current leader address and reroutes the request.

Log Replication & Quorum

A write is not considered committed until the leader receives acknowledgments from a strict majority of the cluster (the quorum).

Caution

If a 5-node cluster suffers a 2-node failure, the remaining 3 nodes maintain quorum and continue serving writes. If 3 nodes fail, the remaining 2 lose quorum and block writes to prevent split-brain data corruption.

Single Raft group, full replication

The cluster currently runs a single Raft group covering every node. Every node holds every partition, and all writes flow through the same log. Three-node clusters are the recommended minimum (quorum = 2). The sharding/ package ships multi-group primitives, but production sharding is unsupported and fail-closed until per-group WAL, Raft metadata, snapshot, install-snapshot, and restart recovery are proven with durable migration chunk ACKs, consensus-backed ownership cutover, and an explicit sharding release gate.

Leader Election

LoomCache hardens standard Raft with several safety mechanisms:

• Pre-vote phase: Before starting a real election, candidates run a pre-vote round. This prevents partitioned nodes from incrementing the global term number and disrupting stable clusters.
• Randomized timeouts: Raft election timers are randomized to prevent simultaneous elections across the cluster.
• Leader lease: The leader maintains a lease based on recent heartbeat acknowledgments from a majority. This enables fast linearizable reads without quorum round-trips on every GET.
• Leadership transfer: Explicit handoff for graceful shutdowns and controlled restarts.

Timing Configuration

The public server/Spring configuration exposes cluster heartbeat and failure-detection settings through ClusterConfig.heartbeatIntervalMs and ClusterConfig.heartbeatTimeoutMs. Spring Boot keeps the legacy loomcache.server.raft.* property names, but they map to these same cluster-level defaults:

Parameter	Default	Purpose
heartbeatIntervalMs / loomcache.server.raft.heartbeat-interval-ms	5000ms	Member heartbeat cadence and peer liveness checks
heartbeatTimeoutMs / loomcache.server.raft.election-timeout-ms	60000ms	Peer failure-detection timeout; must exceed the heartbeat interval
Max entries per append	100	Batch size for AppendEntries RPC

Note

Public server startup uses the ClusterConfig values above for operator-facing heartbeat and timeout behavior.

Write-Ahead Log (WAL)

When dataDir / persistence is enabled, every committed entry is persisted to disk before acknowledgment. With dataDir=null, non-production nodes run in-memory without WAL/snapshots; production startup rejects that mode.

1. The durability layer appends the entry with record metadata and an integrity checksum so recovery can validate and replay committed entries in order.
2. fsync() is called after each committed Raft entry; Raft WALs reject deferred-fsync batching so a committed entry does not sit in a durability window. Disabling fsync (-Dloomcache.wal.disableFsync=true) drops the WAL to WRITE durability and is not permitted for production startup.
3. Compaction triggers once the in-memory log exceeds approximately DEFAULT_COMPACTION_THRESHOLD = 10,000 committed entries. When compaction fires, a snapshot is written in a checksummed binary format that includes the snapshot header, serialized state, optional Raft membership metadata, and a trailing SHA-256 checksum. The WAL is then truncated.
4. On restart, the node recovers Raft metadata, loads the newest valid snapshot when present, replays the WAL, and rejoins the cluster.

Note

Place the WAL on SSD/NVMe storage for low write latency. fsync latency is storage-dependent; measure WAL fsync latency on the target devices and include it in release evidence.

Linearizability

LoomCache’s default production client wire path provides strong consistency (linearizability). Once a write is acknowledged to a client on that path, later leader/read-index reads across the cluster reflect that write, including across leader elections and network partitions that preserve quorum. Non-production embedded/member-local compatibility modes can opt into local backup reads; those reads may be stale and are rejected by loomcache.profile=production.

Reads are CP-pure: every read is served by the current Raft leader. When the leader holds a valid lease, it answers linearizable reads locally with no quorum round-trip on each GET. If the lease has expired, the leader runs a lease-confirmation AppendEntries round first, which adds one round-trip. There is no stale follower-read or eventual-consistency path; reads do not scale by fanning out to followers.

After each leader election, reads and writes stall briefly while the new leader appends and commits a no-op entry in its term. This is a safety property of the Raft protocol: the leader cannot determine the safe commit boundary for entries from prior terms until it has committed at least one entry of its own.

Linearizability is verified under the Java Chaos-style harness. See Chaos testing for details.

Dynamic Membership

LoomCache supports live cluster scaling through single-server Raft configuration changes:

• Learner-first join: A new node is first added as a non-voting learner. It receives a snapshot + WAL catch-up while the cluster keeps serving traffic.
• Add-server promotion: Once the learner’s replication lag is within bounds, a membership change promotes it to a voter that counts toward quorum. Promotion is rejected if the target never joined as a learner or has not caught up.
• RemoveServer: A voter (or learner) is removed from membership. The change is rejected if the target is the last remaining voter.
• Online changes: With quorum healthy and one server changed at a time, the cluster continues serving reads and writes during membership changes without a planned cluster stop.

Note

Membership changes are single-server only. Legacy single-shot multi-voter membership commands are rejected until the learner-first membership path can safely express them. Partition migration is a separate non-production validation capability and is not triggered automatically by RemoveServer.

To learn how keys map to partitions and how clients route requests, see Partitioning & Sharding.