back to research
rustsystemsdistributed

Building a SWIM Gossip Membership Protocol from Scratch in Rust

Implementing the SWIM protocol for failure detection and membership management in distributed systems.

Chaitanya

Source Code: https://github.com/cb7chaitanya/networking/tree/main/gossip-membership

Distributed systems often treat cluster membership as invisible infrastructure — something that just exists inside service meshes, distributed databases, and orchestration platforms.

But behind every node discovery, every failover, every rolling deployment lies a protocol solving a deceptively simple question: Who else in the cluster is alive right now?

This article walks through the design and implementation of a SWIM-style gossip membership protocol written entirely in Rust.

The system includes:

  • Custom binary wire format with protocol versioning
  • Two-phase failure detection (direct + indirect probes)
  • Epidemic gossip dissemination with adaptive fanout
  • Incarnation-based suspicion refutation
  • Anti-entropy full-table synchronization with MTU-safe chunking
  • ChaCha20-Poly1305 encryption with per-node authentication
  • Network simulation (loss, partitions, delay, reordering)
  • Prometheus + Grafana observability
  • Property tests, fuzz testing, and performance benchmarks

This is not a wrapper around an existing library. It is a ground-up implementation of the SWIM protocol.

Why Build a Membership Protocol?

Most distributed systems rely on membership frameworks such as HashiCorp Memberlist, Consul's SWIM implementation, or etcd discovery. These systems handle node discovery, failure detection, state dissemination, and cluster view convergence.

In this project, the protocol performs all of these responsibilities itself. That means probes are constructed and parsed manually, failure detection is implemented as a SWIM state machine, gossip dissemination follows epidemic spreading rules, merge rules enforce ordering on node state, and partitions and failures must be handled explicitly.

Implementing membership from scratch makes the system fully transparent. You can trace cluster state transition down to the exact byte sequence that caused it.

SWIM Architecture Overview

SWIM maintains cluster membership in a fully decentralized way.

Node A                    Node B                    Node C
  │                         │                         │
  │──── GOSSIP ────────────►│                         │
  │     (membership digest) │──── GOSSIP ────────────►│
  │                         │     (merged view)       │
  │◄──── GOSSIP ─────────────────────────────────────│
plain text

Every node maintains a local membership table. Periodically each node: selects a random peer, sends its view of the cluster, the receiver merges the membership entries, and updates spread epidemically across the cluster.

Wire Format: Byte-Level Design

Each gossip message contains a 24-byte fixed header and a variable-length membership payload.

Header Structure

Each message begins with a 24-byte fixed header:

  • Version (1 byte): protocol version, currently v1. Messages with unsupported versions are rejected before decoding.
  • Kind (1 byte): message type identifier such as GOSSIP, PING, or ACK.
  • Payload Length (2 bytes): length of the membership payload.
  • Sender ID (8 bytes): unique identifier of the sending node.
  • Heartbeat (4 bytes): logical clock used to track liveness updates.
  • Incarnation (4 bytes): monotonic counter used to refute suspicion.
  • Flags (1 byte): reserved for optional protocol features.
  • Reserved (1 byte): padding for alignment.
  • Checksum (2 bytes): integrity check using RFC 1071 checksum algorithm.

All multi-byte integers are encoded using big-endian byte order.

Message Types

GOSSIP       (0x01)
PING         (0x02)
PING_REQ     (0x03)
ACK          (0x04)
LEAVE        (0x05)
ANTI_ENTROPY (0x06)
plain text

Membership Entry Encoding

Each membership entry encodes a node's current state. IPv4 entries are 24 bytes. IPv6 entries are 36 bytes.

Bytes 0-7   node_id (u64)
Bytes 8-11  heartbeat (u32)
Bytes 12-15 incarnation (u32)
Byte 16     status
Byte 17     addr_family
Bytes 18+   ip + port
plain text

Status values: 0 = Alive, 1 = Suspect, 2 = Dead.

Failure Detection: Two-Phase Probing

Unlike traditional heartbeat systems, SWIM performs active probing. Each probe cycle: select a random peer, send PING, wait for ACK, and if timeout — escalate to indirect probing.

Indirect probing asks other nodes to verify connectivity:

Node A  ──── PING ──────────────► Node B  (no response)
Node A  ──── PING_REQ ──────────────────────────────► Node C
                                   Node C ◄──── PING ──── Node B
plain text

Without indirect probes, false positives would be extremely common in the presence of a network partition.

Suspicion and Dead Transitions

Nodes are not declared dead immediately. Instead they enter a Suspect state.

Alive → Suspect → Dead
plain text

The suspicion timeout scales with cluster size: timeout ∝ log₂(cluster_size). A deterministic jitter derived from (observer, suspect) ensures that nodes do not simultaneously declare failure.

Incarnation Refutation

If a node learns that it has been suspected, it increments its incarnation number.

Alive (incarnation 4) → suspected → Alive (incarnation 5)
plain text

This allows nodes to refute suspicion without coordination. Incarnation numbers act as a monotonic version counter for node identity.

Merge Rules

When gossip arrives, nodes must decide whether the incoming information is newer. The merge rules form a strict ordering:

  1. Higher incarnation always wins
  2. Dead at same incarnation is terminal
  3. Same incarnation: higher heartbeat wins
  4. Same incarnation + heartbeat: Dead > Suspect > Alive

These rules ensure deterministic convergence across the cluster.

Anti-Entropy: Repairing Gossip Gaps

Regular gossip spreads random subsets of entries. Under heavy packet loss, some nodes may never receive certain updates.

Anti-entropy fixes this by periodically sending the entire membership table. The challenge is packet size — a 1000-node cluster produces ~24 KB of membership data, larger than the UDP MTU. So the table is split into MTU-safe chunks with table_version, chunk_index, total_chunks, and entries fields.

Encryption

Cluster traffic can be encrypted using ChaCha20-Poly1305. The sender ID is bound as Additional Authenticated Data (AAD), preventing ciphertext replay by another node. Nodes with incorrect keys fail to decrypt and never join the cluster.

Adaptive Gossip Fanout

As clusters grow, fixed gossip fanout becomes inefficient. This implementation scales gossip fanout with cluster size:

  • Fanout = base × ceil(log₂(n))
  • Targets per round = base × ceil(log₂(n))

For a 10-node cluster: 1 target, 50 entries. For a 100-node cluster: 7 targets, 350 entries.

Performance Benchmarks

For a 10-node cluster: merging a full gossip digest takes ~1 microsecond, merging a single entry ~780 nanoseconds.

At 100 nodes: digest merging increases to ~11 microseconds, individual entry merges ~7 microseconds.

For a 1000-node cluster: digest merge cost ~128 microseconds, individual entry merges below 100 microseconds.

Message encoding and decoding: encoding a gossip message containing ten entries takes ~320 nanoseconds, decoding ~12 nanoseconds.

CPU overhead is negligible compared to network latency. In practical deployments the dominant cost will almost always be RTT and packet scheduling, not protocol computation.

Testing Strategy

More than 200 unit tests cover individual components. Integration tests simulate multi-node clusters running the full protocol stack. Property-based tests (using proptest) verify: idempotency, commutativity, dead terminality, and convergence invariants.

The binary wire format is fuzz tested using cargo-fuzz with libFuzzer. Both targets executed over eight million iterations within 30 seconds without failures.

What This Project Reveals

Building a gossip membership protocol highlights fundamental distributed systems truths:

  • Failure detection is probabilistic
  • Epidemic dissemination enables scalable convergence
  • Incarnation numbers allow refutation without coordination
  • Dead-terminality prevents stale resurrection
  • Anti-entropy guarantees convergence under loss
  • Protocol versioning protects forward compatibility

Cluster membership is not just discovery. It is a failure detection problem, a state convergence problem, a wire protocol design exercise, and a distributed systems challenge.

Closing Thoughts

This project is not intended to replace production-grade systems. Instead it makes the mechanics of cluster membership transparent.

After implementing SWIM from scratch: every failover has visible mechanics, every partition has a traceable detection path, every gossip round has measurable convergence. Membership stops being invisible infrastructure — it becomes a system you can reason about byte by byte.