Building a SWIM Gossip Membership Protocol from Scratch in Rust

Source Code:

https://github.com/cb7chaitanya/networking/tree/main/gossip-membership

Distributed systems often treat cluster membership as invisible infrastructure.

Something that just exists inside service meshes, distributed databases, and orchestration platforms.

But behind every node discovery, every failover, every rolling deployment lies a protocol solving a deceptively simple question:

Who else in the cluster is alive right now?

This article walks through the design and implementation of a SWIM-style gossip membership protocol written entirely in Rust.

The system includes:

• Custom binary wire format with protocol versioning
• Two-phase failure detection (direct + indirect probes)
• Epidemic gossip dissemination with adaptive fanout
• Incarnation-based suspicion refutation
• Anti-entropy full-table synchronization with MTU-safe chunking
• ChaCha20-Poly1305 encryption with per-node authentication
• Network simulation (loss, partitions, delay, reordering)
• Prometheus + Grafana observability
• Property tests, fuzz testing, and performance benchmarks

This is not a wrapper around an existing library.

It is a ground-up implementation of the SWIM protocol.

Why Build a Membership Protocol?

Most distributed systems rely on membership frameworks such as:

• HashiCorp Memberlist
• Consul’s SWIM implementation
• etcd discovery

These systems handle:

• Node discovery
• Failure detection
• State dissemination
• Cluster view convergence

In this project, the protocol performs all of these responsibilities itself.

That means:

• Probes are constructed and parsed manually
• Failure detection is implemented as a SWIM state machine
• Gossip dissemination follows epidemic spreading rules
• Merge rules enforce ordering on node state
• Partitions and failures must be handled explicitly

Implementing membership from scratch makes the system fully transparent.

You can trace every cluster state transition down to the exact byte sequence that caused it.

SWIM Architecture Overview

SWIM maintains cluster membership in a fully decentralized way.

Node A                    Node B                    Node C
  │                         │                         │
  │──── GOSSIP ────────────►│                         │
  │     (membership digest) │──── GOSSIP ────────────►│
  │                         │     (merged view)       │
  │◄──── GOSSIP ─────────────────────────────────────│
  │     (C's merged view)   │                         │
plain text

Every node maintains a local membership table.

Periodically each node:

1. Selects a random peer
2. Sends its view of the cluster
3. The receiver merges the membership entries
4. Updates spread epidemically across the cluster

There is:

• No leader
• No central coordinator
• No consensus algorithm

Just randomized peer-to-peer communication that eventually converges.

The expected convergence time is:

O(log n) gossip rounds
plain text

Wire Format (Byte-Level Design)

Each gossip message contains:

• 24-byte fixed header
• Variable-length membership payload

Header Structure

Each message begins with a 24-byte fixed header that describes the payload and ensures integrity.

The header fields are laid out sequentially in memory:

• Version (1 byte) — protocol version. Currently v1. Messages with unsupported versions are rejected before decoding.
• Kind (1 byte) — message type identifier such as GOSSIP, PING, or ACK.
• Payload Length (2 bytes) — length of the membership payload that follows the header.
• Sender ID (8 bytes) — unique identifier of the node sending the message.
• Heartbeat (4 bytes) — logical clock used to track liveness updates.
• Incarnation (4 bytes) — monotonic counter used to refute suspicion.
• Flags (1 byte) — reserved for optional protocol features.
• Reserved (1 byte) — padding used to maintain 16-bit alignment for checksum calculation.
• Checksum (2 bytes) — integrity check using the RFC 1071 checksum algorithm.

All multi-byte integers are encoded using big-endian byte order.

The reserved byte ensures the header remains 16-bit aligned, which simplifies checksum computation and matches the alignment expectations of traditional internet protocols like TCP and IP.

Message Types

The protocol currently supports six message kinds:

GOSSIP       (0x01)
PING         (0x02)
PING_REQ     (0x03)
ACK          (0x04)
LEAVE        (0x05)
ANTI_ENTROPY (0x06)
plain text

Because the protocol includes a version field, future changes to the wire format can be introduced without breaking compatibility.

Unsupported versions are rejected before checksum validation.

Membership Entry Encoding

Each membership entry encodes a node’s current state.

IPv4 entries are 24 bytes.

IPv6 entries are 36 bytes.

Bytes 0-7   node_id (u64)
Bytes 8-11  heartbeat (u32)
Bytes 12-15 incarnation (u32)
Byte  16    status
Byte  17    addr_family
Bytes 18+   ip + port
plain text

Status values:

0 = Alive
1 = Suspect
2 = Dead
plain text

With this layout, a single UDP datagram can easily carry dozens of membership updates.

Failure Detection: Two-Phase Probing

Unlike traditional heartbeat systems, SWIM performs active probing.

Each probe cycle:

1. Select a random peer
2. Send PING
3. Wait for ACK
4. If timeout → escalate to indirect probing

Indirect probing asks other nodes to verify connectivity.

Node A                    Node B                    Node C
  │                         │                         │
  │──── PING ──────────────►│  (no response)          │
  │                         │                         │
  │──── PING_REQ ──────────────────────────────────►│
  │                         │◄──── PING ────────────│
  │                         │──── ACK ─────────────►│
  │◄──────────── ACK ──────────────────────────────│
plain text

This mechanism distinguishes between:

• Node failure
• Network partition

Without indirect probes, false positives would be extremely common.

Suspicion and Dead Transitions

Nodes are not declared dead immediately.

Instead they enter a Suspect state.

Alive → Suspect → Dead
plain text

The suspicion timeout scales with cluster size:

timeout ∝ log₂(cluster_size)
plain text

A deterministic jitter derived from (observer, suspect) ensures that nodes do not simultaneously declare failure, preventing cluster-wide cascades.

Incarnation Refutation

If a node learns that it has been suspected, it does not inflate its heartbeat.

Instead it increments its incarnation number.

Alive (incarnation 4)
→ suspected
→ Alive (incarnation 5)
plain text

This allows nodes to refute suspicion without coordination.

Incarnation numbers act as a monotonic version counter for node identity.

Merge Rules

When gossip arrives, nodes must decide whether the incoming information is newer.

The merge rules form a strict ordering:

1. Higher incarnation always wins
2. Dead at same incarnation is terminal
3. Same incarnation → higher heartbeat wins
4. Same incarnation + heartbeat → Dead > Suspect > Alive
plain text

These rules ensure deterministic convergence across the cluster.

Anti-Entropy: Repairing Gossip Gaps

Regular gossip spreads random subsets of entries.

Under heavy packet loss, some nodes may never receive certain updates.

Anti-entropy fixes this by periodically sending the entire membership table.

The challenge is packet size.

A 1000-node cluster produces ~24 KB of membership data — larger than the UDP MTU.

So the table is split into MTU-safe chunks.

table_version
chunk_index
total_chunks
entries
plain text

Receivers reassemble chunks before applying merge rules.

Incomplete chunk sets expire after a timeout.

Encryption

Cluster traffic can be encrypted using ChaCha20-Poly1305.

The sender ID is bound as Additional Authenticated Data (AAD).

[nonce][sender_id][ciphertext][authentication tag]
plain text

This prevents ciphertext replay by another node.

Nodes with incorrect keys fail to decrypt and never join the cluster.

Adaptive Gossip Fanout

As clusters grow, fixed gossip fanout becomes inefficient.

This implementation scales gossip fanout with cluster size.

Entries per message:

fanout = base * ceil(log₂(n))
plain text

Targets per round:

targets = base * ceil(log₂(n))
plain text

For example:

Cluster SizeTargetsEntries10 nodes150100 nodes7350

This maintains rapid convergence as the cluster expands.

Network Simulation

Testing distributed systems against real networks is unreliable.

A built-in simulator introduces:

• Packet loss
• Network partitions
• Artificial delay
• Packet reordering

All randomness uses a deterministic PRNG, making tests reproducible.

This allows simulation of scenarios such as:

• Split-brain clusters
• Partition healing
• Join-during-partition
• Convergence under 50% packet loss

Observability

The protocol exposes operational endpoints:

/healthz
/readyz
/membership
plain text

Prometheus metrics:

swim_gossip_rounds_total
swim_members_alive
swim_probe_failures_total
swim_rate_limited_total
plain text

A Grafana dashboard visualizes:

• Gossip throughput
• Failure detection
• Membership evolution
• Merge outcomes

Performance Benchmarks

Performance measurements were collected using Criterion benchmarks targeting the critical paths of the protocol.

Even for large cluster sizes, the protocol operations remain extremely lightweight.

For a 10-node cluster, merging a full gossip digest takes roughly 1 microsecond, while merging a single membership entry completes in around 780 nanoseconds.

At 100 nodes, digest merging increases to roughly 11 microseconds, with individual entry merges averaging around 7 microseconds.

For a 1000-node cluster, the digest merge cost rises to approximately 128 microseconds, while individual entry merges remain below 100 microseconds.

Message encoding and decoding are significantly cheaper:

• Encoding a gossip message containing ten entries takes roughly 320 nanoseconds.
• Decoding a message typically takes ~12 nanoseconds.
• Encoding and decoding a PING probe takes approximately 22 ns / 12 ns respectively.

These measurements show that CPU overhead is negligible compared to network latency. In practical deployments the dominant cost will almost always be RTT and packet scheduling, not protocol computation.

Testing Strategy

The project uses multiple layers of testing to validate both correctness and robustness.

Unit Tests

More than 200 unit tests cover the individual components of the protocol.

These tests validate:

• membership merge rules
• wire format encoding and decoding
• peer selection logic
• encryption and authentication
• rate limiting behavior
• gossip message construction

Each module is tested independently to ensure correctness at the lowest level.

Integration Tests

Integration tests simulate multi-node clusters running the full protocol stack.

These tests verify system-level behaviors including:

• cluster convergence
• indirect probing
• node joins and graceful departures
• encrypted cluster communication
• failure detection
• recovery after partitions

These tests run multiple simulated nodes and observe whether their membership views eventually converge.

Property-Based Tests

Property testing (using proptest) verifies the mathematical properties of the merge algorithm.

The following invariants are tested:

• Idempotency — merging a state with itself produces the same state
• Commutativity — merging two states in either order produces the same result when incarnations match
• Dead terminality — once a node is dead at a given incarnation, it cannot be revived without a higher incarnation
• Convergence — applying updates in different orders leads to the same final state when incarnations are distinct

These tests help guarantee that the cluster will eventually reach a consistent view regardless of gossip ordering.

Fuzz Testing

The binary wire format is fuzz tested using cargo-fuzz with libFuzzer.

Two targets are exercised:

• Message::decode
• WireNodeEntry::decode

Arbitrary byte streams are fed into the decoders to ensure they never panic or crash when handling malformed input.

During testing, both targets executed over eight million iterations within 30 seconds without failures.

Valid decodes are also re-encoded and decoded again to confirm round-trip correctness.

Benchmarks

Finally, Criterion benchmarks measure the performance of the protocol under varying cluster sizes.

These benchmarks evaluate:

• membership merging performance
• message encoding and decoding
• gossip message construction
• anti-entropy chunk generation

This ensures the protocol remains efficient even as the cluster size scales to hundreds or thousands of nodes.

What This Project Reveals

Building a gossip membership protocol highlights several distributed systems truths:

• Failure detection is probabilistic
• Epidemic dissemination enables scalable convergence
• Incarnation numbers allow refutation without coordination
• Dead-terminality prevents stale resurrection
• Anti-entropy guarantees convergence under loss
• Protocol versioning protects forward compatibility

Cluster membership is not just discovery.

It is:

• A failure detection problem
• A state convergence problem
• A wire protocol design exercise
• A distributed systems challenge

Closing Thoughts

This project is not intended to replace production-grade systems.

Instead it makes the mechanics of cluster membership transparent.

After implementing SWIM from scratch:

• Every failover has visible mechanics
• Every partition has a traceable detection path
• Every gossip round has measurable convergence

Membership stops being invisible infrastructure.

It becomes a system you can reason about byte by byte.