Building a Reliable TCP-like Transport over UDP in Rust

Github link: https://github.com/cb7chaitanya/networking/tree/main/tcp-over-udp

TCP is one of the most foundational protocols in computing, yet many of us use it every day without truly understanding how it works. What makes it reliable? How does it manage packet loss, flow control, and congestion? And most importantly, how would you build something like it yourself?

In this article, we’ll walk through how I implemented a TCP-like transport over UDP from first principles in Rust, including reliability, sliding windows, congestion control, selective acknowledgment, teardown semantics, and even a network simulator for adversarial testing.

Why Build TCP on UDP?

On the surface, TCP just works. But its smooth API hides a complex interplay of timers, windows, and state machines that handle unreliable networks.

Meanwhile, UDP gives you nothing: no guarantee of delivery, order, or even integrity. That’s exactly what makes it a great place to build a transport:

• UDP’s simplicity forces you to handle reliability yourself.
• You control every byte on the wire.
• You can build and test behaviors TCP already has, but with full visibility.
• You can experiment with modern extensions like SACK or alternative congestion control.

Goals of This Project

This project aims to:

Implement TCP-like reliability and connection lifecycle

Handle packet loss, duplication, and reordering

Apply congestion control (TCP Reno)

Support selective repeat and SACK

Provide tooling to simulate real network adversity

Be a teaching implementation, not just a proof-of-concept

Project Anatomy

The implementation lives in ~7,000 lines of Rust across 16 modules. It’s structured cleanly, with each concern separated:

tcp-over-udp
├─ packet.rs
├─ socket.rs
├─ state.rs
├─ rtt.rs
├─ sender.rs
├─ receiver.rs
├─ gbn_sender.rs
├─ gbn_receiver.rs
├─ gbn_connection.rs
├─ congestion_control.rs
├─ persist_timer.rs
├─ simulator.rs
├─ main.rs
└─ lib.rs
plain text

There are 107 unit tests and 54 integration tests, covering corner cases, edge cases, and adversarial network conditions.

Wire Format: The Packet Header

The foundation is in packet.rs.

Transport packets consist of:

• 32-bit sequence & ack numbers
• 8-bit flags (SYN, ACK, FIN, RST)
• 16-bit window size
• 16-bit payload length
• 16-bit Internet checksum (RFC 1071)

Options are supported via a simple TLV extension, enabling features like:

• MSS (Maximum Segment Size) negotiation
• SACK blocks
• Padding / future extensions

Checksum logic ensures integrity without relying on the network.

RTT Estimation (RFC 6298)

Accurate timers are a prerequisite for reliable retransmission.

This stack implements a full adaptive RTO:

• Smoothed RTT (srtt)
• RTT variance (rttvar)
• RTO with exponential backoff
• Karn’s algorithm (ignore RTT samples on retransmit)

This allows the stack to vary timeout intervals based on actual network conditions.

Sliding Windows + Selective Repeat

Sliding windows are the heart of reliable transfer.

This implementation uses:

• Go-Back-N / cumulative ACKs
• Selective Repeat
• BTreeMap to store out-of-order packets efficiently
• Gap-fill cascade logic

Selective Repeat makes transfer robust under loss and reordering — a crucial feature for realistic networks.

Congestion Control (TCP Reno)

Reliable delivery isn’t enough. Sending fast without regard for network health causes collapse.

This stack implements:

• Slow start
• Congestion avoidance
• Fast retransmit
• Fast recovery

Congestion control is abstracted behind a trait, making it easy to swap in alternatives like CUBIC or BBR.

Lifecycle & Teardown (FIN / TIME_WAIT)

Connection teardown is surprisingly nuanced:

• ESTABLISHED → FIN_WAIT_1 → FIN_WAIT_2 → TIME_WAIT → CLOSED
• TIME_WAIT persists for 2×MSL to absorb straggler packets
• Duplicate FINs are absorbed
• Retransmitted FINs are acknowledged

Getting this right separates a pipe from a protocol.

Flow Control and Nagle’s Algorithm

Flow control protects the receiver from overload.

The stack computes dynamic receiver window based on:

rwnd = capacity − (app buffer + OOO buffer)
plain text

Nagle’s algorithm is implemented to reduce tiny packet emission under chatty workloads, buffering small writes until a full MSS or ACK drain.

Fault-Injection Simulator

One of the most powerful parts of this project is the simulator (simulator.rs):

It can inject:

• Packet loss
• Delay
• Reordering
• Corruption
• Duplicates
• Bandwidth throttling

All seeded with a deterministic PRNG for reproducibility.

This lets you test how the stack performs under adverse conditions, something most real stacks don’t expose.

Testing & Validation

The repository includes:

Unit tests

• Validation of packet encode/decode
• RTT estimator correctness
• Congestion control invariants

Integration tests

• Handshake correctness
• Sliding window behavior
• SACK handling
• Simulator-driven loss/reorder tests

Stress test

• Large transfers over lossy/jittery simulated links

Extensibility: Where You Can Take This

Once the core stack is solid, it can be extended in numerous directions:

Protocol Enhancements

• Window scaling (RFC 1323)
• Timestamps (PAWS, RTTM)
• Explicit Congestion Notification (ECN)
• Path MTU Discovery

Robustness Improvements

• RST handling
• Half-open connection validation
• SYN backlog + SYN cookies
• Keep-alive probes

Performance & Research

• Alternative congestion control (CUBIC/BBR)
• Multiplexed streams (QUIC-style)
• True byte-stream API
• Benchmark + comparison suite vs real TCP

What This Project Is and Isn’t

A working reliable transport over UDP

A deterministic testing infrastructure

A teaching implementation of protocol mechanics

Not a drop-in TCP replacement for OS stacks

Not hardened for Internet-wide load

It is a lens into how real transport protocols behave and why they behave that way.

Final Thoughts

This project was both a learning journey and a systems engineering exercise. It taught me:

• Timers must be orthogonal (loss vs flow control)
• Protocol state machines are subtle but deterministic
• Congestion control is more art than code
• Testing under adversity is essential

If you want to build solid networking intuition, implementation beats theory every time.