Introduction
A few months ago, I started my journey of self-hosting and building a homelab. Currently it’s just a Raspberry Pi that serves photos via JellyFin. In the world of home-labs, there is the common problem of: “how do I securely connect to my apps outside of my personal LAN?” I found Tailscale as a rather “out-of-the-box” solution for creating secure tunnels to my devices at home. And it comes with SSH!
But what was happening under the hood of Tailscale? I was quite interested about the implementation of Tailscale and wanted to understand it a bit more. So, like many other curious individuals out there, I queried ChatGPT hehe. How are nodes discovered in a Tailnet? LAN Peer Discovery was a topic that was mentioned by ChatGPT and peaked my interest. In the vain of “learn by doing”, I implemented a LAN Peer Discovery system in Go.
Background
LAN Peer Discovery solves the problem of “Who else is on my LAN?”. In my case, who else is on my home network? How does my personal computer know there are other devices on the network? Devices can broadcast their subnet IPs and setup direct connections with each other. These connections exist within the LAN, which eliminates the need for any connections out to an external server. In the context of Tailscale: are there devices on my tailnet that exist on the same LAN? If so, these devices can connect directly without having to route through the internet!
Architecture
Before delving deeper into the implementation details with Go, it’s important to have a high-level understanding of what I built.
The demo that I’ve setup in the project repository contains three nodes (or peers) that exist within the same Docker network. In this blog post, nodes and peers are used synonymously.
The nodes that send UDP announcement packets to the Broadcast IP found on en0, while also listening for others’ announcements from the Broadcast IP.
Each node holds state of its known peers found by Broadcast. State of each peer is updated by nodes probing its known list of peers.
Probing involves sending a UDP packet to another peer, and subsequently receiving back a UDP packet from the peer as a response. In this way, we can measure round-trip-time (RTT).
Broadcasting
So how do devices discover other devices that are on the same LAN? Similarly in life, devices need to speak up to make themselves known to others!
Technically, this occurs by announcing their subnet IP on a dedicated broadcast address of the chosen network interface.
On MacOS, the wifi runs on interface en0.
There are other interfaces that exist on MacOS on such as lo0, which is the loopback interface used for local traffic to 127.0.0.1 or ::1.
(side note: to list all interfaces on your macbook, run ifconfig in your terminal)
By looking through the en0 interface via ifconfig, the IPv4 address can be found for my device, as well as the broadcast address.
The IPv4 broadcast address is always the highest value of the subnet address space.
Programmatically, the IPv4 broadcast address can always be found by flipping all subnet host bits to 1:
//GOAL: Find broadcast address with ip = 192.168.4.99/22
//1. get the prefix length, ones convey the number of leading 1 bits in the subnet mask (i.e. the network portion)
ones, _ := ipnet.Mask.Size()
prefix := netip.PrefixFrom(netIpAddr, ones)
ones = prefix.Bits()
//2. create the subnet mask (leading 1s and remaining 0s, 0s convey the host address portion)
mask := uint32(0xffffffff << (32 - ones))
//3. get IPv4 address and convert to a big-endian uint32
ip4 := prefix.Addr().As4()
ipInt := binary.BigEndian.Uint32(ip4[:])
//4. turn on all host bits
broadcastInt := ipInt | ^mask
Broadcast Loop
After finding the broadcast address for en0, each node will execute a goroutine that contains an announce loop to periodically send announcements to the broadcast address.
The purpose of an announcement is to say: “Hello! I exist, I live at this IP address!. Remember, this helps with discovering other devices on the same LAN. I’m pretty sure products like Chromecast utilize LAN Peer Discovery to be able to find other eligible Chromecast devices on the network (don’t quote me on this!).
Here in the code, I took advantage of channels, tickers, and the parent Context to create the announce loop. Very refreshing implementation compared to what I do for work (C#, OOP).
//some code was retracted for brevity
func announceLoop(ctx context.Context, interfaces []netx.InterfaceInfo, privateKey ed25519.PrivateKey) {
t := time.NewTicker(AnnounceInterval)
defer t.Stop()
announce := wire.Announce{retracted...}
for {
select {
case <-ctx.Done():
return
case <-t.C:
//1. for each eligible interface (in a home network, it's usually just en0)
for _, iface := range interfaces {
//2. setup the announce body
a := announce
a.Addr = iface.IP
a.EpochMS = time.Now().UnixMilli()
...
//3. Sign the announcement and store signature in body (ed25519)
a.Sign(privateKey)
packet, _ := wire.Encode(a)
//4. Bind UDP socket listen at device's IP, remote address is broadcast address
remoteAddress := &net.UDPAddr{IP: iface.Broadcast.AsSlice(), Port: AnnouncePort}
listenAddress := &net.UDPAddr{IP: iface.IP.AsSlice(), Port: 0}
conn, err := net.ListenUDP("udp4", listenAddress)
...
//5. Send announcement packet to broadcast IP
_, _ = conn.WriteToUDP(packet, remoteAddress)
conn.Close()
}
}
}
}
Listen Loop
After broadcasting, each node will listen for broadcast announcements from other nodes.
We do this by creating a UDP connection at 0.0.0.0, often called the wildcard address.
Listening for UDP packets at 0.0.0.0 implies that we want to listen for any packets that are destined for any local address at the same port.
In this case, the port will be the same port that is used for broadcasting/announcing.
In the code, we use AnnouncePort (8291) which can be settable by an environment variable.
//create the UDP connection to listen to all local addresses at the specified port.
func mustUDPListen(port int) *net.UDPConn {
listenAddr := &net.UDPAddr{IP: net.IPv4zero, Port: port}
conn, err := net.ListenUDP("udp4", listenAddr)
if err != nil {
panic(err)
}
return conn
}
//the listen loop
func listenLoop(ctx context.Context, conn *net.UDPConn, bus *table.Bus, seenCache table.SeenCache) {
buffer := make([]byte, 1024)
for {
//1. timeout read after 5 seconds
_ = conn.SetReadDeadline(time.Now().Add(5 * time.Second))
n, _, err := conn.ReadFromUDP((buffer))
//2. check for deadline timeout or given parent context is done.
if ne, ok := err.(net.Error); ok && ne.Timeout() {
select {
case <-ctx.Done():
return
default:
continue
}
}
if err != nil {
log.Printf("error: %v\n", err)
continue
}
//2. decode/unmarshal data from the read buffer to an announce
announce, err := wire.Decode(buffer[:n])
/* some ommitted code for brevity */
//send announce information to the bus.AnnounceCh chan
bus.AnnounceCh <- table.Announce{ID: announce.ID, Address: netip.AddrPortFrom(announce.Addr, uint16(announce.UDPPort))}
}
}
Note: An alternative would be to bind to the host local IP for en0.
Message Passing through Channels
You may have noticed in the previous code block that some information from a broadcast/announcement was sent to a bus.AnnounceCh channel.
This is rather a fun and unique part of Go, in which concurrency is a first class citizen with the chan type.
So in this project, there are channels to update the Peers table and Seen cache.
The Peers table and Seen cache is owned by a Table struct.
The Table struct listens to multiple channels and the parent Context as a responsibility to update the records in the Peers table and Seen cache.
In other words, the Table struct is an actor with a goroutine to own its private state and channels to update its state.
// channels owned by Table
type Bus struct {
AnnounceCh chan Announce
ProbeRequestCh chan ProbeRequest
ProbeResponseCh chan ProbeResponse
ListPeersRequestCh chan ListPeersRequest
ListPeersResponseCh chan ListPeersResponse
}
type Table struct {
Peers map[string]*Peer
Seen SeenCache
}
func (t *Table) Loop(ctx context.Context, bus *Bus, cfg Config, now func() time.Time) {
tickProbe := time.NewTicker(cfg.ProbeEvery)
tickMaintenance := time.NewTicker((time.Second))
defer tickProbe.Stop()
defer tickMaintenance.Stop()
for {
select {
case a := <-bus.AnnounceCh:
//update a Peer's id, ip, and lastSeen in the Peers table
case r := <-bus.ProbeResponseCh:
//update round-trip-time after probing a Peer
case <-tickProbe.C:
//send a probeRequest to the Probe channel
case <-tickMaintenance.C:
//check and update statuses of Peers at an interval
case <-bus.ListPeersRequestCh:
//send a list of all Peers to the ListPeersResponseCh, used for http GET /peers
case <-ctx.Done():
return
}
}
}
Probing Peers
After discovering other peers in the network with the Broadcast and Listen loops, each node probes its known Peers to keep a record of them.
This one-to-one communication is also known as Unicast.
Specifically, a Peers table is a state that the Table maintains.
The table keeps track of peers’: id (given by the peer), the round-trip-time (RTT), last seen, last probed, and status (unknown, healthy, suspect, down).
So, each node also contains a UDP server to send a UDP packet to its peers while also listening for UDP packets and echoing them back to its sender.
Therefore each node can calculate the RTT for each of its peers.
Specifically, calculations are done with an Exponentially Weighted Moving Average to smooth out any possible RTT outliers.
So there are two components of this workflow: listening then echo back to the sender, and sending UDP packets at a set interval.
Listen and echo is rather a straightforward implementation of setting up a UDP server to listen and send UDP packets:
func StartEchoServer(ctx context.Context, port int) error {
laddr := &net.UDPAddr{IP: net.IPv4zero, Port: port}
conn, err := net.ListenUDP("udp4", laddr)
//ommitted for brevity
//start a goroutine to listen for Context.Done()
//and then set a read deadline for the listen UDP connection and close it
go func() {
<-ctx.Done()
conn.SetReadDeadline(time.Now())
conn.Close()
}()
buf := make([]byte, 2048)
for {
n, raddr, err := conn.ReadFromUDPAddrPort(buf)
//ommitted for brevity
if _, err := conn.WriteToUDPAddrPort(buf[:n], raddr); err != nil {
if ctx.Err() != nil {
return ctx.Err()
}
return err
}
}
}
Sending UDP packets at a set interval involves a pool of Probe workers.
These workers listen for messages from Table via the ProbeRequestCh channel.
Each message in ProbeRequestCh contains the id and address:port for a probe worker.
After each probe, a worker will send back a response to Table via ProbeResponseCh containing: OK, RTT, and When (time of probe).
type ProbeRequest struct {
ID string
Address netip.AddrPort
}
type ProbeResponse struct {
ID string
OK bool
RTT time.Duration
When time.Time
}
func probeWorker(ctx context.Context, bus *table.Bus) {
for {
select {
case request := <-bus.ProbeRequestCh:
duration, success := probe.Probe(request.Address)
resp := table.ProbeResponse{ID: request.ID, OK: success, RTT: duration, When: time.Now()}
select {
case bus.ProbeResponseCh <- resp:
case <-ctx.Done():
return
}
case <-ctx.Done():
return
}
}
}
Extra: Verifying Announcements
There is the question of: “Of the announcements from the Broadcast IP, how do we know if they are legit? In other words, how do we know if these packets weren’t tampered with?” Introducing signatures with ed25519! From my understanding, ed25519 is a method of creating digital signatures and also verifying them. General steps of signing and verifying with a private and public key-pair:
- Create a public and private key pair
- Use the private key to sign a body of data, yielding a signature
- Use the public key to verify the signature, if verification fails, the packet has been tampered with
In the context of this project, ed25519 is used to sign and verify the announcements from each node/peer:
func (a *Announce) Sign(privateKey ed25519.PrivateKey) {
publicKey := privateKey.Public().(ed25519.PublicKey)
if len(a.PublicKey) == 0 {
a.PublicKey = append([]byte(nil), publicKey...)
}
if a.ID == "" {
a.ID = hex.EncodeToString(publicKey)
}
a.Signature = ed25519.Sign(privateKey, a.SignBytes())
}
func (a *Announce) Verify() bool {
if len(a.PublicKey) != ed25519.PublicKeySize || len(a.Signature) != ed25519.SignatureSize {
return false
}
if a.ID != hex.EncodeToString(a.PublicKey) {
return false
}
return ed25519.Verify(ed25519.PublicKey(a.PublicKey), a.SignBytes(), a.Signature)
}
Demo
A self-demo of this project can be done by cloning the project and running the docker environment.
docker compose build && docker compose up -d
Access to a node’s Peer table can be reached at GET /peers.
I know, this isn’t a very authentic way of showcasing LAN Peer Discovery, but how do I debug my code with one machine? haha
Conclusion
Learning about LAN Peer Discovery was quite fun and interesting.
I don’t have much of a background in networks (besides what I learned in my previous post about DNS).
So, all the topics covered in this writing was all new to me: the differentiation of unicast, broadcast, and multicast, network interfaces like en0, computing broadcast addresses with a subnet IP and mask, 0.0.0.0, and ed25519.
This type of system can also be tied to many other technologies out there.
For example, technologies like Spotify Connect and Chromecast utilize LAN Peer Discovery to enable casting media to other devices.
Address Resolution Protocol (ARP) also utilizes LAN Peer Discovery to find MAC addresses of other devices on the network.
This was also the first project I’ve implemented in Go. I’ve done some smaller learnings for Go like “A Tour of Go”. It was really fun and refreshing to dive into Go’s channels. I work with C# and the .NET Stack for my day job, so it was really fun to implement state management and concurrency via message passing with channels.
Currently, this blog holds a trend of learning about network systems topics. It feels rather practical to learn about network systems, due to the fact that networks are everywhere in the world of computing. I may or may not be continuing this trend, but what I know is this: learning about networks is fun, and I’ll continue learning about whatever grabs my curiosity. Thanks for reading, cya later.