How we run a private network across 4 countries without a sysadmin

Our team is 9 people in 4 countries: Uzbekistan, Germany, Georgia, and the UK. We have no full-time sysadmin. I do most of the infrastructure work alongside writing code, and I want our setup to require as little of my attention as possible.

This is what we currently run, in detail, with the tradeoffs I have made and would make again.

The shape of the network

Three Hetzner boxes, one each in: - Falkenstein, Germany (FSN1) — primary application server - Helsinki, Finland (HEL1) — staging environment - Hillsboro, Oregon (HIL1) — for the two team members in the UK and one in Tashkent who get better latency to US East than to FSN1, weirdly

Each box runs the NexGuard VPN server. Each box is also a peer of the other two — they know about each other and route traffic between them when needed.

Each team member has the NexGuard client on their laptop. Their device is a peer of one or more of the servers, depending on what they need access to.

The routing

VPN subnet 100.128.1.0/16. Each server gets a /28: - FSN1: 100.128.1.0/28 - HEL1: 100.128.2.0/28 - HIL1: 100.128.3.0/28

Within each /28, the server is .1 and clients get .2 through .14. With 9 people and a few automated build agents, we have never come close to hitting the limit.

The mesh is configured so that traffic from FSN1's subnet to HEL1's subnet routes through the inter-server tunnel. We do not have to think about this — NexGuard handles route advertisement.

What this gives us

Any team member can SSH to any internal service on any server by hitting its 100.128.x.y address. We have /etc/hosts entries on each laptop that map service names to those IPs (this is the workaround for not having MagicDNS yet — it is not great, I will admit).

Cross-server services (e.g., the build agent in HIL1 needs to push artifacts to a registry in FSN1) work without exposing anything publicly.

Devices roam. My laptop can switch from office WiFi in Tashkent to a hotel WiFi in Berlin without me thinking about it. WireGuard handles the endpoint change. NexGuard's tunnel-mode fallback handles the cases where some hotel firewalls block UDP.

What I had to learn

DNS in a private network is a real problem. We are using /etc/hosts as a stopgap and it is annoying. I have considered running CoreDNS on each server but have not gotten around to it. This is genuinely the weakest part of our setup.

ICMP across the mesh works but you have to think about MTU. We had a strange bug where pings worked but TCP connections to certain ports stalled. Took me three hours to figure out it was MTU mismatch on one of the inter-server links. Now we set MTU explicitly to 1380 on every WireGuard interface.

If you want zero-trust, ACLs are not optional. By default, anyone on the VPN can reach anyone else. We have peer-level firewall rules so contractors can only reach the staging server, not production.

Backups need to be tested. Our VPN servers are stateless (no DB), so restoring one is just re-running the install script with the same config. But I have done this exactly once, in a controlled test, and I should do it again.

What I would do differently

I would set up DNS from day one. Doing it later means migrating dozens of references in our codebase from IPs to hostnames, and that has been on my todo list for six months.

I would put a status page somewhere that monitors each VPN server's reachability and the cross-server mesh. Right now I find out a server is down when someone Slacks me. This is dumb.

I would document the network architecture in a single place. The above is the first time I have written it down.

The unsexy truth

This setup has been running with effectively zero maintenance for eight months. I touched the configuration twice: once to add the HIL1 server, once to onboard a new contractor. Both took less than ten minutes.

That is the whole pitch. I do not want to think about VPN. I want it to work. With NexGuard, it works. With raw WireGuard, I would be writing this post about the time I spent debugging instead.

If you are running something similar (small distributed team, multiple regions, no sysadmin), I would love to hear about it. I am at davron211@gmail.com. We probably have problems in common.