Edge firewall¶

The cluster sits behind a highly-available OPNsense firewall pair that is the single north-south entry point and the out-of-band admin path. Unlike Cilium's host firewall — which polices the nodes themselves — the edge is the perimeter: it terminates the public uplinks, DNATs a fixed set of NodePorts to the workers, serves internal-only services on a floating VIP, and carries the WireGuard admin VPN.

The whole edge is config-as-code, driven declaratively by Ansible over the OPNsense REST API (the edge-ansible repo) rather than by Argo CD. It is the one part of the stack that lives outside the cluster, so it is push-managed, not GitOps-reconciled.

Topology¶

Two nodes — fw01 and fw02 — share a CARP virtual IP on the LAN that is the cluster's default gateway. Each node has its own public WAN uplink; the pair present one floating gateway to the LAN and fail over transparently. One node holds the VIP (the master) at a time; the other stands by.

flowchart TB
  internet(("Internet"))
  subgraph edge["OPNsense edge pair"]
    fw01["fw01 (CARP master)"]
    fw02["fw02 (CARP backup)"]
  end
  vip["LAN CARP VIP = default gateway"]
  subgraph cluster["Kubernetes"]
    workers["workers (NodePorts)"]
    cp["control plane (kube-api)"]
  end
  internet --> fw01
  internet --> fw02
  fw01 -. CARP .- fw02
  fw01 --> vip
  fw02 -. standby .- vip
  vip --> workers
  vip --> cp

What the edge serves¶

Path	Bound on	Purpose
Public HTTPS / HTTP	each node's WAN	terminate at the in-cluster ingress via NodePorts (HAProxy, PROXY-protocol v2)
Public SMTP	each node's WAN	inbound mail to the mail NodePort
kube-api	CARP VIP	external Kubernetes API, proxied to the control-plane nodes
Internal ingress	CARP VIP	split-horizon HTTPS for in-house names (e.g. the SSO login) and the `*.apps.int` admin portals
Recursive DNS	CARP VIP	Unbound resolver for the LAN and the admin VPN, with split-horizon overrides
WireGuard admin VPN	each node's WAN	the management path into the LAN
TURN / STUN	each node's WAN	real-time relay for calls, pinned per node

Inbound public traffic is filtered before it reaches a backend: a GeoIP allow-list and an IP-reputation block-list on the WAN, plus per-source connection-rate limits on the public frontends.

Single-active services on the master¶

Most services run identically on both nodes and follow the CARP VIP, so a failover is transparent. A few are single-active — only the node currently holding the VIP can serve them:

the public HTTPS / SMTP frontends and the external kube-api endpoint, and
the WireGuard admin VPN, which uses OPNsense's Depend on CARP so the tunnel runs only on the master and shuts down on the backup — one active tunnel rather than two racing for the same key.

These ride the master, and public DNS points at the active node. A separate break-glass WireGuard tunnel (its own port and subnet) is an independent way in if the regular path is down.

Config-as-code¶

edge-ansible manages the firewall entirely over the OPNsense REST API, one tagged section per subsystem (CARP, aliases, firewall rules, HAProxy, Unbound, WireGuard, syslog). It is declarative — anything not in the config is purged — so every change is dry-run with --check --diff first, and always applied to one node at a time.

A read-only drift-check runs from inside the cluster on a schedule: it re-runs the playbook in --check mode against both firewalls daily and alerts if the live config has diverged from the repo or a firewall is unreachable.

A few first-boot facts are outside the API's reach and are set once by hand (or baked into a seeded installer image): interface assignment and base IPs, the system hostname and DNS, NAT port-forwards, and the GeoIP licence key. These are covered in the edge-ansible bootstrapping guide.

For adopters¶

The cluster depends on the edge only through the NodePort contract — a fixed set of NodePort numbers the edge must DNAT to the workers, plus the LAN and VIP addresses. Any edge that honours that contract works; the OPNsense config-as-code is a reference implementation you can adopt or replace. If you bring your own firewall, keep its DNAT rules in lockstep with the NodePort map and reproduce the split-horizon DNS for any internal-only names.