AWG2_harness/docs/IMPLEMENTATION_SPEC.md

AWG2 Harness implementation spec

1. Architectural role

AWG2 Harness production package contains two Rust binaries:

• awg2-harness — host-side traffic/firewall/status/swap orchestrator;
• awg2-reboot-scheduler — dedicated reboot scheduler service.

awg2-harness remains the main host-side multi-orchestrator for a dedicated Debian 13 amd64 AmneziaWG/AWG 2.0 server.

It owns the infrastructure state needed by the VPN service:

• Amnezia container discovery;
• container network namespace discovery;
• AWG/WG interface discovery;
• peer discovery;
• traffic-control desired state;
• IFB ingress shaping;
• nftables connection-count anti-torrent guard;
• Docker lifecycle recovery;
• periodic reconciliation;
• status/diagnostics;
• planned reboot;
• emergency swap setup.

It does not replace Amnezia. It controls the surrounding host-side infrastructure state required by the Amnezia service.

2. Process model

Two systemd services run:

/usr/sbin/awg2-harness --config /etc/awg2-harness/config.toml daemon
/usr/sbin/awg2-reboot-scheduler --config /etc/awg2-harness/config.toml

awg2-harness daemon starts as root and keeps running. On startup it performs initial reconcile. During runtime it reacts to Docker events and also runs a periodic reconcile loop.

Reboot scheduling authority belongs to awg2-reboot-scheduler service. awg2-harness does not schedule reboots internally.

Subcommands:

awg2-harness daemon
awg2-harness apply-limits
awg2-harness apply-network-tuning
awg2-harness inspect-network-tuning
awg2-harness apply-firewall
awg2-harness status
awg2-harness ensure-swap
awg2-harness validate-config
awg2-harness validate-runtime

apply-network-tuning/inspect-network-tuning относятся к отдельному host-level модулю sysctl_tuning. Этот модуль:

• управляет только sysctl параметрами хоста;
• пишет только /etc/sysctl.d/90-awg2-harness-vpn.conf;
• применяет runtime-изменения точечно (sysctl -w) только для drifted keys;
• не использует /etc/sysctl.conf и sysctl --system;
• не управляет udp_wmem_min.

3. Module layout

Recommended Rust module split:

src/
├── main.rs
├── cli.rs
├── config.rs
├── daemon.rs
├── lock.rs
├── docker.rs
├── netns.rs
├── iface.rs
├── wg.rs
├── traffic/
│   ├── mod.rs
│   ├── model.rs
│   ├── inspect.rs
│   ├── apply.rs
│   ├── parse_tc.rs
│   └── commands.rs
├── firewall/
│   ├── mod.rs
│   ├── model.rs
│   ├── inspect.rs
│   ├── apply.rs
│   └── nft.rs
├── status.rs
├── swap.rs
└── error.rs

4. Core data model

The traffic module should operate on explicit model structs, not raw strings passed around the codebase.

Conceptual model:

struct RuntimeTarget {
    container_name: String,
    container_id: String,
    container_pid: u32,
    netns_path: PathBuf,
    interface_name: String,
}

struct Peer {
    public_key: Option<String>,
    ipv4: Ipv4Addr,
    cidr: u8, // normally 32
    latest_handshake: Option<HandshakeAge>,
    transfer_rx: Option<ByteSize>,
    transfer_tx: Option<ByteSize>,
}

struct TrafficDesiredState {
    iface: String,
    ifb_name: String,
    root_rate: Rate,
    peer_limit: Rate,
    class_start_id: u16,
    peers: Vec<PeerTrafficClass>,
}

struct PeerTrafficClass {
    peer_ip: Ipv4Addr,
    class_id: u16,
}

Class assignment must be deterministic. Peer IP strings are sorted lexicographically, then assigned class_start_id + index. Do not silently switch this to numeric IPv4 sorting: the production class-id contract maps 10.8.1.10 before 10.8.1.2 because sorting is textual, not numeric.

5. Namespace strategy

Preferred strategy:

1. Detect running container through Docker API or docker ps / docker inspect.
2. Read container PID.
3. Use /proc/<pid>/ns/net as the target namespace.
4. Execute network operations in that namespace.

Implementation options:

• current path: spawn host tools with namespace entry, for example nsenter -t <pid> -n tc ..., nsenter -t <pid> -n ip ...;
• later/native: use netlink crates and explicit namespace switching where practical.

Peer state is the one exception and must never be read with a host-side binary. AmneziaWG is a userspace implementation whose UAPI socket lives under RUNSTATEDIR/amneziawg/<iface>.sock inside the container mount namespace, while host wireguard-tools looks under /var/run/wireguard/. nsenter -n enters only the network namespace, so a host wg reaches neither the socket nor a kernel device and always returns an empty peer list. Peer discovery therefore uses docker exec exclusively, and wireguard-tools is not a host dependency.

docker exec should not be the architectural default. It can remain as compatibility fallback if nsenter/host tools are unavailable, but the design assumption is that the Rust binary has host authority and controls the relevant namespace.

This also reduces operational risk: if the Amnezia container drops tc or iproute2, host-side tools can still apply rules inside the container netns.

6. Discovery flow

discover_target():

1. Load ordered amnezia.container_names.
2. Find the first running matching container.
3. Get container ID, status and PID.
4. Build netns_path.
5. Detect interface in that namespace.
6. Return RuntimeTarget.

Interface detection order:

1. Exact names from amnezia.interface_candidates.
2. Pattern scan from amnezia.interface_patterns.
3. Fallback: first non-loopback interface with IPv4 address, only if it is clearly inside the Amnezia namespace.
4. If ambiguous, return an explicit error and do not apply rules.

7. Peer discovery

Transport is always docker exec <container> — see section 5 for why a host-side wg cannot work.

Binary resolution uses ordered amnezia.wg_binaries, default ["awg", "wg"]. The first binary that exits successfully wins. The official AmneziaWG image ships /usr/bin/awg and symlinks /usr/bin/wg to it, so awg is the real dependency; wg remains for plain WireGuard containers where awg does not exist. There is no probe call: the resolution is a property of the command that is being run anyway.

Primary command:

awg show all allowed-ips

Parsing rules:

• parse lines structurally into peer public key and allowed IPs;
• a peer's allowed ips are space-separated in this command, comma-separated in show all dump and ", "-separated in plain show; all tokens from column 3 onward must be consumed, otherwise peers with more than one allowed IP are shaped only on their first address;
• keep IPv4 /32 addresses within amnezia.peer_ipv4_cidr;
• include offline peers; online state is not a requirement for shaping;
• sort peer IP strings lexicographically before class assignment to preserve the current lexicographic class-id contract.

Fallback:

awg show

The fallback parser should support standard wg show text output by extracting allowed ips: lines.

show all dump line classification:

• a line is a peer line only if it has at least 9 tab-separated columns, column 5 (allowed ips) is (none) or a comma-separated CIDR list, and columns 6-8 (handshake, rx, tx) all parse as integers;
• column count is not a discriminator on its own: the AmneziaWG interface line carries the AWG parameters and is longer than a peer line (14 columns for AWG 2.0, 28 for AWG 3.0), and its handshake/rx/tx positions hold junk-packet sizes that parse as integers;
• this matters beyond correctness — column 2 of an interface line is the device private key, so a misclassified interface line would surface it as a peer public key;
• AWG 3.0 prints protocol timers as ranges (20-30). The peer keepalive column is not read, and a range value must not break parsing.

Execution policy for production:

• do not use shell wrappers (sh -c) for wg invocation;
• do not suppress stderr via redirection in command strings;
• execute the binary directly and handle non-zero exit codes in Rust;
• amnezia.wg_binaries values are executed inside the container and are validated as bare command names: non-empty, no whitespace, no path separators.

8. Desired traffic topology

The expected topology is the AWG2 traffic topology contract.

Egress:

tc qdisc add dev <iface> root handle 1: htb
tc class add dev <iface> parent 1: classid 1:1 htb rate <root_rate> ceil <root_rate>
for each peer:
  tc class add dev <iface> parent 1:1 classid 1:<id> htb rate <peer_limit> ceil <peer_limit>
  tc qdisc add dev <iface> parent 1:<id> fq_codel
  tc filter add dev <iface> parent 1: protocol ip u32 match ip dst <peer_ip> flowid 1:<id>

Ingress through IFB:

modprobe ifb
ip link add name <ifb_name> type ifb
ip link set dev <ifb_name> up
tc qdisc add dev <iface> handle ffff: ingress
tc filter add dev <iface> parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev <ifb_name>
tc qdisc add dev <ifb_name> root handle 2: htb
tc class add dev <ifb_name> parent 2: classid 2:1 htb rate <root_rate> ceil <root_rate>
for each peer:
  tc class add dev <ifb_name> parent 2:1 classid 2:<id> htb rate <peer_limit> ceil <peer_limit>
  tc qdisc add dev <ifb_name> parent 2:<id> fq_codel
  tc filter add dev <ifb_name> parent 2: protocol ip u32 match ip src <peer_ip> flowid 2:<id>

9. Reconcile algorithm

Pseudo-flow:

reconcile_traffic():
  acquire global lock
  target = discover_target()
  peers = discover_peers(target)
  if peers empty:
      log and return NoPeers
  desired = build_desired_state(target, peers, config)
  actual = inspect_actual_state(target, desired)
  drift = diff(desired, actual)
  firewall_desired = build_firewall_desired_state(target, config)
  firewall_actual = inspect_firewall_state(target, firewall_desired)
  firewall_drift = diff(firewall_desired, firewall_actual)

  if drift.none and firewall_drift.none:
      log "rules already up-to-date"
      return NoChange

  if drift.exists:
      teardown_owned_topology(target, desired)
      apply_full_topology(target, desired)
      verify_after_apply(target, desired)

  if firewall_drift.exists:
      apply_owned_firewall_table(target, firewall_desired)
      verify_firewall_after_apply(target, firewall_desired)

For production, controlled full rebuild on drift is the only supported path.

Parser hardening requirements:

• tc filter show must be parsed block-by-block (state-machine), not by naive token scanning across unrelated lines;
• mapping peer_ip -> class_id must be validated for both egress (1:) and ingress (2:) sides;
• verification must fail when classes/filters exist but mapping differs from deterministic assignment.

Apply/verify policy:

• all apply steps are explicit command invocations;
• expected idempotency conflicts (already exists) may be tolerated selectively;
• any other apply error is fatal;
• verify_after_apply is mandatory for both traffic and anti-torrent subsystems.

Anti-torrent observability contract:

• named counter antitorrent_dropped_v4 must be present in owned table;
• status output must expose both dropped packets and dropped bytes.

10. Owned state and teardown rules

The binary owns:

• root qdisc 1: on selected AWG/WG interface;
• ingress qdisc ffff: on selected AWG/WG interface when traffic limits are enabled;
• IFB device named traffic.ifb_name, default ifb0, in the target namespace;
• root qdisc 2: on IFB;
• classes and filters under these handles.

Teardown may delete and recreate these owned objects. Because the server is dedicated, this is acceptable and preferable to leaving stale or partial rules.

The binary should not blindly flush unrelated host firewall rules, Docker NAT rules, or unrelated interfaces. Firewall ownership must be explicit and scoped to the AWG2 Harness table, for example table inet awg2_harness, inside the target network namespace.

11. Anti-torrent firewall guard

This is part of the AWG2 production contract. It fits the multi-orchestrator model because the server is dedicated and the binary is allowed to own host/network rules required for the VPN product. The rule must still be scoped narrowly and observable.

Goal: limit concurrent conntrack flows per VPN peer IP. This is not torrent protocol detection. It is a generic connection-count guard that makes high-peer-count P2P workloads unattractive while leaving normal browsing, streaming, gaming and messaging traffic unaffected.

The rule must not be implemented as a global subnet limit:

ip saddr 10.8.1.0/24 ct count over 200 drop

That form can become one shared counter for the whole matched traffic set, depending on how it is written and where it is placed. For per-peer behavior, key the stateful object by source VPN IP. Recommended shape:

table inet awg2_harness {
  counter antitorrent_dropped_v4 {}

  chain forward_antitorrent {
    type filter hook forward priority filter; policy accept;

    ip saddr <peer_ipv4_cidr> ct state new \
      meter awg2_peer_connlimit_v4 { ip saddr ct count over <max_connections_per_peer> } \
      counter name antitorrent_dropped_v4 \
      drop
  }
}

Target namespace: the same Amnezia network namespace where tc shaping is applied. Do not assume the initial host namespace sees original peer source addresses. If Amnezia performs NAT inside the container namespace, the host Docker bridge may only see the container address, not 10.8.1.x.

Default threshold: 100 concurrent flows per peer IP. 150 is the conservative compatibility setting if there are complaints from unusually connection-heavy WebSocket/WebRTC/trading-browser workloads. 70 is more aggressive and should only be used after real traffic observation.

Rule placement: use forward hook with priority filter or any priority after conntrack lookup. Restrict evaluation to ct state new so established flows continue normally and only new flow attempts beyond the threshold are dropped.

Owned state:

• table inet awg2_harness;
• chain forward_antitorrent;
• counter antitorrent_dropped_v4;
• meter/set state created by the rule.

Status must show whether the table/chain/rule exists, the configured threshold, the peer CIDR, and the drop counter. It should also expose host conntrack pressure from net.netfilter.nf_conntrack_count and net.netfilter.nf_conntrack_max when available.

If anti_torrent.enabled = false, the daemon should remove or leave inactive only the AWG2-owned anti-torrent table according to config policy. It must never flush the global nftables ruleset.

12. Actual-state inspection

inspect_actual_state() must collect enough information to diagnose drift:

• qdisc on <iface>;
• classes on <iface>;
• qdiscs attached to per-peer classes on <iface>;
• filters under parent 1: on <iface>;
• ingress qdisc on <iface>;
• ingress redirect filter on <iface>;
• existence and state of <ifb_name>;
• qdisc on <ifb_name>;
• classes on <ifb_name>;
• filters under parent 2: on <ifb_name>.

status should display a compact summary and optionally a verbose mode for raw tc sections.

13. Docker events and periodic reconciliation

The daemon should use two triggers:

1. Docker events: container start/restart signals cause delayed reconcile after interface readiness.
2. Timer: periodic reconcile every traffic.reconcile_interval_secs.

The timer is mandatory. Events can be missed during daemon restart, Docker restart or transient errors.

Readiness check must not hardcode awg0. It must call the shared interface detector until timeout.

14. Locking

Only one traffic reconcile may run at a time across all subcommands and daemon tasks.

Recommended lock:

/run/awg2-harness/traffic.lock

Behavior:

• daemon skips overlapping reconcile and logs it;
• manual apply-limits either waits briefly or exits with a clear message;
• lock release is automatic when process exits.

15. Status output contract

awg2-harness status should show:

System:
  service: active/inactive/unknown
  docker: available/unavailable
  container: name, id, status, pid
  netns: /proc/<pid>/ns/net
  interface: awg0/wg0/...

Peers:
  total: N
  each peer: ip, optional public key prefix, handshake, rx/tx

Traffic desired:
  peer_limit, root_rate, ifb_name, class_start_id

Traffic actual:
  egress qdisc/classes/filters: ok/drift
  ingress redirect: ok/drift
  ifb topology: ok/drift
  stale peers: list
  missing peers: list

Anti-torrent actual:
  enabled: true/false
  nft table/chain/rule: ok/drift
  max_connections_per_peer
  dropped_new_flows counter
  conntrack_count / conntrack_max if available

Last actions:
  last reconcile result
  last error if any

A generic status banner like “system ready” is not enough. Rust status must be falsifiable: it should clearly say when state is partial.

16. Reboot module integration

Production reboot path is external binary/service awg2-reboot-scheduler:

• window from config, not constants;
• random point inside reboot window (including cross-midnight windows);
• systemctl --no-block reboot remains;
• disabled when reboot.enabled = false;
• planned reboot timestamp is persisted in /run/awg2-harness/reboot.plan.

17. Swap module

ensure-swap desired state:

• path from config, default /swapfile;
• size 2048 MB;
• mode 0600;
• mkswap completed;
• active in swapon --show;
• one /etc/fstab entry with configured priority;
• vm.swappiness = 1 persisted and applied.

When swap.enabled=true, AWG2 Harness owns host swap state exclusively. The desired state is exactly one active swap entry: swap.path. All other active swap devices/files are disabled via swapoff. All non-comment swap entries in /etc/fstab are removed and replaced by one managed AWG2 entry. Regular-file swapfiles that are not the configured swap.path may be deleted only after successful swapoff. Block devices are never deleted, only disabled and removed from fstab. The operation must be idempotent.

18. Testing

Unit tests:

• config validation, including amnezia.wg_binaries shape;
• awg|wg show all allowed-ips parser, including multiple space-separated allowed ips;
• awg|wg show fallback parser;
• awg|wg show all dump peer/interface line classification against AWG 3.0 (28 columns), AWG 2.0 (14 columns) and plain WireGuard (5 columns) interface lines;
• wg binary resolution order and the error raised when no binary is available;
• CIDR filtering;
• deterministic class assignment with lexicographic class-id stability contract;
• nftables ruleset rendering and parsing for the AWG2-owned table;
• tc parser for qdisc/class/filter snapshots;
• drift diff;
• reboot window calculation;
• fstab/sysctl idempotency.

Integration tests on real host:

• initial apply;
• repeated apply;
• partial state recovery after deleting only ingress qdisc;
• partial state recovery after deleting ifb0;
• container restart;
• Docker restart;
• host reboot;
• peer addition;
• peer deletion;
• status output in healthy and broken states;
• anti-torrent rule counter increments when a peer exceeds the configured concurrent-flow threshold;
• anti-torrent disable path does not flush unrelated nftables/Docker/Amnezia rules.