AWG2_harness/docs/IMPLEMENTATION_SPEC.md

# AWG2 Harness implementation spec

## 1. Architectural role

AWG2 Harness production package contains two Rust binaries:

- `awg2-harness` — host-side traffic/firewall/status/swap orchestrator;
- `awg2-reboot-scheduler` — dedicated reboot scheduler service.

`awg2-harness` remains the main host-side multi-orchestrator for a dedicated Debian 13 amd64 AmneziaWG/AWG 2.0 server.

It owns the infrastructure state needed by the VPN service:

- Amnezia container discovery;
- container network namespace discovery;
- AWG/WG interface discovery;
- peer discovery;
- traffic-control desired state;
- IFB ingress shaping;
- nftables connection-count anti-torrent guard;
- Docker lifecycle recovery;
- periodic reconciliation;
- status/diagnostics;
- planned reboot;
- emergency swap setup.

It does not replace Amnezia. It controls the surrounding host-side infrastructure state required by the Amnezia service.

## 2. Process model

Two systemd services run:

```bash
/usr/sbin/awg2-harness --config /etc/awg2-harness/config.toml daemon
/usr/sbin/awg2-reboot-scheduler --config /etc/awg2-harness/config.toml
```

`awg2-harness` daemon starts as root and keeps running. On startup it performs initial reconcile. During runtime it reacts to Docker events and also runs a periodic reconcile loop.

Reboot scheduling authority belongs to `awg2-reboot-scheduler` service. `awg2-harness` does not schedule reboots internally.

Subcommands:

```bash
awg2-harness daemon
awg2-harness apply-limits
awg2-harness apply-network-tuning
awg2-harness inspect-network-tuning
awg2-harness apply-firewall
awg2-harness status
awg2-harness ensure-swap
awg2-harness validate-config
awg2-harness validate-runtime
```

`apply-network-tuning`/`inspect-network-tuning` относятся к отдельному host-level модулю `sysctl_tuning`.
Этот модуль:

- управляет только sysctl параметрами хоста;
- пишет только `/etc/sysctl.d/90-awg2-harness-vpn.conf`;
- применяет runtime-изменения точечно (`sysctl -w`) только для drifted keys;
- не использует `/etc/sysctl.conf` и `sysctl --system`;
- не управляет `udp_wmem_min`.

## 3. Module layout

Recommended Rust module split:

```text
src/
├── main.rs
├── cli.rs
├── config.rs
├── daemon.rs
├── lock.rs
├── docker.rs
├── netns.rs
├── iface.rs
├── wg.rs
├── traffic/
│   ├── mod.rs
│   ├── model.rs
│   ├── inspect.rs
│   ├── apply.rs
│   ├── parse_tc.rs
│   └── commands.rs
├── firewall/
│   ├── mod.rs
│   ├── model.rs
│   ├── inspect.rs
│   ├── apply.rs
│   └── nft.rs
├── status.rs
├── swap.rs
└── error.rs
```

## 4. Core data model

The traffic module should operate on explicit model structs, not raw strings passed around the codebase.

Conceptual model:

```rust
struct RuntimeTarget {
    container_name: String,
    container_id: String,
    container_pid: u32,
    netns_path: PathBuf,
    interface_name: String,
}

struct Peer {
    public_key: Option<String>,
    ipv4: Ipv4Addr,
    cidr: u8, // normally 32
    latest_handshake: Option<HandshakeAge>,
    transfer_rx: Option<ByteSize>,
    transfer_tx: Option<ByteSize>,
}

struct TrafficDesiredState {
    iface: String,
    ifb_name: String,
    root_rate: Rate,
    peer_limit: Rate,
    class_start_id: u16,
    peers: Vec<PeerTrafficClass>,
}

struct PeerTrafficClass {
    peer_ip: Ipv4Addr,
    class_id: u16,
}
```

Class assignment must be deterministic. Peer IP strings are sorted lexicographically, then assigned `class_start_id + index`. Do not silently switch this to numeric IPv4 sorting: the production class-id contract maps `10.8.1.10` before `10.8.1.2` because sorting is textual, not numeric.

## 5. Namespace strategy

Preferred strategy:

1. Detect running container through Docker API or `docker ps` / `docker inspect`.
2. Read container PID.
3. Use `/proc/<pid>/ns/net` as the target namespace.
4. Execute network operations in that namespace.

Implementation options:

- current path: spawn host tools with namespace entry, for example `nsenter -t <pid> -n tc ...`, `nsenter -t <pid> -n ip ...`;
- later/native: use netlink crates and explicit namespace switching where practical.

Peer state is the one exception and must never be read with a host-side binary.
AmneziaWG is a userspace implementation whose UAPI socket lives under
`RUNSTATEDIR/amneziawg/<iface>.sock` inside the container mount namespace, while host
`wireguard-tools` looks under `/var/run/wireguard/`. `nsenter -n` enters only the network
namespace, so a host `wg` reaches neither the socket nor a kernel device and always
returns an empty peer list. Peer discovery therefore uses `docker exec` exclusively, and
`wireguard-tools` is not a host dependency.

`docker exec` should not be the architectural default. It can remain as compatibility fallback if `nsenter`/host tools are unavailable, but the design assumption is that the Rust binary has host authority and controls the relevant namespace.

This also reduces operational risk: if the Amnezia container drops `tc` or `iproute2`, host-side tools can still apply rules inside the container netns.

## 6. Discovery flow

`discover_target()`:

1. Load ordered `amnezia.container_names`.
2. Find the first running matching container.
3. Get container ID, status and PID.
4. Build `netns_path`.
5. Detect interface in that namespace.
6. Return `RuntimeTarget`.

Interface detection order:

1. Exact names from `amnezia.interface_candidates`.
2. Pattern scan from `amnezia.interface_patterns`.
3. Fallback: first non-loopback interface with IPv4 address, only if it is clearly inside the Amnezia namespace.
4. If ambiguous, return an explicit error and do not apply rules.

## 7. Peer discovery

Transport is always `docker exec <container>` — see section 5 for why a host-side `wg`
cannot work.

Binary resolution uses ordered `amnezia.wg_binaries`, default `["awg", "wg"]`. The first
binary that exits successfully wins. The official AmneziaWG image ships `/usr/bin/awg` and
symlinks `/usr/bin/wg` to it, so `awg` is the real dependency; `wg` remains for plain
WireGuard containers where `awg` does not exist. There is no probe call: the resolution is
a property of the command that is being run anyway.

Primary command:

```bash
awg show all allowed-ips
```

Parsing rules:

- parse lines structurally into peer public key and allowed IPs;
- a peer's allowed ips are **space-separated** in this command, comma-separated in
  `show all dump` and `", "`-separated in plain `show`; all tokens from column 3 onward
  must be consumed, otherwise peers with more than one allowed IP are shaped only on their
  first address;
- keep IPv4 `/32` addresses within `amnezia.peer_ipv4_cidr`;
- include offline peers; online state is not a requirement for shaping;
- sort peer IP strings lexicographically before class assignment to preserve the current lexicographic class-id contract.

Fallback:

```bash
awg show
```

The fallback parser should support standard `wg show` text output by extracting `allowed ips:` lines.

`show all dump` line classification:

- a line is a peer line only if it has at least 9 tab-separated columns, column 5
  (allowed ips) is `(none)` or a comma-separated CIDR list, and columns 6-8 (handshake,
  rx, tx) all parse as integers;
- column count is not a discriminator on its own: the AmneziaWG interface line carries the
  AWG parameters and is *longer* than a peer line (14 columns for AWG 2.0, 28 for AWG 3.0),
  and its handshake/rx/tx positions hold junk-packet sizes that parse as integers;
- this matters beyond correctness — column 2 of an interface line is the device private
  key, so a misclassified interface line would surface it as a peer public key;
- AWG 3.0 prints protocol timers as ranges (`20-30`). The peer keepalive column is not
  read, and a range value must not break parsing.

Execution policy for production:

- do not use shell wrappers (`sh -c`) for `wg` invocation;
- do not suppress stderr via redirection in command strings;
- execute the binary directly and handle non-zero exit codes in Rust;
- `amnezia.wg_binaries` values are executed inside the container and are validated as bare
  command names: non-empty, no whitespace, no path separators.

## 8. Desired traffic topology

The expected topology is the AWG2 traffic topology contract.

Egress:

```text
tc qdisc add dev <iface> root handle 1: htb
tc class add dev <iface> parent 1: classid 1:1 htb rate <root_rate> ceil <root_rate>
for each peer:
  tc class add dev <iface> parent 1:1 classid 1:<id> htb rate <peer_limit> ceil <peer_limit>
  tc qdisc add dev <iface> parent 1:<id> fq_codel
  tc filter add dev <iface> parent 1: protocol ip u32 match ip dst <peer_ip> flowid 1:<id>
```

Ingress through IFB:

```text
modprobe ifb
ip link add name <ifb_name> type ifb
ip link set dev <ifb_name> up
tc qdisc add dev <iface> handle ffff: ingress
tc filter add dev <iface> parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev <ifb_name>
tc qdisc add dev <ifb_name> root handle 2: htb
tc class add dev <ifb_name> parent 2: classid 2:1 htb rate <root_rate> ceil <root_rate>
for each peer:
  tc class add dev <ifb_name> parent 2:1 classid 2:<id> htb rate <peer_limit> ceil <peer_limit>
  tc qdisc add dev <ifb_name> parent 2:<id> fq_codel
  tc filter add dev <ifb_name> parent 2: protocol ip u32 match ip src <peer_ip> flowid 2:<id>
```

## 9. Reconcile algorithm

Pseudo-flow:

```text
reconcile_traffic():
  acquire global lock
  target = discover_target()
  peers = discover_peers(target)
  if peers empty:
      log and return NoPeers
  desired = build_desired_state(target, peers, config)
  actual = inspect_actual_state(target, desired)
  drift = diff(desired, actual)
  firewall_desired = build_firewall_desired_state(target, config)
  firewall_actual = inspect_firewall_state(target, firewall_desired)
  firewall_drift = diff(firewall_desired, firewall_actual)

  if drift.none and firewall_drift.none:
      log "rules already up-to-date"
      return NoChange

  if drift.exists:
      teardown_owned_topology(target, desired)
      apply_full_topology(target, desired)
      verify_after_apply(target, desired)

  if firewall_drift.exists:
      apply_owned_firewall_table(target, firewall_desired)
      verify_firewall_after_apply(target, firewall_desired)
```

For production, controlled full rebuild on drift is the only supported path.

Parser hardening requirements:

- `tc filter show` must be parsed block-by-block (state-machine), not by naive token scanning across unrelated lines;
- mapping `peer_ip -> class_id` must be validated for both egress (`1:`) and ingress (`2:`) sides;
- verification must fail when classes/filters exist but mapping differs from deterministic assignment.

Apply/verify policy:

- all apply steps are explicit command invocations;
- expected idempotency conflicts (`already exists`) may be tolerated selectively;
- any other apply error is fatal;
- `verify_after_apply` is mandatory for both traffic and anti-torrent subsystems.

Anti-torrent observability contract:

- named counter `antitorrent_dropped_v4` must be present in owned table;
- status output must expose both dropped packets and dropped bytes.

## 10. Owned state and teardown rules

The binary owns:

- root qdisc `1:` on selected AWG/WG interface;
- ingress qdisc `ffff:` on selected AWG/WG interface when traffic limits are enabled;
- IFB device named `traffic.ifb_name`, default `ifb0`, in the target namespace;
- root qdisc `2:` on IFB;
- classes and filters under these handles.

Teardown may delete and recreate these owned objects. Because the server is dedicated, this is acceptable and preferable to leaving stale or partial rules.

The binary should not blindly flush unrelated host firewall rules, Docker NAT rules, or unrelated interfaces. Firewall ownership must be explicit and scoped to the AWG2 Harness table, for example `table inet awg2_harness`, inside the target network namespace.


## 11. Anti-torrent firewall guard

This is part of the AWG2 production contract. It fits the multi-orchestrator model because the server is dedicated and the binary is allowed to own host/network rules required for the VPN product. The rule must still be scoped narrowly and observable.

Goal: limit concurrent conntrack flows per VPN peer IP. This is not torrent protocol detection. It is a generic connection-count guard that makes high-peer-count P2P workloads unattractive while leaving normal browsing, streaming, gaming and messaging traffic unaffected.

The rule must not be implemented as a global subnet limit:

```nft
ip saddr 10.8.1.0/24 ct count over 200 drop
```

That form can become one shared counter for the whole matched traffic set, depending on how it is written and where it is placed. For per-peer behavior, key the stateful object by source VPN IP. Recommended shape:

```nft
table inet awg2_harness {
  counter antitorrent_dropped_v4 {}

  chain forward_antitorrent {
    type filter hook forward priority filter; policy accept;

    ip saddr <peer_ipv4_cidr> ct state new \
      meter awg2_peer_connlimit_v4 { ip saddr ct count over <max_connections_per_peer> } \
      counter name antitorrent_dropped_v4 \
      drop
  }
}
```

Target namespace: the same Amnezia network namespace where `tc` shaping is applied. Do not assume the initial host namespace sees original peer source addresses. If Amnezia performs NAT inside the container namespace, the host Docker bridge may only see the container address, not `10.8.1.x`.

Default threshold: `100` concurrent flows per peer IP. `150` is the conservative compatibility setting if there are complaints from unusually connection-heavy WebSocket/WebRTC/trading-browser workloads. `70` is more aggressive and should only be used after real traffic observation.

Rule placement: use `forward` hook with priority `filter` or any priority after conntrack lookup. Restrict evaluation to `ct state new` so established flows continue normally and only new flow attempts beyond the threshold are dropped.

Owned state:

- table `inet awg2_harness`;
- chain `forward_antitorrent`;
- counter `antitorrent_dropped_v4`;
- meter/set state created by the rule.

Status must show whether the table/chain/rule exists, the configured threshold, the peer CIDR, and the drop counter. It should also expose host conntrack pressure from `net.netfilter.nf_conntrack_count` and `net.netfilter.nf_conntrack_max` when available.

If `anti_torrent.enabled = false`, the daemon should remove or leave inactive only the AWG2-owned anti-torrent table according to config policy. It must never flush the global nftables ruleset.

## 12. Actual-state inspection

`inspect_actual_state()` must collect enough information to diagnose drift:

- qdisc on `<iface>`;
- classes on `<iface>`;
- qdiscs attached to per-peer classes on `<iface>`;
- filters under parent `1:` on `<iface>`;
- ingress qdisc on `<iface>`;
- ingress redirect filter on `<iface>`;
- existence and state of `<ifb_name>`;
- qdisc on `<ifb_name>`;
- classes on `<ifb_name>`;
- filters under parent `2:` on `<ifb_name>`.

`status` should display a compact summary and optionally a verbose mode for raw `tc` sections.

## 13. Docker events and periodic reconciliation

The daemon should use two triggers:

1. Docker events: container start/restart signals cause delayed reconcile after interface readiness.
2. Timer: periodic reconcile every `traffic.reconcile_interval_secs`.

The timer is mandatory. Events can be missed during daemon restart, Docker restart or transient errors.

Readiness check must not hardcode `awg0`. It must call the shared interface detector until timeout.

## 14. Locking

Only one traffic reconcile may run at a time across all subcommands and daemon tasks.

Recommended lock:

```text
/run/awg2-harness/traffic.lock
```

Behavior:

- `daemon` skips overlapping reconcile and logs it;
- manual `apply-limits` either waits briefly or exits with a clear message;
- lock release is automatic when process exits.

## 15. Status output contract

`awg2-harness status` should show:

```text
System:
  service: active/inactive/unknown
  docker: available/unavailable
  container: name, id, status, pid
  netns: /proc/<pid>/ns/net
  interface: awg0/wg0/...

Peers:
  total: N
  each peer: ip, optional public key prefix, handshake, rx/tx

Traffic desired:
  peer_limit, root_rate, ifb_name, class_start_id

Traffic actual:
  egress qdisc/classes/filters: ok/drift
  ingress redirect: ok/drift
  ifb topology: ok/drift
  stale peers: list
  missing peers: list

Anti-torrent actual:
  enabled: true/false
  nft table/chain/rule: ok/drift
  max_connections_per_peer
  dropped_new_flows counter
  conntrack_count / conntrack_max if available

Last actions:
  last reconcile result
  last error if any
```

A generic status banner like “system ready” is not enough. Rust status must be falsifiable: it should clearly say when state is partial.

## 16. Reboot module integration

Production reboot path is external binary/service `awg2-reboot-scheduler`:

- window from config, not constants;
- random point inside reboot window (including cross-midnight windows);
- `systemctl --no-block reboot` remains;
- disabled when `reboot.enabled = false`;
- planned reboot timestamp is persisted in `/run/awg2-harness/reboot.plan`.

## 17. Swap module

`ensure-swap` desired state:

- path from config, default `/swapfile`;
- size 2048 MB;
- mode `0600`;
- `mkswap` completed;
- active in `swapon --show`;
- one `/etc/fstab` entry with configured priority;
- `vm.swappiness = 1` persisted and applied.

When `swap.enabled=true`, AWG2 Harness owns host swap state exclusively.
The desired state is exactly one active swap entry: `swap.path`.
All other active swap devices/files are disabled via `swapoff`.
All non-comment swap entries in `/etc/fstab` are removed and replaced by one managed AWG2 entry.
Regular-file swapfiles that are not the configured `swap.path` may be deleted only after successful `swapoff`.
Block devices are never deleted, only disabled and removed from fstab.
The operation must be idempotent.

## 18. Testing

Unit tests:

- config validation, including `amnezia.wg_binaries` shape;
- `awg|wg show all allowed-ips` parser, including multiple space-separated allowed ips;
- `awg|wg show` fallback parser;
- `awg|wg show all dump` peer/interface line classification against AWG 3.0 (28 columns),
  AWG 2.0 (14 columns) and plain WireGuard (5 columns) interface lines;
- wg binary resolution order and the error raised when no binary is available;
- CIDR filtering;
- deterministic class assignment with lexicographic class-id stability contract;
- nftables ruleset rendering and parsing for the AWG2-owned table;
- `tc` parser for qdisc/class/filter snapshots;
- drift diff;
- reboot window calculation;
- fstab/sysctl idempotency.

Integration tests on real host:

- initial apply;
- repeated apply;
- partial state recovery after deleting only ingress qdisc;
- partial state recovery after deleting `ifb0`;
- container restart;
- Docker restart;
- host reboot;
- peer addition;
- peer deletion;
- status output in healthy and broken states;
- anti-torrent rule counter increments when a peer exceeds the configured concurrent-flow threshold;
- anti-torrent disable path does not flush unrelated nftables/Docker/Amnezia rules.