AWG2_harness/docs/IMPLEMENTATION_SPEC.md

# AWG2 Harness implementation spec

## 1. Architectural role

AWG2 Harness production package contains two Rust binaries:

- `awg2-harness` — host-side traffic/firewall/status/swap orchestrator;
- `awg2-reboot-scheduler` — dedicated reboot scheduler service.

`awg2-harness` remains the main host-side multi-orchestrator for a dedicated Debian 13 amd64 AmneziaWG/AWG 2.0 server.

It owns the infrastructure state needed by the VPN service:

- Amnezia container discovery;
- container network namespace discovery;
- AWG/WG interface discovery;
- peer discovery;
- traffic-control desired state;
- IFB ingress shaping;
- nftables connection-count anti-torrent guard;
- Docker lifecycle recovery;
- periodic reconciliation;
- status/diagnostics;
- planned reboot;
- emergency swap setup.

It does not replace Amnezia. It controls the surrounding host-side infrastructure state required by the Amnezia service.

## 2. Process model

Two systemd services run:

```bash
/usr/sbin/awg2-harness --config /etc/awg2-harness/config.toml daemon
/usr/sbin/awg2-reboot-scheduler --config /etc/awg2-harness/config.toml
```

`awg2-harness` daemon starts as root and keeps running. On startup it performs initial reconcile. During runtime it reacts to Docker events and also runs a periodic reconcile loop.

Reboot scheduling authority belongs to `awg2-reboot-scheduler` service. `awg2-harness` does not schedule reboots internally.

Subcommands:

```bash
awg2-harness daemon
awg2-harness apply-limits
awg2-harness apply-firewall
awg2-harness status
awg2-harness ensure-swap
awg2-harness validate-config
awg2-harness validate-runtime
```

## 3. Module layout

Recommended Rust module split:

```text
src/
├── main.rs
├── cli.rs
├── config.rs
├── daemon.rs
├── lock.rs
├── docker.rs
├── netns.rs
├── iface.rs
├── wg.rs
├── traffic/
│   ├── mod.rs
│   ├── model.rs
│   ├── inspect.rs
│   ├── apply.rs
│   ├── parse_tc.rs
│   └── commands.rs
├── firewall/
│   ├── mod.rs
│   ├── model.rs
│   ├── inspect.rs
│   ├── apply.rs
│   └── nft.rs
├── status.rs
├── swap.rs
└── error.rs
```

## 4. Core data model

The traffic module should operate on explicit model structs, not raw strings passed around the codebase.

Conceptual model:

```rust
struct RuntimeTarget {
    container_name: String,
    container_id: String,
    container_pid: u32,
    netns_path: PathBuf,
    interface_name: String,
}

struct Peer {
    public_key: Option<String>,
    ipv4: Ipv4Addr,
    cidr: u8, // normally 32
    latest_handshake: Option<HandshakeAge>,
    transfer_rx: Option<ByteSize>,
    transfer_tx: Option<ByteSize>,
}

struct TrafficDesiredState {
    iface: String,
    ifb_name: String,
    root_rate: Rate,
    peer_limit: Rate,
    class_start_id: u16,
    peers: Vec<PeerTrafficClass>,
}

struct PeerTrafficClass {
    peer_ip: Ipv4Addr,
    class_id: u16,
}
```

Class assignment must be deterministic. Peer IP strings are sorted lexicographically, then assigned `class_start_id + index`. Do not silently switch this to numeric IPv4 sorting: the production class-id contract maps `10.8.1.10` before `10.8.1.2` because sorting is textual, not numeric.

## 5. Namespace strategy

Preferred strategy:

1. Detect running container through Docker API or `docker ps` / `docker inspect`.
2. Read container PID.
3. Use `/proc/<pid>/ns/net` as the target namespace.
4. Execute network operations in that namespace.

Implementation options:

- current path: spawn host tools with namespace entry, for example `nsenter -t <pid> -n tc ...`, `nsenter -t <pid> -n ip ...`, `nsenter -t <pid> -n wg ...`;
- later/native: use netlink crates and explicit namespace switching where practical.

`docker exec` should not be the architectural default. It can remain as compatibility fallback if `nsenter`/host tools are unavailable, but the design assumption is that the Rust binary has host authority and controls the relevant namespace.

This also reduces operational risk: if the Amnezia container drops `tc` or `iproute2`, host-side tools can still apply rules inside the container netns.

## 6. Discovery flow

`discover_target()`:

1. Load ordered `amnezia.container_names`.
2. Find the first running matching container.
3. Get container ID, status and PID.
4. Build `netns_path`.
5. Detect interface in that namespace.
6. Return `RuntimeTarget`.

Interface detection order:

1. Exact names from `amnezia.interface_candidates`.
2. Pattern scan from `amnezia.interface_patterns`.
3. Fallback: first non-loopback interface with IPv4 address, only if it is clearly inside the Amnezia namespace.
4. If ambiguous, return an explicit error and do not apply rules.

## 7. Peer discovery

Primary command:

```bash
wg show all allowed-ips
```

Parsing rules:

- parse lines structurally into peer public key and allowed IPs;
- keep IPv4 `/32` addresses within `amnezia.peer_ipv4_cidr`;
- include offline peers; online state is not a requirement for shaping;
- sort peer IP strings lexicographically before class assignment to preserve the current lexicographic class-id contract.

Fallback:

```bash
wg show
```

The fallback parser should support standard `wg show` text output by extracting `allowed ips:` lines.

Execution policy for production:

- do not use shell wrappers (`sh -c`) for `wg` invocation;
- do not suppress stderr via redirection in command strings;
- execute `wg` directly and handle non-zero exit codes in Rust.

## 8. Desired traffic topology

The expected topology is the AWG2 traffic topology contract.

Egress:

```text
tc qdisc add dev <iface> root handle 1: htb
tc class add dev <iface> parent 1: classid 1:1 htb rate <root_rate> ceil <root_rate>
for each peer:
  tc class add dev <iface> parent 1:1 classid 1:<id> htb rate <peer_limit> ceil <peer_limit>
  tc qdisc add dev <iface> parent 1:<id> fq_codel
  tc filter add dev <iface> parent 1: protocol ip u32 match ip dst <peer_ip> flowid 1:<id>
```

Ingress through IFB:

```text
modprobe ifb
ip link add name <ifb_name> type ifb
ip link set dev <ifb_name> up
tc qdisc add dev <iface> handle ffff: ingress
tc filter add dev <iface> parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev <ifb_name>
tc qdisc add dev <ifb_name> root handle 2: htb
tc class add dev <ifb_name> parent 2: classid 2:1 htb rate <root_rate> ceil <root_rate>
for each peer:
  tc class add dev <ifb_name> parent 2:1 classid 2:<id> htb rate <peer_limit> ceil <peer_limit>
  tc qdisc add dev <ifb_name> parent 2:<id> fq_codel
  tc filter add dev <ifb_name> parent 2: protocol ip u32 match ip src <peer_ip> flowid 2:<id>
```

## 9. Reconcile algorithm

Pseudo-flow:

```text
reconcile_traffic():
  acquire global lock
  target = discover_target()
  peers = discover_peers(target)
  if peers empty:
      log and return NoPeers
  desired = build_desired_state(target, peers, config)
  actual = inspect_actual_state(target, desired)
  drift = diff(desired, actual)
  firewall_desired = build_firewall_desired_state(target, config)
  firewall_actual = inspect_firewall_state(target, firewall_desired)
  firewall_drift = diff(firewall_desired, firewall_actual)

  if drift.none and firewall_drift.none:
      log "rules already up-to-date"
      return NoChange

  if drift.exists:
      teardown_owned_topology(target, desired)
      apply_full_topology(target, desired)
      verify_after_apply(target, desired)

  if firewall_drift.exists:
      apply_owned_firewall_table(target, firewall_desired)
      verify_firewall_after_apply(target, firewall_desired)
```

For production, controlled full rebuild on drift is the only supported path.

Parser hardening requirements:

- `tc filter show` must be parsed block-by-block (state-machine), not by naive token scanning across unrelated lines;
- mapping `peer_ip -> class_id` must be validated for both egress (`1:`) and ingress (`2:`) sides;
- verification must fail when classes/filters exist but mapping differs from deterministic assignment.

Apply/verify policy:

- all apply steps are explicit command invocations;
- expected idempotency conflicts (`already exists`) may be tolerated selectively;
- any other apply error is fatal;
- `verify_after_apply` is mandatory for both traffic and anti-torrent subsystems.

Anti-torrent observability contract:

- named counter `antitorrent_dropped_v4` must be present in owned table;
- status output must expose both dropped packets and dropped bytes.

## 10. Owned state and teardown rules

The binary owns:

- root qdisc `1:` on selected AWG/WG interface;
- ingress qdisc `ffff:` on selected AWG/WG interface when traffic limits are enabled;
- IFB device named `traffic.ifb_name`, default `ifb0`, in the target namespace;
- root qdisc `2:` on IFB;
- classes and filters under these handles.

Teardown may delete and recreate these owned objects. Because the server is dedicated, this is acceptable and preferable to leaving stale or partial rules.

The binary should not blindly flush unrelated host firewall rules, Docker NAT rules, or unrelated interfaces. Firewall ownership must be explicit and scoped to the AWG2 Harness table, for example `table inet awg2_harness`, inside the target network namespace.


## 11. Anti-torrent firewall guard

This is part of the AWG2 production contract. It fits the multi-orchestrator model because the server is dedicated and the binary is allowed to own host/network rules required for the VPN product. The rule must still be scoped narrowly and observable.

Goal: limit concurrent conntrack flows per VPN peer IP. This is not torrent protocol detection. It is a generic connection-count guard that makes high-peer-count P2P workloads unattractive while leaving normal browsing, streaming, gaming and messaging traffic unaffected.

The rule must not be implemented as a global subnet limit:

```nft
ip saddr 10.8.1.0/24 ct count over 200 drop
```

That form can become one shared counter for the whole matched traffic set, depending on how it is written and where it is placed. For per-peer behavior, key the stateful object by source VPN IP. Recommended shape:

```nft
table inet awg2_harness {
  counter antitorrent_dropped_v4 {}

  chain forward_antitorrent {
    type filter hook forward priority filter; policy accept;

    ip saddr <peer_ipv4_cidr> ct state new \
      meter awg2_peer_connlimit_v4 { ip saddr ct count over <max_connections_per_peer> } \
      counter name antitorrent_dropped_v4 \
      drop
  }
}
```

Target namespace: the same Amnezia network namespace where `tc` shaping is applied. Do not assume the initial host namespace sees original peer source addresses. If Amnezia performs NAT inside the container namespace, the host Docker bridge may only see the container address, not `10.8.1.x`.

Default threshold: `100` concurrent flows per peer IP. `150` is the conservative compatibility setting if there are complaints from unusually connection-heavy WebSocket/WebRTC/trading-browser workloads. `70` is more aggressive and should only be used after real traffic observation.

Rule placement: use `forward` hook with priority `filter` or any priority after conntrack lookup. Restrict evaluation to `ct state new` so established flows continue normally and only new flow attempts beyond the threshold are dropped.

Owned state:

- table `inet awg2_harness`;
- chain `forward_antitorrent`;
- counter `antitorrent_dropped_v4`;
- meter/set state created by the rule.

Status must show whether the table/chain/rule exists, the configured threshold, the peer CIDR, and the drop counter. It should also expose host conntrack pressure from `net.netfilter.nf_conntrack_count` and `net.netfilter.nf_conntrack_max` when available.

If `anti_torrent.enabled = false`, the daemon should remove or leave inactive only the AWG2-owned anti-torrent table according to config policy. It must never flush the global nftables ruleset.

## 12. Actual-state inspection

`inspect_actual_state()` must collect enough information to diagnose drift:

- qdisc on `<iface>`;
- classes on `<iface>`;
- qdiscs attached to per-peer classes on `<iface>`;
- filters under parent `1:` on `<iface>`;
- ingress qdisc on `<iface>`;
- ingress redirect filter on `<iface>`;
- existence and state of `<ifb_name>`;
- qdisc on `<ifb_name>`;
- classes on `<ifb_name>`;
- filters under parent `2:` on `<ifb_name>`.

`status` should display a compact summary and optionally a verbose mode for raw `tc` sections.

## 13. Docker events and periodic reconciliation

The daemon should use two triggers:

1. Docker events: container start/restart signals cause delayed reconcile after interface readiness.
2. Timer: periodic reconcile every `traffic.reconcile_interval_secs`.

The timer is mandatory. Events can be missed during daemon restart, Docker restart or transient errors.

Readiness check must not hardcode `awg0`. It must call the shared interface detector until timeout.

## 14. Locking

Only one traffic reconcile may run at a time across all subcommands and daemon tasks.

Recommended lock:

```text
/run/awg2-harness/traffic.lock
```

Behavior:

- `daemon` skips overlapping reconcile and logs it;
- manual `apply-limits` either waits briefly or exits with a clear message;
- lock release is automatic when process exits.

## 15. Status output contract

`awg2-harness status` should show:

```text
System:
  service: active/inactive/unknown
  docker: available/unavailable
  container: name, id, status, pid
  netns: /proc/<pid>/ns/net
  interface: awg0/wg0/...

Peers:
  total: N
  each peer: ip, optional public key prefix, handshake, rx/tx

Traffic desired:
  peer_limit, root_rate, ifb_name, class_start_id

Traffic actual:
  egress qdisc/classes/filters: ok/drift
  ingress redirect: ok/drift
  ifb topology: ok/drift
  stale peers: list
  missing peers: list

Anti-torrent actual:
  enabled: true/false
  nft table/chain/rule: ok/drift
  max_connections_per_peer
  dropped_new_flows counter
  conntrack_count / conntrack_max if available

Last actions:
  last reconcile result
  last error if any
```

A generic status banner like “system ready” is not enough. Rust status must be falsifiable: it should clearly say when state is partial.

## 16. Reboot module integration

Production reboot path is external binary/service `awg2-reboot-scheduler`:

- window from config, not constants;
- random point inside reboot window (including cross-midnight windows);
- `systemctl --no-block reboot` remains;
- disabled when `reboot.enabled = false`;
- planned reboot timestamp is persisted in `/run/awg2-harness/reboot.plan`.

## 17. Swap module

`ensure-swap` desired state:

- path from config, default `/swapfile`;
- size 2048 MB;
- mode `0600`;
- `mkswap` completed;
- active in `swapon --show`;
- one `/etc/fstab` entry with configured priority;
- `vm.swappiness = 1` persisted and applied.

When `swap.enabled=true`, AWG2 Harness owns host swap state exclusively.
The desired state is exactly one active swap entry: `swap.path`.
All other active swap devices/files are disabled via `swapoff`.
All non-comment swap entries in `/etc/fstab` are removed and replaced by one managed AWG2 entry.
Regular-file swapfiles that are not the configured `swap.path` may be deleted only after successful `swapoff`.
Block devices are never deleted, only disabled and removed from fstab.
The operation must be idempotent.

## 18. Testing

Unit tests:

- config validation;
- `wg show all allowed-ips` parser;
- `wg show` fallback parser;
- CIDR filtering;
- deterministic class assignment with lexicographic class-id stability contract;
- nftables ruleset rendering and parsing for the AWG2-owned table;
- `tc` parser for qdisc/class/filter snapshots;
- drift diff;
- reboot window calculation;
- fstab/sysctl idempotency.

Integration tests on real host:

- initial apply;
- repeated apply;
- partial state recovery after deleting only ingress qdisc;
- partial state recovery after deleting `ifb0`;
- container restart;
- Docker restart;
- host reboot;
- peer addition;
- peer deletion;
- status output in healthy and broken states;
- anti-torrent rule counter increments when a peer exceeds the configured concurrent-flow threshold;
- anti-torrent disable path does not flush unrelated nftables/Docker/Amnezia rules.