Two release blockers, the first a regression introduced by dd5be47.
1. The anti-torrent batch was applied on the host.
Making the rebuild atomic moved it from run_step(transport, ..) to run_nft_batch(script),
and run_nft_batch spawned `nft` directly with Command::new. An nft transaction belongs to
the network namespace of the process that runs nft, so with the production default
apply_in_amnezia_netns = true the sequence became: inspect inside the container, install
the table on the host, verify inside the container, fail. The guard was silently
ineffective, a stray table accumulated on the host, and the next reconcile could fail on
`add table` because the host copy already existed.
NsTransport gains run_with_stdin, implemented for all three transports - host, nsenter,
and docker exec with -i, which is required or the child sees EOF immediately.
run_nft_batch now takes the transport, and both callers pass the one they inspected
with. A recording fake transport asserts the script goes through the transport rather
than to a host process, which is the regression this could not previously catch.
2. Swap replacement rolled back with the wrong priority and not from every branch.
The rollback reactivated with desired.priority - the value that may be exactly what the
kernel had just rejected - because the previously active priority was dropped at the
match arm and never passed in. The active entry's priority is now carried into
replace_active_swapfile and used by the rollback.
Three early failure branches also returned without undoing anything: a failed swapoff of
the original, a failed rename to .previous, and a failed swapoff of the staging file all
left the staging swap active and the transition half-applied, after which the next
reconcile tried to unlink a file that was still active and could not proceed.
The transition now records what it completed - staging_active, original_active,
original_moved, new_installed - and one rollback undoes exactly that much, in an order
that never leaves the host without swap: restore the original file, reactivate it at its
previous priority, confirm it, and only then release and remove the staging swap. The
returned error carries the original cause plus what the recovery achieved.
Leftovers from a killed process are no longer removed blindly. A .new or .previous that
is still active cannot be unlinked, so it is reported with the swapoff command that
clears it instead of failing the same way on every reconcile.
Also in this commit, from the same audit:
- Requires=docker.service dropped in favour of Wants=. The daemon is built to outlive
Docker being down, but Requires= had systemd stop the harness whenever Docker stopped,
and Restart=on-failure does not cover a dependency-ordered stop - so the reconnect loop
never got to run.
- apply_antitorrent_disabled swallowed every discover_target error, not just "no container
running". Docker being unreachable or a permission error on the socket reported success
for a cleanup that never happened. Only AppError::NotFound is tolerated now.
- FstabTransaction::begin asserts /etc/fstab is a regular non-symlink file. read_to_string
and metadata both follow links, so a symlinked fstab would be read through and then
replaced by the rename, detaching the real file from the path the system boots from.
- append_event serialises its read-modify-write behind a mutex; the reconcile loop and the
Docker events thread both append and could drop an entry.
- tool_exists loses the fallocate, chmod and dd arms, dead since mkswap --file replaced the
allocation chain.
Docs corrected where they no longer matched the code: findmnt is described as mandatory
rather than skippable, the swap creation wording is mkswap --file, and the README package
examples use 0.1.6 to match Cargo.toml. Remaining fallocate mentions are explicitly
historical ("the previous chain"), which is accurate.
Seven tests added: batch routed through the given transport, batch failure reported as
such, rollback state defaults and previous-priority carry, active leftovers reported rather
than deleted, inactive leftovers cleared.
cargo fmt --check, clippy -D warnings, cargo test --locked --workspace --release: 188 + 4
passed.
1. docker events was leaking zombies.
docker_events_loop spawned the child, read its stdout to EOF and returned without
calling wait(). Dropping a Child does not reap it, so every reconnect - which happens
whenever Docker restarts or the stream ends - left a defunct process behind for the
lifetime of the daemon. The reader loop moved into its own function so the wait happens
on every exit path, and the exit status is logged rather than discarded.
2. sysctl status still decided ownership with contains().
The destructive paths were fixed earlier, but inspect_sysctl_tuning kept
first.contains(MANAGED_MARKER) in the disabled branch, so a file whose first line merely
embedded the marker was reported as managed. Diagnostic-only, but it contradicted the
rule the rest of the module follows. Now an exact match on the trimmed first line.
3. Swap was reconciled at the traffic interval.
ensure_swap ran every reconcile_interval_secs, default 15, and each run spawns
swapon --show, sysctl reads, the persistence check and the final verify. A swapfile, its
fstab entry and vm.swappiness change when an operator changes them and at no other time,
so this was process spawning for state that does not drift. Swap now runs on its own
300-second schedule; traffic and firewall keep the configured interval. The startup
reconcile is unchanged, so a fresh daemon still converges immediately.
4. The event log recorded routine success.
Every successful cycle appended a line, and appending means reading the whole file,
rewriting it and truncating. At 15 seconds that is roughly 17k entries a day, and the
entries worth reading were buried among them. Only transitions are written now: a new
failure, a failure whose message changed, and the recovery after one. set_ok and set_err
report whether the state actually changed, which is what makes that possible. journald
still has the full per-cycle picture through the tracing output.
5. Swapfile creation is one command.
mkswap --file <path> --size <N>M creates and formats the file, replacing the
fallocate → dd fallback → chmod → mkswap chain: four processes, three failure branches,
and a window where the file existed but was not yet formatted. util-linux on Debian 13
also handles filesystem-specific requirements - Btrfs in particular - that raw fallocate
does not. fallocate, dd and chmod drop out of the required host tools when swap is
enabled.
cargo test --workspace: 183 + 4 passed. clippy -D warnings clean.
Both owned tables were torn down before the replacement was applied, so a failure in
between left the host with neither.
Host firewall: apply_host_firewall ran `nft delete table` as its own command and only then
piped the new ruleset to `nft -f -`. Any failure in that second step - a syntax error, a
transient netlink error, the process dying - left the host with no INPUT policy and no
allow rules at all. That is fail-open on the host firewall, reached by a single failed
command. The code comment even claimed a rejected batch could not take the existing rules
down, which stopped being true the moment the delete moved out of the batch.
Anti-torrent was worse: delete, then four separate commands to rebuild table, counter,
chain and rule. A failure partway through left a half-built table - a table and counter
with no chain, or a chain with no rule - which the inspector reports as drift but which
provides no protection in the meantime.
Both now emit delete and rebuild as a single `nft -f` script. nft applies a script
atomically, so a rejected ruleset leaves the previous one exactly as it was. The delete
line is only included when the table actually exists, because `delete table` on a missing
table aborts the whole transaction - table_exists in nft.rs answers that, treating a
not-found error as absent and any other error as a real failure.
Rendering the anti-torrent table as a script also collapses four command invocations and
their argument-array plumbing into one string, and removes run_step, which had no other
user.
Four tests: for each table, that the script omits the delete when nothing is there and
that delete precedes the rebuild in the same script when it is; plus that the anti-torrent
script creates every object the inspector later requires, checked element by element
rather than through the listing-format contract matcher.
Not changed, deliberately: tc has no equivalent transaction, so the traffic module keeps
its rebuild-on-drift model. Losing shaping briefly is not comparable to losing the host
firewall.
cargo test --workspace: 183 + 4 passed. clippy -D warnings clean.
1. The state directory was created too late and never verified.
ensure_state_dir only ran when the ownership record was written, well after
assert_swap_path_is_safe had already required every parent of swap.path to exist. On a
clean host that meant a perfectly legal /var/lib/awg2-harness/swapfile was rejected
because the directory this package is supposed to create did not exist yet.
Worse, when the directory did exist the check only confirmed it was a directory. A
pre-existing /var/lib/awg2-harness that was group- or world-writable defeated the whole
point of restricting swap.path to root-owned roots: another user could still swap the
swapfile out from under a root-owned fallocate or mkswap.
ensure_state_dir now runs first in ensure_swap and asserts what the restriction actually
depends on: the parent is a real directory and not a symlink, the state directory itself
is not a symlink, it is owned by uid 0, and it is not writable by group or others. A
mode that is merely narrower than 0700 is tightened rather than rejected; anything
group- or world-writable is a hard error, because the safety argument for the whole
allow-list rests on it.
The units also declare StateDirectory=awg2-harness with mode 0700 and
RuntimeDirectory=awg2-harness, so in the common case systemd creates and owns both and
the binary only has to verify. The runtime checks stay regardless: the CLI subcommands
run outside the service lifecycle.
2. The fstab copies could be more permissive than the file they mirror.
The backup and pristine copies were written at a hardcoded 0644 with a comment claiming
they must not be wider than the original. On a host where /etc/fstab is 0600 they were,
and the rollback then handed that wider mode to the live file too. FstabTransaction now
captures the live mode when it opens and uses it for the backup, the pristine copy, the
candidate and the rollback.
One test added, unix-only since it asserts a mode: a 0600 fstab produces 0600 copies and
stays 0600 after the commit.
cargo test --workspace: 180 + 4 passed. clippy -D warnings clean. Unit files not verified
here - no systemd on this host; tools/build.sh runs systemd-analyze verify on the build
machine.
The previous commit moved the slow part of a resize out of the swapless window but left
the window itself unrecoverable. Both remaining transitions could take a host from a
working state to no swap at all on a single failure.
1. Live resize destroyed the old file before the new one was active.
The sequence ended: swapoff old, remove old, swapoff staging, rename staging into
place, and only later swapon. A failure at the rename, at the ownership write, or at
the activation left the old file already deleted and the staging file inactive - no
swap, and nothing on disk to go back to. On a host given swap because memory is tight,
this is the worst possible outcome of a routine size change.
The old file is now moved aside as <path>.awg2-harness.previous instead of deleted, and
is only removed once the replacement is confirmed present in the active swap list.
Every step past that point rolls back: the staging file is discarded, the previous file
is renamed back, ownership is rewritten and the original priority reactivated. The
returned error carries both the original cause and the outcome of the rollback, so the
operator can see whether the host still has swap.
A related failure mode is closed with it: rename succeeding but the ownership write
failing left a file with a new inode and a stale record, which the next run correctly
classified as StaleOwnership and refused to touch. Ownership is written inside the
rolled-back section, so that state cannot persist.
2. A priority change could leave the host swapless.
swapoff followed by swapon is two steps and the second can fail. Changing one integer
in the config had no recovery path. change_swap_priority now reactivates at the previous
priority if the new one is rejected, and says explicitly in the error whether the
restore worked.
3. A partially created swapfile wedged the module.
In the create-from-nothing branch, a failure after the file existed but before ownership
was recorded left an unowned regular file at the managed path. The next run classified
it as Foreign - correctly - and refused to proceed, so a transient mkswap failure
required manual intervention. The file is created by this run inside an allowed
root-owned location, so it is now removed on failure, leaving the host as it was found.
Activation moved out of ensure_swap into the reconciliation branches. That is what makes
the above possible: each branch owns both the old and the new file for the whole of its
transition, so it can undo it. A transition also refuses to report success while the
expected swap is not in the active list.
Also in this commit, from the same audit: the legacy ownership sidecar was removed on the
basis of being a regular file with the right name, without checking its contents, so a
file an operator happened to create at that path would be deleted. It now goes through the
same marker check as every other managed file - foreign files are left alone with a
warning.
Two tests added: the suffixed path helpers, and that the sidecar is removed only when it
carries the old marker.
cargo test --workspace: 180 + 4 passed. clippy -D warnings clean.
Three diagnostic defects from the follow-up audit.
1. The reboot plan was entirely best-effort.
persist_plan discarded the directory creation, the write and every error. The plan file
is the only thing on the host that says when it is going to reboot, so a failed write
produced a scheduler sitting on a timer that nothing could report. It now creates the
directory, writes through a temp file with sync_all and rename, and returns Result.
The plan also outlived its meaning in two places. With reboot.enabled = false the
scheduler exited without touching a plan written when it was still enabled, so status
kept advertising a reboot that would never happen. And after systemd accepted the reboot
request the plan stayed behind, leaving a past timestamp looking pending. Both paths now
clear it; an absent plan is the desired end state, not an error.
2. sysctl status turned a read failure into "file absent".
inspect_sysctl_tuning used fs::read_to_string(..).ok() for both managed files, so an
unreadable file was reported as missing - a diagnostic stating the opposite of the
truth, in the module whose whole job is to report drift accurately. Reads now go through
ManagedFile::read, which distinguishes absent from unreadable, and an unreadable file
produces its own drift reason naming the path and the error.
3. README described the removed ExecStartPre.
It still said the service runs ensure-swap before starting the daemon. The daemon
performs it as the first step of its own startup sequence.
Docs also state explicitly what survives swap.enabled = false, which was implied but never
written: the swapfile, its activation and the /etc/fstab entry all remain, so swap comes
back after a reboot. Only the vm.swappiness persistence is removed.
Acceptance checklist gained seven checks covering the priority-only path not recreating the
file, swap remaining active throughout a resize, foreign-swap rollback, the unreadable
status, and both reboot-plan lifecycle cases.
cargo test --workspace: 179 + 4 passed. clippy -D warnings clean.
Four defects from the follow-up audit.
1. Legacy migration swallowed every error.
drop_legacy_managed_file was `let _ = file.remove(); Ok(())`, which tolerated not only
the expected "that path is someone else's now" but also PermissionDenied, read errors
and failed unlinks. If /etc/sysctl.d/99-awg2-harness.conf survived, it would keep
winning over the new 90- file - it sorts later - while the reconcile reported success,
so vm.swappiness would silently stay at the old value. The three cases are now
distinguished: Absent is fine, Foreign warns and continues, Managed removes and any
failure propagates.
2. The disable branches ran outside the lock.
Both ensure_swap and ensure_sysctl_tuning checked `enabled` before acquiring their
lock, so the cleanup path - which deletes managed files - was unsynchronised. A manual
CLI invocation and the daemon's periodic reconcile could write and delete the same file
concurrently. The lock is now taken first in both.
3. A failed teardown left foreign swaps half-disabled.
disable_foreign_swaps aborted the loop on the first failure, so a host with two foreign
swaps could end with the first one off and the second still up - a state nobody asked
for and the error message did not describe. It now tracks what it disabled and
re-enables it, in reverse order and with the original priority, before returning the
error. The restore is best-effort and warns per entry rather than replacing the reported
failure with a rollback failure.
4. Resizing dropped swap for the whole allocation.
A live swapfile cannot be resized in place, and the previous code did swapoff, remove,
fallocate/dd, mkswap, swapon. On a host with no other swap that is a swapless window
spanning the slowest and most failure-prone part of the operation - on a machine that
was given swap precisely because memory is tight, with no way back if any step failed.
The replacement is now built and activated at a staging sibling while the old swap is
still serving, and only then is the old file released. The remaining gap is
swapoff/rename/swapon, which allocates nothing. A failure while staging removes the
staging file and leaves the original swap untouched; a failure at the final rename says
explicitly that the file is in place and can be activated with swapon.
Related, same function: changing only swap.priority used to fall through to the full
recreate path, destroying and rebuilding a correct multi-gigabyte file to change one
integer. Size and priority are now separate decisions - a priority-only change is a
reactivation.
The reconciliation is now decided by an explicit match over (size correct, currently
active) rather than a single `geometry_ok` flag, which is what hid the priority case.
Three tests added around the new decision surface: the priority-only case is not a resize,
the staging path is a sibling of the swapfile, and the managed swap is never part of the
foreign teardown set.
cargo test --workspace: 179 + 4 passed. clippy -D warnings clean.
Two defects that together made the previous ownership model unsafe, and reverse an earlier
deviation that the follow-up audit correctly rejected.
1. The sidecar marker identified a path, not a file.
Ownership was `<swap.path>.awg2-harness-owned` sitting next to the swapfile, and the
check was whether that sidecar existed and started with our marker. Nothing tied it to
the file it was supposed to describe. The sequence that loses data: our swapfile is
removed by hand, the sidecar survives, an operator puts an unrelated file at the same
path - and the next reconcile calls it managed and, if the size differs, deletes it.
The claim in the previous commit that a wrong path "can now only fail, never destroy"
was wrong.
Ownership now lives in /var/lib/awg2-harness/swap-ownership.conf, a root-only directory
created 0700, and records version, path, device and inode. A path is not an identity: a
file can be deleted and another created in its place, but (device, inode) cannot be
reused while our file exists. Classification gained StaleOwnership for "right path,
different file", which refuses exactly like Foreign but says which situation the
operator is actually in. Only ManagedValid and ManagedWrongSize ever authorise a
delete, and that is asserted directly.
The record is written after the file exists, from its real metadata, so until it lands
the swapfile is not considered managed. The old sidecar is now only ever removed, never
read.
classify_ownership is a pure function taking the recorded state and the observed
identity, so the cases that matter - stale record, record for another path, no record -
are unit-testable rather than requiring a Linux filesystem.
2. An arbitrary swap.path was still exploitable.
The previous commit kept arbitrary absolute paths and argued the ownership marker was
sufficient protection. It is not, and the audit is right to reject it. Between
assert_swap_path_is_safe and fallocate/dd/mkswap running, a user who can write the
parent directory replaces the final component with a symlink and redirects a root-owned
operation. That is a TOCTOU; stat-ing before use cannot close it, because the check and
the use are separate syscalls. Paths like /tmp/awg.swap or /home/user/awg.swap made this
reachable by any local user.
swap.path is now restricted to /swapfile or a file directly under
/var/lib/awg2-harness/. Both are root-owned, so the window does not exist rather than
being detected. /swapfile is kept so existing deployments need no config change. The
deny-list it replaces only ruled out the paths someone had thought of; what actually
matters is who can write the parent.
Eleven tests: the five classification states, a stale record, a record for another path,
the invariant that no refusal state is managed, record round-trip, malformed and
unsupported-version records, the newly allowed and newly rejected path shapes.
cargo test --workspace: 176 + 4 passed. clippy -D warnings clean.
Three defects from the follow-up audit, all in code that runs as root.
1. The fstab backup was an arbitrary-file-overwrite primitive.
write_file_synced used fs::File::create, which follows symlinks. A symlink planted at
/etc/fstab.awg2-harness.bak would be followed and its target truncated and overwritten
with the contents of /etc/fstab. Same class of bug as the sidecar write in swap.rs,
which the next commit removes entirely.
New module atomic.rs owns the write path for every file this package installs under
/etc and /var/lib: stat the destination with symlink_metadata, refuse anything that is
not a regular file or absent, create a sibling temp with create_new so a symlink at the
temp path cannot be followed either, write, sync_all, set the mode, rename, flush the
directory. fstab.rs and managed_file.rs both use it, which also removes the two
near-identical write_atomic implementations they each carried.
2. Directory fsync failures were swallowed.
Both modules document crash-durable writes, then ignored the result of syncing the
parent directory - the step that makes the rename survive a crash. sync_parent_dir now
returns Result and callers propagate it.
3. findmnt validation was optional in practice.
verify_candidate had `Err(_) => Ok(())`, so a host without util-linux silently skipped
the only structural check on a file that decides whether the machine boots. The package
already depends on util-linux, so this is now a hard error, and findmnt joins the
required host tools when swap.enabled is true - failing preflight rather than at rewrite
time.
To keep the transaction testable off Linux, commit takes the validator through an
injected fn pointer; production always passes the real findmnt check. That seam also
made it possible to test the rejection path directly, which was previously unreachable
in tests.
Also in this commit, since they are the same class of defect:
- fstab rollback used fs::copy straight onto /etc/fstab, which can leave it truncated if
it fails midway. It now writes the original - held in memory since begin, so the
recovery path does not depend on re-reading the backup - through the same atomic
no-follow path as a forward write.
- the pristine .original copy is checked to be a regular file when it already exists; a
symlink or directory there means the recovery copy cannot be trusted, so the commit
refuses instead of proceeding.
- ManagedFile decided ownership with `first.contains(marker)`, so a file whose first line
read "# NOT Managed by awg2-harness sysctl_tuning" was treated as ours and became
eligible for overwrite and removal. Now an exact match on the trimmed first line.
- systemctl daemon-reload after a successful fstab commit, since /etc/fstab feeds
systemd's generators and not only mount. Best-effort with a warning: the change is
already committed and verified, and failing the operation over a missed notification
would be worse than a stale generator state the next boot resolves.
Thirteen tests added: symlinked destination refused with the victim asserted intact,
symlink at the temp path not followed, directory destination refused, mode applied, stale
temp cleared, rejected candidate leaves the target untouched, non-regular pristine copy
refused, symlinked backup path refused with the victim intact, exact marker matching
against three near-miss first lines, and trailing whitespace tolerated on our own marker.
cargo test --workspace: 171 + 4 passed. clippy -D warnings clean.
Both the code comment and the docs said the interface row is "14 columns for AWG 2.0, 28
for AWG 3.0". That reads as a version signal and it is not one. The width of the row is
decided by the version of amneziawg-tools doing the printing, not by which protocol
parameters the tunnel has configured: AWG 3.0 tools emit the wide row for an AWG 2.0
configuration too, with the newer fields simply zero, empty or (none). Any rule of the
form "N columns means version X" is wrong by construction.
The classifier never relied on column count for that purpose - it discriminates on the
allowed-ips column - so this is a correction to what the code claims about itself and to
what the docs would lead a future change to assume.
Five fixtures added from the audit's list, all previously uncovered:
- AWG 3.0 interface row with header protection disabled: the key column is (none) and the
range fields collapse to single values;
- an AWG 2.0 configuration read through AWG 3.0 tools, asserted to have the same width as
a native AWG 3.0 row and to still be rejected while its peer parses;
- a dump containing several interfaces, each with its own peers;
- an interface with no peers at all;
- malformed rows - truncated, non-numeric handshake, non-CIDR allowed-ips - which must be
skipped rather than partially interpreted.
Docs also pin what was verified and against which versions, rather than stating it as
timeless: AmneziaVPN 5.0.0.5 and the master branches of amneziawg-tools and amneziawg-go
at the time of the change.
amnezia-awg3 is now described as what it is - not a confirmed official name. AmneziaVPN
5.0.0.5 added AWG 3.0 support but still exposes only Awg and Awg2 container variants. It
stays in the example list on purpose, and the reasoning is written down: a name that does
not match costs nothing because the search continues down the list, while a name missing
from the list produces a silent "container not found". This is a deliberate deviation from
the audit's suggestion to build defaults only from names in use today.
cargo fmt --check, clippy -D warnings, and cargo test --locked --workspace --release all
clean: 163 + 4 passed.
wireguard-tools was still declared in Depends and in the runtime package list installed by
tools/build.sh --prepare-runtime-env. Since peer discovery moved to docker exec, no code
path invokes a host-side wg, and check_host_tools stopped requiring it. The declaration
was therefore installing a package the harness never calls and misrepresenting the actual
architecture to anyone reading the control file. Removed from both places.
Three items were carrying #[allow(dead_code)], which is a way of silencing the compiler
about a real question rather than answering it:
- SysctlTuningStatus.desired duplicated data that is already in .current - every
SysctlValue holds both the desired and the current value, so the second Vec was the same
information cloned on every inspect call and read by nobody;
- SysctlValue.managed was always set to true and never read. Ownership is decided per file
by the marker, not per key, so the field encoded nothing;
- unix_now() in the reboot scheduler, along with the SystemTime and UNIX_EPOCH imports it
needed. The module works in chrono NaiveDateTime throughout.
All three are removed rather than annotated. The workspace now builds clean under
-D warnings with no dead_code exceptions anywhere.
cargo test --workspace: 158 + 4 passed. clippy -D warnings clean.
The previous implementation had every failure mode a home-grown PID lock has:
- contention was detected by matching the substring "lock already exists" in an error
message, so any change to the message would have turned a busy lock into a hard error;
- stale-lock recovery removed the file and created it again as two separate operations,
so two processes could each decide the lock was stale and each take it;
- a reused PID made a live lock look stale, since the check was only whether /proc/<pid>
exists;
- Drop deleted the lock file, racing with a process that had already opened it and was
sleeping on the retry loop - it would then lock a file that no longer has a directory
entry, and the next arrival would create a fresh one and lock that instead.
Replaced with advisory kernel locking through std::fs::File::try_lock, stabilised in Rust
1.89 and backed by flock on Linux. No new dependency. The kernel releases the lock when
the holder exits by any means, including kill -9, so there is no stale state to reason
about and no recovery heuristic to get wrong. The PID is still written to the file, but
purely as diagnostics - nothing reads it back to make a decision.
The lock file is now never deleted. Its presence carries no meaning; only the lock held
on it does. That removes the race the old Drop introduced.
Five tests: a second acquisition of a live lock fails, timeout reports lock busy timeout,
the lock is free again once the guard drops, the file survives release, and a leftover
file from a previous run does not block acquisition.
Behaviour note for the four call sites (traffic, firewall, sysctl_tuning, swap): timeouts
are unchanged, and a process killed mid-reconcile no longer leaves anything that needs
cleaning up before the next run.
cargo test --workspace: 158 + 4 passed. clippy -D warnings clean.
awg2-harness.service ran ensure-swap from ExecStartPre while the daemon runs it again as
the first step of its own startup. Idempotency kept it harmless, but it cost two locks,
two reads of /proc/swaps, two persistence checks, and split the startup failure path
across two units so an operator reading systemctl status could not tell which invocation
failed. The daemon owns the startup sequence, its error contract and its log stream, so
the ExecStartPre is removed. The standalone ensure-swap subcommand is unaffected.
Type=simple reports a unit as started the moment systemd forks, before the binary is
known to be executable. Type=exec waits for the exec to succeed, so a missing or
non-executable /usr/sbin/awg2-harness fails the unit instead of appearing to start.
Restart policy: the daemon moves from Restart=always to on-failure, matching the
scheduler. A clean exit is deliberate and should not be respawned. For the scheduler this
was already load-bearing and is now documented - it exits 0 both when reboot.enabled is
false and after systemd accepts the reboot request, so Restart=always would have spun it
in a loop on any host with scheduled reboots disabled.
RestartSec=10s with StartLimitBurst=5 over StartLimitIntervalSec=60 could exhaust the
limit in about 50 seconds, which is easy to hit when the daemon is failing because Docker
or the container is still coming up - the restarts burn the budget before the dependency
can recover. Replaced with a stepped backoff, 10 -> 20 -> 40 -> 80 -> 160s via
RestartSteps=4 and RestartMaxDelaySec=160s, and the limit window widened to 15 minutes
with a burst of 10 so a full backoff sequence fits inside it. Both directives require
systemd 254+; Debian 13 ships 257.
Documented alongside, because it is the same lifecycle contract and had no home in the
docs: configuration is read once at process start and a config edit needs a restart of the
affected service. daemon-reload is not involved - it applies to unit files, not to the
application config. ExecReload is deliberately absent, since adding it without real SIGHUP
handling would imply a reload path that does not exist.
Unit files are installed from the repo root by tools/build.sh, which already runs
systemd-analyze verify on the build host when it is available, so these changes are
verified there. Not verifiable in this environment: no Debian, no systemd.
Acceptance checklist gained five checks covering unit verification, the restart-required
contract, single ensure-swap per start, growing backoff, and the scheduler not looping
when reboots are disabled.
parse_cidr_v4 split on '/' and read the first two components without checking for a
third, so "10.8.1.0/24/garbage" parsed as a valid /24 with the tail silently dropped. The
value reaches nftables rules and tc filters, so accepting malformed input meant applying
rules derived from a config the operator did not write. Now a second separator is an
error, and the negative cases are pinned by a test: missing prefix, empty prefix, double
separator, negative and out-of-range prefixes, a malformed octet, an empty string, and a
missing address.
Interface names were only checked for being non-empty, and only for ifb_name. They are
passed straight to ip and tc, so a name that cannot exist produces a confusing runtime
failure instead of a config error. Validation now covers amnezia.interface_candidates and
traffic.ifb_name: non-empty, within the Linux IFNAMSIZ-1 limit of 15 characters, not '.'
or '..', and free of whitespace, control characters, '/' and ':'.
interface_patterns get the same rules plus one specific to the matcher: only a single
trailing '*' is supported, so a pattern like "*wg" or "a*g*" would have silently never
matched anything. A bare "*" is rejected outright - matching every interface is not a
fallback, it is a way to attach shaping to the wrong device.
container_names are matched exactly against docker ps output, so a name outside Docker's
own [a-zA-Z0-9][a-zA-Z0-9_.-]* grammar can never be found. That is now a config error
rather than a container that mysteriously never appears.
All four ordered lists - container_names, interface_candidates, interface_patterns,
wg_binaries - are checked for blank and duplicate entries through one shared helper. A
blank entry becomes an unmatchable candidate; a duplicate makes an ordered list lie about
its priority. Neither is normalized away silently.
Seven tests added, covering accepted and rejected values for each rule.
cargo test --workspace: 155 + 4 passed. clippy -D warnings clean.
Two modules wrote into /etc/sysctl.d and did it differently. sysctl_tuning refused to
touch a file whose first line lacked its marker and cleaned up on disable. swap wrote
/etc/sysctl.d/99-awg2-harness.conf with a different marker, no ownership check at all -
an administrator's file at that path was silently clobbered - and never removed it when
the module was turned off, so vm.swappiness stayed pinned forever.
New module managed_file.rs holds the rules once:
- a read that fails is never an absent file;
- ownership is Absent / Managed / Foreign, decided by the marker on the first line;
- write refuses Foreign, skips when the content already matches, and is otherwise atomic:
temp sibling created with create_new, sync_all, mode 0644, rename, directory flush;
- remove refuses Foreign and treats Absent as success.
Both modules now go through it. sysctl_tuning loses its private write_atomic,
ensure_managed_or_absent and remove_managed_sysctl_file_at.
The swappiness file moves to /etc/sysctl.d/90-awg2-harness-swap.conf with the marker
"# Managed by awg2-harness swap", matching the 90- prefix every other managed file uses.
The old path is removed on first run of the new version, but only when it still carries
the previous "# Managed by awg2-harness" marker; a foreign file there is left alone and
does not fail the migration, since the current file lives elsewhere.
Disable semantics are now explicit: swap.enabled = false removes the managed file and
does nothing else. Runtime vm.swappiness is deliberately not reset - picking a value
would be inventing a policy - and active swap is not deactivated, because merely turning
a feature off should not trigger a destructive action. Previously ensure_swap returned
early and left the persisted file behind.
Also replaced a vacuous test: swappiness_conf_render_is_stable asserted that a string
literal equals a format! of the same literal, touching no production code at all. It now
exercises render_swappiness_conf, and a companion test proves the rendered file is
recognised as managed by the same marker check that guards overwriting it - otherwise
every reconcile would refuse to touch its own output.
Twelve tests: the three ownership states, refusal to overwrite and refusal to remove a
foreign file with content asserted intact, idempotent write, stale temp handling, an
unreadable path reported as an error, plus legacy migration in all three cases - ours,
foreign, absent.
cargo test --workspace: 148 + 4 passed. clippy -D warnings clean.
Two more release blockers in the swap module.
1. Foreign files could be destroyed.
swap.path was validated against a blacklist of exact strings. /etc/../etc/fstab is not
string-equal to /etc/fstab but resolves to it, and nothing normalized the path or
checked the parent chain for symlinks. ensure_desired_swapfile then removed any
existing regular file whose size did not match the configured one, so a mistyped or
hostile path meant deleting /etc/passwd, an application config, or any other regular
file. disable_foreign_swaps additionally deleted foreign swapfiles outright.
Ownership is now explicit. A sidecar marker <swap.path>.awg2-harness-owned sits next to
the managed swapfile, and inspect_swapfile_ownership classifies the path as Missing,
ManagedValid, ManagedWrongSize or Foreign. Foreign is always a hard error - the file is
never resized, rewritten or removed. Foreign active swaps are now only swapped off;
deleting an administrator's swapfile is not this package's call, and swapoff plus the
fstab rewrite already produce the exclusive runtime state.
Path validation is split in two. The lexical half rejects relative paths, '.' and '..'
components, whitespace, control characters and backslashes, the system path list, and
anything under /proc, /sys or /dev; it touches no filesystem, so validate-config stays
side-effect free on a host where the file does not exist yet. It runs after '..' is
rejected, so the string is already normalized and the blacklist cannot be bypassed. The
filesystem half runs just before use and refuses a symlink anywhere in the parent chain,
not only at the final component.
One deliberate deviation from the audit: arbitrary absolute paths stay allowed rather
than being restricted to a fixed managed location. Pinning the path would break
deployments that already configured their own, and the ownership marker is the guard
that actually prevents data loss - a wrong path can now only fail, never destroy.
2. The migration order left the host swapless on failure.
ensure_swap ran disable_foreign_swaps first, then created the swapfile, then activated
it. Any failure in fallocate, dd, mkswap or swapon after that first step left the host
with no swap at all - on a box provisioned with swap precisely because it is short on
memory.
The sequence is now a transaction: preflight, prepare the managed swapfile, swapon,
verify it actually appears in the active list, and only then swapoff the others,
rewrite fstab, apply swappiness and verify. Nothing destructive happens until the
replacement is confirmed live. The one unavoidable gap is deactivating our own swap to
resize it, which cannot be done in place; foreign swap stays up throughout.
chmod 0600 now also happens before mkswap, so the file is never briefly world-readable
while it already holds swap structure.
config.rs no longer carries its own copy of the path blacklist and calls the shared
validator.
Twelve tests: the four ownership states, refusal to recreate a foreign file with its
content asserted intact afterwards, '..' traversal, dangerous and virtual paths,
whitespace and control characters, relative and directory paths, a missing parent
directory, an accepted chain, and a symlinked parent (unix only).
Docs updated: IMPLEMENTATION_SPEC section 17 gained the ownership table, the ordered
transaction and the path rules; CONFIGURATION and the acceptance checklist gained the
operator-facing constraints and nine checks.
cargo test --workspace: 136 + 4 passed. clippy -D warnings clean.
Two release blockers, both in the most destructive operation the package performs.
1. An unreadable /etc/fstab was treated as an empty one.
let content = fs::read_to_string(fstab_path).unwrap_or_default();
PermissionDenied, an I/O error, a damaged filesystem and "file absent" all collapsed
into an empty string. The rewrite then emitted a fstab containing only the managed swap
line and installed it atomically and successfully, dropping the root and boot mounts.
Consequence: a host that does not boot, guarded by one unwrap_or_default. The same
masking was present in read_fstab_swap_entries, so an unreadable fstab was reported by
status as "no swap entries", which reads as healthy drift-free state.
2. The backup was best-effort.
let _ = fs::write(bak_path, &content);
If it failed, the rewrite proceeded anyway. IMPLEMENTATION_SPEC and the acceptance
checklist both asserted the backup exists. The scenario where the backup fails is
exactly the scenario where it is needed.
New module fstab.rs owns the operation:
- read_required distinguishes NotFound from every other error and fails on both. A
missing /etc/fstab on Debian is an anomaly, not a blank slate, so it is never created;
- a blank candidate is refused outright;
- the backup is mandatory and flushed before the temp file is created, so there is no
window where the target is about to be replaced and no recovery copy exists;
- two copies are kept: .awg2-harness.bak is the state before the current rewrite,
.awg2-harness.original is written once with create_new and holds the host as it was
before this package first touched it;
- the temp file is a sibling of the target, created with create_new, sync_all'd, and
inherits the original permissions; a stale temp from a killed run is removed explicitly
rather than reused;
- the candidate is validated with findmnt --verify --tab-file before the swap. The step is
skipped when findmnt cannot be spawned; a rejection is fatal;
- commit is an atomic rename followed by a directory fsync;
- after the rename the target is re-read and compared to what was written. A mismatch
restores the backup and fails.
Also fixed: render_fstab_line interpolated the path raw. fstab fields are
whitespace-separated, so a space or tab in swap.path would shift every following column.
Paths now go through octal escaping (\040, \011, \012, \134).
Ten tests, all portable - the "unreadable but present" case is exercised by pointing the
reader at a directory, which fails as a non-NotFound error on every platform. Covered:
unreadable target is an error rather than an empty file, missing target is an error,
backup failure leaves the target byte-identical, blank candidate refused, unchanged
candidate is a no-op that writes no backup, the pristine copy survives repeated rewrites,
a stale temp does not block commit, and field escaping.
Docs updated: IMPLEMENTATION_SPEC section 17 gained the full rewrite contract, the
acceptance checklist gained seven checks including the two failure modes above.
cargo test --workspace: 124 + 4 passed, ten new. clippy -D warnings clean.
apply_full_topology computed peer class ids inline while inspect_and_diff computed them
through expected_peer_class_map. Two implementations of the same contract, and
apply_full_topology ends by calling inspect_and_diff to verify its own work - so a
divergence between them would surface as verify-after-apply failing on a topology that had
just been applied correctly, with a reason pointing at class mapping rather than at the
real cause. Apply now uses expected_peer_class_map, which makes it structurally impossible
for the two sides to disagree.
Behaviour is identical. Both paths assign class_start_id + index over peers in slice order,
and discover_peers hands them over sorted lexicographically and deduplicated. Iterating the
resulting BTreeMap walks the same keys in the same order, so the emitted tc commands and
their class ids are unchanged; only the loop source moved.
expected_peer_class_map is now the single source of truth for the class-id contract and
gained the tests it never had: lexicographic ordering (10.8.1.10 before 10.8.1.2, the
production contract that must not silently become numeric), a non-default class_start_id,
u16 overflow, and the empty peer list.
Found while splitting traffic.rs; kept out of that commit to keep the move reviewable.
cargo test --workspace: 114 + 4 passed, four new. clippy -D warnings clean.
firewall.rs held two subsystems that share nothing but the nft command layer: the host
INPUT firewall, applied in the host namespace and owning inet awg2_host_firewall, and the
anti-torrent guard, applied in the namespace chosen by config and owning inet
awg2_harness. The split follows that boundary rather than an inspect/apply one:
firewall/mod.rs apply_firewall, namespace selection, shared table constants
firewall/nft.rs text normalization and the nft command wrappers
firewall/host.rs host INPUT firewall and its 5 tests
firewall/antitorrent.rs connection-count guard and its 6 tests
Production logic is unchanged. Function sets before and after are identical: 25 against
25, verified by extracting names from the old blob and the four new files and diffing.
Not a pure move in one respect, stated here so it is not a surprise in review: the test
module contained four near-identical 45-line Config literals, about 180 lines of
copy-paste, differing only in the single field each test actually cared about. Carrying
four copies into the new files would have knowingly propagated the duplication, so they
collapse into one test_config() helper in firewall/mod.rs that tests mutate. This is why
the four files total 875 lines against the old 890 despite gaining doc comments.
Doc comments now record the reasoning that was only implicit before: why nft output is
compared after normalization (rendering differs across versions - quoting, comma-separated
ct state lists, brace layout), why the host firewall batch never contains a delete (a
partially applied INPUT chain could lock out SSH), why ct state new is load-bearing in the
anti-torrent rule, why connlimit_state_present accepts two renderings, and why the
anti-torrent rule defaults to the container namespace.
IMPLEMENTATION_SPEC section 3 is updated: it described the old flat layout and carried the
split as an unfulfilled intention. Both traffic and firewall now match what it documents.
cargo test --workspace: 110 + 4 passed, unchanged count. clippy -D warnings clean.
traffic.rs was 1026 lines of production code, by far the largest module. The split the
spec has described since the beginning is now done, along the responsibility boundaries
that already existed inside the file:
traffic/mod.rs shared models and the three entry points
(inspect_drift_summary, reconcile_traffic, peer_status_table)
traffic/parse_tc.rs pure parsers for tc/ip output, plus all 16 existing tests
traffic/inspect.rs inspect_and_diff and the class-id contract
traffic/apply.rs teardown, full rebuild, and the tolerated-error wrappers
No logic changed. The function set before and after the move is identical - verified by
extracting function names from the old blob and the four new files and diffing them: 26
against 26, no additions, no removals. Bodies moved verbatim; the only edits are module
paths, pub(super) markers on the items that cross a file boundary, and doc comments on
the non-obvious parts (why filter parsing is block-based, why the class-id sort is
textual, what each tolerated-error wrapper is allowed to swallow).
Note on visibility: the shared models stay private in mod.rs. Rust makes private items
visible to descendant modules, so the submodules can construct TrafficDrift and
FilterInspect without widening anything to pub(crate).
Splitting the tests out with their parsers is the practical win: parse_tc.rs is 100% pure
functions and now reads as the tested surface of the module, rather than 16 tests sitting
at the bottom of a thousand-line file next to code that needs a live container.
cargo test --workspace: 110 + 4 passed, same count as before the move, all 16 traffic
tests now reporting under traffic::parse_tc. clippy -D warnings clean.
print_status and print_degraded_sections carried four byte-identical blocks - host
firewall (25 lines), anti-torrent (24), last actions (9) and recent events (10) - plus a
Conntrack block that was the same logic assembled differently and a Traffic desired block
of four printlns. Roughly 80 lines of copy-paste with no test covering either copy, so a
format change applied to one path would have silently diverged from the other. The
divergence would only ever be visible while the container is down, which is the worst
moment to discover the diagnostic output is wrong.
Every shared section is now one function called from both paths:
print_traffic_desired_section, print_host_firewall_section, print_antitorrent_section,
print_conntrack_section, print_last_actions_section, print_recent_events_section. The file
already had this shape for print_swap_status and print_reboot_section; the pattern is now
followed consistently. The Docker/Amnezia port path block keeps two variants because they
genuinely differ - real values against unknown - but the shared FORWARD ownership note is
a single constant.
The runtime state paths were also duplicated across modules: status.rs re-declared the
literals that daemon.rs already owns. daemon::LAST_STATE_FILE and daemon::EVENTS_FILE are
now pub and reused, and the recent-events window is a named constant instead of a bare 10.
Ordering fix in the same function: print_network_tuning_status ran between the service
line and the container lines, so container/container_id/pid/netns/interface appeared under
the "Network tuning:" header at the same two-space indent as its fields. Container
identity is now printed inside the discovery match, before any other section opens, in
both the healthy and degraded paths. Field names are unchanged; only the order of two
blocks moved. IMPLEMENTATION_SPEC section 15 documented the old order and is updated.
print_degraded_sections returned anyhow::Result<()> while never failing; it returns unit
now.
status.rs: 645 -> 602 lines. cargo test --workspace: 110 + 4 passed. clippy -D warnings
clean.
The docs folder had drifted into a mixed-language contract: IMPLEMENTATION_SPEC.md was
entirely in English while the rest was Russian with English headings. Prose is now
Russian throughout; English is kept only where it is technically load-bearing (commands,
tool output, config keys, file/module/table names, tc handles) and for terms with no
settled Russian equivalent in this context (drift, reconcile, peer, namespace, handshake).
docs/README.md now states that contract explicitly.
Rewriting the spec surfaced several places where it described intent rather than the
code. Corrected against traffic_core/src:
- module layout: the spec showed traffic/ and firewall/ as directories with
model/inspect/apply submodules. They are single files, and sysctl_tuning.rs was
missing from the tree entirely.
- data model: Peer was documented as carrying public key, handshake and counters. It
carries only ipv4; runtime state lives in a separate PeerRuntime, and there is no
desired-state struct at all - it is derived inline from config plus sorted peers.
RuntimeTarget.netns_path is a String, not a PathBuf.
- interface detection: the spec listed a third step taking the first non-loopback
interface with an IPv4 address. No such fallback exists; detection bails with an
explicit error, which is safer and is now documented as deliberate.
- reconcile: the pseudo-flow folded firewall into reconcile_traffic. They are two
independent reconciles with separate locks.
- locking: only traffic.lock was documented. There are three locks, and
sysctl_tuning.lock has a 30s timeout rather than 10s. Stale-lock recovery via
/proc/<pid> was undocumented.
- sysctl_tuning: the module also manages /etc/modules-load.d/90-awg2-harness-bbr.conf
when BBR is active, and refuses to touch either managed file without its marker.
Both docs and the acceptance checklist claimed only the sysctl.d file was written.
- status contract: the documented output was outdated. All 15 printed sections are now
listed in the order status.rs emits them, including the degraded path when no container
is found, and the fact that the Network tuning block is printed between the System
service line and the container lines.
- swap: size was documented as a constant 2048 MB instead of swap.size_mb, and the
/etc/fstab.awg2-harness.bak backup was undocumented.
- apply/verify: the tolerated-error lists are now the actual closed sets from the code.
Acceptance checklist gained checks for the daemon startup failure policy (sysctl/swap
fatal, traffic/firewall not), lock contention and stale-lock recovery, degraded status,
the modules-load file, and the marker guard. The stale host wg requirement was dropped
from its host-tools item.
Every path, table, counter and log string quoted in the docs was verified to exist in
the source. cargo test --workspace: 110 + 4 passed.
Peer discovery is now the single owner of how wg is invoked and parsed.
wg.rs:
- resolve the binary through ordered amnezia.wg_binaries (default [awg, wg]);
awg is the real binary in the official image and wg is a symlink to it, so awg is
tried first and wg stays for plain WireGuard containers. No probe call: resolution
is a property of the command being run anyway.
- drop the nsenter fallback. AmneziaWG is userspace and its UAPI socket lives under
RUNSTATEDIR/amneziawg/ in the container mount namespace, while host wireguard-tools
looks under /var/run/wireguard/. nsenter -n enters only the network namespace, so
that path could never return peers.
- classify dump lines structurally instead of by column count. The AWG interface line
is longer than a peer line (14 columns on AWG 2.0, 28 on AWG 3.0) and its
handshake/rx/tx positions hold junk-packet sizes that parse as integers, so only the
allowed-ips column discriminates. Column 1 of an interface line is the device private
key, which the previous heuristic rejected only incidentally.
- fix allowed-ips parsing: `show all allowed-ips` separates a peer's addresses with
spaces, not commas, so peers with more than one allowed IP were shaped on their first
address only and the rest ran unlimited.
- collapse the three duplicated /32-in-CIDR filters into one tokenizer shared by the
allowed-ips, dump and pretty-print parsers.
config.rs:
- add amnezia.wg_binaries with serde default, validated as bare command names
(non-empty, no whitespace, no path separators) since they execute inside the
container. Existing deployed configs keep parsing unchanged.
- remove host wg from the required-tool preflight. It gated daemon startup on
wireguard-tools for a code path that produced nothing.
status.rs:
- discover peers once instead of twice per status call and degrade gracefully like the
neighbouring sections instead of aborting the whole output.
Tests: real AWG 3.0 / AWG 2.0 / plain WireGuard interface-line fixtures, AWG 3.0 range
keepalive, space-separated allowed-ips, binary resolution order, and a test that parses
and validates the shipped config.example.toml.
Добавлен отдельный host-level модуль sysctl_tuning без легаси-подходов: managed sysctl файл, точечный reconcile через sysctl -w только для drifted ключей, режимы tcp_bbr off/auto/require, безопасное удаление только managed-файла и атомарная запись конфигурации.
Интегрированы новые CLI-команды apply-network-tuning/inspect-network-tuning, startup reconcile в daemon и секция Network tuning в status. Расширены config/defaults/валидация, обновлен пример конфига и документация. Добавлены unit-тесты для config и sysctl_tuning.
- Swap: допуск небольшой дельты usable size при verify/compare
- Firewall: поддержка Debian 13 dynamic set/add @set вместо только literal meter
- WG: приоритет docker exec для peer/runtime discovery с безопасным fallback
founder
Crimson