chore: reap docker events, cut per-cycle work, use mkswap --file [P1/P2]

1. docker events was leaking zombies.

   docker_events_loop spawned the child, read its stdout to EOF and returned without
   calling wait(). Dropping a Child does not reap it, so every reconnect - which happens
   whenever Docker restarts or the stream ends - left a defunct process behind for the
   lifetime of the daemon. The reader loop moved into its own function so the wait happens
   on every exit path, and the exit status is logged rather than discarded.

2. sysctl status still decided ownership with contains().

   The destructive paths were fixed earlier, but inspect_sysctl_tuning kept
   first.contains(MANAGED_MARKER) in the disabled branch, so a file whose first line merely
   embedded the marker was reported as managed. Diagnostic-only, but it contradicted the
   rule the rest of the module follows. Now an exact match on the trimmed first line.

3. Swap was reconciled at the traffic interval.

   ensure_swap ran every reconcile_interval_secs, default 15, and each run spawns
   swapon --show, sysctl reads, the persistence check and the final verify. A swapfile, its
   fstab entry and vm.swappiness change when an operator changes them and at no other time,
   so this was process spawning for state that does not drift. Swap now runs on its own
   300-second schedule; traffic and firewall keep the configured interval. The startup
   reconcile is unchanged, so a fresh daemon still converges immediately.

4. The event log recorded routine success.

   Every successful cycle appended a line, and appending means reading the whole file,
   rewriting it and truncating. At 15 seconds that is roughly 17k entries a day, and the
   entries worth reading were buried among them. Only transitions are written now: a new
   failure, a failure whose message changed, and the recovery after one. set_ok and set_err
   report whether the state actually changed, which is what makes that possible. journald
   still has the full per-cycle picture through the tracing output.

5. Swapfile creation is one command.

   mkswap --file <path> --size <N>M creates and formats the file, replacing the
   fallocate → dd fallback → chmod → mkswap chain: four processes, three failure branches,
   and a window where the file existed but was not yet formatted. util-linux on Debian 13
   also handles filesystem-specific requirements - Btrfs in particular - that raw fallocate
   does not. fallocate, dd and chmod drop out of the required host tools when swap is
   enabled.

cargo test --workspace: 183 + 4 passed. clippy -D warnings clean.
This commit is contained in:
2026-08-07 04:50:57 +05:00
parent dd5be47ef7
commit eea787d6f8
5 changed files with 128 additions and 83 deletions
+4 -2
View File
@@ -465,7 +465,7 @@ Status обязан показывать наличие таблицы/chain/п
Демон использует два триггера:
1. Docker events: `docker events --filter event=start --format '{{.Actor.Attributes.name}}'`. Имя сверяется с `amnezia.container_names`, после чего выполняется ожидание готовности интерфейса и reconcile traffic + firewall. Если поток событий завершился, он перезапускается через 10 секунд.
2. Таймер: периодический reconcile каждые `traffic.reconcile_interval_secs`, но не чаще одного раза в 5 секунд.
2. Таймер: периодический reconcile каждые `traffic.reconcile_interval_secs`, но не чаще одного раза в 5 секунд. Он применяется к `traffic` и `firewall`; `swap` идёт по собственному расписанию — раз в 5 минут. Swapfile, запись в `/etc/fstab` и `vm.swappiness` меняются только тогда, когда их меняет оператор, а проверка на traffic-интервале означала несколько вызовов `swapon --show` и `sysctl` каждые 15 секунд ради состояния, которое само по себе не дрейфует.
Таймер обязателен. События могут быть пропущены при рестарте демона, рестарте Docker или транзиентных ошибках.
@@ -485,6 +485,8 @@ Status обязан показывать наличие таблицы/chain/п
- `/run/awg2-harness/last_action.state` — последний успех и последняя ошибка;
- `/run/awg2-harness/events.log` — журнал событий, ограниченный 1000 строками и 256 KiB.
В `events.log` пишутся только **переходы состояния**: первое появление ошибки, изменение текста ошибки и восстановление после неё. Рутинный успех не пишется. Раньше каждый успешный цикл добавлял строку — при интервале 15 секунд это порядка 17 тысяч записей в сутки, каждая с полным чтением, дополнением и перезаписью файла, и на их фоне терялись записи, ради которых журнал и читают. Полная поцикловая картина остаётся в journald через tracing.
## 14. Блокировки
Одновременно может выполняться только один reconcile каждого вида — по всем подкомандам и задачам демона.
@@ -592,7 +594,7 @@ Production-путь перезагрузки — внешний бинарь/с
При `swap.enabled = true` AWG2 Harness владеет состоянием swap на хосте эксклюзивно. Целевое состояние — ровно одна активная swap-запись: `swap.path`. Все прочие активные swap выключаются через `swapoff`, а их записи убираются из `/etc/fstab`. Операция обязана быть идемпотентной.
Создание файла: `fallocate`, при неудаче — fallback на `dd`.
Создание файла: `mkswap --file <path> --size <N>M` — одна команда, которая и создаёт файл, и форматирует его. Прежняя цепочка `fallocate` → fallback `dd` → `chmod` → `mkswap` давала четыре процесса, три ветки отказа и промежуток, в котором файл уже существовал, но ещё не был отформатирован. `util-linux` в Debian 13 дополнительно учитывает требования конкретных ФС (в частности Btrfs), которых голый `fallocate` не знает. Из обязательных host tools при `swap.enabled = true` соответственно ушли `fallocate`, `dd` и `chmod`.
### Владение swapfile
+3 -7
View File
@@ -367,14 +367,10 @@ impl Config {
required.push("systemctl");
}
if self.swap.enabled {
// fallocate, dd and chmod are gone: `mkswap --file --size` creates the file
// itself, so the allocation chain they served no longer exists.
required.extend([
"fallocate",
"chmod",
"mkswap",
"swapon",
"swapoff",
"sysctl",
"dd",
"mkswap", "swapon", "swapoff", "sysctl",
// The candidate /etc/fstab is structurally validated with
// `findmnt --verify` before it replaces the live file. That check is
// mandatory, so its absence must fail preflight rather than be discovered
+92 -30
View File
@@ -23,6 +23,9 @@ pub const LAST_STATE_FILE: &str = "/run/awg2-harness/last_action.state";
pub const EVENTS_FILE: &str = "/run/awg2-harness/events.log";
const EVENTS_MAX_LINES: usize = 1000;
const EVENTS_MAX_BYTES: usize = 256 * 1024;
/// Swap state changes only when an operator changes it, so it is reconciled on its own
/// schedule rather than at the traffic interval.
const SWAP_RECONCILE_INTERVAL: Duration = Duration::from_secs(300);
pub fn run(cfg: Config) -> anyhow::Result<()> {
info!("starting daemon");
@@ -81,40 +84,73 @@ pub fn run(cfg: Config) -> anyhow::Result<()> {
}
});
let mut swap_due = Instant::now() + SWAP_RECONCILE_INTERVAL;
loop {
if let Err(e) = swap::ensure_swap(&cfg) {
error!(error = %e, "swap reconcile error");
set_err(&state, format!("periodic swap reconcile: {e}"));
append_event(&format!("periodic_reconcile swap=error error=\"{e}\""));
} else {
set_ok(&state, "periodic swap reconcile ok");
append_event("periodic_reconcile swap=ok");
// Swap runs on its own, much longer schedule. A swapfile, its /etc/fstab entry and
// vm.swappiness do not drift on their own, and checking them at the traffic
// interval meant several `swapon --show` and `sysctl` invocations every 15 seconds
// for state that changes when an operator changes it and at no other time.
if Instant::now() >= swap_due {
swap_due = Instant::now() + SWAP_RECONCILE_INTERVAL;
report(
&state,
"swap",
"periodic swap reconcile",
swap::ensure_swap(&cfg),
);
}
if let Err(e) = traffic::reconcile_traffic(&cfg) {
error!(error = %e, "traffic reconcile error");
set_err(&state, format!("periodic traffic reconcile: {e}"));
append_event(&format!("periodic_reconcile traffic=error error=\"{e}\""));
} else {
set_ok(&state, "periodic traffic reconcile ok");
append_event("periodic_reconcile traffic=ok");
}
if let Err(e) = firewall::apply_firewall(&cfg) {
error!(error = %e, "firewall reconcile error");
set_err(&state, format!("periodic firewall reconcile: {e}"));
append_event(&format!("periodic_reconcile firewall=error error=\"{e}\""));
} else {
set_ok(&state, "periodic firewall reconcile ok");
append_event("periodic_reconcile firewall=ok");
}
report(
&state,
"traffic",
"periodic traffic reconcile",
traffic::reconcile_traffic(&cfg),
);
report(
&state,
"firewall",
"periodic firewall reconcile",
firewall::apply_firewall(&cfg),
);
thread::sleep(Duration::from_secs(
cfg.traffic.reconcile_interval_secs.max(5),
));
}
}
/// Records the outcome of one reconcile, writing an event only when something changed.
///
/// A routine success used to append a line every cycle: at a 15 second interval that is
/// roughly 17k entries a day, each one a full read-modify-write of the events file, and it
/// buried the entries that actually matter. Only transitions are written now - a failure,
/// and the recovery after one - which is what an operator reads the log for. journald still
/// has the full per-cycle picture through the tracing output.
fn report(
state: &Arc<Mutex<LastActionState>>,
subsystem: &str,
what: &str,
outcome: anyhow::Result<()>,
) {
match outcome {
Ok(()) => {
let recovered = set_ok(state, &format!("{what} ok"));
if recovered {
append_event(&format!("{subsystem}=recovered"));
}
}
Err(e) => {
error!(error = %e, "{subsystem} reconcile error");
let changed = set_err(state, format!("{what}: {e}"));
if changed {
append_event(&format!("{subsystem}=error error=\"{e}\""));
}
}
}
}
fn docker_events_loop(cfg: &Config, state: &Arc<Mutex<LastActionState>>) -> anyhow::Result<()> {
use std::io::{BufRead, BufReader};
use std::io::BufReader;
use std::process::{Command, Stdio};
let mut child = Command::new("docker")
@@ -134,6 +170,23 @@ fn docker_events_loop(cfg: &Config, state: &Arc<Mutex<LastActionState>>) -> anyh
.ok_or_else(|| anyhow::anyhow!("failed to capture docker events stdout"))?;
let reader = BufReader::new(stdout);
let result = consume_docker_events(cfg, state, reader);
// Without this the finished `docker events` process stays a zombie: dropping a Child
// does not reap it. The loop reconnects roughly every time Docker restarts, so the
// leak accumulates over the lifetime of the daemon.
match child.wait() {
Ok(status) => tracing::info!(target: "daemon", "docker events exited: {status}"),
Err(e) => tracing::warn!(target: "daemon", "failed to reap docker events: {e}"),
}
result
}
fn consume_docker_events(
cfg: &Config,
state: &Arc<Mutex<LastActionState>>,
reader: impl std::io::BufRead,
) -> anyhow::Result<()> {
for line in reader.lines() {
let name = line?.trim().to_string();
if cfg.amnezia.container_names.iter().any(|n| n == &name)
@@ -177,21 +230,30 @@ fn wait_for_interface(cfg: &Config, container_name: &str, timeout_secs: u64) ->
false
}
fn set_ok(state: &Arc<Mutex<LastActionState>>, msg: &str) {
if let Ok(mut s) = state.lock() {
/// Returns whether this success followed a failure, i.e. whether it is a recovery.
fn set_ok(state: &Arc<Mutex<LastActionState>>, msg: &str) -> bool {
let Ok(mut s) = state.lock() else {
return false;
};
let recovered = s.last_error.is_some();
s.last_ok_at = Some(now_rfc3339());
s.last_ok = Some(msg.to_string());
s.last_error = None;
persist_state(&s);
}
recovered
}
fn set_err(state: &Arc<Mutex<LastActionState>>, msg: String) {
if let Ok(mut s) = state.lock() {
/// Returns whether the failure differs from the one already recorded, so a subsystem
/// failing identically every cycle produces one entry rather than thousands.
fn set_err(state: &Arc<Mutex<LastActionState>>, msg: String) -> bool {
let Ok(mut s) = state.lock() else {
return false;
};
let changed = s.last_error.as_deref() != Some(msg.as_str());
s.last_error_at = Some(now_rfc3339());
s.last_error = Some(msg);
persist_state(&s);
}
changed
}
fn persist_state(state: &LastActionState) {
+20 -35
View File
@@ -1020,12 +1020,27 @@ fn suffixed_path(path: &Path, suffix: &str) -> anyhow::Result<PathBuf> {
Ok(path.with_file_name(out))
}
/// Allocates a swapfile and formats it, with the restrictive mode applied before `mkswap`
/// so the file is never briefly world-readable while it already holds swap structure.
/// Allocates a swapfile and formats it in one step.
///
/// `mkswap --file --size` creates the file itself, which replaces the previous
/// `fallocate`, fall back to `dd`, `chmod`, `mkswap` chain: four processes, three failure
/// branches and a window where the file existed but was not yet formatted. util-linux in
/// Debian 13 also handles filesystem-specific requirements - notably Btrfs - that raw
/// `fallocate` does not.
///
/// The mode is set again afterwards because the file must be 0600 before it is activated,
/// and this way it never sits world-readable while already holding swap structure.
fn build_swapfile(path: &Path, size_bytes: u64) -> anyhow::Result<()> {
create_swapfile(path, size_bytes)?;
set_swapfile_mode(path)?;
run("mkswap", &[path.to_string_lossy().as_ref()])?;
let size_mb = size_bytes / (1024 * 1024);
run(
"mkswap",
&[
"--file",
path.to_string_lossy().as_ref(),
"--size",
&format!("{size_mb}M"),
],
)?;
set_swapfile_mode(path)
}
@@ -1040,36 +1055,6 @@ fn set_swapfile_mode(_path: &Path) -> anyhow::Result<()> {
Ok(())
}
fn create_swapfile(path: &Path, size_bytes: u64) -> anyhow::Result<()> {
let size_mb = size_bytes / (1024 * 1024);
let p = path.to_string_lossy().to_string();
let out = Command::new("fallocate")
.args(["-l", &format!("{}M", size_mb), &p])
.output()?;
if out.status.success() {
return Ok(());
}
let out_dd = Command::new("dd")
.args([
"if=/dev/zero",
&format!("of={p}"),
"bs=1M",
&format!("count={size_mb}"),
"status=none",
])
.output()?;
if !out_dd.status.success() {
anyhow::bail!(
"swapfile create failed (fallocate+dd): fallocate={}, dd={}",
String::from_utf8_lossy(&out.stderr),
String::from_utf8_lossy(&out_dd.stderr)
);
}
Ok(())
}
fn rewrite_fstab_exclusive(desired: &DesiredSwap) -> anyhow::Result<()> {
let tx = FstabTransaction::begin(Path::new(FSTAB_PATH))?;
let rewritten = rewrite_fstab_content(tx.original(), &desired.path, desired.priority);
+2 -2
View File
@@ -203,7 +203,7 @@ pub fn inspect_sysctl_tuning(cfg: &Config) -> anyhow::Result<SysctlTuningStatus>
let managed = persisted_raw
.as_deref()
.and_then(|raw| raw.lines().next())
.map(|first| first.contains(MANAGED_MARKER))
.map(|first| first.trim_end() == MANAGED_MARKER)
.unwrap_or(false);
if managed {
@@ -217,7 +217,7 @@ pub fn inspect_sysctl_tuning(cfg: &Config) -> anyhow::Result<SysctlTuningStatus>
let managed = modules_load_raw
.as_deref()
.and_then(|raw| raw.lines().next())
.map(|first| first.contains(MANAGED_MARKER))
.map(|first| first.trim_end() == MANAGED_MARKER)
.unwrap_or(false);
if managed {