chore: reap docker events, cut per-cycle work, use mkswap --file [P1/P2]
1. docker events was leaking zombies. docker_events_loop spawned the child, read its stdout to EOF and returned without calling wait(). Dropping a Child does not reap it, so every reconnect - which happens whenever Docker restarts or the stream ends - left a defunct process behind for the lifetime of the daemon. The reader loop moved into its own function so the wait happens on every exit path, and the exit status is logged rather than discarded. 2. sysctl status still decided ownership with contains(). The destructive paths were fixed earlier, but inspect_sysctl_tuning kept first.contains(MANAGED_MARKER) in the disabled branch, so a file whose first line merely embedded the marker was reported as managed. Diagnostic-only, but it contradicted the rule the rest of the module follows. Now an exact match on the trimmed first line. 3. Swap was reconciled at the traffic interval. ensure_swap ran every reconcile_interval_secs, default 15, and each run spawns swapon --show, sysctl reads, the persistence check and the final verify. A swapfile, its fstab entry and vm.swappiness change when an operator changes them and at no other time, so this was process spawning for state that does not drift. Swap now runs on its own 300-second schedule; traffic and firewall keep the configured interval. The startup reconcile is unchanged, so a fresh daemon still converges immediately. 4. The event log recorded routine success. Every successful cycle appended a line, and appending means reading the whole file, rewriting it and truncating. At 15 seconds that is roughly 17k entries a day, and the entries worth reading were buried among them. Only transitions are written now: a new failure, a failure whose message changed, and the recovery after one. set_ok and set_err report whether the state actually changed, which is what makes that possible. journald still has the full per-cycle picture through the tracing output. 5. Swapfile creation is one command. mkswap --file <path> --size <N>M creates and formats the file, replacing the fallocate → dd fallback → chmod → mkswap chain: four processes, three failure branches, and a window where the file existed but was not yet formatted. util-linux on Debian 13 also handles filesystem-specific requirements - Btrfs in particular - that raw fallocate does not. fallocate, dd and chmod drop out of the required host tools when swap is enabled. cargo test --workspace: 183 + 4 passed. clippy -D warnings clean.
This commit is contained in:
@@ -465,7 +465,7 @@ Status обязан показывать наличие таблицы/chain/п
|
|||||||
Демон использует два триггера:
|
Демон использует два триггера:
|
||||||
|
|
||||||
1. Docker events: `docker events --filter event=start --format '{{.Actor.Attributes.name}}'`. Имя сверяется с `amnezia.container_names`, после чего выполняется ожидание готовности интерфейса и reconcile traffic + firewall. Если поток событий завершился, он перезапускается через 10 секунд.
|
1. Docker events: `docker events --filter event=start --format '{{.Actor.Attributes.name}}'`. Имя сверяется с `amnezia.container_names`, после чего выполняется ожидание готовности интерфейса и reconcile traffic + firewall. Если поток событий завершился, он перезапускается через 10 секунд.
|
||||||
2. Таймер: периодический reconcile каждые `traffic.reconcile_interval_secs`, но не чаще одного раза в 5 секунд.
|
2. Таймер: периодический reconcile каждые `traffic.reconcile_interval_secs`, но не чаще одного раза в 5 секунд. Он применяется к `traffic` и `firewall`; `swap` идёт по собственному расписанию — раз в 5 минут. Swapfile, запись в `/etc/fstab` и `vm.swappiness` меняются только тогда, когда их меняет оператор, а проверка на traffic-интервале означала несколько вызовов `swapon --show` и `sysctl` каждые 15 секунд ради состояния, которое само по себе не дрейфует.
|
||||||
|
|
||||||
Таймер обязателен. События могут быть пропущены при рестарте демона, рестарте Docker или транзиентных ошибках.
|
Таймер обязателен. События могут быть пропущены при рестарте демона, рестарте Docker или транзиентных ошибках.
|
||||||
|
|
||||||
@@ -485,6 +485,8 @@ Status обязан показывать наличие таблицы/chain/п
|
|||||||
- `/run/awg2-harness/last_action.state` — последний успех и последняя ошибка;
|
- `/run/awg2-harness/last_action.state` — последний успех и последняя ошибка;
|
||||||
- `/run/awg2-harness/events.log` — журнал событий, ограниченный 1000 строками и 256 KiB.
|
- `/run/awg2-harness/events.log` — журнал событий, ограниченный 1000 строками и 256 KiB.
|
||||||
|
|
||||||
|
В `events.log` пишутся только **переходы состояния**: первое появление ошибки, изменение текста ошибки и восстановление после неё. Рутинный успех не пишется. Раньше каждый успешный цикл добавлял строку — при интервале 15 секунд это порядка 17 тысяч записей в сутки, каждая с полным чтением, дополнением и перезаписью файла, и на их фоне терялись записи, ради которых журнал и читают. Полная поцикловая картина остаётся в journald через tracing.
|
||||||
|
|
||||||
## 14. Блокировки
|
## 14. Блокировки
|
||||||
|
|
||||||
Одновременно может выполняться только один reconcile каждого вида — по всем подкомандам и задачам демона.
|
Одновременно может выполняться только один reconcile каждого вида — по всем подкомандам и задачам демона.
|
||||||
@@ -592,7 +594,7 @@ Production-путь перезагрузки — внешний бинарь/с
|
|||||||
|
|
||||||
При `swap.enabled = true` AWG2 Harness владеет состоянием swap на хосте эксклюзивно. Целевое состояние — ровно одна активная swap-запись: `swap.path`. Все прочие активные swap выключаются через `swapoff`, а их записи убираются из `/etc/fstab`. Операция обязана быть идемпотентной.
|
При `swap.enabled = true` AWG2 Harness владеет состоянием swap на хосте эксклюзивно. Целевое состояние — ровно одна активная swap-запись: `swap.path`. Все прочие активные swap выключаются через `swapoff`, а их записи убираются из `/etc/fstab`. Операция обязана быть идемпотентной.
|
||||||
|
|
||||||
Создание файла: `fallocate`, при неудаче — fallback на `dd`.
|
Создание файла: `mkswap --file <path> --size <N>M` — одна команда, которая и создаёт файл, и форматирует его. Прежняя цепочка `fallocate` → fallback `dd` → `chmod` → `mkswap` давала четыре процесса, три ветки отказа и промежуток, в котором файл уже существовал, но ещё не был отформатирован. `util-linux` в Debian 13 дополнительно учитывает требования конкретных ФС (в частности Btrfs), которых голый `fallocate` не знает. Из обязательных host tools при `swap.enabled = true` соответственно ушли `fallocate`, `dd` и `chmod`.
|
||||||
|
|
||||||
### Владение swapfile
|
### Владение swapfile
|
||||||
|
|
||||||
|
|||||||
@@ -367,14 +367,10 @@ impl Config {
|
|||||||
required.push("systemctl");
|
required.push("systemctl");
|
||||||
}
|
}
|
||||||
if self.swap.enabled {
|
if self.swap.enabled {
|
||||||
|
// fallocate, dd and chmod are gone: `mkswap --file --size` creates the file
|
||||||
|
// itself, so the allocation chain they served no longer exists.
|
||||||
required.extend([
|
required.extend([
|
||||||
"fallocate",
|
"mkswap", "swapon", "swapoff", "sysctl",
|
||||||
"chmod",
|
|
||||||
"mkswap",
|
|
||||||
"swapon",
|
|
||||||
"swapoff",
|
|
||||||
"sysctl",
|
|
||||||
"dd",
|
|
||||||
// The candidate /etc/fstab is structurally validated with
|
// The candidate /etc/fstab is structurally validated with
|
||||||
// `findmnt --verify` before it replaces the live file. That check is
|
// `findmnt --verify` before it replaces the live file. That check is
|
||||||
// mandatory, so its absence must fail preflight rather than be discovered
|
// mandatory, so its absence must fail preflight rather than be discovered
|
||||||
|
|||||||
+92
-30
@@ -23,6 +23,9 @@ pub const LAST_STATE_FILE: &str = "/run/awg2-harness/last_action.state";
|
|||||||
pub const EVENTS_FILE: &str = "/run/awg2-harness/events.log";
|
pub const EVENTS_FILE: &str = "/run/awg2-harness/events.log";
|
||||||
const EVENTS_MAX_LINES: usize = 1000;
|
const EVENTS_MAX_LINES: usize = 1000;
|
||||||
const EVENTS_MAX_BYTES: usize = 256 * 1024;
|
const EVENTS_MAX_BYTES: usize = 256 * 1024;
|
||||||
|
/// Swap state changes only when an operator changes it, so it is reconciled on its own
|
||||||
|
/// schedule rather than at the traffic interval.
|
||||||
|
const SWAP_RECONCILE_INTERVAL: Duration = Duration::from_secs(300);
|
||||||
|
|
||||||
pub fn run(cfg: Config) -> anyhow::Result<()> {
|
pub fn run(cfg: Config) -> anyhow::Result<()> {
|
||||||
info!("starting daemon");
|
info!("starting daemon");
|
||||||
@@ -81,40 +84,73 @@ pub fn run(cfg: Config) -> anyhow::Result<()> {
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
let mut swap_due = Instant::now() + SWAP_RECONCILE_INTERVAL;
|
||||||
loop {
|
loop {
|
||||||
if let Err(e) = swap::ensure_swap(&cfg) {
|
// Swap runs on its own, much longer schedule. A swapfile, its /etc/fstab entry and
|
||||||
error!(error = %e, "swap reconcile error");
|
// vm.swappiness do not drift on their own, and checking them at the traffic
|
||||||
set_err(&state, format!("periodic swap reconcile: {e}"));
|
// interval meant several `swapon --show` and `sysctl` invocations every 15 seconds
|
||||||
append_event(&format!("periodic_reconcile swap=error error=\"{e}\""));
|
// for state that changes when an operator changes it and at no other time.
|
||||||
} else {
|
if Instant::now() >= swap_due {
|
||||||
set_ok(&state, "periodic swap reconcile ok");
|
swap_due = Instant::now() + SWAP_RECONCILE_INTERVAL;
|
||||||
append_event("periodic_reconcile swap=ok");
|
report(
|
||||||
|
&state,
|
||||||
|
"swap",
|
||||||
|
"periodic swap reconcile",
|
||||||
|
swap::ensure_swap(&cfg),
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
if let Err(e) = traffic::reconcile_traffic(&cfg) {
|
report(
|
||||||
error!(error = %e, "traffic reconcile error");
|
&state,
|
||||||
set_err(&state, format!("periodic traffic reconcile: {e}"));
|
"traffic",
|
||||||
append_event(&format!("periodic_reconcile traffic=error error=\"{e}\""));
|
"periodic traffic reconcile",
|
||||||
} else {
|
traffic::reconcile_traffic(&cfg),
|
||||||
set_ok(&state, "periodic traffic reconcile ok");
|
);
|
||||||
append_event("periodic_reconcile traffic=ok");
|
report(
|
||||||
}
|
&state,
|
||||||
if let Err(e) = firewall::apply_firewall(&cfg) {
|
"firewall",
|
||||||
error!(error = %e, "firewall reconcile error");
|
"periodic firewall reconcile",
|
||||||
set_err(&state, format!("periodic firewall reconcile: {e}"));
|
firewall::apply_firewall(&cfg),
|
||||||
append_event(&format!("periodic_reconcile firewall=error error=\"{e}\""));
|
);
|
||||||
} else {
|
|
||||||
set_ok(&state, "periodic firewall reconcile ok");
|
|
||||||
append_event("periodic_reconcile firewall=ok");
|
|
||||||
}
|
|
||||||
thread::sleep(Duration::from_secs(
|
thread::sleep(Duration::from_secs(
|
||||||
cfg.traffic.reconcile_interval_secs.max(5),
|
cfg.traffic.reconcile_interval_secs.max(5),
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Records the outcome of one reconcile, writing an event only when something changed.
|
||||||
|
///
|
||||||
|
/// A routine success used to append a line every cycle: at a 15 second interval that is
|
||||||
|
/// roughly 17k entries a day, each one a full read-modify-write of the events file, and it
|
||||||
|
/// buried the entries that actually matter. Only transitions are written now - a failure,
|
||||||
|
/// and the recovery after one - which is what an operator reads the log for. journald still
|
||||||
|
/// has the full per-cycle picture through the tracing output.
|
||||||
|
fn report(
|
||||||
|
state: &Arc<Mutex<LastActionState>>,
|
||||||
|
subsystem: &str,
|
||||||
|
what: &str,
|
||||||
|
outcome: anyhow::Result<()>,
|
||||||
|
) {
|
||||||
|
match outcome {
|
||||||
|
Ok(()) => {
|
||||||
|
let recovered = set_ok(state, &format!("{what} ok"));
|
||||||
|
if recovered {
|
||||||
|
append_event(&format!("{subsystem}=recovered"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
error!(error = %e, "{subsystem} reconcile error");
|
||||||
|
let changed = set_err(state, format!("{what}: {e}"));
|
||||||
|
if changed {
|
||||||
|
append_event(&format!("{subsystem}=error error=\"{e}\""));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
fn docker_events_loop(cfg: &Config, state: &Arc<Mutex<LastActionState>>) -> anyhow::Result<()> {
|
fn docker_events_loop(cfg: &Config, state: &Arc<Mutex<LastActionState>>) -> anyhow::Result<()> {
|
||||||
use std::io::{BufRead, BufReader};
|
use std::io::BufReader;
|
||||||
use std::process::{Command, Stdio};
|
use std::process::{Command, Stdio};
|
||||||
|
|
||||||
let mut child = Command::new("docker")
|
let mut child = Command::new("docker")
|
||||||
@@ -134,6 +170,23 @@ fn docker_events_loop(cfg: &Config, state: &Arc<Mutex<LastActionState>>) -> anyh
|
|||||||
.ok_or_else(|| anyhow::anyhow!("failed to capture docker events stdout"))?;
|
.ok_or_else(|| anyhow::anyhow!("failed to capture docker events stdout"))?;
|
||||||
let reader = BufReader::new(stdout);
|
let reader = BufReader::new(stdout);
|
||||||
|
|
||||||
|
let result = consume_docker_events(cfg, state, reader);
|
||||||
|
|
||||||
|
// Without this the finished `docker events` process stays a zombie: dropping a Child
|
||||||
|
// does not reap it. The loop reconnects roughly every time Docker restarts, so the
|
||||||
|
// leak accumulates over the lifetime of the daemon.
|
||||||
|
match child.wait() {
|
||||||
|
Ok(status) => tracing::info!(target: "daemon", "docker events exited: {status}"),
|
||||||
|
Err(e) => tracing::warn!(target: "daemon", "failed to reap docker events: {e}"),
|
||||||
|
}
|
||||||
|
result
|
||||||
|
}
|
||||||
|
|
||||||
|
fn consume_docker_events(
|
||||||
|
cfg: &Config,
|
||||||
|
state: &Arc<Mutex<LastActionState>>,
|
||||||
|
reader: impl std::io::BufRead,
|
||||||
|
) -> anyhow::Result<()> {
|
||||||
for line in reader.lines() {
|
for line in reader.lines() {
|
||||||
let name = line?.trim().to_string();
|
let name = line?.trim().to_string();
|
||||||
if cfg.amnezia.container_names.iter().any(|n| n == &name)
|
if cfg.amnezia.container_names.iter().any(|n| n == &name)
|
||||||
@@ -177,21 +230,30 @@ fn wait_for_interface(cfg: &Config, container_name: &str, timeout_secs: u64) ->
|
|||||||
false
|
false
|
||||||
}
|
}
|
||||||
|
|
||||||
fn set_ok(state: &Arc<Mutex<LastActionState>>, msg: &str) {
|
/// Returns whether this success followed a failure, i.e. whether it is a recovery.
|
||||||
if let Ok(mut s) = state.lock() {
|
fn set_ok(state: &Arc<Mutex<LastActionState>>, msg: &str) -> bool {
|
||||||
|
let Ok(mut s) = state.lock() else {
|
||||||
|
return false;
|
||||||
|
};
|
||||||
|
let recovered = s.last_error.is_some();
|
||||||
s.last_ok_at = Some(now_rfc3339());
|
s.last_ok_at = Some(now_rfc3339());
|
||||||
s.last_ok = Some(msg.to_string());
|
s.last_ok = Some(msg.to_string());
|
||||||
s.last_error = None;
|
s.last_error = None;
|
||||||
persist_state(&s);
|
persist_state(&s);
|
||||||
}
|
recovered
|
||||||
}
|
}
|
||||||
|
|
||||||
fn set_err(state: &Arc<Mutex<LastActionState>>, msg: String) {
|
/// Returns whether the failure differs from the one already recorded, so a subsystem
|
||||||
if let Ok(mut s) = state.lock() {
|
/// failing identically every cycle produces one entry rather than thousands.
|
||||||
|
fn set_err(state: &Arc<Mutex<LastActionState>>, msg: String) -> bool {
|
||||||
|
let Ok(mut s) = state.lock() else {
|
||||||
|
return false;
|
||||||
|
};
|
||||||
|
let changed = s.last_error.as_deref() != Some(msg.as_str());
|
||||||
s.last_error_at = Some(now_rfc3339());
|
s.last_error_at = Some(now_rfc3339());
|
||||||
s.last_error = Some(msg);
|
s.last_error = Some(msg);
|
||||||
persist_state(&s);
|
persist_state(&s);
|
||||||
}
|
changed
|
||||||
}
|
}
|
||||||
|
|
||||||
fn persist_state(state: &LastActionState) {
|
fn persist_state(state: &LastActionState) {
|
||||||
|
|||||||
+20
-35
@@ -1020,12 +1020,27 @@ fn suffixed_path(path: &Path, suffix: &str) -> anyhow::Result<PathBuf> {
|
|||||||
Ok(path.with_file_name(out))
|
Ok(path.with_file_name(out))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Allocates a swapfile and formats it, with the restrictive mode applied before `mkswap`
|
/// Allocates a swapfile and formats it in one step.
|
||||||
/// so the file is never briefly world-readable while it already holds swap structure.
|
///
|
||||||
|
/// `mkswap --file --size` creates the file itself, which replaces the previous
|
||||||
|
/// `fallocate`, fall back to `dd`, `chmod`, `mkswap` chain: four processes, three failure
|
||||||
|
/// branches and a window where the file existed but was not yet formatted. util-linux in
|
||||||
|
/// Debian 13 also handles filesystem-specific requirements - notably Btrfs - that raw
|
||||||
|
/// `fallocate` does not.
|
||||||
|
///
|
||||||
|
/// The mode is set again afterwards because the file must be 0600 before it is activated,
|
||||||
|
/// and this way it never sits world-readable while already holding swap structure.
|
||||||
fn build_swapfile(path: &Path, size_bytes: u64) -> anyhow::Result<()> {
|
fn build_swapfile(path: &Path, size_bytes: u64) -> anyhow::Result<()> {
|
||||||
create_swapfile(path, size_bytes)?;
|
let size_mb = size_bytes / (1024 * 1024);
|
||||||
set_swapfile_mode(path)?;
|
run(
|
||||||
run("mkswap", &[path.to_string_lossy().as_ref()])?;
|
"mkswap",
|
||||||
|
&[
|
||||||
|
"--file",
|
||||||
|
path.to_string_lossy().as_ref(),
|
||||||
|
"--size",
|
||||||
|
&format!("{size_mb}M"),
|
||||||
|
],
|
||||||
|
)?;
|
||||||
set_swapfile_mode(path)
|
set_swapfile_mode(path)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1040,36 +1055,6 @@ fn set_swapfile_mode(_path: &Path) -> anyhow::Result<()> {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
fn create_swapfile(path: &Path, size_bytes: u64) -> anyhow::Result<()> {
|
|
||||||
let size_mb = size_bytes / (1024 * 1024);
|
|
||||||
let p = path.to_string_lossy().to_string();
|
|
||||||
|
|
||||||
let out = Command::new("fallocate")
|
|
||||||
.args(["-l", &format!("{}M", size_mb), &p])
|
|
||||||
.output()?;
|
|
||||||
if out.status.success() {
|
|
||||||
return Ok(());
|
|
||||||
}
|
|
||||||
|
|
||||||
let out_dd = Command::new("dd")
|
|
||||||
.args([
|
|
||||||
"if=/dev/zero",
|
|
||||||
&format!("of={p}"),
|
|
||||||
"bs=1M",
|
|
||||||
&format!("count={size_mb}"),
|
|
||||||
"status=none",
|
|
||||||
])
|
|
||||||
.output()?;
|
|
||||||
if !out_dd.status.success() {
|
|
||||||
anyhow::bail!(
|
|
||||||
"swapfile create failed (fallocate+dd): fallocate={}, dd={}",
|
|
||||||
String::from_utf8_lossy(&out.stderr),
|
|
||||||
String::from_utf8_lossy(&out_dd.stderr)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
fn rewrite_fstab_exclusive(desired: &DesiredSwap) -> anyhow::Result<()> {
|
fn rewrite_fstab_exclusive(desired: &DesiredSwap) -> anyhow::Result<()> {
|
||||||
let tx = FstabTransaction::begin(Path::new(FSTAB_PATH))?;
|
let tx = FstabTransaction::begin(Path::new(FSTAB_PATH))?;
|
||||||
let rewritten = rewrite_fstab_content(tx.original(), &desired.path, desired.priority);
|
let rewritten = rewrite_fstab_content(tx.original(), &desired.path, desired.priority);
|
||||||
|
|||||||
@@ -203,7 +203,7 @@ pub fn inspect_sysctl_tuning(cfg: &Config) -> anyhow::Result<SysctlTuningStatus>
|
|||||||
let managed = persisted_raw
|
let managed = persisted_raw
|
||||||
.as_deref()
|
.as_deref()
|
||||||
.and_then(|raw| raw.lines().next())
|
.and_then(|raw| raw.lines().next())
|
||||||
.map(|first| first.contains(MANAGED_MARKER))
|
.map(|first| first.trim_end() == MANAGED_MARKER)
|
||||||
.unwrap_or(false);
|
.unwrap_or(false);
|
||||||
|
|
||||||
if managed {
|
if managed {
|
||||||
@@ -217,7 +217,7 @@ pub fn inspect_sysctl_tuning(cfg: &Config) -> anyhow::Result<SysctlTuningStatus>
|
|||||||
let managed = modules_load_raw
|
let managed = modules_load_raw
|
||||||
.as_deref()
|
.as_deref()
|
||||||
.and_then(|raw| raw.lines().next())
|
.and_then(|raw| raw.lines().next())
|
||||||
.map(|first| first.contains(MANAGED_MARKER))
|
.map(|first| first.trim_end() == MANAGED_MARKER)
|
||||||
.unwrap_or(false);
|
.unwrap_or(false);
|
||||||
|
|
||||||
if managed {
|
if managed {
|
||||||
|
|||||||
Reference in New Issue
Block a user