dev #227
@@ -499,6 +499,8 @@ sudo python3 /opt/fedeo/scripts/selfhost-update.py status
|
||||
|
||||
Beim ersten Update die erzeugte Compose-Differenz prüfen, eigene Anpassungen in eine Override-Datei übertragen und den Austausch mit `--accept-compose-replacement` bestätigen. Bestehende Installationen müssen das Update-Skript zunächst aus einem passenden Release übernehmen.
|
||||
|
||||
Optional führt `fedeo-updater.service` die Aufträge unabhängig von der anfragenden Verbindung aus. Nach Bereitstellung der zugehörigen Dienstdateien installiert `sudo bash /opt/fedeo/scripts/selfhost-update-service-install.sh` den systemd-Dienst. Die lokale Unix-Socket-API bietet Prüfung, Update-Start und Status; Backend-Anbindung und Frontend-Button folgen separat. Einrichtung und API-Beispiele stehen in der folgenden Anleitung.
|
||||
|
||||
Voraussetzungen, unterstützte Konfigurationen und Wiederherstellung stehen in [Vollständige Selfhost-Updates](docs/selfhost-updates.md). Für ältere Releases steht der manuelle Ablauf in [Versionierung und Releases](docs/releases.md).
|
||||
|
||||
## Backup-Empfehlung
|
||||
|
||||
@@ -53,7 +53,7 @@ Backend, Frontend, Dokumentation, Website und zentrale Dienste werden als gemein
|
||||
- Bewegliche Kanäle: `latest` und `main` zeigen nach erfolgreicher Veröffentlichung auf denselben Release.
|
||||
- Entwicklung: `dev` sowie unveränderliche `dev-sha-<vollständiger Commit>-<UUID>`-Images. Alle Komponenten eines erfolgreichen Laufs erhalten denselben Snapshot-Tag. Auch eine Wiederholung desselben Commits baut einen eigenen Snapshot. Die eingebettete Entwicklungsversion enthält Commit-Anzahl und Commit-ID.
|
||||
- Release-Builds erhalten zunächst einen eindeutigen temporären `build-<UUID>`-Tag. Diese Tags können später nach einer Aufbewahrungsfrist bereinigt werden; stabile Versionen und deren Digests müssen erhalten bleiben.
|
||||
- Release-Dateien: `CHANGELOG.md`, `release-manifest.json`, `docker-compose.selfhost.yml`, `.env.example`, `selfhost-install.sh`, `selfhost-setup.sh`, `selfhost-update.py`, `SELFHOST-UPDATES.md` und `SHA256SUMS`.
|
||||
- Release-Dateien: `CHANGELOG.md`, `release-manifest.json`, `docker-compose.selfhost.yml`, `.env.example`, `selfhost-install.sh`, `selfhost-setup.sh`, `selfhost-update.py`, `selfhost-update-service.py`, `selfhost-update-service-install.sh`, `fedeo-updater.service`, `SELFHOST-UPDATES.md` und `SHA256SUMS`.
|
||||
|
||||
Das Manifest dokumentiert die Image-Digests, den Commit und SHA-256-Prüfsummen aller enthaltenen SQL-Migrationen. Es beschreibt den ausgelieferten Migrationsstand; den tatsächlich angewendeten Stand einer Installation bestimmt deren Datenbank.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Vollständige Selfhost-Updates
|
||||
|
||||
`scripts/selfhost-update.py` ist der gemeinsame Update-Ablauf für die Kommandozeile und einen zukünftigen Update-Dienst. Diese Implementierung installiert noch keinen Dienst und stellt noch keinen Update-Button bereit. Der Prozess muss auf dem Docker-Host außerhalb der FEDEO-Container laufen.
|
||||
`scripts/selfhost-update.py` ist der gemeinsame Update-Ablauf für die Kommandozeile und den optionalen Host-Dienst `fedeo-updater.service`. Beide laufen auf dem Docker-Host außerhalb der FEDEO-Container und verwenden dieselbe exklusive Sperre. Der Dienst bietet bereits eine lokale Schnittstelle; Backend-Anbindung und Update-Button im Frontend folgen separat.
|
||||
|
||||
## Voraussetzungen und Umfang
|
||||
|
||||
@@ -40,7 +40,7 @@ sudo python3 /opt/fedeo/scripts/selfhost-update.py check --version 3.1.0
|
||||
|
||||
`3.1.0` ist ein Beispiel und muss durch einen tatsächlich veröffentlichten Zielrelease ersetzt werden. Ein anderes Betriebsverzeichnis wird mit `--directory /pfad/zur/instanz` angegeben.
|
||||
|
||||
Die Prüfung lädt die Betriebsdateien und erstellt einen Plan einschließlich `compose.diff` und Release Notes unter `/opt/fedeo/.updates/<Auftrag>/`. Sie verändert die laufende Installation nicht und lädt noch keine Docker-Images. Die JSON-Ausgabe nennt Zielversion, Commit, betroffene Dienste, geschätzten Backup-Platz und die Pfade der Vorschau. Unveröffentlichte Entwürfe und Vorabversionen werden ignoriert.
|
||||
Die Prüfung lädt die Betriebsdateien und erstellt einen Plan einschließlich `compose.diff` und Release Notes unter `/opt/fedeo/.updates/<Auftrag>/`. Sie verändert die laufende Installation nicht und lädt noch keine Docker-Images. Die JSON-Ausgabe nennt Zielversion, Commit, betroffene Dienste, geschätzten Backup-Platz und die Pfade der Vorschau. `updateAvailable: false` bedeutet, dass die geprüfte Version nicht neuer als die installierte ist. Unveröffentlichte Entwürfe und Vorabversionen werden ignoriert.
|
||||
|
||||
Beim ersten Update kennt der Updater die Herkunft der vorhandenen Compose-Datei noch nicht. Die erzeugte `compose.diff` prüfen und eigene Anpassungen in die Override-Datei übertragen. Dann den Austausch ausdrücklich akzeptieren:
|
||||
|
||||
@@ -55,9 +55,72 @@ sudo python3 /opt/fedeo/scripts/selfhost-update.py apply --version 3.2.0
|
||||
sudo python3 /opt/fedeo/scripts/selfhost-update.py status
|
||||
```
|
||||
|
||||
Eine explizite Zielversion ist beim Anwenden erforderlich. `apply` prüft die Release-Dateien und die tatsächliche Installation nochmals. Bei gleichen oder älteren stabilen Zielversionen wird abgebrochen. Änderungen an `.env`, Basis-Compose oder der Override-Datei während des Downloads führen ebenfalls vor der Wartungsphase zum Abbruch.
|
||||
Eine explizite Zielversion ist beim Anwenden erforderlich. `apply` prüft die Release-Dateien und die tatsächliche Installation nochmals. Beim Anwenden gleicher oder älterer stabiler Zielversionen wird abgebrochen. Änderungen an `.env`, Basis-Compose oder der Override-Datei während des Downloads führen ebenfalls vor der Wartungsphase zum Abbruch.
|
||||
|
||||
Während des ersten Ausbaus den Befehl in einer bestehenden persistenten Terminalsitzung ausführen. Der spätere Host-Dienst übernimmt die vom Browser unabhängige Ausführung. Ein Neustart des FEDEO-Stacks beendet diesen Host-Prozess nicht; ein Neustart des Hosts unterbricht ihn.
|
||||
Direkte CLI-Aufrufe in einer persistenten Terminalsitzung ausführen. Alternativ übernimmt der folgende Host-Dienst die von der Verbindung unabhängige Ausführung. Ein Neustart des FEDEO-Stacks beendet diese Host-Prozesse nicht. Ein erzwungener Host-Neustart kann sie unterbrechen; beim normalen Dienst-Stop wartet der Dienst auf den laufenden Auftrag.
|
||||
|
||||
## Host-Dienst installieren
|
||||
|
||||
Benötigt zusätzlich einen Linux-Host mit systemd und Docker als `docker.service`. Die Dateien `selfhost-update.py`, `selfhost-update-service.py`, `selfhost-update-service-install.sh` und `fedeo-updater.service` müssen aus demselben geprüften Stand im selben Verzeichnis liegen, beispielsweise `/opt/fedeo/scripts/`. Neuere Installer laden sie dorthin; neuere Releases enthalten alle vier Dateien samt Prüfsummen. Release `3.1.0` enthält den Dienst noch nicht. Zum Nachrüsten können die vier Dateien aus dem geprüften Repository-Commit kopiert werden, ohne die laufende FEDEO-Version zu ändern.
|
||||
|
||||
```bash
|
||||
sudo bash /opt/fedeo/scripts/selfhost-update-service-install.sh --directory /opt/fedeo
|
||||
sudo systemctl status fedeo-updater.service
|
||||
sudo curl --unix-socket /run/fedeo-updater/control.sock http://localhost/v1/status
|
||||
```
|
||||
|
||||
Der Installer legt die Python-Module root-eigen unter `/usr/local/lib/fedeo-updater/` ab, installiert die systemd-Unit und aktiviert den Dienst beim Systemstart. Bestehende FEDEO-Container werden durch die Dienstinstallation nicht neu gestartet. Bei einer erneuten Dienstinstallation wartet ein normales `systemctl stop` einen laufenden Auftrag ab; ein aktiver CLI-Updater blockiert den Dateiaustausch. Die Unit kann vorab ohne Installation angezeigt werden:
|
||||
|
||||
```bash
|
||||
bash /opt/fedeo/scripts/selfhost-update-service-install.sh --directory /opt/fedeo --print-unit
|
||||
```
|
||||
|
||||
Der Dienst läuft wegen Docker und der Backups als root. Seine HTTP-Schnittstelle lauscht ausschließlich auf `/run/fedeo-updater/control.sock`, nicht auf einem TCP-Port. Verzeichnis und Socket gehören `root:fedeo-updater` mit `0750` bzw. `0660`. Zugriff auf diese Gruppe erlaubt instanzweite Updates; nur vertrauenswürdigen Host-Prozessen Zugriff geben. Die spätere Backend-Anbindung muss Systemadministratoren prüfen und darf nur diese kontrollierte Schnittstelle verwenden. Es wird kein Docker-Socket in Backend oder Frontend eingebunden. Dieser Stand richtet noch keine Backend-Route und keinen Frontend-Button ein.
|
||||
|
||||
Registry-Anmeldung bei privaten Images als root durchführen. Für private Releases kann `/etc/fedeo/updater.env` root-lesbar (`0600`) `FEDEO_RELEASE_TOKEN=...` enthalten; danach den Dienst neu starten. Dieser Token wird nicht an API-Clients ausgegeben. Die Unit verwendet `/root` als Home-Verzeichnis für die Docker-Anmeldung. Der Dienst ersetzt sich bei einem FEDEO-Update nicht selbst; neue Dienstdateien zusammen bereitstellen und den Dienstinstaller erneut ausführen.
|
||||
|
||||
## Lokale Dienst-API
|
||||
|
||||
Alle Antworten sind JSON mit `Cache-Control: no-store`. Mutationen benötigen `Content-Type: application/json`, höchstens 4096 Byte und eine vom Client erzeugte `requestId` (8–80 Zeichen, Buchstaben/Ziffern/Unterstrich/Bindestrich). Für jede neue Aktion eine neue ID verwenden; bei einem Verbindungsfehler dieselbe ID und denselben Inhalt wiederholen. Das liefert denselben Auftrag statt eines zweiten Updates, auch nach einem Dienstneustart. Eine andere Anfrage mit derselben ID wird mit `409 REQUEST_CONFLICT` abgelehnt.
|
||||
|
||||
| Aufruf | Bedeutung |
|
||||
| --- | --- |
|
||||
| `GET /v1/status` | Installierter Stand, Zustand der Update-Engine und aktueller Dienstauftrag |
|
||||
| `POST /v1/check` | Asynchrone Prüfung; `requestId`, optional `version` ohne `v` |
|
||||
| `GET /v1/requests/<requestId>` | Auftrag, bereinigter Prüfplan und aktueller Fortschritt |
|
||||
| `POST /v1/apply` | Asynchrones Update; `requestId`, feste `version` und erfolgreiche `checkRequestId` |
|
||||
|
||||
Beispiel für eine Prüfung des neuesten Releases:
|
||||
|
||||
```bash
|
||||
CHECK_ID="$(python3 -c 'import uuid; print(uuid.uuid4())')"
|
||||
curl --unix-socket /run/fedeo-updater/control.sock \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"requestId\":\"$CHECK_ID\"}" http://localhost/v1/check
|
||||
|
||||
curl --unix-socket /run/fedeo-updater/control.sock \
|
||||
"http://localhost/v1/requests/$CHECK_ID"
|
||||
```
|
||||
|
||||
Neue Aufträge antworten sofort mit `202`. `queued` und `running` bedeuten, dass die Arbeit noch läuft; `checked` bedeutet erfolgreiche Prüfung und `up_to_date`, dass keine neuere Version vorliegt. Der bereinigte Plan enthält Version, Commit, Platzbedarf und stillzulegende Dienste. Vollständige Release Notes und Compose-Diff bleiben auf dem Host, da insbesondere bestehende Compose-Dateien Geheimnisse enthalten können.
|
||||
|
||||
Nach `checked`, `updateAvailable: true` und Prüfung des Plans eine konkrete Zielversion einsetzen:
|
||||
|
||||
```bash
|
||||
APPLY_ID="$(python3 -c 'import uuid; print(uuid.uuid4())')"
|
||||
TARGET_VERSION=X.Y.Z
|
||||
curl --unix-socket /run/fedeo-updater/control.sock \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"requestId\":\"$APPLY_ID\",\"version\":\"$TARGET_VERSION\",\"checkRequestId\":\"$CHECK_ID\"}" \
|
||||
http://localhost/v1/apply
|
||||
|
||||
curl --unix-socket /run/fedeo-updater/control.sock \
|
||||
"http://localhost/v1/requests/$APPLY_ID"
|
||||
```
|
||||
|
||||
`succeeded` bestätigt den Abschluss. Der Dienst bindet den Start an genau den geprüften Release und die geprüften Konfigurationsdateien. Bei zwischenzeitlichen Änderungen bricht er mit `PLAN_CHANGED` vor Image-Pull/Wartungsphase ab. `requiresComposeAdoption: true` benötigt zunächst eine manuell geprüfte Übernahme über die CLI; die API bietet dafür keinen Freigabeschalter. Ebenso bleiben `resolve`, freie Shell-Befehle, alternative Betriebsverzeichnisse und Download-Adressen außerhalb der API. Ein zweiter Auftrag bzw. ein laufender CLI-Updater führt zu `409 BUSY`, ein offener Wiederherstellungsfall zu `409 RECOVERY_REQUIRED`.
|
||||
|
||||
Aufträge liegen dauerhaft und root-lesbar unter `.updates/service/`. Nach einem harten Dienstabbruch werden offene Aufträge als `interrupted` markiert und nicht automatisch erneut gestartet. Bei normalem Stop/Restart nimmt der Dienst keine neue Arbeit mehr an und lässt den aktiven Auftrag fertiglaufen (`KillMode=mixed`, `TimeoutStopSec=infinity`). Auch ein reguläres Herunterfahren des Hosts kann dadurch bis zum Auftragsende warten. Nach erzwungenem Abbruch oder Host-Neustart den Status prüfen; unterbrochene Updates folgen den bestehenden Wiederherstellungsprüfungen. Auftragsspezifische Fehlerdetails stehen in `.updates/service/<requestId>.log`, Werkzeugfehler weiterhin im geschützten `commands.log` des Engine-Auftrags. Über die API werden nur feste Fehlermeldungen und erlaubte Statusfelder ausgegeben, keine Rohlogs, Konfiguration, Dateipfade oder Prüffingerprints.
|
||||
|
||||
## Ablauf und Speicherung
|
||||
|
||||
|
||||
33
scripts/fedeo-updater.service
Normal file
33
scripts/fedeo-updater.service
Normal file
@@ -0,0 +1,33 @@
|
||||
[Unit]
|
||||
Description=FEDEO host update service
|
||||
Wants=network-online.target
|
||||
After=network-online.target docker.service
|
||||
Requires=docker.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
Group=fedeo-updater
|
||||
Environment=HOME=/root
|
||||
Environment=PYTHONDONTWRITEBYTECODE=1
|
||||
EnvironmentFile=-/etc/fedeo/updater.env
|
||||
ExecStart=/usr/bin/python3 -B /usr/local/lib/fedeo-updater/selfhost-update-service.py serve --directory @FEDEO_DIRECTORY@
|
||||
Restart=on-failure
|
||||
RestartSec=5s
|
||||
KillMode=mixed
|
||||
TimeoutStopSec=infinity
|
||||
UMask=0077
|
||||
RuntimeDirectory=fedeo-updater
|
||||
RuntimeDirectoryMode=0750
|
||||
RuntimeDirectoryPreserve=yes
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=full
|
||||
ProtectKernelTunables=true
|
||||
ProtectKernelModules=true
|
||||
ProtectControlGroups=true
|
||||
RestrictSUIDSGID=true
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -33,6 +33,9 @@ export function releaseFiles(manifest, notes) {
|
||||
['selfhost-install.sh', Buffer.from(readFileSync('scripts/selfhost-install.sh', 'utf8').replace('FEDEO_REF:-dev', `FEDEO_REF:-${manifest.tag}`))],
|
||||
['selfhost-setup.sh', Buffer.from(readFileSync('scripts/selfhost-setup.sh', 'utf8').replaceAll('FEDEO_VERSION:-dev', `FEDEO_VERSION:-${manifest.version}`))],
|
||||
['selfhost-update.py', Buffer.from(readFileSync('scripts/selfhost-update.py', 'utf8'))],
|
||||
['selfhost-update-service.py', Buffer.from(readFileSync('scripts/selfhost-update-service.py', 'utf8'))],
|
||||
['selfhost-update-service-install.sh', Buffer.from(readFileSync('scripts/selfhost-update-service-install.sh', 'utf8'))],
|
||||
['fedeo-updater.service', Buffer.from(readFileSync('scripts/fedeo-updater.service', 'utf8'))],
|
||||
['SELFHOST-UPDATES.md', Buffer.from(readFileSync('docs/selfhost-updates.md', 'utf8'))],
|
||||
]);
|
||||
files.set('SHA256SUMS', Buffer.from([...files].map(([name, data]) => `${createHash('sha256').update(data).digest('hex')} ${name}`).join('\n') + '\n'));
|
||||
|
||||
@@ -152,6 +152,10 @@ test('release assets pin compose, environment and installer to one version', ()
|
||||
assert.doesNotMatch(files.get('docker-compose.selfhost.yml').toString(), /FEDEO_VERSION:-dev/);
|
||||
assert.match(files.get('.env.example').toString(), /^FEDEO_VERSION=1\.2\.0$/m);
|
||||
assert.match(files.get('selfhost-install.sh').toString(), /FEDEO_REF:-v1\.2\.0/);
|
||||
for (const name of ['selfhost-update-service.py', 'selfhost-update-service-install.sh', 'fedeo-updater.service']) {
|
||||
assert.ok(files.get(name)?.length > 0, `missing service asset: ${name}`);
|
||||
assert.ok(files.get('SHA256SUMS').toString().includes(` ${name}\n`));
|
||||
}
|
||||
assert.equal(files.get('SHA256SUMS').toString().trim().split('\n').length, files.size - 1);
|
||||
});
|
||||
|
||||
|
||||
@@ -186,6 +186,9 @@ download_selfhost_files() {
|
||||
download_file "$FEDEO_RAW_BASE_URL/.env.example" "$FEDEO_DIR/.env.example"
|
||||
download_file "$FEDEO_RAW_BASE_URL/scripts/selfhost-setup.sh" "$FEDEO_DIR/scripts/selfhost-setup.sh"
|
||||
download_file "$FEDEO_RAW_BASE_URL/scripts/selfhost-update.py" "$FEDEO_DIR/scripts/selfhost-update.py"
|
||||
download_file "$FEDEO_RAW_BASE_URL/scripts/selfhost-update-service.py" "$FEDEO_DIR/scripts/selfhost-update-service.py"
|
||||
download_file "$FEDEO_RAW_BASE_URL/scripts/selfhost-update-service-install.sh" "$FEDEO_DIR/scripts/selfhost-update-service-install.sh"
|
||||
download_file "$FEDEO_RAW_BASE_URL/scripts/fedeo-updater.service" "$FEDEO_DIR/scripts/fedeo-updater.service"
|
||||
download_file "$FEDEO_RAW_BASE_URL/docs/selfhost-updates.md" "$FEDEO_DIR/SELFHOST-UPDATES.md"
|
||||
chmod +x "$FEDEO_DIR/scripts/selfhost-setup.sh"
|
||||
}
|
||||
|
||||
93
scripts/selfhost-update-service-install.sh
Normal file
93
scripts/selfhost-update-service-install.sh
Normal file
@@ -0,0 +1,93 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
FEDEO_DIR=/opt/fedeo
|
||||
PRINT_UNIT=false
|
||||
SOURCE_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--directory) FEDEO_DIR="${2:?Verzeichnis fehlt}"; shift 2 ;;
|
||||
--print-unit) PRINT_UNIT=true; shift ;;
|
||||
--help|-h) echo "sudo bash $0 [--directory /opt/fedeo] [--print-unit]"; exit 0 ;;
|
||||
*) echo "Unbekanntes Argument: $1" >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
render_unit() {
|
||||
python3 - "$SOURCE_DIR/fedeo-updater.service" "$FEDEO_DIR" <<'PY'
|
||||
import pathlib, sys
|
||||
directory = sys.argv[2]
|
||||
if not directory.startswith('/') or any(c in directory for c in '\n\r\0'):
|
||||
raise SystemExit('Ein gültiges absolutes Betriebsverzeichnis angeben.')
|
||||
quoted = '"' + directory.replace('\\', '\\\\').replace('"', '\\"').replace('%', '%%').replace('$', '$$') + '"'
|
||||
sys.stdout.write(pathlib.Path(sys.argv[1]).read_text().replace('@FEDEO_DIRECTORY@', quoted))
|
||||
PY
|
||||
}
|
||||
if [[ "$PRINT_UNIT" == true ]]; then
|
||||
render_unit
|
||||
exit 0
|
||||
fi
|
||||
|
||||
[[ "$(id -u)" -eq 0 ]] || { echo "Als root ausführen." >&2; exit 1; }
|
||||
[[ -d /run/systemd/system ]] || { echo "Ein Linux-Host mit laufendem systemd wird benötigt." >&2; exit 1; }
|
||||
FEDEO_DIR="$(realpath -e -- "$FEDEO_DIR")"
|
||||
[[ -f "$FEDEO_DIR/.env" && -f "$FEDEO_DIR/docker-compose.yml" && ! -e "$FEDEO_DIR/.git" ]] || {
|
||||
echo "Ein bestehendes Selfhost-Betriebsverzeichnis ohne Git-Checkout angeben." >&2; exit 1;
|
||||
}
|
||||
python3 -c 'import sys; sys.exit(0 if sys.version_info >= (3, 10) else "Python ab 3.10 wird benötigt.")'
|
||||
for name in selfhost-update.py selfhost-update-service.py fedeo-updater.service; do
|
||||
[[ -f "$SOURCE_DIR/$name" ]] || { echo "Fehlende Installationsdatei: $name" >&2; exit 1; }
|
||||
done
|
||||
python3 - "$SOURCE_DIR" <<'PY'
|
||||
import importlib.util, inspect, pathlib, sys
|
||||
source = pathlib.Path(sys.argv[1])
|
||||
spec = importlib.util.spec_from_file_location('fedeo_update_install_check', source / 'selfhost-update.py')
|
||||
engine = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(engine)
|
||||
if not {'expected_fingerprint', 'request_id'}.issubset(inspect.signature(engine.Updater.execute).parameters):
|
||||
raise SystemExit('Updater und Dienst passen nicht zusammen. Alle Dateien aus demselben Stand bereitstellen.')
|
||||
compile((source / 'selfhost-update-service.py').read_text(), 'selfhost-update-service.py', 'exec')
|
||||
PY
|
||||
command -v flock >/dev/null
|
||||
command -v docker >/dev/null
|
||||
command -v curl >/dev/null
|
||||
docker compose version >/dev/null
|
||||
UNIT_CONTENT="$(render_unit)"
|
||||
|
||||
# Drain an active service job before replacing either Python module. No containers are restarted here.
|
||||
if systemctl is-active --quiet fedeo-updater.service; then
|
||||
systemctl stop fedeo-updater.service
|
||||
fi
|
||||
[[ ! -L "$FEDEO_DIR/.updates" ]] || { echo ".updates darf kein symbolischer Link sein." >&2; exit 1; }
|
||||
install -d -m 0700 "$FEDEO_DIR/.updates"
|
||||
exec 9>"$FEDEO_DIR/.updates/lock"
|
||||
flock -n 9 || { echo "Ein CLI-Update läuft noch. Dienstinstallation später wiederholen." >&2; exit 1; }
|
||||
|
||||
getent group fedeo-updater >/dev/null || groupadd --system fedeo-updater
|
||||
install -d -o root -g root -m 0700 /usr/local/lib/fedeo-updater /etc/fedeo
|
||||
install -o root -g root -m 0700 "$SOURCE_DIR/selfhost-update.py" /usr/local/lib/fedeo-updater/selfhost-update.py
|
||||
install -o root -g root -m 0700 "$SOURCE_DIR/selfhost-update-service.py" /usr/local/lib/fedeo-updater/selfhost-update-service.py
|
||||
if [[ ! -e /etc/fedeo/updater.env ]]; then
|
||||
(umask 077; printf '%s\n' '# Optional: FEDEO_RELEASE_TOKEN=... (nur Leserechte)' > /etc/fedeo/updater.env)
|
||||
fi
|
||||
chmod 0600 /etc/fedeo/updater.env
|
||||
UNIT_TMP="$(mktemp /etc/systemd/system/fedeo-updater.service.XXXXXX)"
|
||||
trap 'rm -f -- "$UNIT_TMP"' EXIT
|
||||
printf '%s\n' "$UNIT_CONTENT" > "$UNIT_TMP"
|
||||
chmod 0644 "$UNIT_TMP"
|
||||
mv -- "$UNIT_TMP" /etc/systemd/system/fedeo-updater.service
|
||||
flock -u 9
|
||||
exec 9>&-
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now fedeo-updater.service
|
||||
READY=false
|
||||
for attempt in {1..50}; do
|
||||
if curl --silent --fail --max-time 1 --unix-socket /run/fedeo-updater/control.sock http://localhost/v1/status >/dev/null; then
|
||||
READY=true
|
||||
break
|
||||
fi
|
||||
sleep 0.1
|
||||
done
|
||||
systemctl --no-pager --full status fedeo-updater.service
|
||||
[[ "$READY" == true ]] || { echo "Dienstschnittstelle noch nicht erreichbar; journalctl -u fedeo-updater.service prüfen." >&2; exit 1; }
|
||||
echo "Lokale Schnittstelle: /run/fedeo-updater/control.sock"
|
||||
373
scripts/selfhost-update-service.py
Normal file
373
scripts/selfhost-update-service.py
Normal file
@@ -0,0 +1,373 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Local HTTP/Unix-socket control service for the FEDEO host updater. No TCP listener."""
|
||||
import argparse
|
||||
import contextlib
|
||||
import fcntl
|
||||
from http.server import BaseHTTPRequestHandler
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import signal
|
||||
import socketserver
|
||||
import stat
|
||||
import threading
|
||||
import traceback
|
||||
|
||||
spec = importlib.util.spec_from_file_location("fedeo_updater", Path(__file__).with_name("selfhost-update.py"))
|
||||
engine = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(engine)
|
||||
|
||||
REQUEST_ID = re.compile(r"[A-Za-z0-9][A-Za-z0-9_-]{7,79}\Z")
|
||||
PHASES = {"preparing", "validating", "checked", "pulling", "maintenance", "backed_up", "applying",
|
||||
"retiring_matrix", "migrating", "starting", "verifying", "complete", "resolved"}
|
||||
STATUSES = {"idle", "queued", "running", "checked", "up_to_date", "succeeded", "failed", "interrupted", "recovered"}
|
||||
ERRORS = {
|
||||
"INVALID_REQUEST": "Ungültige Anfrage.",
|
||||
"NOT_FOUND": "Auftrag oder Endpunkt nicht gefunden.",
|
||||
"BUSY": "Eine Prüfung oder ein Update läuft bereits.",
|
||||
"SHUTTING_DOWN": "Der Dienst wird beendet.",
|
||||
"REQUEST_CONFLICT": "Diese Auftrags-ID wurde für eine andere Anfrage verwendet.",
|
||||
"CHECK_REQUIRED": "Zuerst genau diese Zielversion erfolgreich prüfen.",
|
||||
"COMPOSE_REVIEW_REQUIRED": "Die Compose-Datei muss zuerst auf dem Host geprüft und übernommen werden.",
|
||||
"RECOVERY_REQUIRED": "Ein vorheriges Update benötigt Wiederherstellung auf dem Host.",
|
||||
"PLAN_CHANGED": "Release oder Konfiguration haben sich geändert. Erneut prüfen.",
|
||||
"UPDATE_FAILED": "Auftrag fehlgeschlagen. Geschützte Protokolle auf dem Host prüfen.",
|
||||
"INTERRUPTED": "Der Auftrag wurde unterbrochen. Status auf dem Host prüfen.",
|
||||
}
|
||||
|
||||
|
||||
class ApiError(Exception):
|
||||
def __init__(self, code, status=400):
|
||||
self.code, self.status = code, status
|
||||
|
||||
|
||||
def identifier(value):
|
||||
if not isinstance(value, str) or not REQUEST_ID.fullmatch(value):
|
||||
raise ApiError("INVALID_REQUEST")
|
||||
return value
|
||||
|
||||
|
||||
def public_state(state):
|
||||
"""An allowlist, never raw command logs, exceptions, host paths, config or diff contents."""
|
||||
result = {"status": state.get("status") if state.get("status") in STATUSES else "idle"}
|
||||
if state.get("phase") in PHASES:
|
||||
result["phase"] = state["phase"]
|
||||
for key in ("recoveryRequired", "maintenance", "mutationStarted"):
|
||||
if key in state:
|
||||
result[key] = bool(state[key])
|
||||
for key in ("version", "previousVersion"):
|
||||
if isinstance(state.get(key), str) and engine.VERSION.fullmatch(state[key]):
|
||||
result[key] = state[key]
|
||||
if re.fullmatch(r"[a-f0-9]{40}", str(state.get("revision", ""))):
|
||||
result["revision"] = state["revision"]
|
||||
for key in ("startedAt", "updatedAt"):
|
||||
if isinstance(state.get(key), str) and re.fullmatch(r"[0-9T:+.Z-]{10,40}", state[key]):
|
||||
result[key] = state[key]
|
||||
if state.get("requestId") and REQUEST_ID.fullmatch(str(state["requestId"])):
|
||||
result["requestId"] = state["requestId"]
|
||||
if state.get("command") in ("check", "apply"):
|
||||
result["command"] = state["command"]
|
||||
return result
|
||||
|
||||
|
||||
def public_plan(plan):
|
||||
result = {k: v for k, v in public_state(plan).items() if k != "status"}
|
||||
for key in ("updateAvailable", "requiresComposeAdoption"):
|
||||
result[key] = bool(plan.get(key))
|
||||
if type(plan.get("estimatedBackupBytes")) is int:
|
||||
result["estimatedBackupBytes"] = plan["estimatedBackupBytes"]
|
||||
result["retiredServices"] = sorted(set(plan.get("retiredServices", [])) & engine.LEGACY_MATRIX_SERVICES)
|
||||
return result
|
||||
|
||||
|
||||
class Controller:
|
||||
def __init__(self, directory, *, updater_factory=engine.Updater):
|
||||
self.directory = Path(directory).resolve()
|
||||
self.factory = updater_factory
|
||||
updater = self.factory(self.directory)
|
||||
self.home = updater.home / "service"
|
||||
self.home.mkdir(mode=0o700, exist_ok=True)
|
||||
if self.home.is_symlink():
|
||||
raise engine.UpdateError("Dienstverzeichnis darf kein symbolischer Link sein.")
|
||||
os.chmod(self.home, 0o700)
|
||||
self.gate = threading.Lock()
|
||||
self.worker = None
|
||||
self.closing = False
|
||||
self.active = None
|
||||
self.daemon_lock = (self.home / "daemon.lock").open("a")
|
||||
try:
|
||||
fcntl.flock(self.daemon_lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except OSError:
|
||||
self.daemon_lock.close()
|
||||
raise engine.UpdateError("Für diese Instanz läuft bereits ein Update-Dienst.") from None
|
||||
# A persisted request is an audit record, not a queue that silently replays updates.
|
||||
try:
|
||||
for path in self.home.glob("*.json"):
|
||||
record = engine.read_json(path)
|
||||
if record.get("status") in ("queued", "running"):
|
||||
record.update(status="interrupted", errorCode="INTERRUPTED", updatedAt=engine.now())
|
||||
engine.write_json(path, record)
|
||||
updater.status() # Detect an abandoned engine job without touching an active CLI job.
|
||||
except BaseException:
|
||||
self.daemon_lock.close()
|
||||
raise
|
||||
|
||||
def path(self, request_id):
|
||||
return self.home / (identifier(request_id) + ".json")
|
||||
|
||||
def read(self, request_id):
|
||||
record = engine.read_json(self.path(request_id))
|
||||
if not record:
|
||||
raise ApiError("NOT_FOUND", 404)
|
||||
return record
|
||||
|
||||
def request_status(self, request_id):
|
||||
record = self.read(request_id)
|
||||
result = public_state(record)
|
||||
if record.get("plan"):
|
||||
result["plan"] = public_plan(record["plan"])
|
||||
code = record.get("errorCode")
|
||||
if code in ERRORS:
|
||||
result["error"] = {"code": code, "message": ERRORS[code]}
|
||||
if record["status"] == "running":
|
||||
state = self.factory(self.directory).status()
|
||||
if state.get("requestId") == request_id:
|
||||
result["progress"] = public_state(state)
|
||||
return result
|
||||
|
||||
def status(self):
|
||||
updater = self.factory(self.directory)
|
||||
result = {"apiVersion": 1, "engine": public_state(updater.status()), "acceptingRequests": not self.closing}
|
||||
installed = engine.read_json(updater.home / "installed.json", {}).get("manifest", {})
|
||||
result["installed"] = {k: v for k, v in public_state({"version": installed.get("version"),
|
||||
"revision": installed.get("commit")}).items() if k != "status"}
|
||||
if self.active:
|
||||
result["request"] = self.request_status(self.active)
|
||||
return result
|
||||
|
||||
def submit(self, command, body):
|
||||
if not isinstance(body, dict) or command not in ("check", "apply"):
|
||||
raise ApiError("INVALID_REQUEST")
|
||||
allowed = {"requestId", "version"} | ({"checkRequestId"} if command == "apply" else set())
|
||||
if set(body) - allowed:
|
||||
raise ApiError("INVALID_REQUEST")
|
||||
request_id = identifier(body.get("requestId"))
|
||||
version = body.get("version")
|
||||
if version is not None and (not isinstance(version, str) or not engine.VERSION.fullmatch(version)):
|
||||
raise ApiError("INVALID_REQUEST")
|
||||
if command == "apply" and not version:
|
||||
raise ApiError("INVALID_REQUEST")
|
||||
check_id = identifier(body.get("checkRequestId")) if command == "apply" else None
|
||||
payload = {"command": command, "version": version, "checkRequestId": check_id}
|
||||
with self.gate:
|
||||
previous = engine.read_json(self.path(request_id))
|
||||
if previous:
|
||||
if previous["input"] != payload:
|
||||
raise ApiError("REQUEST_CONFLICT", 409)
|
||||
return 200, self.request_status(request_id)
|
||||
if self.closing:
|
||||
raise ApiError("SHUTTING_DOWN", 503)
|
||||
if self.worker and self.worker.is_alive():
|
||||
raise ApiError("BUSY", 409)
|
||||
state = self.factory(self.directory).status()
|
||||
if state.get("recoveryRequired"):
|
||||
raise ApiError("RECOVERY_REQUIRED", 409)
|
||||
if state.get("status") == "running":
|
||||
raise ApiError("BUSY", 409)
|
||||
fingerprint = None
|
||||
if command == "apply":
|
||||
try:
|
||||
check = self.read(check_id)
|
||||
except ApiError:
|
||||
raise ApiError("CHECK_REQUIRED", 409) from None
|
||||
plan = check.get("plan", {})
|
||||
if (check.get("command") != "check" or check.get("status") != "checked"
|
||||
or plan.get("version") != version or not plan.get("updateAvailable")
|
||||
or not re.fullmatch(r"[a-f0-9]{64}", str(plan.get("approvalFingerprint", "")))):
|
||||
raise ApiError("CHECK_REQUIRED", 409)
|
||||
if plan.get("requiresComposeAdoption"):
|
||||
raise ApiError("COMPOSE_REVIEW_REQUIRED", 409)
|
||||
fingerprint = plan["approvalFingerprint"]
|
||||
record = {"requestId": request_id, "command": command, "input": payload,
|
||||
"status": "queued", "startedAt": engine.now(), "updatedAt": engine.now()}
|
||||
engine.write_json(self.path(request_id), record)
|
||||
self.active = request_id
|
||||
self.worker = threading.Thread(target=self.run, args=(record, fingerprint), daemon=False)
|
||||
try:
|
||||
self.worker.start()
|
||||
except BaseException:
|
||||
record.update(status="failed", errorCode="UPDATE_FAILED", updatedAt=engine.now())
|
||||
engine.write_json(self.path(request_id), record)
|
||||
raise
|
||||
return 202, public_state(record)
|
||||
|
||||
def run(self, record, fingerprint):
|
||||
request_id = record["requestId"]
|
||||
try:
|
||||
record.update(status="running", updatedAt=engine.now())
|
||||
engine.write_json(self.path(request_id), record)
|
||||
updater = self.factory(self.directory)
|
||||
result = updater.execute(record["command"], record["input"]["version"],
|
||||
expected_fingerprint=fingerprint, request_id=request_id)
|
||||
record.update(status=result["status"], plan=result["plan"])
|
||||
if record["command"] == "check" and not result["plan"]["updateAvailable"]:
|
||||
record["status"] = "up_to_date"
|
||||
except BaseException as error:
|
||||
code = "PLAN_CHANGED" if isinstance(error, engine.PlanChangedError) else "UPDATE_FAILED"
|
||||
record.update(status="failed", errorCode=code)
|
||||
with (self.home / (request_id + ".log")).open("a") as log:
|
||||
os.chmod(log.name, 0o600)
|
||||
traceback.print_exc(file=log)
|
||||
finally:
|
||||
record["updatedAt"] = engine.now()
|
||||
engine.write_json(self.path(request_id), record)
|
||||
|
||||
def close(self):
|
||||
with self.gate:
|
||||
self.closing = True
|
||||
worker = self.worker
|
||||
if worker:
|
||||
worker.join() # Normal service stop drains the active update instead of interrupting a migration.
|
||||
self.daemon_lock.close()
|
||||
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
protocol_version = "HTTP/1.0"
|
||||
server_version = "FEDEO-Updater/1"
|
||||
sys_version = ""
|
||||
|
||||
def setup(self):
|
||||
super().setup()
|
||||
self.connection.settimeout(5)
|
||||
|
||||
def log_message(self, *_):
|
||||
pass
|
||||
|
||||
def send_json(self, status, body):
|
||||
data = json.dumps(body, ensure_ascii=False).encode()
|
||||
self.close_connection = True
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", "application/json; charset=utf-8")
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.send_header("Cache-Control", "no-store")
|
||||
self.send_header("Connection", "close")
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
def dispatch(self):
|
||||
try:
|
||||
if self.headers.get("Origin") or self.headers.get("Transfer-Encoding"):
|
||||
raise ApiError("INVALID_REQUEST")
|
||||
controller = self.server.controller
|
||||
if self.command == "GET" and self.path == "/v1/status":
|
||||
self.send_json(200, controller.status())
|
||||
elif self.command == "GET" and self.path.startswith("/v1/requests/"):
|
||||
self.send_json(200, controller.request_status(self.path.removeprefix("/v1/requests/")))
|
||||
elif self.command == "POST" and self.path in ("/v1/check", "/v1/apply"):
|
||||
lengths = self.headers.get_all("Content-Length", [])
|
||||
if (len(lengths) != 1 or not lengths[0].isdigit() or not 1 <= int(lengths[0]) <= 4096
|
||||
or self.headers.get_content_type() != "application/json"):
|
||||
raise ApiError("INVALID_REQUEST")
|
||||
body = json.loads(self.rfile.read(int(lengths[0])))
|
||||
status, result = controller.submit(self.path.rsplit("/", 1)[-1], body)
|
||||
self.send_json(status, result)
|
||||
else:
|
||||
raise ApiError("NOT_FOUND", 404)
|
||||
except ApiError as error:
|
||||
self.send_json(error.status, {"error": {"code": error.code, "message": ERRORS[error.code]}})
|
||||
except (ValueError, UnicodeError):
|
||||
self.send_json(400, {"error": {"code": "INVALID_REQUEST", "message": ERRORS["INVALID_REQUEST"]}})
|
||||
except (OSError, engine.UpdateError, KeyError, TypeError):
|
||||
with contextlib.suppress(OSError):
|
||||
self.send_json(500, {"error": {"code": "UPDATE_FAILED", "message": ERRORS["UPDATE_FAILED"]}})
|
||||
|
||||
do_GET = dispatch
|
||||
do_POST = dispatch
|
||||
|
||||
|
||||
class Server(socketserver.ThreadingMixIn, socketserver.UnixStreamServer):
|
||||
daemon_threads = True
|
||||
request_queue_size = 8
|
||||
|
||||
def __init__(self, path, controller):
|
||||
self.controller = controller
|
||||
self.slots = threading.BoundedSemaphore(16)
|
||||
super().__init__(str(path), Handler)
|
||||
os.chmod(path, 0o660)
|
||||
|
||||
def process_request(self, request, client_address):
|
||||
if not self.slots.acquire(blocking=False):
|
||||
self.shutdown_request(request)
|
||||
return
|
||||
try:
|
||||
super().process_request(request, client_address)
|
||||
except BaseException:
|
||||
self.slots.release()
|
||||
raise
|
||||
|
||||
def process_request_thread(self, request, client_address):
|
||||
try:
|
||||
super().process_request_thread(request, client_address)
|
||||
finally:
|
||||
self.slots.release()
|
||||
|
||||
def handle_error(self, *_):
|
||||
pass # Do not send request data or tracebacks to the journal.
|
||||
|
||||
|
||||
def serve(directory, socket_path):
|
||||
controller = None
|
||||
path = Path(socket_path)
|
||||
stop = threading.Event()
|
||||
server = None
|
||||
socket_lock = None
|
||||
try:
|
||||
path.parent.mkdir(mode=0o750, parents=True, exist_ok=True)
|
||||
parent = path.parent.stat()
|
||||
if path.parent.is_symlink() or parent.st_uid != 0 or parent.st_mode & 0o022:
|
||||
raise engine.UpdateError("Socket-Verzeichnis muss root gehören und darf nicht durch andere beschreibbar sein.")
|
||||
socket_lock = path.with_name(path.name + ".lock").open("a")
|
||||
try:
|
||||
fcntl.flock(socket_lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except OSError:
|
||||
raise engine.UpdateError("An diesem Socket läuft bereits ein Update-Dienst.") from None
|
||||
controller = Controller(directory)
|
||||
if path.is_symlink() or (path.exists() and not stat.S_ISSOCK(path.stat().st_mode)):
|
||||
raise engine.UpdateError("Socket-Pfad ist bereits anderweitig belegt.")
|
||||
# Both the instance and endpoint are locked before replacing a stale socket.
|
||||
path.unlink(missing_ok=True)
|
||||
server = Server(path, controller)
|
||||
for sig in (signal.SIGTERM, signal.SIGINT):
|
||||
signal.signal(sig, lambda *_: stop.set())
|
||||
server.timeout = 0.5
|
||||
while not stop.is_set():
|
||||
server.handle_request()
|
||||
finally:
|
||||
if controller:
|
||||
controller.close()
|
||||
if server:
|
||||
server.server_close()
|
||||
path.unlink(missing_ok=True)
|
||||
if socket_lock:
|
||||
socket_lock.close()
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("command", choices=("serve",))
|
||||
parser.add_argument("--directory", default="/opt/fedeo")
|
||||
parser.add_argument("--socket", default="/run/fedeo-updater/control.sock")
|
||||
args = parser.parse_args()
|
||||
if os.geteuid() != 0:
|
||||
parser.error("Der Host-Dienst muss als root laufen.")
|
||||
os.umask(0o077)
|
||||
try:
|
||||
serve(args.directory, args.socket)
|
||||
except (OSError, engine.UpdateError) as error:
|
||||
parser.exit(1, f"Update-Dienst konnte nicht gestartet werden: {error}\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -37,6 +37,10 @@ class UpdateError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class PlanChangedError(UpdateError):
|
||||
pass
|
||||
|
||||
|
||||
def now():
|
||||
return datetime.datetime.now(datetime.timezone.utc).isoformat()
|
||||
|
||||
@@ -288,7 +292,7 @@ class Updater:
|
||||
if shutil.disk_usage(directory).free < required:
|
||||
raise UpdateError(f"Zu wenig freier Speicher auf {directory}: mindestens {required // GIB + 1} GiB erforderlich.")
|
||||
|
||||
def prepare(self, version):
|
||||
def prepare(self, version, *, allow_current=False):
|
||||
for name in (".env", "docker-compose.yml"):
|
||||
path = self.root / name
|
||||
if not path.is_file() or path.is_symlink():
|
||||
@@ -318,7 +322,8 @@ class Updater:
|
||||
self.target = self.manifest["version"]
|
||||
self.save("validating", version=self.target, revision=self.manifest["commit"])
|
||||
old_version = self.current["services"]["backend"].get("environment", {}).get("FEDEO_VERSION", "dev")
|
||||
if VERSION.fullmatch(old_version) and tuple(map(int, old_version.split("."))) >= tuple(map(int, self.target.split("."))):
|
||||
update_available = not VERSION.fullmatch(old_version) or tuple(map(int, old_version.split("."))) < tuple(map(int, self.target.split(".")))
|
||||
if not allow_current and not update_available:
|
||||
raise UpdateError("Zielversion muss neuer sein; Downgrades und erneute Installation sind gesperrt.")
|
||||
previous = read_json(self.home / "installed.json", {})
|
||||
write_json(self.job / "original-installed.json", previous)
|
||||
@@ -381,7 +386,11 @@ class Updater:
|
||||
"images": self.images, "composeDiff": str(self.job / "compose.diff"),
|
||||
"releaseNotes": str(self.job / "CHANGELOG.md"), "requiresComposeAdoption": not self.managed,
|
||||
"estimatedBackupBytes": self.backup_size, "services": list(self.candidate["services"]),
|
||||
"retiredServices": self.retired_services, "retainedDataPaths": self.matrix_backup_paths}
|
||||
"retiredServices": self.retired_services, "retainedDataPaths": self.matrix_backup_paths,
|
||||
"updateAvailable": update_available,
|
||||
# Kept in private job records; the service never exposes configuration fingerprints.
|
||||
"approvalFingerprint": sha(json.dumps({"manifest": self.manifest, "configuration": self.before,
|
||||
"targetCompose": sha((self.job / "target.compose.yml").read_bytes())}, sort_keys=True).encode())}
|
||||
write_json(self.job / "plan.json", plan)
|
||||
return plan
|
||||
|
||||
@@ -593,7 +602,7 @@ class Updater:
|
||||
write_json(self.home / "installed.json", {"manifest": self.manifest, "composeSha256": sha((self.root / "docker-compose.yml").read_bytes()), "installedAt": now()})
|
||||
self.save("complete", status="succeeded", recoveryRequired=False, maintenance=False)
|
||||
|
||||
def execute(self, command, version=None, accept_compose=False):
|
||||
def execute(self, command, version=None, accept_compose=False, *, expected_fingerprint=None, request_id=None):
|
||||
with self.lock():
|
||||
previous = read_json(self.home / "status.json", {})
|
||||
if previous.get("recoveryRequired"):
|
||||
@@ -602,9 +611,13 @@ class Updater:
|
||||
self.job.mkdir(mode=0o700)
|
||||
self.state = {"status": "running", "phase": "preparing", "job": str(self.job), "command": command,
|
||||
"startedAt": now(), "recoveryRequired": False, "mutationStarted": False}
|
||||
if request_id:
|
||||
self.state["requestId"] = request_id
|
||||
self.save()
|
||||
try:
|
||||
plan = self.prepare(version)
|
||||
plan = self.prepare(version, allow_current=command == "check")
|
||||
if expected_fingerprint is not None and plan["approvalFingerprint"] != expected_fingerprint:
|
||||
raise PlanChangedError("Release oder Konfiguration haben sich seit der Prüfung geändert; erneut prüfen.")
|
||||
if command == "check":
|
||||
self.save("checked", status="checked")
|
||||
else:
|
||||
|
||||
268
scripts/update/tests/test_service.py
Normal file
268
scripts/update/tests/test_service.py
Normal file
@@ -0,0 +1,268 @@
|
||||
import concurrent.futures
|
||||
import http.client
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
import socket
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[3]
|
||||
spec = importlib.util.spec_from_file_location('update_service', ROOT / 'scripts/selfhost-update-service.py')
|
||||
s = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(s)
|
||||
|
||||
|
||||
class Factory:
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
self.started = threading.Event()
|
||||
self.release = threading.Event()
|
||||
self.release.set()
|
||||
self.failure = None
|
||||
self.adoption = False
|
||||
self.available = True
|
||||
self.fingerprint = 'a' * 64
|
||||
|
||||
def __call__(self, directory):
|
||||
factory = self
|
||||
|
||||
class FakeUpdater(s.engine.Updater):
|
||||
def execute(self, command, version, *, expected_fingerprint, request_id):
|
||||
with self.lock():
|
||||
factory.calls.append((command, version, expected_fingerprint, request_id))
|
||||
self.job = self.home / ('job-' + request_id)
|
||||
self.job.mkdir()
|
||||
self.state = {'requestId': request_id, 'status': 'running', 'phase': 'validating',
|
||||
'job': str(self.job), 'recoveryRequired': False}
|
||||
self.save()
|
||||
factory.started.set()
|
||||
if not factory.release.wait(10):
|
||||
raise AssertionError('test worker timed out')
|
||||
if factory.failure:
|
||||
self.save(status='failed')
|
||||
raise factory.failure
|
||||
if expected_fingerprint and expected_fingerprint != factory.fingerprint:
|
||||
self.save(status='failed')
|
||||
raise s.engine.PlanChangedError('private changed configuration')
|
||||
plan = {'version': version or '3.2.0', 'revision': 'b' * 40,
|
||||
'updateAvailable': factory.available, 'requiresComposeAdoption': factory.adoption,
|
||||
'estimatedBackupBytes': 1000, 'approvalFingerprint': factory.fingerprint,
|
||||
'composeDiff': '/private/path/secret.diff', 'retiredServices': ['matrix-db']}
|
||||
self.save(status='checked' if command == 'check' else 'succeeded', phase='complete')
|
||||
return {**self.state, 'plan': plan}
|
||||
|
||||
return FakeUpdater(directory)
|
||||
|
||||
|
||||
class UnixConnection(http.client.HTTPConnection):
|
||||
def __init__(self, path):
|
||||
super().__init__('localhost', timeout=3)
|
||||
self.path = str(path)
|
||||
|
||||
def connect(self):
|
||||
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
self.sock.settimeout(self.timeout)
|
||||
self.sock.connect(self.path)
|
||||
|
||||
|
||||
class ServiceTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.root = Path(self.temp.name)
|
||||
self.factory = Factory()
|
||||
self.controller = s.Controller(self.root, updater_factory=self.factory)
|
||||
self.socket = self.root / 'control.sock'
|
||||
self.server = s.Server(self.socket, self.controller)
|
||||
self.thread = threading.Thread(target=self.server.serve_forever, kwargs={'poll_interval': 0.01}, daemon=True)
|
||||
self.thread.start()
|
||||
|
||||
def tearDown(self):
|
||||
self.factory.release.set()
|
||||
self.server.shutdown()
|
||||
self.server.server_close()
|
||||
self.thread.join(3)
|
||||
self.controller.close()
|
||||
self.temp.cleanup()
|
||||
|
||||
def request(self, method, path, body=None, headers=None):
|
||||
connection = UnixConnection(self.socket)
|
||||
connection.request(method, path, None if body is None else json.dumps(body),
|
||||
headers or ({'Content-Type': 'application/json'} if body is not None else {}))
|
||||
response = connection.getresponse()
|
||||
result = response.status, json.loads(response.read())
|
||||
self.assertEqual(response.getheader('Cache-Control'), 'no-store')
|
||||
connection.close()
|
||||
return result
|
||||
|
||||
def finish(self):
|
||||
self.controller.worker.join(3)
|
||||
self.assertFalse(self.controller.worker.is_alive())
|
||||
|
||||
def check(self, request_id='check-request-01'):
|
||||
status, _ = self.request('POST', '/v1/check', {'requestId': request_id, 'version': '3.2.0'})
|
||||
self.assertEqual(status, 202)
|
||||
self.finish()
|
||||
return request_id
|
||||
|
||||
def test_async_job_survives_client_disconnect_and_status_stays_responsive(self):
|
||||
self.factory.release.clear()
|
||||
connection = UnixConnection(self.socket)
|
||||
connection.request('POST', '/v1/check', json.dumps({'requestId': 'check-disconnected'}),
|
||||
{'Content-Type': 'application/json'})
|
||||
connection.close()
|
||||
self.assertTrue(self.factory.started.wait(3))
|
||||
status, result = self.request('GET', '/v1/status')
|
||||
self.assertEqual(status, 200)
|
||||
self.assertEqual(result['request']['status'], 'running')
|
||||
self.assertEqual(result['request']['progress']['phase'], 'validating')
|
||||
self.factory.release.set()
|
||||
self.finish()
|
||||
self.assertEqual(self.request('GET', '/v1/requests/check-disconnected')[1]['status'], 'checked')
|
||||
|
||||
def test_duplicate_request_is_idempotent_even_after_completion_and_restart(self):
|
||||
request_id = self.check()
|
||||
status, result = self.request('POST', '/v1/check', {'requestId': request_id, 'version': '3.2.0'})
|
||||
self.assertEqual(status, 200)
|
||||
self.assertEqual(result['status'], 'checked')
|
||||
self.controller.close()
|
||||
self.controller = s.Controller(self.root, updater_factory=self.factory)
|
||||
self.server.controller = self.controller
|
||||
status, _ = self.request('POST', '/v1/check', {'requestId': request_id, 'version': '3.2.0'})
|
||||
self.assertEqual(status, 200)
|
||||
self.assertEqual(len(self.factory.calls), 1)
|
||||
status, result = self.request('POST', '/v1/check', {'requestId': request_id, 'version': '3.3.0'})
|
||||
self.assertEqual(status, 409)
|
||||
self.assertEqual(result['error']['code'], 'REQUEST_CONFLICT')
|
||||
|
||||
def test_concurrent_submissions_start_exactly_one_worker(self):
|
||||
self.factory.release.clear()
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as pool:
|
||||
responses = list(pool.map(lambda i: self.request('POST', '/v1/check', {'requestId': f'parallel-{i}'}), range(2)))
|
||||
self.assertEqual(sorted(r[0] for r in responses), [202, 409])
|
||||
self.assertTrue(self.factory.started.wait(3))
|
||||
self.assertEqual(len(self.factory.calls), 1)
|
||||
|
||||
def test_apply_requires_reviewed_version_and_passes_private_fingerprint(self):
|
||||
body = {'requestId': 'apply-request-01', 'version': '3.2.0', 'checkRequestId': 'check-request-01'}
|
||||
self.assertEqual(self.request('POST', '/v1/apply', body)[1]['error']['code'], 'CHECK_REQUIRED')
|
||||
self.check()
|
||||
self.assertEqual(self.request('POST', '/v1/apply', {**body, 'version': '3.3.0'})[0], 409)
|
||||
self.assertEqual(self.request('POST', '/v1/apply', body)[0], 202)
|
||||
self.finish()
|
||||
self.assertEqual(self.factory.calls[-1], ('apply', '3.2.0', 'a' * 64, 'apply-request-01'))
|
||||
self.assertEqual(self.request('GET', '/v1/requests/apply-request-01')[1]['status'], 'succeeded')
|
||||
|
||||
def test_unmanaged_compose_and_changed_plan_cannot_be_applied_remotely(self):
|
||||
self.factory.adoption = True
|
||||
self.check()
|
||||
body = {'requestId': 'apply-request-01', 'version': '3.2.0', 'checkRequestId': 'check-request-01'}
|
||||
self.assertEqual(self.request('POST', '/v1/apply', body)[1]['error']['code'], 'COMPOSE_REVIEW_REQUIRED')
|
||||
self.factory.adoption = False
|
||||
self.check('check-request-02')
|
||||
self.factory.fingerprint = 'b' * 64
|
||||
self.assertEqual(self.request('POST', '/v1/apply', {**body, 'checkRequestId': 'check-request-02'})[0], 202)
|
||||
self.finish()
|
||||
self.assertEqual(self.request('GET', '/v1/requests/apply-request-01')[1]['error']['code'], 'PLAN_CHANGED')
|
||||
|
||||
def test_already_current_is_a_successful_check_without_apply(self):
|
||||
self.factory.available = False
|
||||
self.check()
|
||||
self.assertEqual(self.request('GET', '/v1/requests/check-request-01')[1]['status'], 'up_to_date')
|
||||
body = {'requestId': 'apply-request-01', 'version': '3.2.0', 'checkRequestId': 'check-request-01'}
|
||||
self.assertEqual(self.request('POST', '/v1/apply', body)[0], 409)
|
||||
|
||||
def test_restart_marks_pending_requests_interrupted_without_replaying(self):
|
||||
for status in ('queued', 'running'):
|
||||
s.engine.write_json(self.controller.path('abandoned-' + status),
|
||||
{'requestId': 'abandoned-' + status, 'command': 'apply', 'status': status})
|
||||
self.controller.close()
|
||||
self.controller = s.Controller(self.root, updater_factory=self.factory)
|
||||
self.server.controller = self.controller
|
||||
for status in ('queued', 'running'):
|
||||
record = self.request('GET', '/v1/requests/abandoned-' + status)[1]
|
||||
self.assertEqual(record['status'], 'interrupted')
|
||||
self.assertEqual(self.factory.calls, [])
|
||||
|
||||
def test_shutdown_drains_worker_and_rejects_new_work(self):
|
||||
self.factory.release.clear()
|
||||
self.assertEqual(self.request('POST', '/v1/check', {'requestId': 'check-request-01'})[0], 202)
|
||||
self.assertTrue(self.factory.started.wait(3))
|
||||
closing = threading.Thread(target=self.controller.close)
|
||||
closing.start()
|
||||
# Acquire the gate after close has entered it, without relying on a wall-clock sleep.
|
||||
for _ in range(1000):
|
||||
with self.controller.gate:
|
||||
if self.controller.closing:
|
||||
break
|
||||
threading.Event().wait(0.001)
|
||||
self.assertTrue(closing.is_alive())
|
||||
self.assertEqual(self.request('POST', '/v1/check', {'requestId': 'check-request-02'})[1]['error']['code'], 'SHUTTING_DOWN')
|
||||
self.factory.release.set()
|
||||
closing.join(3)
|
||||
self.assertFalse(closing.is_alive())
|
||||
self.assertEqual(self.controller.request_status('check-request-01')['status'], 'checked')
|
||||
|
||||
def test_cli_lock_and_recovery_block_new_requests(self):
|
||||
updater = self.factory(self.root)
|
||||
job = updater.home / 'cli-job'
|
||||
job.mkdir()
|
||||
s.engine.write_json(updater.home / 'status.json', {'status': 'running', 'job': str(job)})
|
||||
with updater.lock():
|
||||
self.assertEqual(self.request('POST', '/v1/check', {'requestId': 'check-request-01'})[1]['error']['code'], 'BUSY')
|
||||
s.engine.write_json(updater.home / 'status.json', {'status': 'failed', 'recoveryRequired': True})
|
||||
self.assertEqual(self.request('POST', '/v1/check', {'requestId': 'check-request-01'})[1]['error']['code'], 'RECOVERY_REQUIRED')
|
||||
|
||||
def test_status_never_returns_raw_errors_paths_logs_or_configuration_fingerprints(self):
|
||||
self.factory.failure = ValueError('password-secret /opt/private/.env')
|
||||
self.check()
|
||||
result = self.request('GET', '/v1/requests/check-request-01')[1]
|
||||
self.assertEqual(result['error']['code'], 'UPDATE_FAILED')
|
||||
self.assertNotIn('password-secret', json.dumps(result))
|
||||
self.factory.failure = None
|
||||
self.check('check-request-02')
|
||||
payload = json.dumps(self.request('GET', '/v1/requests/check-request-02')[1])
|
||||
self.assertNotIn('/private', payload)
|
||||
self.assertNotIn('approvalFingerprint', payload)
|
||||
self.assertNotIn('a' * 64, payload)
|
||||
self.assertEqual(self.request('GET', '/v1/requests/../../secret')[0], 400)
|
||||
self.assertEqual(self.request('GET', '/v1/logs')[0], 404)
|
||||
self.assertEqual((self.controller.home / 'check-request-01.log').stat().st_mode & 0o777, 0o600)
|
||||
|
||||
def test_api_rejects_unsupported_commands_options_bodies_and_browser_requests(self):
|
||||
for body in [[], {'requestId': 'bad/path'}, {'requestId': 'check-request-01', 'version': ';reboot'},
|
||||
{'requestId': 'check-request-01', 'directory': '/etc'},
|
||||
{'requestId': 'check-request-01', 'acceptComposeReplacement': True}]:
|
||||
with self.subTest(body=body):
|
||||
self.assertEqual(self.request('POST', '/v1/check', body)[0], 400)
|
||||
self.assertEqual(self.request('POST', '/v1/resolve', {'requestId': 'resolve-request'})[0], 404)
|
||||
self.assertEqual(self.request('POST', '/v1/check', {'requestId': 'check-request-01'},
|
||||
{'Content-Type': 'application/json', 'Origin': 'https://example.test'})[0], 400)
|
||||
self.assertEqual(self.request('POST', '/v1/check', {'requestId': 'check-request-01', 'padding': 'x' * 5000})[0], 400)
|
||||
self.assertEqual(self.factory.calls, [])
|
||||
|
||||
def test_second_controller_cannot_take_over_active_instance(self):
|
||||
self.assertEqual(self.socket.stat().st_mode & 0o777, 0o660)
|
||||
with self.assertRaisesRegex(s.engine.UpdateError, 'bereits ein Update-Dienst'):
|
||||
s.Controller(self.root, updater_factory=self.factory)
|
||||
|
||||
|
||||
class InstallerTests(unittest.TestCase):
|
||||
def test_unit_render_is_read_only_and_escapes_systemd_expansions(self):
|
||||
installer = ROOT / 'scripts/selfhost-update-service-install.sh'
|
||||
subprocess.run(['bash', '-n', str(installer)], check=True)
|
||||
result = subprocess.run(['bash', str(installer), '--print-unit', '--directory', '/opt/fedeo test%$"'],
|
||||
text=True, capture_output=True, check=True)
|
||||
self.assertIn('serve --directory "/opt/fedeo test%%$$\\\""', result.stdout)
|
||||
self.assertIn('KillMode=mixed', result.stdout)
|
||||
self.assertIn('TimeoutStopSec=infinity', result.stdout)
|
||||
self.assertNotIn('@FEDEO_DIRECTORY@', result.stdout)
|
||||
invalid = subprocess.run(['bash', str(installer), '--print-unit', '--directory', '/opt/fedeo\nExecStart=reboot'],
|
||||
capture_output=True)
|
||||
self.assertNotEqual(invalid.returncode, 0)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -567,6 +567,37 @@ class UpdateTests(unittest.TestCase):
|
||||
self.run_update()
|
||||
self.assertFalse(self.called('stop'))
|
||||
|
||||
def test_check_current_release_reports_no_available_update(self):
|
||||
(self.root / '.env').write_bytes(u.set_version(self.original_env, '3.1.0'))
|
||||
result = self.updater.execute('check', '3.1.0')
|
||||
self.assertEqual(result['status'], 'checked')
|
||||
self.assertFalse(result['plan']['updateAvailable'])
|
||||
self.assertFalse(self.called('pull'))
|
||||
with self.assertRaisesRegex(u.UpdateError, 'erneute Installation'):
|
||||
self.run_update()
|
||||
|
||||
def test_reviewed_plan_can_be_applied_but_configuration_changes_require_recheck(self):
|
||||
result = self.updater.execute('check', '3.1.0', request_id='check-request')
|
||||
self.assertEqual(self.updater.status()['requestId'], 'check-request')
|
||||
fingerprint = result['plan']['approvalFingerprint']
|
||||
(self.root / '.env').write_bytes(self.original_env + b'\nCUSTOM_SETTING=changed\n')
|
||||
with self.assertRaises(u.PlanChangedError):
|
||||
self.run_update(expected_fingerprint=fingerprint)
|
||||
self.assertFalse(self.called('pull'))
|
||||
self.assertFalse(self.called('stop'))
|
||||
result = self.updater.execute('check', '3.1.0')
|
||||
self.assertEqual(self.run_update(expected_fingerprint=result['plan']['approvalFingerprint'])['status'], 'succeeded')
|
||||
|
||||
def test_changed_release_digests_cannot_reuse_an_approved_plan(self):
|
||||
result = self.updater.execute('check', '3.1.0')
|
||||
self.http.manifest['images'][0]['digest'] = 'sha256:' + 'c' * 64
|
||||
self.http.files['release-manifest.json'] = data(self.http.manifest)
|
||||
self.http.refresh()
|
||||
with self.assertRaises(u.PlanChangedError):
|
||||
self.run_update(expected_fingerprint=result['plan']['approvalFingerprint'])
|
||||
self.assert_untouched()
|
||||
self.assertFalse(self.called('pull'))
|
||||
|
||||
def test_backup_integrity_manifest_is_complete(self):
|
||||
result = self.run_update()
|
||||
backup = Path(result['backup'])
|
||||
|
||||
Reference in New Issue
Block a user