From b4777a5e4d77f8a86b2eb8752957a7f9ad9bc906 Mon Sep 17 00:00:00 2001 From: Lord Race Date: Tue, 11 Aug 2026 12:09:26 -0500 Subject: [PATCH] rewire 2026-08-11: single-mount NFS architecture MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - topology.svg, container-tree.svg, data-flow.svg: show /srv/nc-files bound directly into nextcloud container at /mnt/ncdata (no intermediate /mnt/nc-data/nextcloud-data layer). - architecture.html: explain rewire, update storage table, remove pre-rewire 'why two layers' section. - procedure.html: NEXTCLOUD_DATADIR=/srv/nc-files, no mastercontainer bind, add §2.5 Rewire 2026-08-11 with recovery procedure, update backup script to also tar local nextcloud app volume, fix 4.3 verification (302 -> /login, status.php, file visibility). - operations.html: 4 backup files now (added aio-nextcloud-app.tar.gz), restore procedures updated, single-file restore uses new tar layout. - troubleshooting.html: add 3 new sections — appdata-missing, nextcloud-not-spawning, collabora-discovery-warning. - request-flow.svg: remove nextcloud/ subdir from NFS write path. - index.html, README.md: update paths and metadata. --- README.md | 10 +- architecture.html | 54 ++++++--- assets/diagrams/container-tree.svg | 22 ++-- assets/diagrams/data-flow.svg | 33 +++--- assets/diagrams/request-flow.svg | 4 +- assets/diagrams/topology.svg | 73 ++++++------ index.html | 6 +- operations.html | 37 +++++-- procedure.html | 134 ++++++++++++++++++---- troubleshooting.html | 171 ++++++++++++++++++++++++++++- 10 files changed, 423 insertions(+), 121 deletions(-) diff --git a/README.md b/README.md index 8afc875..020e7a6 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,11 @@ # Nextcloud Office -Session log + runbook for the 2026-08-10 deployment of `office.rmf44.xyz` -as a Nextcloud All-in-One stack with Collabora + Whiteboard on -`homework03`, replacing the retired OnlyOffice container on `hawker`. +Session log + runbook for the 2026-08-10 deployment and 2026-08-11 +rewire of `office.rmf44.xyz` as a Nextcloud All-in-One stack with +Collabora + Whiteboard on `homework03`, replacing the retired +OnlyOffice container on `hawker`. The 2026-08-11 rewire collapsed the +storage layer to a single NFS bind (post-mortem in +`troubleshooting.html#appdata-missing`). ## Files @@ -16,7 +19,6 @@ as a Nextcloud All-in-One stack with Collabora + Whiteboard on - `assets/diagrams/container-tree.svg` — 8 AIO containers + bind mounts - `assets/diagrams/data-flow.svg` — NFS vs ext4 split, database on host - `assets/diagrams/request-flow.svg` — swimlane sequence of a `git clone`-equivalent -- `assets/diagrams/backup-pipeline.svg` — hector timer → homework03 → desslok NFS ## How to view diff --git a/architecture.html b/architecture.html index b57210f..4b3de60 100644 --- a/architecture.html +++ b/architecture.html @@ -23,8 +23,9 @@

Three layers to understand: network (public → Caddy → NetBird → AIO Apache), containers (8 AIO processes on - one host, single docker network), and data flow - (NFS for everything, plus a postgres dump that hits NFS too). + one host, host network namespace shared with mastercontainer), and + data flow (NFS for user files, local ext4 for + everything else, plus a daily postgres dump that lands on NFS).

Topology — public ingress

@@ -58,9 +59,11 @@
  • Storage — NFSv4.1 from desslok:/slab/container_storage/office mounted at - /srv/nc-files/ on homework03. Subdirs: - nextcloud/ for user files, backups/ for - daily pgdump + config + user-files tars. + /srv/nc-files on homework03. User files live at + the NFS export root (no intermediate nextcloud/ + subdir): admin/, race/, + appdata_*/, plus backups/ for the + daily backup pipeline.
  • Retired (torn down) — OnlyOffice container @@ -99,7 +102,7 @@ nextcloud-aio-nextcloud PHP-FPM + Nextcloud app code - ./nextcloud-aio-nextcloud/ + /mnt/nc-data/nextcloud-data via NEXTCLOUD_DATADIR + ./nextcloud-aio-nextcloud/ (local volume) + /srv/nc-files (NFS) → /mnt/ncdata root (entrypoint) :9000 (PHP-FPM) @@ -172,20 +175,21 @@

    Data flow — where each piece lives

    - Most of AIO's data is on NFS (/srv/nc-files on - homework03). The Postgres database is inside the database - container; its data directory is a docker named-volume bind, not - on NFS — keeping PostgreSQL's WAL writes off NFS is critical for - durability. + Most of AIO's user data is on NFS + (/srv/nc-files on homework03 → container bind at + /mnt/ncdata). Postgres, Redis, and the AIO + mastercontainer's configuration live on local ext4 (named + volumes) — keeping PostgreSQL's WAL writes off NFS is critical + for durability.

    Data flow — NFS for user files, named volumes for container state

    - + - + @@ -193,9 +197,9 @@ - + - + @@ -204,6 +208,24 @@
    PathFilesystemWhy
    /srv/nc-files/nextcloud//srv/nc-files/ (NFS root on homework03) NFSv4.1 from desslokUser-uploaded files. Snapshotted daily via desslok's existing ZFS path.User-uploaded files live at the export root: admin/, race/, appdata_*/, etc. The nextcloud container binds this path to /mnt/ncdata directly. Snapshotted daily via desslok's ZFS path.
    /srv/nc-files/backups/Daily pgdump + AIO config tar + user-files tar. 14-day retention.
    /mnt/nc-data/nextcloud-data//usr/local/containers/nextcloudaio/nextcloud-aio-nextcloud/_data/ ext4 (local)Bind mount, mounted INTO the nextcloud container as /nextcloud-aio. Holds app config, theme, install state.Local named-volume bind for the nextcloud container. AIO-managed; NEXTCLOUD_DATADIR points at /srv/nc-files separately, NOT at this volume.
    /usr/local/containers/nextcloudaio/nextcloud-aio-{mastercontainer,database,redis,apache,...}/
    +
    +

    + Rewire 2026-08-11: before the rewire, AIO + bind-mounted a third host layer + (/mnt/nc-data/nextcloud-data) into the nextcloud + container, with NEXTCLOUD_DATADIR=/mnt/nc-data/nextcloud-data. + That double-bind worked but had two sharp edges: (a) the + NEXTCLOUD_DATADIR host path and the actual NFS + mount path had to be kept in sync manually; (b) a typo would + silently create an empty datadir, masking the real NFS data + and triggering an "Appdata is not present" crash on next + start. The rewire collapses both to a single layer: + /srv/nc-files is the NFS mount AND the + NEXTCLOUD_DATADIR value — AIO binds it directly + into the nextcloud container at /mnt/ncdata. +

    +
    +

    Why isn't the postgres data on NFS? @@ -232,7 +254,7 @@

  • NetBird tunnel encapsulates the request in WireGuard (UDP 51820), P2P from hawker to homework03.
  • AIO Apache (nextcloud-aio-apache container) terminates the TLS-stripped HTTP and forwards to 127.0.0.1:9000 (PHP-FPM in nextcloud-aio-nextcloud).
  • PHP-FPM (Nextcloud) authenticates the user via the session cookie, authorizes the path under /admin/files/, and writes the file via WebDAV.
    • -
  • NFS write of the file to /srv/nc-files/nextcloud/admin/files/smoke-test.md on homework03 → /slab/container_storage/office/nextcloud/admin/files/smoke-test.md on desslok.
    • +
  • NFS write of the file to /srv/nc-files/admin/files/smoke-test.md on homework03 → /slab/container_storage/office/admin/files/smoke-test.md on desslok (nextcloud container's /mnt/ncdata is bound to /srv/nc-files directly).
  • Response: HTTP/2 201 Created with empty body (WebDAV semantics).
    • diff --git a/assets/diagrams/container-tree.svg b/assets/diagrams/container-tree.svg index 0a7531c..df50b9e 100644 --- a/assets/diagrams/container-tree.svg +++ b/assets/diagrams/container-tree.svg @@ -11,6 +11,7 @@ .box-emp { fill: #1f2c3a; stroke: #4a78b5; stroke-width: 2; } .box-host { fill: #2a2535; stroke: #6b5aa0; stroke-width: 2; } .box-storage { fill: #1f2e28; stroke: #4a8a6b; stroke-width: 2; } + .box-bind { fill: #1f2e28; stroke: #6cba92; stroke-width: 2; stroke-dasharray: 3 3; } .arrow { stroke: #7aa2f7; stroke-width: 2; fill: none; marker-end: url(#arr-b); } .arrow-sto { stroke: #6cba92; stroke-width: 2; fill: none; marker-end: url(#arr-g); } .arrow-back { stroke: #d9a96b; stroke-width: 2; fill: none; marker-end: url(#arr-o); } @@ -29,14 +30,14 @@ Container Tree — AIO on homework03 - network_mode: host on mastercontainer · 8 active · 5 disabled + network_mode: host on mastercontainer · 8 active · 5 disabled · rewire 2026-08-11 homework03 (10.0.0.73) · Debian 13 · 15 GB Docker 29.6.2 · host network namespace shared with mastercontainer - + nextcloud-aio-mastercontainer all-in-one:latest @@ -52,11 +53,12 @@ 33:33 (www-data) - - nextcloud-aio-nextcloud + + nextcloud-aio-nextcloud PHP-FPM 8.3 + Nextcloud :9000 (PHP-FPM listen) - NEXTCLOUD_DATADIR bind + bind: /srv/nc-files → /mnt/ncdata + vol: nextcloud_aio_nextcloud → /var/www/html @@ -109,7 +111,7 @@ desslok NFS /slab/container_storage/office - NFSv4.1 · /srv/nc-files/ + NFSv4.1 · /srv/nc-files @@ -130,10 +132,10 @@ ?php-fpm - + - + @@ -147,5 +149,5 @@ writes pgdump - 2026-08-10 · nextcloud_office - \ No newline at end of file + 2026-08-11 · nextcloud_office · rewire + diff --git a/assets/diagrams/data-flow.svg b/assets/diagrams/data-flow.svg index 947e5b5..62a718a 100644 --- a/assets/diagrams/data-flow.svg +++ b/assets/diagrams/data-flow.svg @@ -11,9 +11,11 @@ .fs-local { fill: #1f2c3a; stroke: #4a78b5; stroke-width: 2; } .fs-nfs { fill: #1f2e28; stroke: #4a8a6b; stroke-width: 2; } .fs-named { fill: #2a2535; stroke: #6b5aa0; stroke-width: 2; } + .fs-bind { fill: #1f2e28; stroke: #6cba92; stroke-width: 2; stroke-dasharray: 3 3; } .arrow-nfs { stroke: #6cba92; stroke-width: 2; fill: none; marker-end: url(#arr-g); } .arrow-local { stroke: #7aa2f7; stroke-width: 2; fill: none; marker-end: url(#arr-b); } .arrow-named { stroke: #9d7ad9; stroke-width: 2; fill: none; marker-end: url(#arr-p); } + .arrow-bind { stroke: #6cba92; stroke-width: 2.5; fill: none; stroke-dasharray: 3 3; marker-end: url(#arr-g); } @@ -29,7 +31,7 @@ Data Flow — where each piece lives - NFS for user data + backups · ext4 for container state · local named volumes for postgres + NFS at /srv/nc-files → direct bind into nextcloud container at /mnt/ncdata · rewire 2026-08-11 homework03 (local ext4) desslok (NFS v4.1) @@ -38,9 +40,10 @@ / (ext4) - - /mnt/nc-data/nextcloud-data - → nextcloud-aio-nextcloud:/nextcloud-aio/data + + + /srv/nc-files + NFS v4.1 mount → nextcloud-aio-nextcloud:/mnt/ncdata mastercontainer @@ -79,9 +82,9 @@ /slab/container_storage/office (NFS) - nextcloud/ - user-uploaded files - mounted at /srv/nc-files/nextcloud/ on homework03 + office/ (NFS root) + user-uploaded files at NFS ROOT (no nextcloud/ subdir) + admin/ · race/ · appdata_*/ · *.log · .htaccess backups/ @@ -93,14 +96,18 @@ ZFS snapshot policy (desslok) /slab is on a ZFS pool with periodic snapshots - nightly snapshot → nextcloud/ and backups/ both covered + nightly snapshot → office/ and backups/ both covered → true point-in-time recovery available independent of our daily backup daily backup is belt-and-suspenders, ZFS snapshots are the primary - - - read/write + + + NFS mount + + + + bind @@ -117,5 +124,5 @@ postgres WAL writes stay LOCAL (named volume) — PostgreSQL is sensitive to NFS close-to-open consistency - 2026-08-10 · nextcloud_office - \ No newline at end of file + 2026-08-11 · nextcloud_office · rewire + diff --git a/assets/diagrams/request-flow.svg b/assets/diagrams/request-flow.svg index 65129a3..c7937fd 100644 --- a/assets/diagrams/request-flow.svg +++ b/assets/diagrams/request-flow.svg @@ -99,7 +99,7 @@ NFS write → /slab/container_storage/office/ - nextcloud/admin/files/ + admin/files/ @@ -134,5 +134,5 @@ Tip: WOPI (Collabora file open) follows a parallel path — browser iframe → apache → nextcloud PHP → wopi URL → apache proxy → collabora → WOPI read the WOPI URL is verified as http://nextcloud-aio-apache.nextcloud-aio:23973 (internal docker network only) - 2026-08-10 · nextcloud_office + 2026-08-11 · nextcloud_office · rewire \ No newline at end of file diff --git a/assets/diagrams/topology.svg b/assets/diagrams/topology.svg index 8f8df9f..ad734fc 100644 --- a/assets/diagrams/topology.svg +++ b/assets/diagrams/topology.svg @@ -1,4 +1,4 @@ - + @@ -33,10 +35,10 @@ - + Nextcloud Office — Public Topology - office.rmf44.xyz · Caddy on hawker · AIO on homework03 · NFS on desslok + office.rmf44.xyz · Caddy on hawker · AIO on homework03 · NFS on desslok · rewire 2026-08-11 Public Internet @@ -71,12 +73,6 @@ NetBird (wt0) 100.79.4.103 · P2P mesh - - + retired OnlyOffice torn down - see procedure.html §4.4 - - [retired] onlyoffice-files-1 - homework03 (10.0.0.73) @@ -98,16 +94,16 @@ nextcloud-aio-apache :11000 → PHP-FPM - - - 5 sidecars - nextcloud · database · redis - collabora · whiteboard - notify-push · database-dump + + + nextcloud-aio-nextcloud + /srv/nc-files → /mnt/ncdata + (named vol) → /var/www/html + NFS user data · local app - + - /srv/nc-files/ + /srv/nc-files NFS v4.1 mount @@ -120,18 +116,18 @@ office/ - nextcloud/ + admin/ race/ backups/ - - Daily backup pipeline (hector → homework03 → NFS) - 03:30 UTC, systemd timer on hector - ssh homework03 sudo /usr/local/bin/office-backup.sh - → pg_dumpall → /srv/nc-files/backups/office-YYYYMMDD-*.gz - retention: 14 days, prunes via find -mtime +14 + + Daily backup pipeline (hector → homework03 → NFS) + 03:30 UTC, systemd timer on hector + ssh homework03 sudo /usr/local/bin/office-backup.sh + → pg_dumpall → /srv/nc-files/backups/office-YYYYMMDD-*.gz + retention: 14 days, prunes via find -mtime +14 @@ -142,39 +138,42 @@ 2. HTTPS - - - WireGuard P2P · UDP 51820 - - - upstream :11000 - + NFS + + + bind + :mnt/ncdata + + + + app vol + - + writes - + public mesh storage - - retired + + NFS bind - 2026-08-10 · nextcloud_office - \ No newline at end of file + 2026-08-11 · nextcloud_office · rewire + diff --git a/index.html b/index.html index 72e56f9..06bceec 100644 --- a/index.html +++ b/index.html @@ -119,9 +119,9 @@ PathHostWhat /usr/local/containers/nextcloudaio/docker-compose.yamlhomework03Mastercontainer with network_mode: host /usr/local/containers/nextcloudaio/nextcloud-aio-{mastercontainer,database,redis,apache,nextcloud,collabora,whiteboard,notify-push,database-dump}/homework03Named docker volume bind targets - /srv/nc-files/homework03NFS mount of desslok:/slab/container_storage/office - /mnt/nc-data/nextcloud-data/homework03Bind into nextcloud container at /nextcloud-aio/data - /usr/local/bin/office-backup.shhomework03Daily backup script (pgdump + config tar + user files tar) + /srv/nc-files/homework03NFS mount of desslok:/slab/container_storage/office; AIO binds this directly into the nextcloud container at /mnt/ncdata via NEXTCLOUD_DATADIR + /usr/local/containers/nextcloudaio/nextcloud-aio-nextcloud/homework03Local docker volume bind for the nextcloud container's /var/www/html (AIO-managed app code) + /usr/local/bin/office-backup.shhomework03Daily backup script (pgdump + AIO config tar + app volume tar + user files tar) /etc/systemd/system/office-backup.{service,timer}hectorDaily 03:30 UTC trigger, SSH to homework03 /etc/caddy/CaddyfilehawkerReverse proxy block: office.rmf44.xyz → 100.79.142.164:11000 /slab/container_storage/office/desslokLive data + backups/ subdir diff --git a/operations.html b/operations.html index 44b078c..e3ca9d2 100644 --- a/operations.html +++ b/operations.html @@ -28,7 +28,7 @@

    Backup pipeline

    - Three files written daily to + Four files written daily to /srv/nc-files/backups/ on homework03 (NFS, real path /slab/container_storage/office/backups/ on desslok):

    @@ -47,9 +47,15 @@ ~6 KB Reconstruct the AIO install state without going through the setup wizard again. + + office-YYYYMMDD-aio-nextcloud-app.tar.gz + Tar of /usr/local/containers/nextcloudaio/nextcloud-aio-nextcloud/ — the AIO-managed local app volume (Nextcloud app code, installed apps, config/). + ~200-500 MB depending on installed apps + Survives a fresh AIO install: restore this AND the config tarball to skip the entire setup wizard and preserve installed apps. + office-YYYYMMDD-ncdata.tar.gz - Tar of /srv/nc-files/nextcloud/ (user-uploaded files) — excludes backups/ to avoid recursion. + Tar of /srv/nc-files/ root (user files: admin/, race/, appdata_*/, etc.) — excludes backups/ to avoid recursion. Empty (~100 B) until users upload files, then grows Restore user files after data loss. @@ -93,12 +99,12 @@ systemctl status office-backup.service | head -5
  • NFS quiescence — the tar reads - /srv/nc-files/nextcloud/ while the filesystem is - actively being written to by the nextcloud container. The tar - will see a consistent enough snapshot for crash-consistent - recovery; for true point-in-time recovery, you'd want to - quiesce Nextcloud (set maintenance mode) for the duration of - the tar, which we haven't done. + /srv/nc-files/ (the NFS root) while the + filesystem is actively being written to by the nextcloud + container. The tar will see a consistent enough snapshot for + crash-consistent recovery; for true point-in-time recovery, + you'd want to quiesce Nextcloud (set maintenance mode) for the + duration of the tar, which we haven't done.
    • @@ -142,9 +148,15 @@ sudo -n docker compose down tar -C /usr/local/containers/nextcloudaio -xzf "$LATEST"
  • - Restore user files (overwrites the NFS share's nextcloud/): + Restore the local nextcloud app volume (AIO-managed code + + installed apps): + 
    LATEST=$(ls -t /srv/nc-files/backups/office-*-aio-nextcloud-app.tar.gz | head -1)
+tar -C /usr/local/containers/nextcloudaio -xzf "$LATEST"
    +
    • +
  • + Restore user files (NFS root, no intermediate nextcloud/): 
    LATEST=$(ls -t /srv/nc-files/backups/office-*-ncdata.tar.gz | head -1)
-# Tar contains /nextcloud/ at root
+# Tar contains files at root (admin/, race/, appdata_*/, ...)
 tar -C /srv/nc-files -xzf "$LATEST"
  • @@ -172,8 +184,9 @@ curl -skI https://office.rmf44.xyz/login 

    ssh desslok
 LATEST=$(ls -t /slab/container_storage/office/backups/office-*-ncdata.tar.gz | head -1)
-tar -C / -xzf "$LATEST" nextcloud/admin/files/path/to/file
-# Adjust for the user + path
    +# Tar contains files at root; restore one user's file: +tar -C / -xzf "$LATEST" race/files/path/to/file +# Adjust for the user + path; user dirs are at NFS root (no nextcloud/ prefix)

    Re-initialize the admin user

    diff --git a/procedure.html b/procedure.html index fac336f..60f0d90 100644 --- a/procedure.html +++ b/procedure.html @@ -123,7 +123,7 @@ sudo -n mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600 \ desslok:/slab/container_storage/office /srv/nc-files df -h /srv/nc-files ls -la /srv/nc-files -# Expect: nextcloud/ backups/ +# Expect: admin/ race/ backups/ appdata_*/ nextcloud.log ...

    Add to /etc/fstab for boot persistence: @@ -146,8 +146,9 @@ for sub in mastercontainer database database-dump redis apache nextcloud \ sudo -n mkdir -p "/usr/local/containers/nextcloudaio/nextcloud-aio-$sub" done -# /mnt/nc-data for the Nextcloud data dir (lives on local ext4) -sudo -n mkdir -p /mnt/nc-data/nextcloud-data +# /srv/nc-files is the NFS mount. AIO bind-mounts it directly into +# the nextcloud container at /mnt/ncdata via NEXTCLOUD_DATADIR. +# No intermediate /mnt/nc-data layer — see "Rewire 2026-08-11" note. # Local backup stash (so the script can write the pgdump into NFS without recursion) ls -la /usr/local/containers/nextcloudaio/ @@ -166,6 +167,79 @@ sudo -n chown -R 999:999 /usr/local/containers/nextcloudaio/nextcloud-aio-redis sudo -n chown -R root:root /usr/local/containers/nextcloudaio/nextcloud-aio-nextcloud sudo -n chown -R 100:101 /usr/local/containers/nextcloudaio/nextcloud-aio-collabora +

    2.5 Rewire 2026-08-11 — single-mount architecture

    +

    + After the initial deploy we discovered a sharp edge: when + NEXTCLOUD_DATADIR is a different host path + than the actual NFS mount (the original design used + /srv/nc-files for NFS and + /mnt/nc-data/nextcloud-data for AIO), a typo on + either side silently creates an empty datadir inside the + nextcloud container. Nextcloud then refuses to start with + "Appdata directory is not present!" — but the empty datadir + is real, so the NFS data is still there, just not mounted. + That's the failure mode that took the office suite offline on + 2026-08-11. +

    +

    + The rewire removes the intermediate + /mnt/nc-data/nextcloud-data bind entirely: +

    +
      +
    • + NFS export desslok:/slab/container_storage/office + mounts at /srv/nc-files on homework03 (unchanged). +
      • +
    • + NEXTCLOUD_DATADIR=/srv/nc-files in compose AND + configuration.json's nextcloud_datadir + field both point at the same path. +
      • +
    • + AIO's containers.json template substitutes + %NEXTCLOUD_DATADIR% with that path and creates a + bind mount directly: host /srv/nc-files → + container /mnt/ncdata. +
      • +
    • + The mastercontainer no longer has the + /srv/nc-files or + /mnt/nc-data/nextcloud-data binds in its + compose volumes: section. +
      • +
    +

    + Recovery if it ever breaks again: +

    + 
    # 1. Confirm what's in the NFS export
+ssh desslok ls -la /slab/container_storage/office
+# Expect: admin/  race/  appdata_*/  ...
+
+# 2. Confirm the mount is healthy on homework03
+ssh homework03 df -h /srv/nc-files
+ssh homework03 ls -la /srv/nc-files
+# Expect: same admin/, race/, ... as step 1
+
+# 3. Check what AIO thinks the datadir is
+ssh homework03 sudo cat \
+  /var/lib/docker/volumes/nextcloud_aio_mastercontainer/_data/configuration.json \
+  | jq -r .nextcloud_datadir
+# Expect: "/srv/nc-files" — if not, fix with jq (see ~/.hermes
+# creds or the rewire script notes in this repo's history)
+
+# 4. Inspect what the running nextcloud container actually has bound
+ssh homework03 sudo docker inspect nextcloud-aio-nextcloud \
+  | jq -r '.[0].Mounts[] | "\(.Source) -> \(.Destination)"'
+# Expect: "/srv/nc-files -> /mnt/ncdata" AND
+#         "/var/lib/docker/volumes/nextcloud_aio_nextcloud/_data -> /var/www/html"
+# If /mnt/ncdata is bound to something else, the container has stale config.
+
+# 5. If the bind source is wrong, force AIO to re-spawn nextcloud:
+ssh homework03 sudo docker rm -f nextcloud-aio-nextcloud
+# Then trigger /api/docker/start from the admin UI (Apache must
+# be stopped first; the nextcloud container does NOT auto-spawn
+# on mastercontainer restart). See "Phase 6: Spawn lifecycle" below.
    +

    Phase 3 — AIO mastercontainer + setup wizard

    Write the compose file, start the mastercontainer, and walk the @@ -184,7 +258,7 @@ services: environment: APACHE_PORT: "11000" APACHE_DISABLE_REWRITE_IP: "1" - NEXTCLOUD_DATADIR: "/mnt/nc-data/nextcloud-data" + NEXTCLOUD_DATADIR: "/srv/nc-files" NEXTCLOUD_UPLOAD_LIMIT: "10G" NEXTCLOUD_MAX_TIME: "3600" AIO_DISABLE_BACKUP: "true" @@ -201,8 +275,11 @@ services: volumes: - ./nextcloud-aio-mastercontainer:/container-volume - /var/run/docker.sock:/var/run/docker.sock:ro - - /srv/nc-files:/srv/nc-files - - /mnt/nc-data/nextcloud-data:/mnt/nc-data/nextcloud-data + # NOTE: do NOT bind /srv/nc-files into the mastercontainer. + # AIO bind-mounts it directly into the nextcloud container via + # NEXTCLOUD_DATADIR. (Pre-rewire this entry also bound + # /mnt/nc-data/nextcloud-data — that intermediate layer was + # removed 2026-08-11.) EOF

    3.2 Start mastercontainer + pull the AIO passphrase

    @@ -239,7 +316,7 @@ hostname -I | awk '{print $1}' 
    ssh homework03
 sudo -n cat /usr/local/containers/nextcloudaio/nextcloud-aio-mastercontainer/configuration.json
 # Expect: "officeSuite": "collabora", "isWhiteboardEnabled": true,
-# "domain": "office.rmf44.xyz", "nextcloud_datadir": "/mnt/nc-data/nextcloud-data"
    +# "domain": "office.rmf44.xyz", "nextcloud_datadir": "/srv/nc-files"

    Phase 4 — Public ingress + cutover

    @@ -276,17 +353,24 @@ sudo -n docker exec caddy-caddy-1 caddy reload \ --config /etc/caddy/Caddyfile --adapter caddyfile

    4.3 Verify the cutover

    - 
    curl -skI https://office.rmf44.xyz/
-# HTTP/2 200
-# content-type: text/html; charset=UTF-8
-# ...
-curl -s https://office.rmf44.xyz/ | grep -oE '[^<]+'
-# <title>Login – Nextcloud</title>
+    
    curl -skI https://office.rmf44.xyz/
+  # HTTP/2 302
+  # location: /login
+  curl -sk https://office.rmf44.xyz/login | grep -oE '<title>[^<]+</title>'
+  # <title>Login - AIO</title>
 
-# Test login
-# 1. GET /login → grab requesttoken + cookies
-# 2. POST /login with user=admin + password + requesttoken
-# 3. Expect HTTP 303 → /apps/dashboard/
    
+  curl -sk https://office.rmf44.xyz/status.php
+  # {"installed":true,"version":"34.0.2.1","...","maintenance":false}
+
+  # Test login
+  # 1. GET /login → grab requesttoken + cookies
+  # 2. POST /login with user=admin + password + requesttoken
+  # 3. Expect HTTP 303 → /apps/dashboard/
+
+  # Verify the nextcloud container can see NFS user files
+  ssh homework03 sudo docker exec nextcloud-aio-nextcloud \
+    ls -la /mnt/ncdata/race/files/ | head
+  # Expect: Documents/  Photos/  Templates/  ...

    4.4 Tear down the old OnlyOffice

    ssh tigo@hawker
@@ -343,16 +427,24 @@ tar -C /usr/local/containers/nextcloudaio \
   -czf "${BACKUP_DIR}/${NAME}-aio-config.tar.gz" \
   nextcloud-aio-mastercontainer nextcloud-aio-database-dump
 
-# 3. Tar user files (excluding the backups/ subdir to avoid recursion)
+# 3. Tar the local nextcloud app volume (AIO-managed app code +
+#    config — survives a fresh AIO install if we ever need to
+#    restore from a corrupt mastercontainer state).
+tar -C /usr/local/containers/nextcloudaio \
+  -czf "${BACKUP_DIR}/${NAME}-aio-nextcloud-app.tar.gz" \
+  nextcloud-aio-nextcloud
+
+# 4. Tar user files (NFS root: admin/, race/, appdata_*/, etc.)
+#    Exclude backups/ to avoid recursion.
 tar -C /srv/nc-files \
   --exclude='backups' \
   -czf "${BACKUP_DIR}/${NAME}-ncdata.tar.gz" \
-  nextcloud
+  .
 
-# 4. Prune anything older than 14 days
+# 5. Prune anything older than 14 days
 find "${BACKUP_DIR}" -maxdepth 1 -type f -name 'office-*' -mtime +14 -delete
 
-echo "OK: wrote ${NAME}-{{pgdump.sql.gz,aio-config.tar.gz,ncdata.tar.gz}} to ${BACKUP_DIR}"
+echo "OK: wrote ${NAME}-{pgdump.sql.gz,aio-config.tar.gz,aio-nextcloud-app.tar.gz,ncdata.tar.gz} to ${BACKUP_DIR}"
 EOF
 
 sudo -n chmod 755 /usr/local/bin/office-backup.sh
diff --git a/troubleshooting.html b/troubleshooting.html
index d582c89..550f93c 100644
--- a/troubleshooting.html
+++ b/troubleshooting.html
@@ -21,9 +21,10 @@
 
   
    Troubleshooting
    
   
    
-    Every pitfall hit during the 2026-08-10 deployment, with root cause
-    and resolution. Order is roughly chronological — these are what
-    blocked progress at each stage.
+    Every pitfall hit during the 2026-08-10 deployment plus the
+    2026-08-11 rewire, with root cause and resolution. Order is
+    roughly chronological — these are what blocked progress at each
+    stage.
   
    
 
   
    
@@ -38,6 +39,9 @@
       
  • "OnlyOffice" rejected by AIO (must use Collabora or office flag)
    • 
       
  • curl login returns 303 with empty user — CSRF cookie dance
    • 
       
  • Backup script won't run as tigo — root-owned 755 instead
    • 
+      
  • "Appdata directory is not present" — wrong datadir (rewire 2026-08-11)
    • 
+      
  • nextcloud container not appearing after config change
    • 
+      
  • Collabora logs "Could not create path .../richdocuments/remoteData/discovery"
    • 
     
   
    
 
@@ -351,6 +355,167 @@ sudo -n bash -n /usr/local/bin/office-backup.sh  # syntax check
    ExecStart=/usr/bin/ssh -o BatchMode=yes -o ConnectTimeout=30 \ homework03 sudo /usr/local/bin/office-backup.sh +

    "Appdata directory is not present"

    + +
    +

    Symptom: docker logs nextcloud-aio-nextcloud + shows Cannot write into directory "/mnt/ncdata/appdata_*" + or Appdata directory is not present!. The login page + may load but logins loop or fail. Sometimes the nextcloud + container restart-loops.

    +
    + +

    Root cause

    +

    + AIO uses the value of NEXTCLOUD_DATADIR (or the + nextcloud_datadir field in + configuration.json) as the host bind-mount + source that goes into the nextcloud container's + /mnt/ncdata. If that path is not the actual NFS + mount (or is a different host path than the NFS mount), AIO + happily bind-mounts whatever the path resolves to — including an + empty local directory — and Nextcloud boots against an empty + datadir that has no appdata_* directory in it. +

    +

    + The data is still on NFS. It's just not being mounted. This + took office.rmf44.xyz offline on 2026-08-11 for ~15 minutes. +

    + +

    Fix (the 2026-08-11 rewire)

    +
      +
    1. + Pick one host path that is the NFS mount. + We picked /srv/nc-files. +
      2. +
    2. + Set both NEXTCLOUD_DATADIR in compose AND + nextcloud_datadir in + configuration.json to that same path. +
      3. +
    3. + Remove any intermediate bind in the mastercontainer's + compose volumes: section — AIO doesn't need it + and it adds a layer that can drift out of sync. +
      4. +
    4. + Force the nextcloud container to respawn with the new bind: + docker rm -f nextcloud-aio-nextcloud, then + trigger /api/docker/start via the admin UI (with + Apache stopped first). See + nextcloud container not appearing + below for the spawn dance. +
      5. +
    + +

    + The full sequence (mount, compose, config, respawn) is documented + in procedure §2.5 + Rewire 2026-08-11. +

    + +

    nextcloud container not appearing after config change

    + +
    +

    Symptom: you updated + configuration.json (changed + nextcloud_datadir, enabled an extra container, etc.) + and restarted the mastercontainer, but the + nextcloud-aio-nextcloud container is missing or + running with the old config.

    +
    + +

    Root cause

    +

    + AIO does NOT auto-spawn the nextcloud container on + mastercontainer restart. The mastercontainer only + orchestrates the lifecycle of containers it spawns. If the + nextcloud container was already running, the mastercontainer + just observes it. If you docker rm -f an old + broken container and restart the mastercontainer, the + mastercontainer has no awareness that you want a new one — you + must explicitly trigger /api/docker/start. +

    +

    + Additionally, isLoginAllowed() in + DockerActionManager.php returns false + when Apache is starting or running and its port is open. So + /api/docker/start only fires when Apache is stopped. +

    + +

    Fix — the spawn dance

    +

    + Use /tmp/aio-flow.sh on homework03 — it runs the + four steps atomically: +

    + 
    ssh homework03
+sudo /tmp/aio-flow.sh
+# 1. POST /api/docker/stop         (stops Apache — login allowed)
+# 2. GET /login + POST /api/auth/login (saves cookies + CSRF)
+# 3. POST /api/docker/start        (triggers spawn)
+# 4. POST /api/docker/stop /start  (re-runs for nextcloud container, restart Apache)
    + +

    + Or manually via curl: +

    + 
    JAR=/tmp/cookies.txt
+BASE=https://office.rmf44.xyz:8080
+# 1. Stop Apache
+curl -skb $JAR -X POST "$BASE/api/docker/stop"
+# 2. Login
+curl -skc $JAR -o /tmp/login.html "$BASE/login"
+TOKEN=$(grep -oE 'data-requesttoken="[^"]+"' /tmp/login.html | head -1 | sed 's/data-requesttoken="//;s/"$//')
+curl -skb $JAR -c $JAR \
+  -H "requesttoken: $TOKEN" \
+  -d "password=$ADMIN_PASSWORD" \
+  -X POST "$BASE/api/auth/login"
+# 3. Trigger spawn
+curl -skb $JAR -c $JAR \
+  -H "requesttoken: $TOKEN" \
+  -X POST "$BASE/api/docker/start"
+# 4. Restart Apache
+curl -skb $JAR -X POST "$BASE/api/docker/start"
    + +

    Collabora logs "Could not create path .../richdocuments/remoteData/discovery"

    + +
    +

    Symptom: after starting Collabora, the + nextcloud container logs lines like:

    + 
    Could not create path /mnt/ncdata/appdata_*/richdocuments/remoteData/discovery
+Failed to fetch discovery endpoint
    +
    + +

    Root cause

    +

    + The nextcloud container runs as www-data, but the + parent appdata_*/ directory on NFS was created by an + earlier process (possibly the AIO entrypoint as root during first + init) and the per-app richdocuments/ subdir doesn't + exist yet. The nextcloud container can't create it because it + doesn't own the parent. +

    + +

    Resolution

    +

    + Wait it out. Collabora generates the discovery + JSON on first use (when a user opens a Word/Excel file in the + Collabora iframe). The warning is logged but the discovery is + cached on first successful WOPI round-trip. If Collabora is + actually broken (the iframe stays blank), check perms: +

    + 
    ssh homework03
+sudo docker exec nextcloud-aio-nextcloud \
+  ls -la /mnt/ncdata/appdata_*/richdocuments/remoteData/
+# If missing, force creation as www-data:
+sudo docker exec -u www-data nextcloud-aio-nextcloud \
+  mkdir -p /mnt/ncdata/appdata_*/richdocuments/remoteData/
    + +

    + We have not seen this fail on a live Collabora open since the + 2026-08-11 rewire — the warning appears once at startup and + then Collabora works normally. +

    + \ No newline at end of file