nextcloud-aio-nextcloud./nextcloud-aio-nextcloud/ + /mnt/nc-data/nextcloud-data via NEXTCLOUD_DATADIR./nextcloud-aio-nextcloud/ (local volume) + /srv/nc-files (NFS) → /mnt/ncdataData flow — where each piece lives
- Most of AIO's data is on NFS (/srv/nc-files on
- homework03). The Postgres database is inside the database
- container; its data directory is a docker named-volume bind, not
- on NFS — keeping PostgreSQL's WAL writes off NFS is critical for
- durability.
+ Most of AIO's user data is on NFS
+ (/srv/nc-files on homework03 → container bind at
+ /mnt/ncdata). Postgres, Redis, and the AIO
+ mastercontainer's configuration live on local ext4 (named
+ volumes) — keeping PostgreSQL's WAL writes off NFS is critical
+ for durability.
| Path | Filesystem | Why | ||
|---|---|---|---|---|
/srv/nc-files/nextcloud/ |
+ /srv/nc-files/ (NFS root on homework03) |
NFSv4.1 from desslok | -User-uploaded files. Snapshotted daily via desslok's existing ZFS path. | +User-uploaded files live at the export root: admin/, race/, appdata_*/, etc. The nextcloud container binds this path to /mnt/ncdata directly. Snapshotted daily via desslok's ZFS path. |
/srv/nc-files/backups/ |
@@ -193,9 +197,9 @@
Daily pgdump + AIO config tar + user-files tar. 14-day retention. | |||
/mnt/nc-data/nextcloud-data/ |
+ /usr/local/containers/nextcloudaio/nextcloud-aio-nextcloud/_data/ |
ext4 (local) | -Bind mount, mounted INTO the nextcloud container as /nextcloud-aio. Holds app config, theme, install state. |
+ Local named-volume bind for the nextcloud container. AIO-managed; NEXTCLOUD_DATADIR points at /srv/nc-files separately, NOT at this volume. |
/usr/local/containers/nextcloudaio/nextcloud-aio-{mastercontainer,database,redis,apache,...}/ |
@@ -204,6 +208,24 @@
+ Rewire 2026-08-11: before the rewire, AIO
+ bind-mounted a third host layer
+ (/mnt/nc-data/nextcloud-data) into the nextcloud
+ container, with NEXTCLOUD_DATADIR=/mnt/nc-data/nextcloud-data.
+ That double-bind worked but had two sharp edges: (a) the
+ NEXTCLOUD_DATADIR host path and the actual NFS
+ mount path had to be kept in sync manually; (b) a typo would
+ silently create an empty datadir, masking the real NFS data
+ and triggering an "Appdata is not present" crash on next
+ start. The rewire collapses both to a single layer:
+ /srv/nc-files is the NFS mount AND the
+ NEXTCLOUD_DATADIR value — AIO binds it directly
+ into the nextcloud container at /mnt/ncdata.
+
Why isn't the postgres data on NFS? @@ -232,7 +254,7 @@
127.0.0.1:9000 (PHP-FPM in nextcloud-aio-nextcloud)./admin/files/, and writes the file via WebDAV./srv/nc-files/nextcloud/admin/files/smoke-test.md on homework03 → /slab/container_storage/office/nextcloud/admin/files/smoke-test.md on desslok./srv/nc-files/admin/files/smoke-test.md on homework03 → /slab/container_storage/office/admin/files/smoke-test.md on desslok (nextcloud container's /mnt/ncdata is bound to /srv/nc-files directly).HTTP/2 201 Created with empty body (WebDAV semantics)./usr/local/containers/nextcloudaio/docker-compose.yamlnetwork_mode: host/usr/local/containers/nextcloudaio/nextcloud-aio-{mastercontainer,database,redis,apache,nextcloud,collabora,whiteboard,notify-push,database-dump}//srv/nc-files/desslok:/slab/container_storage/office/mnt/nc-data/nextcloud-data//nextcloud-aio/data/usr/local/bin/office-backup.sh/srv/nc-files/desslok:/slab/container_storage/office; AIO binds this directly into the nextcloud container at /mnt/ncdata via NEXTCLOUD_DATADIR/usr/local/containers/nextcloudaio/nextcloud-aio-nextcloud//var/www/html (AIO-managed app code)/usr/local/bin/office-backup.sh/etc/systemd/system/office-backup.{service,timer}/etc/caddy/Caddyfileoffice.rmf44.xyz → 100.79.142.164:11000/slab/container_storage/office/backups/ subdirBackup pipeline
- Three files written daily to
+ Four files written daily to
/srv/nc-files/backups/ on homework03 (NFS, real path
/slab/container_storage/office/backups/ on desslok):
office-YYYYMMDD-aio-nextcloud-app.tar.gz/usr/local/containers/nextcloudaio/nextcloud-aio-nextcloud/ — the AIO-managed local app volume (Nextcloud app code, installed apps, config/).office-YYYYMMDD-ncdata.tar.gz/srv/nc-files/nextcloud/ (user-uploaded files) — excludes backups/ to avoid recursion./srv/nc-files/ root (user files: admin/, race/, appdata_*/, etc.) — excludes backups/ to avoid recursion./srv/nc-files/nextcloud/ while the filesystem is
- actively being written to by the nextcloud container. The tar
- will see a consistent enough snapshot for crash-consistent
- recovery; for true point-in-time recovery, you'd want to
- quiesce Nextcloud (set maintenance mode) for the duration of
- the tar, which we haven't done.
+ /srv/nc-files/ (the NFS root) while the
+ filesystem is actively being written to by the nextcloud
+ container. The tar will see a consistent enough snapshot for
+ crash-consistent recovery; for true point-in-time recovery,
+ you'd want to quiesce Nextcloud (set maintenance mode) for the
+ duration of the tar, which we haven't done.
nextcloud/):
+ Restore the local nextcloud app volume (AIO-managed code +
+ installed apps):
+ LATEST=$(ls -t /srv/nc-files/backups/office-*-aio-nextcloud-app.tar.gz | head -1)
+tar -C /usr/local/containers/nextcloudaio -xzf "$LATEST"nextcloud/):
LATEST=$(ls -t /srv/nc-files/backups/office-*-ncdata.tar.gz | head -1)
-# Tar contains /nextcloud/ at root
+# Tar contains files at root (admin/, race/, appdata_*/, ...)
tar -C /srv/nc-files -xzf "$LATEST"ssh desslok
LATEST=$(ls -t /slab/container_storage/office/backups/office-*-ncdata.tar.gz | head -1)
-tar -C / -xzf "$LATEST" nextcloud/admin/files/path/to/file
-# Adjust for the user + pathRe-initialize the admin user
diff --git a/procedure.html b/procedure.html index fac336f..60f0d90 100644 --- a/procedure.html +++ b/procedure.html @@ -123,7 +123,7 @@ sudo -n mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600 \ desslok:/slab/container_storage/office /srv/nc-files df -h /srv/nc-files ls -la /srv/nc-files -# Expect: nextcloud/ backups/ +# Expect: admin/ race/ backups/ appdata_*/ nextcloud.log ...
Add to /etc/fstab for boot persistence:
@@ -146,8 +146,9 @@ for sub in mastercontainer database database-dump redis apache nextcloud \
sudo -n mkdir -p "/usr/local/containers/nextcloudaio/nextcloud-aio-$sub"
done
-# /mnt/nc-data for the Nextcloud data dir (lives on local ext4)
-sudo -n mkdir -p /mnt/nc-data/nextcloud-data
+# /srv/nc-files is the NFS mount. AIO bind-mounts it directly into
+# the nextcloud container at /mnt/ncdata via NEXTCLOUD_DATADIR.
+# No intermediate /mnt/nc-data layer — see "Rewire 2026-08-11" note.
# Local backup stash (so the script can write the pgdump into NFS without recursion)
ls -la /usr/local/containers/nextcloudaio/
@@ -166,6 +167,79 @@ sudo -n chown -R 999:999 /usr/local/containers/nextcloudaio/nextcloud-aio-redis
sudo -n chown -R root:root /usr/local/containers/nextcloudaio/nextcloud-aio-nextcloud
sudo -n chown -R 100:101 /usr/local/containers/nextcloudaio/nextcloud-aio-collabora
+
2.5 Rewire 2026-08-11 — single-mount architecture
+
+ After the initial deploy we discovered a sharp edge: when
+ NEXTCLOUD_DATADIR is a different host path
+ than the actual NFS mount (the original design used
+ /srv/nc-files for NFS and
+ /mnt/nc-data/nextcloud-data for AIO), a typo on
+ either side silently creates an empty datadir inside the
+ nextcloud container. Nextcloud then refuses to start with
+ "Appdata directory is not present!" — but the empty datadir
+ is real, so the NFS data is still there, just not mounted.
+ That's the failure mode that took the office suite offline on
+ 2026-08-11.
+
+ The rewire removes the intermediate
+ /mnt/nc-data/nextcloud-data bind entirely:
+
-
+
-
+ NFS export
desslok:/slab/container_storage/office+ mounts at/srv/nc-fileson homework03 (unchanged). +
+ -
+
NEXTCLOUD_DATADIR=/srv/nc-filesin compose AND +configuration.json'snextcloud_datadir+ field both point at the same path. +
+ -
+ AIO's
containers.jsontemplate substitutes +%NEXTCLOUD_DATADIR%with that path and creates a + bind mount directly: host/srv/nc-files→ + container/mnt/ncdata. +
+ -
+ The mastercontainer no longer has the
+
/srv/nc-filesor +/mnt/nc-data/nextcloud-databinds in its + composevolumes:section. +
+
+ Recovery if it ever breaks again: +
+# 1. Confirm what's in the NFS export
+ssh desslok ls -la /slab/container_storage/office
+# Expect: admin/ race/ appdata_*/ ...
+
+# 2. Confirm the mount is healthy on homework03
+ssh homework03 df -h /srv/nc-files
+ssh homework03 ls -la /srv/nc-files
+# Expect: same admin/, race/, ... as step 1
+
+# 3. Check what AIO thinks the datadir is
+ssh homework03 sudo cat \
+ /var/lib/docker/volumes/nextcloud_aio_mastercontainer/_data/configuration.json \
+ | jq -r .nextcloud_datadir
+# Expect: "/srv/nc-files" — if not, fix with jq (see ~/.hermes
+# creds or the rewire script notes in this repo's history)
+
+# 4. Inspect what the running nextcloud container actually has bound
+ssh homework03 sudo docker inspect nextcloud-aio-nextcloud \
+ | jq -r '.[0].Mounts[] | "\(.Source) -> \(.Destination)"'
+# Expect: "/srv/nc-files -> /mnt/ncdata" AND
+# "/var/lib/docker/volumes/nextcloud_aio_nextcloud/_data -> /var/www/html"
+# If /mnt/ncdata is bound to something else, the container has stale config.
+
+# 5. If the bind source is wrong, force AIO to re-spawn nextcloud:
+ssh homework03 sudo docker rm -f nextcloud-aio-nextcloud
+# Then trigger /api/docker/start from the admin UI (Apache must
+# be stopped first; the nextcloud container does NOT auto-spawn
+# on mastercontainer restart). See "Phase 6: Spawn lifecycle" below.Phase 3 — AIO mastercontainer + setup wizard
Write the compose file, start the mastercontainer, and walk the @@ -184,7 +258,7 @@ services: environment: APACHE_PORT: "11000" APACHE_DISABLE_REWRITE_IP: "1" - NEXTCLOUD_DATADIR: "/mnt/nc-data/nextcloud-data" + NEXTCLOUD_DATADIR: "/srv/nc-files" NEXTCLOUD_UPLOAD_LIMIT: "10G" NEXTCLOUD_MAX_TIME: "3600" AIO_DISABLE_BACKUP: "true" @@ -201,8 +275,11 @@ services: volumes: - ./nextcloud-aio-mastercontainer:/container-volume - /var/run/docker.sock:/var/run/docker.sock:ro - - /srv/nc-files:/srv/nc-files - - /mnt/nc-data/nextcloud-data:/mnt/nc-data/nextcloud-data + # NOTE: do NOT bind /srv/nc-files into the mastercontainer. + # AIO bind-mounts it directly into the nextcloud container via + # NEXTCLOUD_DATADIR. (Pre-rewire this entry also bound + # /mnt/nc-data/nextcloud-data — that intermediate layer was + # removed 2026-08-11.) EOF
3.2 Start mastercontainer + pull the AIO passphrase
@@ -239,7 +316,7 @@ hostname -I | awk '{print $1}'ssh homework03
sudo -n cat /usr/local/containers/nextcloudaio/nextcloud-aio-mastercontainer/configuration.json
# Expect: "officeSuite": "collabora", "isWhiteboardEnabled": true,
-# "domain": "office.rmf44.xyz", "nextcloud_datadir": "/mnt/nc-data/nextcloud-data"Phase 4 — Public ingress + cutover
@@ -276,17 +353,24 @@ sudo -n docker exec caddy-caddy-1 caddy reload \ --config /etc/caddy/Caddyfile --adapter caddyfile
4.3 Verify the cutover
-curl -skI https://office.rmf44.xyz/
-# HTTP/2 200
-# content-type: text/html; charset=UTF-8
-# ...
-curl -s https://office.rmf44.xyz/ | grep -oE '[^<]+ '
-# <title>Login – Nextcloud</title>
+ curl -skI https://office.rmf44.xyz/
+ # HTTP/2 302
+ # location: /login
+ curl -sk https://office.rmf44.xyz/login | grep -oE '<title>[^<]+</title>'
+ # <title>Login - AIO</title>
-# Test login
-# 1. GET /login → grab requesttoken + cookies
-# 2. POST /login with user=admin + password + requesttoken
-# 3. Expect HTTP 303 → /apps/dashboard/
+ curl -sk https://office.rmf44.xyz/status.php
+ # {"installed":true,"version":"34.0.2.1","...","maintenance":false}
+
+ # Test login
+ # 1. GET /login → grab requesttoken + cookies
+ # 2. POST /login with user=admin + password + requesttoken
+ # 3. Expect HTTP 303 → /apps/dashboard/
+
+ # Verify the nextcloud container can see NFS user files
+ ssh homework03 sudo docker exec nextcloud-aio-nextcloud \
+ ls -la /mnt/ncdata/race/files/ | head
+ # Expect: Documents/ Photos/ Templates/ ...4.4 Tear down the old OnlyOffice
ssh tigo@hawker
@@ -343,16 +427,24 @@ tar -C /usr/local/containers/nextcloudaio \
-czf "${BACKUP_DIR}/${NAME}-aio-config.tar.gz" \
nextcloud-aio-mastercontainer nextcloud-aio-database-dump
-# 3. Tar user files (excluding the backups/ subdir to avoid recursion)
+# 3. Tar the local nextcloud app volume (AIO-managed app code +
+# config — survives a fresh AIO install if we ever need to
+# restore from a corrupt mastercontainer state).
+tar -C /usr/local/containers/nextcloudaio \
+ -czf "${BACKUP_DIR}/${NAME}-aio-nextcloud-app.tar.gz" \
+ nextcloud-aio-nextcloud
+
+# 4. Tar user files (NFS root: admin/, race/, appdata_*/, etc.)
+# Exclude backups/ to avoid recursion.
tar -C /srv/nc-files \
--exclude='backups' \
-czf "${BACKUP_DIR}/${NAME}-ncdata.tar.gz" \
- nextcloud
+ .
-# 4. Prune anything older than 14 days
+# 5. Prune anything older than 14 days
find "${BACKUP_DIR}" -maxdepth 1 -type f -name 'office-*' -mtime +14 -delete
-echo "OK: wrote ${NAME}-{{pgdump.sql.gz,aio-config.tar.gz,ncdata.tar.gz}} to ${BACKUP_DIR}"
+echo "OK: wrote ${NAME}-{pgdump.sql.gz,aio-config.tar.gz,aio-nextcloud-app.tar.gz,ncdata.tar.gz} to ${BACKUP_DIR}"
EOF
sudo -n chmod 755 /usr/local/bin/office-backup.sh
diff --git a/troubleshooting.html b/troubleshooting.html
index d582c89..550f93c 100644
--- a/troubleshooting.html
+++ b/troubleshooting.html
@@ -21,9 +21,10 @@
Troubleshooting
- Every pitfall hit during the 2026-08-10 deployment, with root cause
- and resolution. Order is roughly chronological — these are what
- blocked progress at each stage.
+ Every pitfall hit during the 2026-08-10 deployment plus the
+ 2026-08-11 rewire, with root cause and resolution. Order is
+ roughly chronological — these are what blocked progress at each
+ stage.
@@ -38,6 +39,9 @@
"OnlyOffice" rejected by AIO (must use Collabora or office flag)
curl login returns 303 with empty user — CSRF cookie dance
Backup script won't run as tigo — root-owned 755 instead
+ "Appdata directory is not present" — wrong datadir (rewire 2026-08-11)
+ nextcloud container not appearing after config change
+ Collabora logs "Could not create path .../richdocuments/remoteData/discovery"
@@ -351,6 +355,167 @@ sudo -n bash -n /usr/local/bin/office-backup.sh # syntax check"Appdata directory is not present"
+ +Symptom: docker logs nextcloud-aio-nextcloud
+ shows Cannot write into directory "/mnt/ncdata/appdata_*"
+ or Appdata directory is not present!. The login page
+ may load but logins loop or fail. Sometimes the nextcloud
+ container restart-loops.
Root cause
+
+ AIO uses the value of NEXTCLOUD_DATADIR (or the
+ nextcloud_datadir field in
+ configuration.json) as the host bind-mount
+ source that goes into the nextcloud container's
+ /mnt/ncdata. If that path is not the actual NFS
+ mount (or is a different host path than the NFS mount), AIO
+ happily bind-mounts whatever the path resolves to — including an
+ empty local directory — and Nextcloud boots against an empty
+ datadir that has no appdata_* directory in it.
+
+ The data is still on NFS. It's just not being mounted. This + took office.rmf44.xyz offline on 2026-08-11 for ~15 minutes. +
+ +Fix (the 2026-08-11 rewire)
+-
+
-
+ Pick one host path that is the NFS mount.
+ We picked
/srv/nc-files. +
+ -
+ Set both
NEXTCLOUD_DATADIRin compose AND +nextcloud_datadirin +configuration.jsonto that same path. +
+ -
+ Remove any intermediate bind in the mastercontainer's
+ compose
volumes:section — AIO doesn't need it + and it adds a layer that can drift out of sync. +
+ -
+ Force the nextcloud container to respawn with the new bind:
+
docker rm -f nextcloud-aio-nextcloud, then + trigger/api/docker/startvia the admin UI (with + Apache stopped first). See + nextcloud container not appearing + below for the spawn dance. +
+
+ The full sequence (mount, compose, config, respawn) is documented + in procedure §2.5 + Rewire 2026-08-11. +
+ +nextcloud container not appearing after config change
+ +Symptom: you updated
+ configuration.json (changed
+ nextcloud_datadir, enabled an extra container, etc.)
+ and restarted the mastercontainer, but the
+ nextcloud-aio-nextcloud container is missing or
+ running with the old config.
Root cause
+
+ AIO does NOT auto-spawn the nextcloud container on
+ mastercontainer restart. The mastercontainer only
+ orchestrates the lifecycle of containers it spawns. If the
+ nextcloud container was already running, the mastercontainer
+ just observes it. If you docker rm -f an old
+ broken container and restart the mastercontainer, the
+ mastercontainer has no awareness that you want a new one — you
+ must explicitly trigger /api/docker/start.
+
+ Additionally, isLoginAllowed() in
+ DockerActionManager.php returns false
+ when Apache is starting or running and its port is open. So
+ /api/docker/start only fires when Apache is stopped.
+
Fix — the spawn dance
+
+ Use /tmp/aio-flow.sh on homework03 — it runs the
+ four steps atomically:
+
ssh homework03
+sudo /tmp/aio-flow.sh
+# 1. POST /api/docker/stop (stops Apache — login allowed)
+# 2. GET /login + POST /api/auth/login (saves cookies + CSRF)
+# 3. POST /api/docker/start (triggers spawn)
+# 4. POST /api/docker/stop /start (re-runs for nextcloud container, restart Apache)+ Or manually via curl: +
+JAR=/tmp/cookies.txt
+BASE=https://office.rmf44.xyz:8080
+# 1. Stop Apache
+curl -skb $JAR -X POST "$BASE/api/docker/stop"
+# 2. Login
+curl -skc $JAR -o /tmp/login.html "$BASE/login"
+TOKEN=$(grep -oE 'data-requesttoken="[^"]+"' /tmp/login.html | head -1 | sed 's/data-requesttoken="//;s/"$//')
+curl -skb $JAR -c $JAR \
+ -H "requesttoken: $TOKEN" \
+ -d "password=$ADMIN_PASSWORD" \
+ -X POST "$BASE/api/auth/login"
+# 3. Trigger spawn
+curl -skb $JAR -c $JAR \
+ -H "requesttoken: $TOKEN" \
+ -X POST "$BASE/api/docker/start"
+# 4. Restart Apache
+curl -skb $JAR -X POST "$BASE/api/docker/start"Collabora logs "Could not create path .../richdocuments/remoteData/discovery"
+ +Symptom: after starting Collabora, the + nextcloud container logs lines like:
+Could not create path /mnt/ncdata/appdata_*/richdocuments/remoteData/discovery
+Failed to fetch discovery endpointRoot cause
+
+ The nextcloud container runs as www-data, but the
+ parent appdata_*/ directory on NFS was created by an
+ earlier process (possibly the AIO entrypoint as root during first
+ init) and the per-app richdocuments/ subdir doesn't
+ exist yet. The nextcloud container can't create it because it
+ doesn't own the parent.
+
Resolution
++ Wait it out. Collabora generates the discovery + JSON on first use (when a user opens a Word/Excel file in the + Collabora iframe). The warning is logged but the discovery is + cached on first successful WOPI round-trip. If Collabora is + actually broken (the iframe stays blank), check perms: +
+ssh homework03
+sudo docker exec nextcloud-aio-nextcloud \
+ ls -la /mnt/ncdata/appdata_*/richdocuments/remoteData/
+# If missing, force creation as www-data:
+sudo docker exec -u www-data nextcloud-aio-nextcloud \
+ mkdir -p /mnt/ncdata/appdata_*/richdocuments/remoteData/+ We have not seen this fail on a live Collabora open since the + 2026-08-11 rewire — the warning appears once at startup and + then Collabora works normally. +
+