rewire 2026-08-11: single-mount NFS architecture

- topology.svg, container-tree.svg, data-flow.svg: show /srv/nc-files bound directly into nextcloud container at /mnt/ncdata (no intermediate /mnt/nc-data/nextcloud-data layer). - architecture.html: explain rewire, update storage table, remove pre-rewire 'why two layers' section. - procedure.html: NEXTCLOUD_DATADIR=/srv/nc-files, no mastercontainer bind, add §2.5 Rewire 2026-08-11 with recovery procedure, update backup script to also tar local nextcloud app volume, fix 4.3 verification (302 -> /login, status.php, file visibility). - operations.html: 4 backup files now (added aio-nextcloud-app.tar.gz), restore procedures updated, single-file restore uses new tar layout. - troubleshooting.html: add 3 new sections — appdata-missing, nextcloud-not-spawning, collabora-discovery-warning. - request-flow.svg: remove nextcloud/ subdir from NFS write path. - index.html, README.md: update paths and metadata.
2026-08-11 12:09:26 -05:00
parent d172771aa1
commit b4777a5e4d
10 changed files with 423 additions and 121 deletions
@@ -123,7 +123,7 @@ sudo -n mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600 \
  desslok:/slab/container_storage/office /srv/nc-files
 df -h /srv/nc-files
 ls -la /srv/nc-files
-# Expect: nextcloud/  backups/</code></pre>
+# Expect: admin/  race/  backups/  appdata_*/  nextcloud.log  ...</code></pre>

  <p>
    Add to <code>/etc/fstab</code> for boot persistence:
@@ -146,8 +146,9 @@ for sub in mastercontainer database database-dump redis apache nextcloud \
  sudo -n mkdir -p "/usr/local/containers/nextcloudaio/nextcloud-aio-$sub"
 done

-# /mnt/nc-data for the Nextcloud data dir (lives on local ext4)
-sudo -n mkdir -p /mnt/nc-data/nextcloud-data
+# /srv/nc-files is the NFS mount. AIO bind-mounts it directly into
+# the nextcloud container at /mnt/ncdata via NEXTCLOUD_DATADIR.
+# No intermediate /mnt/nc-data layer — see "Rewire 2026-08-11" note.

 # Local backup stash (so the script can write the pgdump into NFS without recursion)
 ls -la /usr/local/containers/nextcloudaio/</code></pre>
@@ -166,6 +167,79 @@ sudo -n chown -R 999:999 /usr/local/containers/nextcloudaio/nextcloud-aio-redis
 sudo -n chown -R root:root /usr/local/containers/nextcloudaio/nextcloud-aio-nextcloud
 sudo -n chown -R 100:101 /usr/local/containers/nextcloudaio/nextcloud-aio-collabora</code></pre>

+  <h3>2.5 Rewire 2026-08-11 — single-mount architecture</h3>
+  <p>
+    After the initial deploy we discovered a sharp edge: when
+    <code>NEXTCLOUD_DATADIR</code> is a <em>different</em> host path
+    than the actual NFS mount (the original design used
+    <code>/srv/nc-files</code> for NFS and
+    <code>/mnt/nc-data/nextcloud-data</code> for AIO), a typo on
+    either side silently creates an empty datadir inside the
+    nextcloud container. Nextcloud then refuses to start with
+    "Appdata directory is not present!" — but the empty datadir
+    is real, so the NFS data is still there, just not mounted.
+    That's the failure mode that took the office suite offline on
+    2026-08-11.
+  </p>
+  <p>
+    The rewire removes the intermediate
+    <code>/mnt/nc-data/nextcloud-data</code> bind entirely:
+  </p>
+  <ul>
+    <li>
+      NFS export <code>desslok:/slab/container_storage/office</code>
+      mounts at <code>/srv/nc-files</code> on homework03 (unchanged).
+    </li>
+    <li>
+      <code>NEXTCLOUD_DATADIR=/srv/nc-files</code> in compose AND
+      <code>configuration.json</code>'s <code>nextcloud_datadir</code>
+      field both point at the same path.
+    </li>
+    <li>
+      AIO's <code>containers.json</code> template substitutes
+      <code>%NEXTCLOUD_DATADIR%</code> with that path and creates a
+      bind mount directly: host <code>/srv/nc-files</code> →
+      container <code>/mnt/ncdata</code>.
+    </li>
+    <li>
+      The mastercontainer no longer has the
+      <code>/srv/nc-files</code> or
+      <code>/mnt/nc-data/nextcloud-data</code> binds in its
+      compose <code>volumes:</code> section.
+    </li>
+  </ul>
+  <p>
+    <strong>Recovery if it ever breaks again:</strong>
+  </p>
+  <pre><code># 1. Confirm what's in the NFS export
+ssh desslok ls -la /slab/container_storage/office
+# Expect: admin/  race/  appdata_*/  ...
+
+# 2. Confirm the mount is healthy on homework03
+ssh homework03 df -h /srv/nc-files
+ssh homework03 ls -la /srv/nc-files
+# Expect: same admin/, race/, ... as step 1
+
+# 3. Check what AIO thinks the datadir is
+ssh homework03 sudo cat \
+  /var/lib/docker/volumes/nextcloud_aio_mastercontainer/_data/configuration.json \
+  | jq -r .nextcloud_datadir
+# Expect: "/srv/nc-files" — if not, fix with jq (see ~/.hermes
+# creds or the rewire script notes in this repo's history)
+
+# 4. Inspect what the running nextcloud container actually has bound
+ssh homework03 sudo docker inspect nextcloud-aio-nextcloud \
+  | jq -r '.[0].Mounts[] | "\(.Source) -> \(.Destination)"'
+# Expect: "/srv/nc-files -> /mnt/ncdata" AND
+#         "/var/lib/docker/volumes/nextcloud_aio_nextcloud/_data -> /var/www/html"
+# If /mnt/ncdata is bound to something else, the container has stale config.
+
+# 5. If the bind source is wrong, force AIO to re-spawn nextcloud:
+ssh homework03 sudo docker rm -f nextcloud-aio-nextcloud
+# Then trigger /api/docker/start from the admin UI (Apache must
+# be stopped first; the nextcloud container does NOT auto-spawn
+# on mastercontainer restart). See "Phase 6: Spawn lifecycle" below.</code></pre>
+
  <h2 id="phase-3">Phase 3 — AIO mastercontainer + setup wizard</h2>
  <p>
    Write the compose file, start the mastercontainer, and walk the
@@ -184,7 +258,7 @@ services:
    environment:
      APACHE_PORT: "11000"
      APACHE_DISABLE_REWRITE_IP: "1"
-      NEXTCLOUD_DATADIR: "/mnt/nc-data/nextcloud-data"
+      NEXTCLOUD_DATADIR: "/srv/nc-files"
      NEXTCLOUD_UPLOAD_LIMIT: "10G"
      NEXTCLOUD_MAX_TIME: "3600"
      AIO_DISABLE_BACKUP: "true"
@@ -201,8 +275,11 @@ services:
    volumes:
      - ./nextcloud-aio-mastercontainer:/container-volume
      - /var/run/docker.sock:/var/run/docker.sock:ro
-      - /srv/nc-files:/srv/nc-files
-      - /mnt/nc-data/nextcloud-data:/mnt/nc-data/nextcloud-data
+      # NOTE: do NOT bind /srv/nc-files into the mastercontainer.
+      # AIO bind-mounts it directly into the nextcloud container via
+      # NEXTCLOUD_DATADIR. (Pre-rewire this entry also bound
+      # /mnt/nc-data/nextcloud-data — that intermediate layer was
+      # removed 2026-08-11.)
 EOF</code></pre>

  <h3>3.2 Start mastercontainer + pull the AIO passphrase</h3>
@@ -239,7 +316,7 @@ hostname -I | awk '{print $1}'
  <pre><code>ssh homework03
 sudo -n cat /usr/local/containers/nextcloudaio/nextcloud-aio-mastercontainer/configuration.json
 # Expect: "officeSuite": "collabora", "isWhiteboardEnabled": true,
-# "domain": "office.rmf44.xyz", "nextcloud_datadir": "/mnt/nc-data/nextcloud-data"</code></pre>
+# "domain": "office.rmf44.xyz", "nextcloud_datadir": "/srv/nc-files"</code></pre>

  <h2 id="phase-4">Phase 4 — Public ingress + cutover</h2>
  <p>
@@ -276,17 +353,24 @@ sudo -n docker exec caddy-caddy-1 caddy reload \
  --config /etc/caddy/Caddyfile --adapter caddyfile</code></pre>

  <h3>4.3 Verify the cutover</h3>
-  <pre><code>curl -skI https://office.rmf44.xyz/
-# HTTP/2 200
-# content-type: text/html; charset=UTF-8
-# ...
-curl -s https://office.rmf44.xyz/ | grep -oE '<title>[^<]+</title>'
-# &lt;title&gt;Login – Nextcloud&lt;/title&gt;
+    <pre><code>curl -skI https://office.rmf44.xyz/
+  # HTTP/2 302
+  # location: /login
+  curl -sk https://office.rmf44.xyz/login | grep -oE '&lt;title&gt;[^&lt;]+&lt;/title&gt;'
+  # &lt;title&gt;Login - AIO&lt;/title&gt;

-# Test login
-# 1. GET /login → grab requesttoken + cookies
-# 2. POST /login with user=admin + password + requesttoken
-# 3. Expect HTTP 303 → /apps/dashboard/</code></pre>
+  curl -sk https://office.rmf44.xyz/status.php
+  # {"installed":true,"version":"34.0.2.1","...","maintenance":false}
+
+  # Test login
+  # 1. GET /login → grab requesttoken + cookies
+  # 2. POST /login with user=admin + password + requesttoken
+  # 3. Expect HTTP 303 → /apps/dashboard/
+
+  # Verify the nextcloud container can see NFS user files
+  ssh homework03 sudo docker exec nextcloud-aio-nextcloud \
+    ls -la /mnt/ncdata/race/files/ | head
+  # Expect: Documents/  Photos/  Templates/  ...</code></pre>

  <h3>4.4 Tear down the old OnlyOffice</h3>
  <pre><code>ssh tigo@hawker
@@ -343,16 +427,24 @@ tar -C /usr/local/containers/nextcloudaio \
  -czf "${BACKUP_DIR}/${NAME}-aio-config.tar.gz" \
  nextcloud-aio-mastercontainer nextcloud-aio-database-dump

-# 3. Tar user files (excluding the backups/ subdir to avoid recursion)
+# 3. Tar the local nextcloud app volume (AIO-managed app code +
+#    config — survives a fresh AIO install if we ever need to
+#    restore from a corrupt mastercontainer state).
+tar -C /usr/local/containers/nextcloudaio \
+  -czf "${BACKUP_DIR}/${NAME}-aio-nextcloud-app.tar.gz" \
+  nextcloud-aio-nextcloud
+
+# 4. Tar user files (NFS root: admin/, race/, appdata_*/, etc.)
+#    Exclude backups/ to avoid recursion.
 tar -C /srv/nc-files \
  --exclude='backups' \
  -czf "${BACKUP_DIR}/${NAME}-ncdata.tar.gz" \
-  nextcloud
+  .

-# 4. Prune anything older than 14 days
+# 5. Prune anything older than 14 days
 find "${BACKUP_DIR}" -maxdepth 1 -type f -name 'office-*' -mtime +14 -delete

-echo "OK: wrote ${NAME}-{{pgdump.sql.gz,aio-config.tar.gz,ncdata.tar.gz}} to ${BACKUP_DIR}"
+echo "OK: wrote ${NAME}-{pgdump.sql.gz,aio-config.tar.gz,aio-nextcloud-app.tar.gz,ncdata.tar.gz} to ${BACKUP_DIR}"
 EOF

 sudo -n chmod 755 /usr/local/bin/office-backup.sh