rewire 2026-08-11: single-mount NFS architecture

- topology.svg, container-tree.svg, data-flow.svg: show /srv/nc-files
  bound directly into nextcloud container at /mnt/ncdata (no
  intermediate /mnt/nc-data/nextcloud-data layer).
- architecture.html: explain rewire, update storage table, remove
  pre-rewire 'why two layers' section.
- procedure.html: NEXTCLOUD_DATADIR=/srv/nc-files, no mastercontainer
  bind, add §2.5 Rewire 2026-08-11 with recovery procedure,
  update backup script to also tar local nextcloud app volume,
  fix 4.3 verification (302 -> /login, status.php, file visibility).
- operations.html: 4 backup files now (added aio-nextcloud-app.tar.gz),
  restore procedures updated, single-file restore uses new tar layout.
- troubleshooting.html: add 3 new sections — appdata-missing,
  nextcloud-not-spawning, collabora-discovery-warning.
- request-flow.svg: remove nextcloud/ subdir from NFS write path.
- index.html, README.md: update paths and metadata.

README.md

@@ -1,8 +1,11 @@
 # Nextcloud Office
 
-Session log + runbook for the 2026-08-10 deployment of `office.rmf44.xyz`
+Session log + runbook for the 2026-08-10 deployment and 2026-08-11
-as a Nextcloud All-in-One stack with Collabora + Whiteboard on
+rewire of `office.rmf44.xyz` as a Nextcloud All-in-One stack with
-`homework03`, replacing the retired OnlyOffice container on `hawker`.
+Collabora + Whiteboard on `homework03`, replacing the retired
+OnlyOffice container on `hawker`. The 2026-08-11 rewire collapsed the
+storage layer to a single NFS bind (post-mortem in
+`troubleshooting.html#appdata-missing`).
 
 ## Files
 
@@ -16,7 +19,6 @@ as a Nextcloud All-in-One stack with Collabora + Whiteboard on
 - `assets/diagrams/container-tree.svg` — 8 AIO containers + bind mounts
 - `assets/diagrams/data-flow.svg` — NFS vs ext4 split, database on host
 - `assets/diagrams/request-flow.svg` — swimlane sequence of a `git clone`-equivalent
-- `assets/diagrams/backup-pipeline.svg` — hector timer → homework03 → desslok NFS
 
 ## How to view
 

architecture.html

@@ -23,8 +23,9 @@
   <p>
     Three layers to understand: <strong>network</strong> (public → Caddy →
     NetBird → AIO Apache), <strong>containers</strong> (8 AIO processes on
-    one host, single docker network), and <strong>data flow</strong>
+    one host, host network namespace shared with mastercontainer), and
-    (NFS for everything, plus a postgres dump that hits NFS too).
+    <strong>data flow</strong> (NFS for user files, local ext4 for
+    everything else, plus a daily postgres dump that lands on NFS).
   </p>
 
   <h2>Topology — public ingress</h2>
@@ -58,9 +59,11 @@
     <li>
       <strong>Storage</strong> — NFSv4.1 from
       <code>desslok:/slab/container_storage/office</code> mounted at
-      <code>/srv/nc-files/</code> on homework03. Subdirs:
+      <code>/srv/nc-files</code> on homework03. User files live at
-      <code>nextcloud/</code> for user files, <code>backups/</code> for
+      the NFS export root (no intermediate <code>nextcloud/</code>
-      daily pgdump + config + user-files tars.
+      subdir): <code>admin/</code>, <code>race/</code>,
+      <code>appdata_*/</code>, plus <code>backups/</code> for the
+      daily backup pipeline.
     </li>
     <li>
       <strong>Retired (torn down)</strong> — OnlyOffice container
@@ -99,7 +102,7 @@
     <tr>
       <td><code>nextcloud-aio-nextcloud</code></td>
       <td>PHP-FPM + Nextcloud app code</td>
-      <td><code>./nextcloud-aio-nextcloud/</code> + <code>/mnt/nc-data/nextcloud-data</code> via <code>NEXTCLOUD_DATADIR</code></td>
+      <td><code>./nextcloud-aio-nextcloud/</code> (local volume) + <code>/srv/nc-files</code> (NFS) → <code>/mnt/ncdata</code></td>
       <td>root (entrypoint)</td>
       <td>:9000 (PHP-FPM)</td>
     </tr>
@@ -172,20 +175,21 @@
 
   <h2>Data flow — where each piece lives</h2>
   <p>
-    Most of AIO's data is on NFS (<code>/srv/nc-files</code> on
+    Most of AIO's user data is on NFS
-    homework03). The Postgres database is inside the database
+    (<code>/srv/nc-files</code> on homework03 → container bind at
-    container; its data directory is a docker named-volume bind, not
+    <code>/mnt/ncdata</code>). Postgres, Redis, and the AIO
-    on NFS — keeping PostgreSQL's WAL writes off NFS is critical for
+    mastercontainer's configuration live on local ext4 (named
-    durability.
+    volumes) — keeping PostgreSQL's WAL writes off NFS is critical
+    for durability.
   </p>
   <p><img src="assets/diagrams/data-flow.svg" alt="Data flow — NFS for user files, named volumes for container state" class="diagram"></p>
 
   <table>
     <tr><th>Path</th><th>Filesystem</th><th>Why</th></tr>
     <tr>
-      <td><code>/srv/nc-files/nextcloud/</code></td>
+      <td><code>/srv/nc-files/</code> (NFS root on homework03)</td>
       <td>NFSv4.1 from desslok</td>
-      <td>User-uploaded files. Snapshotted daily via desslok's existing ZFS path.</td>
+      <td>User-uploaded files live at the export root: <code>admin/</code>, <code>race/</code>, <code>appdata_*/</code>, etc. The nextcloud container binds this path to <code>/mnt/ncdata</code> directly. Snapshotted daily via desslok's ZFS path.</td>
     </tr>
     <tr>
       <td><code>/srv/nc-files/backups/</code></td>
@@ -193,9 +197,9 @@
       <td>Daily pgdump + AIO config tar + user-files tar. 14-day retention.</td>
     </tr>
     <tr>
-      <td><code>/mnt/nc-data/nextcloud-data/</code></td>
+      <td><code>/usr/local/containers/nextcloudaio/nextcloud-aio-nextcloud/_data/</code></td>
       <td>ext4 (local)</td>
-      <td>Bind mount, mounted INTO the nextcloud container as <code>/nextcloud-aio</code>. Holds app config, theme, install state.</td>
+      <td>Local named-volume bind for the nextcloud container. AIO-managed; <code>NEXTCLOUD_DATADIR</code> points at <code>/srv/nc-files</code> separately, NOT at this volume.</td>
     </tr>
     <tr>
       <td><code>/usr/local/containers/nextcloudaio/nextcloud-aio-{mastercontainer,database,redis,apache,...}/</code></td>
@@ -204,6 +208,24 @@
     </tr>
   </table>
 
+  <div class="callout info">
+    <p>
+      <strong>Rewire 2026-08-11:</strong> before the rewire, AIO
+      bind-mounted a <em>third</em> host layer
+      (<code>/mnt/nc-data/nextcloud-data</code>) into the nextcloud
+      container, with <code>NEXTCLOUD_DATADIR=/mnt/nc-data/nextcloud-data</code>.
+      That double-bind worked but had two sharp edges: (a) the
+      <code>NEXTCLOUD_DATADIR</code> host path and the actual NFS
+      mount path had to be kept in sync manually; (b) a typo would
+      silently create an empty datadir, masking the real NFS data
+      and triggering an "Appdata is not present" crash on next
+      start. The rewire collapses both to a single layer:
+      <code>/srv/nc-files</code> is the NFS mount AND the
+      <code>NEXTCLOUD_DATADIR</code> value — AIO binds it directly
+      into the nextcloud container at <code>/mnt/ncdata</code>.
+    </p>
+  </div>
+
   <div class="callout warn">
     <p>
       <strong>Why isn't the postgres data on NFS?</strong>
@@ -232,7 +254,7 @@
     <li><strong>NetBird tunnel</strong> encapsulates the request in WireGuard (UDP 51820), P2P from hawker to homework03.</li>
     <li><strong>AIO Apache</strong> (nextcloud-aio-apache container) terminates the TLS-stripped HTTP and forwards to <code>127.0.0.1:9000</code> (PHP-FPM in nextcloud-aio-nextcloud).</li>
     <li><strong>PHP-FPM (Nextcloud)</strong> authenticates the user via the session cookie, authorizes the path under <code>/admin/files/</code>, and writes the file via WebDAV.</li>
-    <li><strong>NFS write</strong> of the file to <code>/srv/nc-files/nextcloud/admin/files/smoke-test.md</code> on homework03 → <code>/slab/container_storage/office/nextcloud/admin/files/smoke-test.md</code> on desslok.</li>
+    <li><strong>NFS write</strong> of the file to <code>/srv/nc-files/admin/files/smoke-test.md</code> on homework03 → <code>/slab/container_storage/office/admin/files/smoke-test.md</code> on desslok (nextcloud container's <code>/mnt/ncdata</code> is bound to <code>/srv/nc-files</code> directly).</li>
     <li><strong>Response</strong>: <code>HTTP/2 201 Created</code> with empty body (WebDAV semantics).</li>
   </ol>
 

assets/diagrams/container-tree.svg

@@ -11,6 +11,7 @@
       .box-emp { fill: #1f2c3a; stroke: #4a78b5; stroke-width: 2; }
       .box-host { fill: #2a2535; stroke: #6b5aa0; stroke-width: 2; }
       .box-storage { fill: #1f2e28; stroke: #4a8a6b; stroke-width: 2; }
+      .box-bind { fill: #1f2e28; stroke: #6cba92; stroke-width: 2; stroke-dasharray: 3 3; }
       .arrow { stroke: #7aa2f7; stroke-width: 2; fill: none; marker-end: url(#arr-b); }
       .arrow-sto { stroke: #6cba92; stroke-width: 2; fill: none; marker-end: url(#arr-g); }
       .arrow-back { stroke: #d9a96b; stroke-width: 2; fill: none; marker-end: url(#arr-o); }
@@ -29,14 +30,14 @@
   <rect width="1100" height="600" fill="#0f1117"/>
 
   <text x="40" y="38" class="ttl">Container Tree — AIO on homework03</text>
-  <text x="40" y="58" class="lbl-sm">network_mode: host on mastercontainer · 8 active · 5 disabled</text>
+  <text x="40" y="58" class="lbl-sm">network_mode: host on mastercontainer · 8 active · 5 disabled · rewire 2026-08-11</text>
 
   <!-- homework03 host frame -->
   <rect x="320" y="100" width="760" height="450" rx="10" class="box-host"/>
   <text x="335" y="124" class="ttl" fill="#bda3e8">homework03 (10.0.0.73) · Debian 13 · 15 GB</text>
   <text x="335" y="142" class="lbl-sm">Docker 29.6.2 · host network namespace shared with mastercontainer</text>
 
-  <!-- Mastercontainer -->
+  <!-- Mastercontainer (no nc-data bind anymore) -->
   <rect x="345" y="170" width="200" height="100" rx="6" class="box-emp"/>
   <text x="445" y="194" text-anchor="middle" class="lbl">nextcloud-aio-mastercontainer</text>
   <text x="445" y="212" text-anchor="middle" class="lbl-sm">all-in-one:latest</text>
@@ -52,11 +53,12 @@
   <text x="685" y="244" text-anchor="middle" class="lbl-tiny">33:33 (www-data)</text>
 
   <!-- nextcloud-fcgi -->
-  <rect x="825" y="170" width="235" height="80" rx="6" class="box-internal"/>
+  <rect x="825" y="170" width="235" height="100" rx="6" class="box-bind"/>
-  <text x="942" y="194" text-anchor="middle" class="lbl">nextcloud-aio-nextcloud</text>
+  <text x="942" y="194" text-anchor="middle" class="lbl" fill="#6cba92">nextcloud-aio-nextcloud</text>
   <text x="942" y="212" text-anchor="middle" class="lbl-sm">PHP-FPM 8.3 + Nextcloud</text>
   <text x="942" y="228" text-anchor="middle" class="lbl-tiny">:9000 (PHP-FPM listen)</text>
-  <text x="942" y="244" text-anchor="middle" class="lbl-tiny">NEXTCLOUD_DATADIR bind</text>
+  <text x="942" y="244" text-anchor="middle" class="lbl-tiny" fill="#6cba92">bind: /srv/nc-files → /mnt/ncdata</text>
+  <text x="942" y="260" text-anchor="middle" class="lbl-tiny" fill="#6cba92">vol: nextcloud_aio_nextcloud → /var/www/html</text>
 
   <!-- Middle row: backing services -->
   <rect x="345" y="295" width="160" height="60" rx="6" class="box"/>
@@ -109,7 +111,7 @@
   <rect x="40" y="380" width="220" height="80" rx="6" class="box-storage"/>
   <text x="150" y="404" text-anchor="middle" class="lbl">desslok NFS</text>
   <text x="150" y="422" text-anchor="middle" class="lbl-sm">/slab/container_storage/office</text>
-  <text x="150" y="438" text-anchor="middle" class="lbl-tiny">NFSv4.1 · /srv/nc-files/</text>
+  <text x="150" y="438" text-anchor="middle" class="lbl-tiny">NFSv4.1 · /srv/nc-files</text>
 
   <!-- External: backup -->
   <rect x="40" y="500" width="220" height="60" rx="6" class="box" stroke="#d9a96b" stroke-width="1.5"/>
@@ -130,10 +132,10 @@
   <line x1="685" y1="250" x2="685" y2="295" class="arrow"/>
   <text x="691" y="278" class="lbl-tiny" fill="#7aa2f7">?php-fpm</text>
 
-  <line x1="942" y1="250" x2="945" y2="295" class="arrow"/>
+  <line x1="942" y1="270" x2="945" y2="295" class="arrow"/>
 
   <line x1="505" y1="480" x2="685" y2="380" class="arrow" stroke-dasharray="3 3"/>
-  <line x1="870" y1="380" x2="942" y2="250" class="arrow" stroke-dasharray="3 3"/>
+  <line x1="870" y1="380" x2="942" y2="270" class="arrow" stroke-dasharray="3 3"/>
 
   <line x1="425" y1="355" x2="425" y2="380" class="arrow"/>
   <line x1="945" y1="355" x2="945" y2="380" class="arrow"/>
@@ -147,5 +149,5 @@
   <line x1="425" y1="355" x2="260" y2="420" class="arrow-sto" stroke-dasharray="4 4"/>
   <text x="355" y="398" class="lbl-tiny" fill="#6cba92">writes pgdump</text>
 
-  <text x="1060" y="592" text-anchor="end" class="lbl-tiny">2026-08-10 · nextcloud_office</text>
+  <text x="1060" y="592" text-anchor="end" class="lbl-tiny">2026-08-11 · nextcloud_office · rewire</text>
 </svg>

assets/diagrams/data-flow.svg

@@ -11,9 +11,11 @@
       .fs-local { fill: #1f2c3a; stroke: #4a78b5; stroke-width: 2; }
       .fs-nfs { fill: #1f2e28; stroke: #4a8a6b; stroke-width: 2; }
       .fs-named { fill: #2a2535; stroke: #6b5aa0; stroke-width: 2; }
+      .fs-bind { fill: #1f2e28; stroke: #6cba92; stroke-width: 2; stroke-dasharray: 3 3; }
       .arrow-nfs { stroke: #6cba92; stroke-width: 2; fill: none; marker-end: url(#arr-g); }
       .arrow-local { stroke: #7aa2f7; stroke-width: 2; fill: none; marker-end: url(#arr-b); }
       .arrow-named { stroke: #9d7ad9; stroke-width: 2; fill: none; marker-end: url(#arr-p); }
+      .arrow-bind { stroke: #6cba92; stroke-width: 2.5; fill: none; stroke-dasharray: 3 3; marker-end: url(#arr-g); }
     </style>
     <marker id="arr-g" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto">
       <path d="M0,0 L10,5 L0,10 z" fill="#6cba92"/>
@@ -29,7 +31,7 @@
   <rect width="1100" height="600" fill="#0f1117"/>
 
   <text x="40" y="38" class="ttl">Data Flow — where each piece lives</text>
-  <text x="40" y="58" class="lbl-sm">NFS for user data + backups · ext4 for container state · local named volumes for postgres</text>
+  <text x="40" y="58" class="lbl-sm">NFS at /srv/nc-files → direct bind into nextcloud container at /mnt/ncdata · rewire 2026-08-11</text>
 
   <text x="80"  y="105" class="section">homework03 (local ext4)</text>
   <text x="540" y="105" class="section">desslok (NFS v4.1)</text>
@@ -38,9 +40,10 @@
   <rect x="60" y="130" width="430" height="380" rx="8" class="fs-local"/>
   <text x="80" y="155" class="lbl">/ (ext4)</text>
 
-  <rect x="80" y="180" width="390" height="48" rx="4" class="box-internal"/>
+  <!-- /srv/nc-files mount (NFS on homework03 host) -->
-  <text x="275" y="200" text-anchor="middle" class="lbl-sm">/mnt/nc-data/nextcloud-data</text>
+  <rect x="80" y="180" width="390" height="48" rx="4" class="fs-bind"/>
-  <text x="275" y="216" text-anchor="middle" class="lbl-tiny">→ nextcloud-aio-nextcloud:/nextcloud-aio/data</text>
+  <text x="275" y="200" text-anchor="middle" class="lbl-sm" fill="#6cba92">/srv/nc-files</text>
+  <text x="275" y="216" text-anchor="middle" class="lbl-tiny" fill="#6cba92">NFS v4.1 mount → nextcloud-aio-nextcloud:/mnt/ncdata</text>
 
   <rect x="80" y="240" width="190" height="80" rx="4" class="box-internal"/>
   <text x="175" y="262" text-anchor="middle" class="lbl-sm">mastercontainer</text>
@@ -79,9 +82,9 @@
   <text x="560" y="155" class="lbl">/slab/container_storage/office (NFS)</text>
 
   <rect x="560" y="180" width="460" height="80" rx="4" class="box-internal"/>
-  <text x="790" y="202" text-anchor="middle" class="lbl">nextcloud/</text>
+  <text x="790" y="202" text-anchor="middle" class="lbl">office/ (NFS root)</text>
-  <text x="790" y="220" text-anchor="middle" class="lbl-sm">user-uploaded files</text>
+  <text x="790" y="220" text-anchor="middle" class="lbl-sm">user-uploaded files at NFS ROOT (no nextcloud/ subdir)</text>
-  <text x="790" y="238" text-anchor="middle" class="lbl-tiny">mounted at /srv/nc-files/nextcloud/ on homework03</text>
+  <text x="790" y="238" text-anchor="middle" class="lbl-tiny">admin/ · race/ · appdata_*/ · *.log · .htaccess</text>
 
   <rect x="560" y="272" width="460" height="100" rx="4" class="box-internal"/>
   <text x="790" y="294" text-anchor="middle" class="lbl">backups/</text>
@@ -93,14 +96,18 @@
   <rect x="560" y="384" width="460" height="110" rx="4" class="box-internal"/>
   <text x="790" y="406" text-anchor="middle" class="lbl-sm">ZFS snapshot policy (desslok)</text>
   <text x="790" y="424" text-anchor="middle" class="lbl-tiny">/slab is on a ZFS pool with periodic snapshots</text>
-  <text x="790" y="440" text-anchor="middle" class="lbl-tiny">nightly snapshot → nextcloud/ and backups/ both covered</text>
+  <text x="790" y="440" text-anchor="middle" class="lbl-tiny">nightly snapshot → office/ and backups/ both covered</text>
   <text x="790" y="456" text-anchor="middle" class="lbl-tiny">→ true point-in-time recovery available independent of our daily backup</text>
   <text x="790" y="478" text-anchor="middle" class="lbl-tiny">daily backup is belt-and-suspenders, ZFS snapshots are the primary</text>
 
   <!-- Arrows -->
-  <!-- nextcloud-data → nextcloud/ (NFS read/write) -->
+  <!-- NFS export → /srv/nc-files on homework03 -->
-  <line x1="470" y1="204" x2="540" y2="220" class="arrow-nfs"/>
+  <line x1="540" y1="204" x2="470" y2="204" class="arrow-nfs"/>
-  <text x="505" y="206" text-anchor="middle" class="lbl-tiny" fill="#6cba92">read/write</text>
+  <text x="505" y="200" text-anchor="middle" class="lbl-tiny" fill="#6cba92">NFS mount</text>
+
+  <!-- /srv/nc-files → /mnt/ncdata (nextcloud container) -->
+  <line x1="380" y1="228" x2="380" y2="240" class="arrow-bind"/>
+  <text x="388" y="234" class="lbl-tiny" fill="#6cba92">bind</text>
 
   <!-- mastercontainer ↔ configuration.json → AIO config read by apache -->
   <line x1="270" y1="280" x2="380" y2="280" class="arrow-local"/>
@@ -117,5 +124,5 @@
   <rect x="60" y="525" width="980" height="40" rx="4" class="box-internal" stroke="#d9a96b"/>
   <text x="550" y="546" text-anchor="middle" class="lbl-sm" fill="#d9a96b">postgres WAL writes stay LOCAL (named volume) — PostgreSQL is sensitive to NFS close-to-open consistency</text>
 
-  <text x="1060" y="592" text-anchor="end" class="lbl-tiny">2026-08-10 · nextcloud_office</text>
+  <text x="1060" y="592" text-anchor="end" class="lbl-tiny">2026-08-11 · nextcloud_office · rewire</text>
 </svg>

assets/diagrams/request-flow.svg

@@ -99,7 +99,7 @@
   <rect x="920" y="465" width="160" height="60" rx="4" class="ok"/>
   <text x="1000" y="485" text-anchor="middle" class="lbl" fill="#9d7ad9">NFS write</text>
   <text x="1000" y="503" text-anchor="middle" class="lbl-tiny" fill="#9d7ad9">→ /slab/container_storage/office/</text>
-  <text x="1000" y="518" text-anchor="middle" class="lbl-tiny" fill="#9d7ad9">nextcloud/admin/files/</text>
+  <text x="1000" y="518" text-anchor="middle" class="lbl-tiny" fill="#9d7ad9">admin/files/</text>
 
   <!-- 10. 201 Created returned -->
   <rect x="20" y="555" width="160" height="50" rx="4" class="ok"/>
@@ -134,5 +134,5 @@
   <text x="550" y="681" text-anchor="middle" class="lbl-sm" fill="#d9a96b">Tip: WOPI (Collabora file open) follows a parallel path — browser iframe → apache → nextcloud PHP → wopi URL → apache proxy → collabora → WOPI read</text>
   <text x="550" y="694" text-anchor="middle" class="lbl-tiny" fill="#d9a96b">the WOPI URL is verified as http://nextcloud-aio-apache.nextcloud-aio:23973 (internal docker network only)</text>
 
-  <text x="1060" y="715" text-anchor="end" class="lbl-tiny">2026-08-10 · nextcloud_office</text>
+  <text x="1060" y="715" text-anchor="end" class="lbl-tiny">2026-08-11 · nextcloud_office · rewire</text>
 </svg>

assets/diagrams/topology.svg

@@ -1,4 +1,4 @@
-<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1140 720" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif">
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1140 740" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif">
   <defs>
     <style>
       .lbl { fill: #e6e6e6; font-size: 13px; }
@@ -11,11 +11,13 @@
       .box-public { fill: #1f2c3a; stroke: #4a78b5; stroke-width: 2; }
       .box-netbird { fill: #2a2535; stroke: #6b5aa0; stroke-width: 2; }
       .box-storage { fill: #1f2e28; stroke: #4a8a6b; stroke-width: 2; }
+      .box-bind { fill: #1f2e28; stroke: #6cba92; stroke-width: 2; stroke-dasharray: 3 3; }
       .box-retired { fill: #1f2230; stroke: #5a3a3a; stroke-width: 1.5; stroke-dasharray: 4 3; }
       .x-link { stroke: #5a6072; stroke-width: 1.5; fill: none; }
       .x-link-primary { stroke: #7aa2f7; stroke-width: 2; fill: none; marker-end: url(#arr-b); }
       .x-link-mesh { stroke: #9d7ad9; stroke-width: 2; fill: none; stroke-dasharray: 6 4; marker-end: url(#arr-p); }
       .x-link-storage { stroke: #6cba92; stroke-width: 2; fill: none; marker-end: url(#arr-g); }
+      .x-link-bind { stroke: #6cba92; stroke-width: 2; fill: none; stroke-dasharray: 3 3; marker-end: url(#arr-g); }
       .x-link-retired { stroke: #a86b6b; stroke-width: 1.5; fill: none; stroke-dasharray: 4 3; marker-end: url(#arr-r); }
       .x-link-fail { stroke: #d97a7a; stroke-width: 2; fill: none; marker-end: url(#arr-r); }
     </style>
@@ -33,10 +35,10 @@
     </marker>
   </defs>
 
-  <rect width="1100" height="720" fill="#0f1117"/>
+  <rect width="1100" height="740" fill="#0f1117"/>
 
   <text x="40" y="38" class="ttl">Nextcloud Office — Public Topology</text>
-  <text x="40" y="58" class="lbl-sm">office.rmf44.xyz · Caddy on hawker · AIO on homework03 · NFS on desslok</text>
+  <text x="40" y="58" class="lbl-sm">office.rmf44.xyz · Caddy on hawker · AIO on homework03 · NFS on desslok · rewire 2026-08-11</text>
 
   <!-- Section labels -->
   <text x="80"  y="110" class="section">Public Internet</text>
@@ -71,12 +73,6 @@
   <text x="490" y="316" text-anchor="middle" class="lbl">NetBird (wt0)</text>
   <text x="490" y="333" text-anchor="middle" class="lbl-tiny">100.79.4.103 · P2P mesh</text>
 
-  <!-- Note -->
-  <text x="490" y="370" text-anchor="middle" class="lbl-sm" fill="#9d7ad9">+ retired OnlyOffice torn down</text>
-  <text x="490" y="386" text-anchor="middle" class="lbl-tiny" fill="#9d7ad9">see procedure.html §4.4</text>
-  <rect x="400" y="395" width="180" height="18" rx="3" class="box-retired"/>
-  <text x="490" y="408" text-anchor="middle" class="lbl-sm" fill="#a86b6b">[retired] onlyoffice-files-1</text>
-
   <!-- Compute homework03 -->
   <rect x="660" y="140" width="240" height="380" rx="8" class="box-public"/>
   <text x="780" y="166" text-anchor="middle" class="lbl">homework03 (10.0.0.73)</text>
@@ -98,16 +94,16 @@
   <text x="780" y="380" text-anchor="middle" class="lbl">nextcloud-aio-apache</text>
   <text x="780" y="397" text-anchor="middle" class="lbl-tiny">:11000 → PHP-FPM</text>
 
-  <!-- Sidecar group -->
+  <!-- nextcloud-aio-nextcloud -->
-  <rect x="680" y="428" width="200" height="80" rx="6" class="box"/>
+  <rect x="680" y="428" width="200" height="80" rx="6" class="box-bind"/>
-  <text x="780" y="448" text-anchor="middle" class="lbl-sm">5 sidecars</text>
+  <text x="780" y="450" text-anchor="middle" class="lbl" fill="#6cba92">nextcloud-aio-nextcloud</text>
-  <text x="780" y="466" text-anchor="middle" class="lbl-tiny">nextcloud · database · redis</text>
+  <text x="780" y="466" text-anchor="middle" class="lbl-tiny" fill="#6cba92">/srv/nc-files → /mnt/ncdata</text>
-  <text x="780" y="481" text-anchor="middle" class="lbl-tiny">collabora · whiteboard</text>
+  <text x="780" y="482" text-anchor="middle" class="lbl-tiny" fill="#6cba92">(named vol) → /var/www/html</text>
-  <text x="780" y="497" text-anchor="middle" class="lbl-tiny">notify-push · database-dump</text>
+  <text x="780" y="498" text-anchor="middle" class="lbl-tiny" fill="#6cba92">NFS user data · local app</text>
 
-  <!-- NFS mount -->
+  <!-- NFS mount on homework03 -->
   <rect x="940" y="358" width="140" height="50" rx="6" class="box-storage"/>
-  <text x="1010" y="378" text-anchor="middle" class="lbl-sm">/srv/nc-files/</text>
+  <text x="1010" y="378" text-anchor="middle" class="lbl-sm">/srv/nc-files</text>
   <text x="1010" y="395" text-anchor="middle" class="lbl-tiny">NFS v4.1 mount</text>
 
   <!-- Storage desslok -->
@@ -120,18 +116,18 @@
   <text x="1010" y="244" text-anchor="middle" class="lbl-tiny">office/</text>
 
   <rect x="955" y="260" width="110" height="30" rx="3" class="box"/>
-  <text x="1010" y="279" text-anchor="middle" class="lbl-tiny">nextcloud/</text>
+  <text x="1010" y="279" text-anchor="middle" class="lbl-tiny">admin/ race/</text>
 
   <rect x="955" y="294" width="110" height="30" rx="3" class="box"/>
   <text x="1010" y="313" text-anchor="middle" class="lbl-tiny">backups/</text>
 
   <!-- Backup pipeline note -->
-  <rect x="660" y="540" width="420" height="120" rx="6" class="box"/>
+  <rect x="660" y="560" width="420" height="120" rx="6" class="box"/>
-  <text x="870" y="562" text-anchor="middle" class="lbl">Daily backup pipeline (hector → homework03 → NFS)</text>
+  <text x="870" y="582" text-anchor="middle" class="lbl">Daily backup pipeline (hector → homework03 → NFS)</text>
-  <text x="700" y="585" class="lbl-sm">03:30 UTC, systemd timer on hector</text>
+  <text x="700" y="605" class="lbl-sm">03:30 UTC, systemd timer on hector</text>
-  <text x="700" y="605" class="lbl-sm">ssh homework03 sudo /usr/local/bin/office-backup.sh</text>
+  <text x="700" y="625" class="lbl-sm">ssh homework03 sudo /usr/local/bin/office-backup.sh</text>
-  <text x="700" y="625" class="lbl-sm">  → pg_dumpall → /srv/nc-files/backups/office-YYYYMMDD-*.gz</text>
+  <text x="700" y="645" class="lbl-sm">  → pg_dumpall → /srv/nc-files/backups/office-YYYYMMDD-*.gz</text>
-  <text x="700" y="645" class="lbl-tiny">retention: 14 days, prunes via find -mtime +14</text>
+  <text x="700" y="665" class="lbl-tiny">retention: 14 days, prunes via find -mtime +14</text>
 
   <!-- Edges -->
   <!-- user → DNS -->
@@ -142,39 +138,42 @@
   <line x1="260" y1="285" x2="400" y2="244" class="x-link-primary"/>
   <text x="280" y="260" class="lbl-tiny">2. HTTPS</text>
 
-  <!-- Caddy → NetBird -->
-  <line x1="490" y1="278" x2="490" y2="298" class="x-link"/>
-
   <!-- NetBird hawker → NetBird homework03 (mesh) -->
   <line x1="580" y1="234" x2="680" y2="234" class="x-link-mesh"/>
   <text x="630" y="225" text-anchor="middle" class="lbl-tiny" fill="#9d7ad9">WireGuard P2P · UDP 51820</text>
 
-  <!-- NetBird homework03 → Apache -->
-  <line x1="780" y1="258" x2="780" y2="358" class="x-link"/>
-
   <!-- Caddy → Apache (logically) -->
   <line x1="600" y1="244" x2="680" y2="383" class="x-link-primary" stroke-dasharray="3 3"/>
   <text x="630" y="320" class="lbl-tiny" fill="#7aa2f7" transform="rotate(60 630 320)">upstream :11000</text>
 
-  <!-- NFS from compute → storage -->
+  <!-- NFS mount edge -->
   <line x1="880" y1="383" x2="940" y2="383" class="x-link-storage"/>
   <text x="910" y="377" text-anchor="middle" class="lbl-tiny" fill="#6cba92">NFS</text>
 
+  <!-- Bind arrow: /srv/nc-files → /mnt/ncdata in nextcloud container -->
+  <line x1="940" y1="408" x2="880" y2="448" class="x-link-bind"/>
+  <text x="950" y="430" class="lbl-tiny" fill="#6cba92">bind</text>
+  <text x="945" y="442" class="lbl-tiny" fill="#6cba92">:mnt/ncdata</text>
+
+  <!-- Local volume bind (dashed, small) -->
+  <line x1="880" y1="468" x2="940" y2="408" class="x-link"/>
+  <text x="900" y="465" class="lbl-tiny" fill="#8088a0">app vol</text>
+
   <!-- Backup arrow from backup pipeline → storage -->
-  <line x1="1050" y1="580" x2="1010" y2="340" class="x-link-storage" stroke-dasharray="4 4"/>
+  <line x1="1050" y1="600" x2="1010" y2="340" class="x-link-storage" stroke-dasharray="4 4"/>
   <text x="1050" y="450" class="lbl-tiny" fill="#6cba92">writes</text>
 
   <!-- Legend -->
-  <g transform="translate(40, 690)">
+  <g transform="translate(40, 710)">
     <rect x="0" y="-10" width="14" height="14" rx="2" class="box-public"/>
     <text x="22" y="2" class="lbl-tiny">public</text>
     <rect x="80" y="-10" width="14" height="14" rx="2" class="box-netbird"/>
     <text x="102" y="2" class="lbl-tiny">mesh</text>
     <rect x="160" y="-10" width="14" height="14" rx="2" class="box-storage"/>
     <text x="182" y="2" class="lbl-tiny">storage</text>
-    <rect x="260" y="-10" width="14" height="14" rx="2" class="box-retired"/>
+    <rect x="260" y="-10" width="14" height="14" rx="2" class="box-bind"/>
-    <text x="282" y="2" class="lbl-tiny">retired</text>
+    <text x="282" y="2" class="lbl-tiny">NFS bind</text>
   </g>
 
-  <text x="1060" y="694" text-anchor="end" class="lbl-tiny">2026-08-10 · nextcloud_office</text>
+  <text x="1060" y="714" text-anchor="end" class="lbl-tiny">2026-08-11 · nextcloud_office · rewire</text>
 </svg>

index.html

@@ -119,9 +119,9 @@
     <tr><th>Path</th><th>Host</th><th>What</th></tr>
     <tr><td><code>/usr/local/containers/nextcloudaio/docker-compose.yaml</code></td><td>homework03</td><td>Mastercontainer with <code>network_mode: host</code></td></tr>
     <tr><td><code>/usr/local/containers/nextcloudaio/nextcloud-aio-{mastercontainer,database,redis,apache,nextcloud,collabora,whiteboard,notify-push,database-dump}/</code></td><td>homework03</td><td>Named docker volume bind targets</td></tr>
-    <tr><td><code>/srv/nc-files/</code></td><td>homework03</td><td>NFS mount of <code>desslok:/slab/container_storage/office</code></td></tr>
+    <tr><td><code>/srv/nc-files/</code></td><td>homework03</td><td>NFS mount of <code>desslok:/slab/container_storage/office</code>; AIO binds this directly into the nextcloud container at <code>/mnt/ncdata</code> via <code>NEXTCLOUD_DATADIR</code></td></tr>
-    <tr><td><code>/mnt/nc-data/nextcloud-data/</code></td><td>homework03</td><td>Bind into nextcloud container at <code>/nextcloud-aio/data</code></td></tr>
+    <tr><td><code>/usr/local/containers/nextcloudaio/nextcloud-aio-nextcloud/</code></td><td>homework03</td><td>Local docker volume bind for the nextcloud container's <code>/var/www/html</code> (AIO-managed app code)</td></tr>
-    <tr><td><code>/usr/local/bin/office-backup.sh</code></td><td>homework03</td><td>Daily backup script (pgdump + config tar + user files tar)</td></tr>
+    <tr><td><code>/usr/local/bin/office-backup.sh</code></td><td>homework03</td><td>Daily backup script (pgdump + AIO config tar + app volume tar + user files tar)</td></tr>
     <tr><td><code>/etc/systemd/system/office-backup.{service,timer}</code></td><td>hector</td><td>Daily 03:30 UTC trigger, SSH to homework03</td></tr>
     <tr><td><code>/etc/caddy/Caddyfile</code></td><td>hawker</td><td>Reverse proxy block: <code>office.rmf44.xyz → 100.79.142.164:11000</code></td></tr>
     <tr><td><code>/slab/container_storage/office/</code></td><td>desslok</td><td>Live data + <code>backups/</code> subdir</td></tr>

operations.html

@@ -28,7 +28,7 @@
   <h2>Backup pipeline</h2>
 
   <p>
-    Three files written daily to
+    Four files written daily to
     <code>/srv/nc-files/backups/</code> on homework03 (NFS, real path
     <code>/slab/container_storage/office/backups/</code> on desslok):
   </p>
@@ -47,9 +47,15 @@
       <td>~6 KB</td>
       <td>Reconstruct the AIO install state without going through the setup wizard again.</td>
     </tr>
+    <tr>
+      <td><code>office-YYYYMMDD-aio-nextcloud-app.tar.gz</code></td>
+      <td>Tar of <code>/usr/local/containers/nextcloudaio/nextcloud-aio-nextcloud/</code> — the AIO-managed local app volume (Nextcloud app code, installed apps, <code>config/</code>).</td>
+      <td>~200-500 MB depending on installed apps</td>
+      <td>Survives a fresh AIO install: restore this AND the config tarball to skip the entire setup wizard and preserve installed apps.</td>
+    </tr>
     <tr>
       <td><code>office-YYYYMMDD-ncdata.tar.gz</code></td>
-      <td>Tar of <code>/srv/nc-files/nextcloud/</code> (user-uploaded files) — excludes <code>backups/</code> to avoid recursion.</td>
+      <td>Tar of <code>/srv/nc-files/</code> root (user files: <code>admin/</code>, <code>race/</code>, <code>appdata_*/</code>, etc.) — excludes <code>backups/</code> to avoid recursion.</td>
       <td>Empty (~100 B) until users upload files, then grows</td>
       <td>Restore user files after data loss.</td>
     </tr>
@@ -93,12 +99,12 @@ systemctl status office-backup.service | head -5
     </li>
     <li>
       <strong>NFS quiescence</strong> — the tar reads
-      <code>/srv/nc-files/nextcloud/</code> while the filesystem is
+      <code>/srv/nc-files/</code> (the NFS root) while the
-      actively being written to by the nextcloud container. The tar
+      filesystem is actively being written to by the nextcloud
-      will see a consistent enough snapshot for crash-consistent
+      container. The tar will see a consistent enough snapshot for
-      recovery; for true point-in-time recovery, you'd want to
+      crash-consistent recovery; for true point-in-time recovery,
-      quiesce Nextcloud (set maintenance mode) for the duration of
+      you'd want to quiesce Nextcloud (set maintenance mode) for the
-      the tar, which we haven't done.
+      duration of the tar, which we haven't done.
     </li>
   </ul>
 
@@ -142,9 +148,15 @@ sudo -n docker compose down</code></pre>
 tar -C /usr/local/containers/nextcloudaio -xzf "$LATEST"</code></pre>
     </li>
     <li>
-      Restore user files (overwrites the NFS share's <code>nextcloud/</code>):
+      Restore the local nextcloud app volume (AIO-managed code +
+      installed apps):
+      <pre><code>LATEST=$(ls -t /srv/nc-files/backups/office-*-aio-nextcloud-app.tar.gz | head -1)
+tar -C /usr/local/containers/nextcloudaio -xzf "$LATEST"</code></pre>
+    </li>
+    <li>
+      Restore user files (NFS root, no intermediate <code>nextcloud/</code>):
       <pre><code>LATEST=$(ls -t /srv/nc-files/backups/office-*-ncdata.tar.gz | head -1)
-# Tar contains /nextcloud/ at root
+# Tar contains files at root (admin/, race/, appdata_*/, ...)
 tar -C /srv/nc-files -xzf "$LATEST"</code></pre>
     </li>
     <li>
@@ -172,8 +184,9 @@ curl -skI https://office.rmf44.xyz/login
   </p>
   <pre><code>ssh desslok
 LATEST=$(ls -t /slab/container_storage/office/backups/office-*-ncdata.tar.gz | head -1)
-tar -C / -xzf "$LATEST" nextcloud/admin/files/path/to/file
+# Tar contains files at root; restore one user's file:
-# Adjust for the user + path</code></pre>
+tar -C / -xzf "$LATEST" race/files/path/to/file
+# Adjust for the user + path; user dirs are at NFS root (no nextcloud/ prefix)</code></pre>
 
   <h3>Re-initialize the admin user</h3>
   <p>

procedure.html

@@ -123,7 +123,7 @@ sudo -n mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600 \
   desslok:/slab/container_storage/office /srv/nc-files
 df -h /srv/nc-files
 ls -la /srv/nc-files
-# Expect: nextcloud/  backups/</code></pre>
+# Expect: admin/  race/  backups/  appdata_*/  nextcloud.log  ...</code></pre>
 
   <p>
     Add to <code>/etc/fstab</code> for boot persistence:
@@ -146,8 +146,9 @@ for sub in mastercontainer database database-dump redis apache nextcloud \
   sudo -n mkdir -p "/usr/local/containers/nextcloudaio/nextcloud-aio-$sub"
 done
 
-# /mnt/nc-data for the Nextcloud data dir (lives on local ext4)
+# /srv/nc-files is the NFS mount. AIO bind-mounts it directly into
-sudo -n mkdir -p /mnt/nc-data/nextcloud-data
+# the nextcloud container at /mnt/ncdata via NEXTCLOUD_DATADIR.
+# No intermediate /mnt/nc-data layer — see "Rewire 2026-08-11" note.
 
 # Local backup stash (so the script can write the pgdump into NFS without recursion)
 ls -la /usr/local/containers/nextcloudaio/</code></pre>
@@ -166,6 +167,79 @@ sudo -n chown -R 999:999 /usr/local/containers/nextcloudaio/nextcloud-aio-redis
 sudo -n chown -R root:root /usr/local/containers/nextcloudaio/nextcloud-aio-nextcloud
 sudo -n chown -R 100:101 /usr/local/containers/nextcloudaio/nextcloud-aio-collabora</code></pre>
 
+  <h3>2.5 Rewire 2026-08-11 — single-mount architecture</h3>
+  <p>
+    After the initial deploy we discovered a sharp edge: when
+    <code>NEXTCLOUD_DATADIR</code> is a <em>different</em> host path
+    than the actual NFS mount (the original design used
+    <code>/srv/nc-files</code> for NFS and
+    <code>/mnt/nc-data/nextcloud-data</code> for AIO), a typo on
+    either side silently creates an empty datadir inside the
+    nextcloud container. Nextcloud then refuses to start with
+    "Appdata directory is not present!" — but the empty datadir
+    is real, so the NFS data is still there, just not mounted.
+    That's the failure mode that took the office suite offline on
+    2026-08-11.
+  </p>
+  <p>
+    The rewire removes the intermediate
+    <code>/mnt/nc-data/nextcloud-data</code> bind entirely:
+  </p>
+  <ul>
+    <li>
+      NFS export <code>desslok:/slab/container_storage/office</code>
+      mounts at <code>/srv/nc-files</code> on homework03 (unchanged).
+    </li>
+    <li>
+      <code>NEXTCLOUD_DATADIR=/srv/nc-files</code> in compose AND
+      <code>configuration.json</code>'s <code>nextcloud_datadir</code>
+      field both point at the same path.
+    </li>
+    <li>
+      AIO's <code>containers.json</code> template substitutes
+      <code>%NEXTCLOUD_DATADIR%</code> with that path and creates a
+      bind mount directly: host <code>/srv/nc-files</code> →
+      container <code>/mnt/ncdata</code>.
+    </li>
+    <li>
+      The mastercontainer no longer has the
+      <code>/srv/nc-files</code> or
+      <code>/mnt/nc-data/nextcloud-data</code> binds in its
+      compose <code>volumes:</code> section.
+    </li>
+  </ul>
+  <p>
+    <strong>Recovery if it ever breaks again:</strong>
+  </p>
+  <pre><code># 1. Confirm what's in the NFS export
+ssh desslok ls -la /slab/container_storage/office
+# Expect: admin/  race/  appdata_*/  ...
+
+# 2. Confirm the mount is healthy on homework03
+ssh homework03 df -h /srv/nc-files
+ssh homework03 ls -la /srv/nc-files
+# Expect: same admin/, race/, ... as step 1
+
+# 3. Check what AIO thinks the datadir is
+ssh homework03 sudo cat \
+  /var/lib/docker/volumes/nextcloud_aio_mastercontainer/_data/configuration.json \
+  | jq -r .nextcloud_datadir
+# Expect: "/srv/nc-files" — if not, fix with jq (see ~/.hermes
+# creds or the rewire script notes in this repo's history)
+
+# 4. Inspect what the running nextcloud container actually has bound
+ssh homework03 sudo docker inspect nextcloud-aio-nextcloud \
+  | jq -r '.[0].Mounts[] | "\(.Source) -> \(.Destination)"'
+# Expect: "/srv/nc-files -> /mnt/ncdata" AND
+#         "/var/lib/docker/volumes/nextcloud_aio_nextcloud/_data -> /var/www/html"
+# If /mnt/ncdata is bound to something else, the container has stale config.
+
+# 5. If the bind source is wrong, force AIO to re-spawn nextcloud:
+ssh homework03 sudo docker rm -f nextcloud-aio-nextcloud
+# Then trigger /api/docker/start from the admin UI (Apache must
+# be stopped first; the nextcloud container does NOT auto-spawn
+# on mastercontainer restart). See "Phase 6: Spawn lifecycle" below.</code></pre>
+
   <h2 id="phase-3">Phase 3 — AIO mastercontainer + setup wizard</h2>
   <p>
     Write the compose file, start the mastercontainer, and walk the
@@ -184,7 +258,7 @@ services:
     environment:
       APACHE_PORT: "11000"
       APACHE_DISABLE_REWRITE_IP: "1"
-      NEXTCLOUD_DATADIR: "/mnt/nc-data/nextcloud-data"
+      NEXTCLOUD_DATADIR: "/srv/nc-files"
       NEXTCLOUD_UPLOAD_LIMIT: "10G"
       NEXTCLOUD_MAX_TIME: "3600"
       AIO_DISABLE_BACKUP: "true"
@@ -201,8 +275,11 @@ services:
     volumes:
       - ./nextcloud-aio-mastercontainer:/container-volume
       - /var/run/docker.sock:/var/run/docker.sock:ro
-      - /srv/nc-files:/srv/nc-files
+      # NOTE: do NOT bind /srv/nc-files into the mastercontainer.
-      - /mnt/nc-data/nextcloud-data:/mnt/nc-data/nextcloud-data
+      # AIO bind-mounts it directly into the nextcloud container via
+      # NEXTCLOUD_DATADIR. (Pre-rewire this entry also bound
+      # /mnt/nc-data/nextcloud-data — that intermediate layer was
+      # removed 2026-08-11.)
 EOF</code></pre>
 
   <h3>3.2 Start mastercontainer + pull the AIO passphrase</h3>
@@ -239,7 +316,7 @@ hostname -I | awk '{print $1}'
   <pre><code>ssh homework03
 sudo -n cat /usr/local/containers/nextcloudaio/nextcloud-aio-mastercontainer/configuration.json
 # Expect: "officeSuite": "collabora", "isWhiteboardEnabled": true,
-# "domain": "office.rmf44.xyz", "nextcloud_datadir": "/mnt/nc-data/nextcloud-data"</code></pre>
+# "domain": "office.rmf44.xyz", "nextcloud_datadir": "/srv/nc-files"</code></pre>
 
   <h2 id="phase-4">Phase 4 — Public ingress + cutover</h2>
   <p>
@@ -277,16 +354,23 @@ sudo -n docker exec caddy-caddy-1 caddy reload \
 
   <h3>4.3 Verify the cutover</h3>
     <pre><code>curl -skI https://office.rmf44.xyz/
-# HTTP/2 200
+  # HTTP/2 302
-# content-type: text/html; charset=UTF-8
+  # location: /login
-# ...
+  curl -sk https://office.rmf44.xyz/login | grep -oE '&lt;title&gt;[^&lt;]+&lt;/title&gt;'
-curl -s https://office.rmf44.xyz/ | grep -oE '<title>[^<]+</title>'
+  # &lt;title&gt;Login - AIO&lt;/title&gt;
-# &lt;title&gt;Login – Nextcloud&lt;/title&gt;
+
+  curl -sk https://office.rmf44.xyz/status.php
+  # {"installed":true,"version":"34.0.2.1","...","maintenance":false}
 
   # Test login
   # 1. GET /login → grab requesttoken + cookies
   # 2. POST /login with user=admin + password + requesttoken
-# 3. Expect HTTP 303 → /apps/dashboard/</code></pre>
+  # 3. Expect HTTP 303 → /apps/dashboard/
+
+  # Verify the nextcloud container can see NFS user files
+  ssh homework03 sudo docker exec nextcloud-aio-nextcloud \
+    ls -la /mnt/ncdata/race/files/ | head
+  # Expect: Documents/  Photos/  Templates/  ...</code></pre>
 
   <h3>4.4 Tear down the old OnlyOffice</h3>
   <pre><code>ssh tigo@hawker
@@ -343,16 +427,24 @@ tar -C /usr/local/containers/nextcloudaio \
   -czf "${BACKUP_DIR}/${NAME}-aio-config.tar.gz" \
   nextcloud-aio-mastercontainer nextcloud-aio-database-dump
 
-# 3. Tar user files (excluding the backups/ subdir to avoid recursion)
+# 3. Tar the local nextcloud app volume (AIO-managed app code +
+#    config — survives a fresh AIO install if we ever need to
+#    restore from a corrupt mastercontainer state).
+tar -C /usr/local/containers/nextcloudaio \
+  -czf "${BACKUP_DIR}/${NAME}-aio-nextcloud-app.tar.gz" \
+  nextcloud-aio-nextcloud
+
+# 4. Tar user files (NFS root: admin/, race/, appdata_*/, etc.)
+#    Exclude backups/ to avoid recursion.
 tar -C /srv/nc-files \
   --exclude='backups' \
   -czf "${BACKUP_DIR}/${NAME}-ncdata.tar.gz" \
-  nextcloud
+  .
 
-# 4. Prune anything older than 14 days
+# 5. Prune anything older than 14 days
 find "${BACKUP_DIR}" -maxdepth 1 -type f -name 'office-*' -mtime +14 -delete
 
-echo "OK: wrote ${NAME}-{{pgdump.sql.gz,aio-config.tar.gz,ncdata.tar.gz}} to ${BACKUP_DIR}"
+echo "OK: wrote ${NAME}-{pgdump.sql.gz,aio-config.tar.gz,aio-nextcloud-app.tar.gz,ncdata.tar.gz} to ${BACKUP_DIR}"
 EOF
 
 sudo -n chmod 755 /usr/local/bin/office-backup.sh

troubleshooting.html

@@ -21,9 +21,10 @@
 
   <h1>Troubleshooting</h1>
   <p>
-    Every pitfall hit during the 2026-08-10 deployment, with root cause
+    Every pitfall hit during the 2026-08-10 deployment plus the
-    and resolution. Order is roughly chronological — these are what
+    2026-08-11 rewire, with root cause and resolution. Order is
-    blocked progress at each stage.
+    roughly chronological — these are what blocked progress at each
+    stage.
   </p>
 
   <div class="toc">
@@ -38,6 +39,9 @@
       <li><a href="#onlyoffice-rejected">"OnlyOffice" rejected by AIO (must use Collabora or office flag)</a></li>
       <li><a href="#nextcloud-login-flow">curl login returns 303 with empty user — CSRF cookie dance</a></li>
       <li><a href="#backup-script-ownership">Backup script won't run as tigo — root-owned 755 instead</a></li>
+      <li><a href="#appdata-missing">"Appdata directory is not present" — wrong datadir (rewire 2026-08-11)</a></li>
+      <li><a href="#nextcloud-not-spawning">nextcloud container not appearing after config change</a></li>
+      <li><a href="#collabora-discovery-warning">Collabora logs "Could not create path .../richdocuments/remoteData/discovery"</a></li>
     </ul>
   </div>
 
@@ -351,6 +355,167 @@ sudo -n bash -n /usr/local/bin/office-backup.sh  # syntax check</code></pre>
 ExecStart=/usr/bin/ssh -o BatchMode=yes -o ConnectTimeout=30 \
     homework03 sudo /usr/local/bin/office-backup.sh</code></pre>
 
+  <h2 id="appdata-missing">"Appdata directory is not present"</h2>
+
+  <div class="callout danger">
+    <p><strong>Symptom:</strong> <code>docker logs nextcloud-aio-nextcloud</code>
+    shows <code>Cannot write into directory "/mnt/ncdata/appdata_*"</code>
+    or <code>Appdata directory is not present!</code>. The login page
+    may load but logins loop or fail. Sometimes the nextcloud
+    container restart-loops.</p>
+  </div>
+
+  <h3>Root cause</h3>
+  <p>
+    AIO uses the value of <code>NEXTCLOUD_DATADIR</code> (or the
+    <code>nextcloud_datadir</code> field in
+    <code>configuration.json</code>) as the <strong>host bind-mount
+    source</strong> that goes into the nextcloud container's
+    <code>/mnt/ncdata</code>. If that path is not the actual NFS
+    mount (or is a different host path than the NFS mount), AIO
+    happily bind-mounts whatever the path resolves to — including an
+    empty local directory — and Nextcloud boots against an empty
+    datadir that has no <code>appdata_*</code> directory in it.
+  </p>
+  <p>
+    The data is still on NFS. It's just not being mounted. This
+    took office.rmf44.xyz offline on 2026-08-11 for ~15 minutes.
+  </p>
+
+  <h3>Fix (the 2026-08-11 rewire)</h3>
+  <ol>
+    <li>
+      Pick <strong>one</strong> host path that is the NFS mount.
+      We picked <code>/srv/nc-files</code>.
+    </li>
+    <li>
+      Set both <code>NEXTCLOUD_DATADIR</code> in compose AND
+      <code>nextcloud_datadir</code> in
+      <code>configuration.json</code> to that same path.
+    </li>
+    <li>
+      Remove any intermediate bind in the mastercontainer's
+      compose <code>volumes:</code> section — AIO doesn't need it
+      and it adds a layer that can drift out of sync.
+    </li>
+    <li>
+      Force the nextcloud container to respawn with the new bind:
+      <code>docker rm -f nextcloud-aio-nextcloud</code>, then
+      trigger <code>/api/docker/start</code> via the admin UI (with
+      Apache stopped first). See
+      <a href="#nextcloud-not-spawning">nextcloud container not appearing</a>
+      below for the spawn dance.
+    </li>
+  </ol>
+
+  <p>
+    The full sequence (mount, compose, config, respawn) is documented
+    in <a href="procedure.html#rewire-2026-08-11">procedure §2.5
+    Rewire 2026-08-11</a>.
+  </p>
+
+  <h2 id="nextcloud-not-spawning">nextcloud container not appearing after config change</h2>
+
+  <div class="callout warn">
+    <p><strong>Symptom:</strong> you updated
+    <code>configuration.json</code> (changed
+    <code>nextcloud_datadir</code>, enabled an extra container, etc.)
+    and restarted the mastercontainer, but the
+    <code>nextcloud-aio-nextcloud</code> container is missing or
+    running with the old config.</p>
+  </div>
+
+  <h3>Root cause</h3>
+  <p>
+    <strong>AIO does NOT auto-spawn the nextcloud container on
+    mastercontainer restart.</strong> The mastercontainer only
+    orchestrates the lifecycle of containers it spawns. If the
+    nextcloud container was already running, the mastercontainer
+    just observes it. If you <code>docker rm -f</code> an old
+    broken container and restart the mastercontainer, the
+    mastercontainer has no awareness that you want a new one — you
+    must explicitly trigger <code>/api/docker/start</code>.
+  </p>
+  <p>
+    Additionally, <code>isLoginAllowed()</code> in
+    <code>DockerActionManager.php</code> returns <code>false</code>
+    when Apache is starting or running and its port is open. So
+    <code>/api/docker/start</code> only fires when Apache is stopped.
+  </p>
+
+  <h3>Fix — the spawn dance</h3>
+  <p>
+    Use <code>/tmp/aio-flow.sh</code> on homework03 — it runs the
+    four steps atomically:
+  </p>
+  <pre><code>ssh homework03
+sudo /tmp/aio-flow.sh
+# 1. POST /api/docker/stop         (stops Apache — login allowed)
+# 2. GET /login + POST /api/auth/login (saves cookies + CSRF)
+# 3. POST /api/docker/start        (triggers spawn)
+# 4. POST /api/docker/stop /start  (re-runs for nextcloud container, restart Apache)</code></pre>
+
+  <p>
+    Or manually via curl:
+  </p>
+  <pre><code>JAR=/tmp/cookies.txt
+BASE=https://office.rmf44.xyz:8080
+# 1. Stop Apache
+curl -skb $JAR -X POST "$BASE/api/docker/stop"
+# 2. Login
+curl -skc $JAR -o /tmp/login.html "$BASE/login"
+TOKEN=$(grep -oE 'data-requesttoken="[^"]+"' /tmp/login.html | head -1 | sed 's/data-requesttoken="//;s/"$//')
+curl -skb $JAR -c $JAR \
+  -H "requesttoken: $TOKEN" \
+  -d "password=$ADMIN_PASSWORD" \
+  -X POST "$BASE/api/auth/login"
+# 3. Trigger spawn
+curl -skb $JAR -c $JAR \
+  -H "requesttoken: $TOKEN" \
+  -X POST "$BASE/api/docker/start"
+# 4. Restart Apache
+curl -skb $JAR -X POST "$BASE/api/docker/start"</code></pre>
+
+  <h2 id="collabora-discovery-warning">Collabora logs "Could not create path .../richdocuments/remoteData/discovery"</h2>
+
+  <div class="callout info">
+    <p><strong>Symptom:</strong> after starting Collabora, the
+    nextcloud container logs lines like:</p>
+    <pre><code>Could not create path /mnt/ncdata/appdata_*/richdocuments/remoteData/discovery
+Failed to fetch discovery endpoint</code></pre>
+  </div>
+
+  <h3>Root cause</h3>
+  <p>
+    The nextcloud container runs as <code>www-data</code>, but the
+    parent <code>appdata_*/</code> directory on NFS was created by an
+    earlier process (possibly the AIO entrypoint as root during first
+    init) and the per-app <code>richdocuments/</code> subdir doesn't
+    exist yet. The nextcloud container can't create it because it
+    doesn't own the parent.
+  </p>
+
+  <h3>Resolution</h3>
+  <p>
+    <strong>Wait it out.</strong> Collabora generates the discovery
+    JSON on first use (when a user opens a Word/Excel file in the
+    Collabora iframe). The warning is logged but the discovery is
+    cached on first successful WOPI round-trip. If Collabora is
+    actually broken (the iframe stays blank), check perms:
+  </p>
+  <pre><code>ssh homework03
+sudo docker exec nextcloud-aio-nextcloud \
+  ls -la /mnt/ncdata/appdata_*/richdocuments/remoteData/
+# If missing, force creation as www-data:
+sudo docker exec -u www-data nextcloud-aio-nextcloud \
+  mkdir -p /mnt/ncdata/appdata_*/richdocuments/remoteData/</code></pre>
+
+  <p>
+    We have not seen this fail on a live Collabora open since the
+    2026-08-11 rewire — the warning appears once at startup and
+    then Collabora works normally.
+  </p>
+
 </div>
 </body>
 </html>