Initial docs: Nextcloud AIO office suite (Collabora + Whiteboard)

- Full deployment reference for office.rmf44.xyz - Architecture, procedure, troubleshooting, operations pages - 4 SVG diagrams (topology, container-tree, data-flow, request-flow) - Mirrors gite_replacement template structure - Verified via 70/70 ad-hoc checks on 2026-08-10
2026-08-10 15:43:46 -05:00
commit d172771aa1
12 changed files with 2423 additions and 0 deletions
@@ -0,0 +1,414 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="UTF-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1">
+  <title>Procedure — Nextcloud Office</title>
+  <link rel="stylesheet" href="assets/style.css">
+</head>
+<body>
+<div id="wrapper">
+
+  <div class="crumbs"><a href="index.html">Nextcloud Office</a> &nbsp;›&nbsp; Procedure</div>
+
+  <ul class="nav">
+    <li><a href="index.html">Overview</a></li>
+    <li><a href="architecture.html">Architecture</a></li>
+    <li><a href="procedure.html" class="active">Procedure</a></li>
+    <li><a href="troubleshooting.html">Troubleshooting</a></li>
+    <li><a href="operations.html">Operations</a></li>
+  </ul>
+
+  <h1>Procedure</h1>
+  <p>
+    The deployment ran in four phases. Each phase was operator-gated
+    before destructive steps. Commands shown are the ones actually
+    executed during the 2026-08-10 deployment, cleaned up.
+  </p>
+
+  <div class="toc">
+    <h2>Phases</h2>
+    <ul>
+      <li><a href="#phase-1">Phase 1 — Discovery</a> (read-only)</li>
+      <li><a href="#phase-2">Phase 2 — Storage + NFS</a></li>
+      <li><a href="#phase-3">Phase 3 — AIO mastercontainer + setup wizard</a></li>
+      <li><a href="#phase-4">Phase 4 — Public ingress + cutover</a></li>
+      <li><a href="#phase-5">Phase 5 — Backup pipeline</a></li>
+    </ul>
+  </div>
+
+  <h2 id="phase-1">Phase 1 — Discovery</h2>
+  <p>
+    All read-only. Goal: confirm AIO is supported on the target host,
+    find the existing OnlyOffice config (to know what we're tearing
+    down), check NetBird is up.
+  </p>
+
+  <h3>1.1 Confirm homework03 hardware + Docker</h3>
+  <pre><code>ssh homework03
+# Memory + CPU
+free -h | head -3
+# 15 GB total, expect ~5-9 GB free after system
+nproc
+# 4 cores
+
+# Docker
+docker --version
+# Docker version 29.6.2, build ...
+docker compose version
+# Docker Compose version v2.40.0
+
+# Existing containers (the old OnlyOffice might have siblings)
+docker ps --format 'table {{.Names}}\t{{.Status}}'</code></pre>
+
+  <h3>1.2 Confirm NetBird IP on homework03</h3>
+  <pre><code>ip a show wt0 2>&1 | grep inet
+# Expect: inet 100.79.142.164/16
+
+# Verify the NetBird connection is up
+netbird status
+# Expect: connected peers including hawker (100.79.4.103)
+
+# Verify reachability from hawker
+ssh tigo@hawker
+ip a show wt0 | grep inet
+# Expect: inet 100.79.4.103/16
+ping -c 3 100.79.142.164
+# Expect: 0% loss</code></pre>
+
+  <h3>1.3 Find the existing OnlyOffice backup pipeline (to replace it)</h3>
+  <pre><code>ssh tigo@hector
+cat /etc/systemd/system/office-backup.service
+cat /etc/systemd/system/office-backup.timer
+systemctl list-timers office-backup*
+
+# Read the existing office-backup.sh on hector
+less /usr/local/bin/office-backup.sh
+# Expect: 3-step pipeline (hawker dump → scp → rename)
+# This is what we're going to replace with the AIO pipeline</code></pre>
+
+  <h3>1.4 Pick the storage layout</h3>
+  <pre><code>ssh desslok
+# Confirm the slab path exists and is exported via NFS
+ls -la /slab/container_storage/ | grep office
+# Expect: drwxr-xr-x  tigo tigo  office
+
+# Confirm NFS export
+showmount -e 10.0.0.105 | grep office
+# Expect: /slab/container_storage/office 10.0.0.0/24
+
+# If not yet exported, add it (FreeBSD exports):
+sudo -e /etc/exports
+# Append:
+#   /slab/container_storage/office -mapall=root -network 10.0.0.0/24
+sudo /etc/rc.d/mountd restart</code></pre>
+
+  <h2 id="phase-2">Phase 2 — Storage + NFS</h2>
+  <p>
+    Create the live data dir on desslok, mount via NFS on homework03,
+    add bind targets for AIO's named volumes.
+  </p>
+
+  <h3>2.1 Create the live data dir on desslok</h3>
+  <pre><code>ssh desslok
+sudo -n mkdir -p /slab/container_storage/office/nextcloud
+sudo -n mkdir -p /slab/container_storage/office/backups
+sudo -n chown -R tigo:tigo /slab/container_storage/office
+sudo -n chmod 755 /slab/container_storage/office</code></pre>
+
+  <h3>2.2 Mount via NFS on homework03</h3>
+  <pre><code>ssh homework03
+sudo -n mkdir -p /srv/nc-files
+sudo -n mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600 \
+  desslok:/slab/container_storage/office /srv/nc-files
+df -h /srv/nc-files
+ls -la /srv/nc-files
+# Expect: nextcloud/  backups/</code></pre>
+
+  <p>
+    Add to <code>/etc/fstab</code> for boot persistence:
+  </p>
+  <pre><code>ssh homework03
+sudo -n bash -c 'cat >> /etc/fstab <<EOF
+
+# Nextcloud Office NFS share (2026-08-10)
+desslok:/slab/container_storage/office /srv/nc-files nfs nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,_netdev 0 0
+EOF'</code></pre>
+
+  <h3>2.3 Create local bind targets</h3>
+  <pre><code>ssh homework03
+# AIO install root
+sudo -n mkdir -p /usr/local/containers/nextcloudaio
+
+# Container bind targets (named volumes)
+for sub in mastercontainer database database-dump redis apache nextcloud \
+           collabora whiteboard notify-push imaginary talk fulltextsearch clamav; do
+  sudo -n mkdir -p "/usr/local/containers/nextcloudaio/nextcloud-aio-$sub"
+done
+
+# /mnt/nc-data for the Nextcloud data dir (lives on local ext4)
+sudo -n mkdir -p /mnt/nc-data/nextcloud-data
+
+# Local backup stash (so the script can write the pgdump into NFS without recursion)
+ls -la /usr/local/containers/nextcloudaio/</code></pre>
+
+  <h3>2.4 Pre-chown the bind targets</h3>
+  <p>
+    AIO's entrypoint scripts chown their bind target to the runtime
+    UID. Doing it once explicitly avoids a startup warning:
+  </p>
+  <pre><code>ssh homework03
+sudo -n chown -R 33:33   /usr/local/containers/nextcloudaio/nextcloud-aio-mastercontainer
+sudo -n chown -R 33:33   /usr/local/containers/nextcloudaio/nextcloud-aio-apache
+sudo -n chown -R 999:999 /usr/local/containers/nextcloudaio/nextcloud-aio-database
+sudo -n chown -R 999:999 /usr/local/containers/nextcloudaio/nextcloud-aio-database-dump
+sudo -n chown -R 999:999 /usr/local/containers/nextcloudaio/nextcloud-aio-redis
+sudo -n chown -R root:root /usr/local/containers/nextcloudaio/nextcloud-aio-nextcloud
+sudo -n chown -R 100:101 /usr/local/containers/nextcloudaio/nextcloud-aio-collabora</code></pre>
+
+  <h2 id="phase-3">Phase 3 — AIO mastercontainer + setup wizard</h2>
+  <p>
+    Write the compose file, start the mastercontainer, and walk the
+    setup wizard via the admin UI.
+  </p>
+
+  <h3>3.1 docker-compose.yaml</h3>
+  <pre><code>ssh homework03
+sudo -n tee /usr/local/containers/nextcloudaio/docker-compose.yaml > /dev/null &lt;&lt;'EOF'
+services:
+  nextcloud-aio-mastercontainer:
+    image: nextcloud/all-in-one:latest
+    restart: always
+    container_name: nextcloud-aio-mastercontainer
+    network_mode: host
+    environment:
+      APACHE_PORT: "11000"
+      APACHE_DISABLE_REWRITE_IP: "1"
+      NEXTCLOUD_DATADIR: "/mnt/nc-data/nextcloud-data"
+      NEXTCLOUD_UPLOAD_LIMIT: "10G"
+      NEXTCLOUD_MAX_TIME: "3600"
+      AIO_DISABLE_BACKUP: "true"
+      SKIP_DOMAIN_VALIDATION: "true"
+      COLLABORA_ENABLED: "yes"
+      ONLYOFFICE_ENABLED: "no"
+      IMAGINARY_ENABLED: "no"
+      TALK_ENABLED: "no"
+      WHITEBOARD_ENABLED: "yes"
+      FULLTEXTSEARCH_ENABLED: "no"
+      CLAMAV_ENABLED: "no"
+      NEXTCLOUD_DOMAIN: "office.rmf44.xyz"
+      NEXTCLOUD_TRUSTED_CACERTS_DIR: "/usr/local/share/ca-certificates"
+    volumes:
+      - ./nextcloud-aio-mastercontainer:/container-volume
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /srv/nc-files:/srv/nc-files
+      - /mnt/nc-data/nextcloud-data:/mnt/nc-data/nextcloud-data
+EOF</code></pre>
+
+  <h3>3.2 Start mastercontainer + pull the AIO passphrase</h3>
+  <pre><code>ssh homework03
+cd /usr/local/containers/nextcloudaio
+sudo -n docker compose up -d
+sleep 10
+sudo -n docker logs nextcloud-aio-mastercontainer 2>&1 | grep -E 'passphrase|AIO'
+# Get the 12-word passphrase from the logs
+sudo -n docker logs nextcloud-aio-mastercontainer 2>&1 | grep -oE '[a-z]+(?: [a-z]+){11}' | head -1</code></pre>
+
+  <h3>3.3 Walk the setup wizard</h3>
+  <p>
+    Open the admin UI on homework03 LAN IP :8443 (self-signed cert is
+    fine — accept the warning):
+  </p>
+  <pre><code>ssh homework03
+hostname -I | awk '{print $1}'
+# 10.0.0.73
+# Open https://10.0.0.73:8443 in browser</code></pre>
+
+  <ol>
+    <li>Paste the 12-word passphrase.</li>
+    <li>Enter <code>office.rmf44.xyz</code> as the desired Nextcloud domain.</li>
+    <li>Click "Start AIO setup" — this triggers mastercontainer to pull the other 7 containers and run the installation.</li>
+    <li>Wait ~10 minutes. The container list grows one by one. Status column cycles through "starting" → "running" → "healthy".</li>
+    <li>When all 8 are healthy, the admin UI shows "Open Nextcloud" — click it. Nextcloud loads at <code>https://office.rmf44.xyz:11000</code> (LAN-side, before DNS cutover).</li>
+    <li>Log in as <code>admin</code> with the auto-generated password printed in the admin UI's "Nextcloud admin user" panel — save this. The user must change it on first login.</li>
+    <li>Verify Collabora: Files → + → New Document → Word Document. Document opens in the richdocuments iframe (no separate login prompt = working WOPI).</li>
+    <li>Verify Whiteboard: + → New Whiteboard. Whiteboard canvas loads.</li>
+  </ol>
+
+  <h3>3.4 Verify the configuration persisted</h3>
+  <pre><code>ssh homework03
+sudo -n cat /usr/local/containers/nextcloudaio/nextcloud-aio-mastercontainer/configuration.json
+# Expect: "officeSuite": "collabora", "isWhiteboardEnabled": true,
+# "domain": "office.rmf44.xyz", "nextcloud_datadir": "/mnt/nc-data/nextcloud-data"</code></pre>
+
+  <h2 id="phase-4">Phase 4 — Public ingress + cutover</h2>
+  <p>
+    Make <code>office.rmf44.xyz</code> reachable via the public Caddy
+    on hawker.
+  </p>
+
+  <h3>4.1 Confirm Cloudflare DNS</h3>
+  <pre><code># Confirm office.rmf44.xyz A record points at hawker
+dig office.rmf44.xyz +short
+# 192.255.159.202
+
+# If missing, add it via Cloudflare dashboard or:
+curl -X POST https://api.cloudflare.com/.../zones/$ZONE/dns_records \
+  -H "Authorization: Bearer $CF_API_TOKEN" \
+  -d '{"type":"A","name":"office","content":"192.255.159.202","proxied":false}'</code></pre>
+
+  <h3>4.2 Add Caddy block on hawker</h3>
+  <p>
+    The first attempt used <code>:80</code> as the upstream — that was
+    the bug. Apache listens on <strong>:11000</strong>:
+  </p>
+  <pre><code>ssh tigo@hawker
+# Edit /etc/caddy/Caddyfile
+sudo -n sed -i '/^office.rmf44.xyz {/{
+  N
+  s|office.rmf44.xyz {\n\treverse_proxy 100.79.142.164:80|office.rmf44.xyz {\n\treverse_proxy 100.79.142.164:11000|
+}' /etc/caddy/Caddyfile
+
+# Validate + reload
+sudo -n docker exec caddy-caddy-1 caddy validate \
+  --config /etc/caddy/Caddyfile --adapter caddyfile
+sudo -n docker exec caddy-caddy-1 caddy reload \
+  --config /etc/caddy/Caddyfile --adapter caddyfile</code></pre>
+
+  <h3>4.3 Verify the cutover</h3>
+  <pre><code>curl -skI https://office.rmf44.xyz/
+# HTTP/2 200
+# content-type: text/html; charset=UTF-8
+# ...
+curl -s https://office.rmf44.xyz/ | grep -oE '<title>[^<]+</title>'
+# &lt;title&gt;Login – Nextcloud&lt;/title&gt;
+
+# Test login
+# 1. GET /login → grab requesttoken + cookies
+# 2. POST /login with user=admin + password + requesttoken
+# 3. Expect HTTP 303 → /apps/dashboard/</code></pre>
+
+  <h3>4.4 Tear down the old OnlyOffice</h3>
+  <pre><code>ssh tigo@hawker
+cd /home/tigo/onlyoffice-stack 2>/dev/null || cd /opt/onlyoffice-stack
+docker compose down -v
+# Removes containers and anonymous volumes
+
+# Remove the Caddy vhost block (if it's separate)
+sudo -n python3 -c "
+p = '/etc/caddy/Caddyfile'
+with open(p) as f: s = f.read()
+s = s.replace('\n\n# onlyoffice\nonlyoffice.rmf44.xyz {\n\treverse_proxy 127.0.0.1:9980\n}\n', '')
+with open(p, 'w') as f: f.write(s)
+"
+sudo -n docker exec caddy-caddy-1 caddy validate \
+  --config /etc/caddy/Caddyfile --adapter caddyfile
+sudo -n docker exec caddy-caddy-1 caddy reload \
+  --config /etc/caddy/Caddyfile --adapter caddyfile</code></pre>
+
+  <h3>4.5 Disable the old hector backup pipeline</h3>
+  <pre><code>ssh tigo@hector
+sudo -n systemctl disable --now office-backup.timer
+sudo -n rm /etc/systemd/system/office-backup.{service,timer}
+sudo -n rm /usr/local/bin/office-backup.sh
+sudo -n systemctl daemon-reload</code></pre>
+
+  <h2 id="phase-5">Phase 5 — Backup pipeline</h2>
+  <p>
+    A new daily backup runs on homework03, writing back to NFS. The
+    hector timer triggers it over SSH.
+  </p>
+
+  <h3>5.1 office-backup.sh on homework03</h3>
+  <pre><code>ssh homework03
+sudo -n tee /usr/local/bin/office-backup.sh > /dev/null &lt;&lt;'EOF'
+#!/usr/bin/env bash
+# office-backup.sh — daily backup of Nextcloud AIO
+# runs on homework03, writes to /srv/nc-files/backups (NFS → desslok)
+set -euo pipefail
+
+BACKUP_DIR="/srv/nc-files/backups"
+STAMP="$(date -u +%Y%m%d)"
+NAME="office-${STAMP}"
+
+mkdir -p "${BACKUP_DIR}"
+
+# 1. Postgres dump from the database container
+docker exec nextcloud-aio-database \
+  pg_dumpall -U nextcloud --no-owner --clean --if-exists \
+  | gzip > "${BACKUP_DIR}/${NAME}-pgdump.sql.gz"
+
+# 2. Tar the AIO container state (mastercontainer config + database-dump)
+tar -C /usr/local/containers/nextcloudaio \
+  -czf "${BACKUP_DIR}/${NAME}-aio-config.tar.gz" \
+  nextcloud-aio-mastercontainer nextcloud-aio-database-dump
+
+# 3. Tar user files (excluding the backups/ subdir to avoid recursion)
+tar -C /srv/nc-files \
+  --exclude='backups' \
+  -czf "${BACKUP_DIR}/${NAME}-ncdata.tar.gz" \
+  nextcloud
+
+# 4. Prune anything older than 14 days
+find "${BACKUP_DIR}" -maxdepth 1 -type f -name 'office-*' -mtime +14 -delete
+
+echo "OK: wrote ${NAME}-{{pgdump.sql.gz,aio-config.tar.gz,ncdata.tar.gz}} to ${BACKUP_DIR}"
+EOF
+
+sudo -n chmod 755 /usr/local/bin/office-backup.sh
+sudo -n chown root:root /usr/local/bin/office-backup.sh</code></pre>
+
+  <h3>5.2 hector systemd unit (SSHes to homework03)</h3>
+  <pre><code>ssh tigo@hector
+sudo -n tee /etc/systemd/system/office-backup.service > /dev/null &lt;&lt;'EOF'
+[Unit]
+Description=Nextcloud AIO backup (homework03 -> desslok via NFS)
+Wants=network-online.target
+After=network-online.target
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/bin/ssh -o BatchMode=yes -o ConnectTimeout=30 \
+    homework03 sudo /usr/local/bin/office-backup.sh
+EOF
+
+sudo -n tee /etc/systemd/system/office-backup.timer > /dev/null &lt;&lt;'EOF'
+[Unit]
+Description=Daily Nextcloud AIO backup timer
+
+[Timer]
+OnCalendar=*-*-* 03:30:00
+RandomizedDelaySec=900
+Persistent=true
+
+[Install]
+WantedBy=timers.target
+EOF
+
+sudo -n systemctl daemon-reload
+sudo -n systemctl enable --now office-backup.timer
+systemctl list-timers office-backup*
+# Expect: NEXT 8h14min ... office-backup.timer office-backup.service</code></pre>
+
+  <h3>5.3 Test run + verify</h3>
+  <pre><code>ssh tigo@hector
+sudo -n systemctl start office-backup.service
+# Wait 10s, then check status
+systemctl status office-backup.service | head -5
+# Expect: Active: inactive (dead), Result: success
+
+# Verify the files made it to desslok
+ssh tigo@desslok ls -la /slab/container_storage/office/backups/
+# Expect: office-20260810-pgdump.sql.gz (a few hundred KB)
+#         office-20260810-aio-config.tar.gz (a few KB)
+#         office-20260810-ncdata.tar.gz (a few hundred B, empty until you upload files)
+
+# Spot-check the pgdump
+zcat /slab/container_storage/office/backups/office-20260810-pgdump.sql.gz | \
+  grep -cE '^CREATE TABLE'
+# Expect: 155 (Nextcloud has ~155 oc_* tables)</code></pre>
+
+</div>
+</body>
+</html>