office/procedure.html

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Procedure — Nextcloud Office</title>
  <link rel="stylesheet" href="assets/style.css">
</head>
<body>
<div id="wrapper">

  <div class="crumbs"><a href="index.html">Nextcloud Office</a> &nbsp;›&nbsp; Procedure</div>

  <ul class="nav">
    <li><a href="index.html">Overview</a></li>
    <li><a href="architecture.html">Architecture</a></li>
    <li><a href="procedure.html" class="active">Procedure</a></li>
    <li><a href="troubleshooting.html">Troubleshooting</a></li>
    <li><a href="operations.html">Operations</a></li>
  </ul>

  <h1>Procedure</h1>
  <p>
    The deployment ran in four phases. Each phase was operator-gated
    before destructive steps. Commands shown are the ones actually
    executed during the 2026-08-10 deployment, cleaned up.
  </p>

  <div class="toc">
    <h2>Phases</h2>
    <ul>
      <li><a href="#phase-1">Phase 1 — Discovery</a> (read-only)</li>
      <li><a href="#phase-2">Phase 2 — Storage + NFS</a></li>
      <li><a href="#phase-3">Phase 3 — AIO mastercontainer + setup wizard</a></li>
      <li><a href="#phase-4">Phase 4 — Public ingress + cutover</a></li>
      <li><a href="#phase-5">Phase 5 — Backup pipeline</a></li>
    </ul>
  </div>

  <h2 id="phase-1">Phase 1 — Discovery</h2>
  <p>
    All read-only. Goal: confirm AIO is supported on the target host,
    find the existing OnlyOffice config (to know what we're tearing
    down), check NetBird is up.
  </p>

  <h3>1.1 Confirm homework03 hardware + Docker</h3>
  <pre><code>ssh homework03
# Memory + CPU
free -h | head -3
# 15 GB total, expect ~5-9 GB free after system
nproc
# 4 cores

# Docker
docker --version
# Docker version 29.6.2, build ...
docker compose version
# Docker Compose version v2.40.0

# Existing containers (the old OnlyOffice might have siblings)
docker ps --format 'table {{.Names}}\t{{.Status}}'</code></pre>

  <h3>1.2 Confirm NetBird IP on homework03</h3>
  <pre><code>ip a show wt0 2>&1 | grep inet
# Expect: inet 100.79.142.164/16

# Verify the NetBird connection is up
netbird status
# Expect: connected peers including hawker (100.79.4.103)

# Verify reachability from hawker
ssh tigo@hawker
ip a show wt0 | grep inet
# Expect: inet 100.79.4.103/16
ping -c 3 100.79.142.164
# Expect: 0% loss</code></pre>

  <h3>1.3 Find the existing OnlyOffice backup pipeline (to replace it)</h3>
  <pre><code>ssh tigo@hector
cat /etc/systemd/system/office-backup.service
cat /etc/systemd/system/office-backup.timer
systemctl list-timers office-backup*

# Read the existing office-backup.sh on hector
less /usr/local/bin/office-backup.sh
# Expect: 3-step pipeline (hawker dump → scp → rename)
# This is what we're going to replace with the AIO pipeline</code></pre>

  <h3>1.4 Pick the storage layout</h3>
  <pre><code>ssh desslok
# Confirm the slab path exists and is exported via NFS
ls -la /slab/container_storage/ | grep office
# Expect: drwxr-xr-x  tigo tigo  office

# Confirm NFS export
showmount -e 10.0.0.105 | grep office
# Expect: /slab/container_storage/office 10.0.0.0/24

# If not yet exported, add it (FreeBSD exports):
sudo -e /etc/exports
# Append:
#   /slab/container_storage/office -mapall=root -network 10.0.0.0/24
sudo /etc/rc.d/mountd restart</code></pre>

  <h2 id="phase-2">Phase 2 — Storage + NFS</h2>
  <p>
    Create the live data dir on desslok, mount via NFS on homework03,
    add bind targets for AIO's named volumes.
  </p>

  <h3>2.1 Create the live data dir on desslok</h3>
  <pre><code>ssh desslok
sudo -n mkdir -p /slab/container_storage/office/nextcloud
sudo -n mkdir -p /slab/container_storage/office/backups
sudo -n chown -R tigo:tigo /slab/container_storage/office
sudo -n chmod 755 /slab/container_storage/office</code></pre>

  <h3>2.2 Mount via NFS on homework03</h3>
  <pre><code>ssh homework03
sudo -n mkdir -p /srv/nc-files
sudo -n mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600 \
  desslok:/slab/container_storage/office /srv/nc-files
df -h /srv/nc-files
ls -la /srv/nc-files
# Expect: admin/  race/  backups/  appdata_*/  nextcloud.log  ...</code></pre>

  <p>
    Add to <code>/etc/fstab</code> for boot persistence:
  </p>
  <pre><code>ssh homework03
sudo -n bash -c 'cat >> /etc/fstab <<EOF

# Nextcloud Office NFS share (2026-08-10)
desslok:/slab/container_storage/office /srv/nc-files nfs nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,_netdev 0 0
EOF'</code></pre>

  <h3>2.3 Create local bind targets</h3>
  <pre><code>ssh homework03
# AIO install root
sudo -n mkdir -p /usr/local/containers/nextcloudaio

# Container bind targets (named volumes)
for sub in mastercontainer database database-dump redis apache nextcloud \
           collabora whiteboard notify-push imaginary talk fulltextsearch clamav; do
  sudo -n mkdir -p "/usr/local/containers/nextcloudaio/nextcloud-aio-$sub"
done

# /srv/nc-files is the NFS mount. AIO bind-mounts it directly into
# the nextcloud container at /mnt/ncdata via NEXTCLOUD_DATADIR.
# No intermediate /mnt/nc-data layer — see "Rewire 2026-08-11" note.

# Local backup stash (so the script can write the pgdump into NFS without recursion)
ls -la /usr/local/containers/nextcloudaio/</code></pre>

  <h3>2.4 Pre-chown the bind targets</h3>
  <p>
    AIO's entrypoint scripts chown their bind target to the runtime
    UID. Doing it once explicitly avoids a startup warning:
  </p>
  <pre><code>ssh homework03
sudo -n chown -R 33:33   /usr/local/containers/nextcloudaio/nextcloud-aio-mastercontainer
sudo -n chown -R 33:33   /usr/local/containers/nextcloudaio/nextcloud-aio-apache
sudo -n chown -R 999:999 /usr/local/containers/nextcloudaio/nextcloud-aio-database
sudo -n chown -R 999:999 /usr/local/containers/nextcloudaio/nextcloud-aio-database-dump
sudo -n chown -R 999:999 /usr/local/containers/nextcloudaio/nextcloud-aio-redis
sudo -n chown -R root:root /usr/local/containers/nextcloudaio/nextcloud-aio-nextcloud
sudo -n chown -R 100:101 /usr/local/containers/nextcloudaio/nextcloud-aio-collabora</code></pre>

  <h3>2.5 Rewire 2026-08-11 — single-mount architecture</h3>
  <p>
    After the initial deploy we discovered a sharp edge: when
    <code>NEXTCLOUD_DATADIR</code> is a <em>different</em> host path
    than the actual NFS mount (the original design used
    <code>/srv/nc-files</code> for NFS and
    <code>/mnt/nc-data/nextcloud-data</code> for AIO), a typo on
    either side silently creates an empty datadir inside the
    nextcloud container. Nextcloud then refuses to start with
    "Appdata directory is not present!" — but the empty datadir
    is real, so the NFS data is still there, just not mounted.
    That's the failure mode that took the office suite offline on
    2026-08-11.
  </p>
  <p>
    The rewire removes the intermediate
    <code>/mnt/nc-data/nextcloud-data</code> bind entirely:
  </p>
  <ul>
    <li>
      NFS export <code>desslok:/slab/container_storage/office</code>
      mounts at <code>/srv/nc-files</code> on homework03 (unchanged).
    </li>
    <li>
      <code>NEXTCLOUD_DATADIR=/srv/nc-files</code> in compose AND
      <code>configuration.json</code>'s <code>nextcloud_datadir</code>
      field both point at the same path.
    </li>
    <li>
      AIO's <code>containers.json</code> template substitutes
      <code>%NEXTCLOUD_DATADIR%</code> with that path and creates a
      bind mount directly: host <code>/srv/nc-files</code> →
      container <code>/mnt/ncdata</code>.
    </li>
    <li>
      The mastercontainer no longer has the
      <code>/srv/nc-files</code> or
      <code>/mnt/nc-data/nextcloud-data</code> binds in its
      compose <code>volumes:</code> section.
    </li>
  </ul>
  <p>
    <strong>Recovery if it ever breaks again:</strong>
  </p>
  <pre><code># 1. Confirm what's in the NFS export
ssh desslok ls -la /slab/container_storage/office
# Expect: admin/  race/  appdata_*/  ...

# 2. Confirm the mount is healthy on homework03
ssh homework03 df -h /srv/nc-files
ssh homework03 ls -la /srv/nc-files
# Expect: same admin/, race/, ... as step 1

# 3. Check what AIO thinks the datadir is
ssh homework03 sudo cat \
  /var/lib/docker/volumes/nextcloud_aio_mastercontainer/_data/configuration.json \
  | jq -r .nextcloud_datadir
# Expect: "/srv/nc-files" — if not, fix with jq (see ~/.hermes
# creds or the rewire script notes in this repo's history)

# 4. Inspect what the running nextcloud container actually has bound
ssh homework03 sudo docker inspect nextcloud-aio-nextcloud \
  | jq -r '.[0].Mounts[] | "\(.Source) -> \(.Destination)"'
# Expect: "/srv/nc-files -> /mnt/ncdata" AND
#         "/var/lib/docker/volumes/nextcloud_aio_nextcloud/_data -> /var/www/html"
# If /mnt/ncdata is bound to something else, the container has stale config.

# 5. If the bind source is wrong, force AIO to re-spawn nextcloud:
ssh homework03 sudo docker rm -f nextcloud-aio-nextcloud
# Then trigger /api/docker/start from the admin UI (Apache must
# be stopped first; the nextcloud container does NOT auto-spawn
# on mastercontainer restart). See "Phase 6: Spawn lifecycle" below.</code></pre>

  <h2 id="phase-3">Phase 3 — AIO mastercontainer + setup wizard</h2>
  <p>
    Write the compose file, start the mastercontainer, and walk the
    setup wizard via the admin UI.
  </p>

  <h3>3.1 docker-compose.yaml</h3>
  <pre><code>ssh homework03
sudo -n tee /usr/local/containers/nextcloudaio/docker-compose.yaml > /dev/null &lt;&lt;'EOF'
services:
  nextcloud-aio-mastercontainer:
    image: nextcloud/all-in-one:latest
    restart: always
    container_name: nextcloud-aio-mastercontainer
    network_mode: host
    environment:
      APACHE_PORT: "11000"
      APACHE_DISABLE_REWRITE_IP: "1"
      NEXTCLOUD_DATADIR: "/srv/nc-files"
      NEXTCLOUD_UPLOAD_LIMIT: "10G"
      NEXTCLOUD_MAX_TIME: "3600"
      AIO_DISABLE_BACKUP: "true"
      SKIP_DOMAIN_VALIDATION: "true"
      COLLABORA_ENABLED: "yes"
      ONLYOFFICE_ENABLED: "no"
      IMAGINARY_ENABLED: "no"
      TALK_ENABLED: "no"
      WHITEBOARD_ENABLED: "yes"
      FULLTEXTSEARCH_ENABLED: "no"
      CLAMAV_ENABLED: "no"
      NEXTCLOUD_DOMAIN: "office.rmf44.xyz"
      NEXTCLOUD_TRUSTED_CACERTS_DIR: "/usr/local/share/ca-certificates"
    volumes:
      - ./nextcloud-aio-mastercontainer:/container-volume
      - /var/run/docker.sock:/var/run/docker.sock:ro
      # NOTE: do NOT bind /srv/nc-files into the mastercontainer.
      # AIO bind-mounts it directly into the nextcloud container via
      # NEXTCLOUD_DATADIR. (Pre-rewire this entry also bound
      # /mnt/nc-data/nextcloud-data — that intermediate layer was
      # removed 2026-08-11.)
EOF</code></pre>

  <h3>3.2 Start mastercontainer + pull the AIO passphrase</h3>
  <pre><code>ssh homework03
cd /usr/local/containers/nextcloudaio
sudo -n docker compose up -d
sleep 10
sudo -n docker logs nextcloud-aio-mastercontainer 2>&1 | grep -E 'passphrase|AIO'
# Get the 12-word passphrase from the logs
sudo -n docker logs nextcloud-aio-mastercontainer 2>&1 | grep -oE '[a-z]+(?: [a-z]+){11}' | head -1</code></pre>

  <h3>3.3 Walk the setup wizard</h3>
  <p>
    Open the admin UI on homework03 LAN IP :8443 (self-signed cert is
    fine — accept the warning):
  </p>
  <pre><code>ssh homework03
hostname -I | awk '{print $1}'
# 10.0.0.73
# Open https://10.0.0.73:8443 in browser</code></pre>

  <ol>
    <li>Paste the 12-word passphrase.</li>
    <li>Enter <code>office.rmf44.xyz</code> as the desired Nextcloud domain.</li>
    <li>Click "Start AIO setup" — this triggers mastercontainer to pull the other 7 containers and run the installation.</li>
    <li>Wait ~10 minutes. The container list grows one by one. Status column cycles through "starting" → "running" → "healthy".</li>
    <li>When all 8 are healthy, the admin UI shows "Open Nextcloud" — click it. Nextcloud loads at <code>https://office.rmf44.xyz:11000</code> (LAN-side, before DNS cutover).</li>
    <li>Log in as <code>admin</code> with the auto-generated password printed in the admin UI's "Nextcloud admin user" panel — save this. The user must change it on first login.</li>
    <li>Verify Collabora: Files → + → New Document → Word Document. Document opens in the richdocuments iframe (no separate login prompt = working WOPI).</li>
    <li>Verify Whiteboard: + → New Whiteboard. Whiteboard canvas loads.</li>
  </ol>

  <h3>3.4 Verify the configuration persisted</h3>
  <pre><code>ssh homework03
sudo -n cat /usr/local/containers/nextcloudaio/nextcloud-aio-mastercontainer/configuration.json
# Expect: "officeSuite": "collabora", "isWhiteboardEnabled": true,
# "domain": "office.rmf44.xyz", "nextcloud_datadir": "/srv/nc-files"</code></pre>

  <h2 id="phase-4">Phase 4 — Public ingress + cutover</h2>
  <p>
    Make <code>office.rmf44.xyz</code> reachable via the public Caddy
    on hawker.
  </p>

  <h3>4.1 Confirm Cloudflare DNS</h3>
  <pre><code># Confirm office.rmf44.xyz A record points at hawker
dig office.rmf44.xyz +short
# 192.255.159.202

# If missing, add it via Cloudflare dashboard or:
curl -X POST https://api.cloudflare.com/.../zones/$ZONE/dns_records \
  -H "Authorization: Bearer $CF_API_TOKEN" \
  -d '{"type":"A","name":"office","content":"192.255.159.202","proxied":false}'</code></pre>

  <h3>4.2 Add Caddy block on hawker</h3>
  <p>
    The first attempt used <code>:80</code> as the upstream — that was
    the bug. Apache listens on <strong>:11000</strong>:
  </p>
  <pre><code>ssh tigo@hawker
# Edit /etc/caddy/Caddyfile
sudo -n sed -i '/^office.rmf44.xyz {/{
  N
  s|office.rmf44.xyz {\n\treverse_proxy 100.79.142.164:80|office.rmf44.xyz {\n\treverse_proxy 100.79.142.164:11000|
}' /etc/caddy/Caddyfile

# Validate + reload
sudo -n docker exec caddy-caddy-1 caddy validate \
  --config /etc/caddy/Caddyfile --adapter caddyfile
sudo -n docker exec caddy-caddy-1 caddy reload \
  --config /etc/caddy/Caddyfile --adapter caddyfile</code></pre>

  <h3>4.3 Verify the cutover</h3>
    <pre><code>curl -skI https://office.rmf44.xyz/
  # HTTP/2 302
  # location: /login
  curl -sk https://office.rmf44.xyz/login | grep -oE '&lt;title&gt;[^&lt;]+&lt;/title&gt;'
  # &lt;title&gt;Login - AIO&lt;/title&gt;

  curl -sk https://office.rmf44.xyz/status.php
  # {"installed":true,"version":"34.0.2.1","...","maintenance":false}

  # Test login
  # 1. GET /login → grab requesttoken + cookies
  # 2. POST /login with user=admin + password + requesttoken
  # 3. Expect HTTP 303 → /apps/dashboard/

  # Verify the nextcloud container can see NFS user files
  ssh homework03 sudo docker exec nextcloud-aio-nextcloud \
    ls -la /mnt/ncdata/race/files/ | head
  # Expect: Documents/  Photos/  Templates/  ...</code></pre>

  <h3>4.4 Tear down the old OnlyOffice</h3>
  <pre><code>ssh tigo@hawker
cd /home/tigo/onlyoffice-stack 2>/dev/null || cd /opt/onlyoffice-stack
docker compose down -v
# Removes containers and anonymous volumes

# Remove the Caddy vhost block (if it's separate)
sudo -n python3 -c "
p = '/etc/caddy/Caddyfile'
with open(p) as f: s = f.read()
s = s.replace('\n\n# onlyoffice\nonlyoffice.rmf44.xyz {\n\treverse_proxy 127.0.0.1:9980\n}\n', '')
with open(p, 'w') as f: f.write(s)
"
sudo -n docker exec caddy-caddy-1 caddy validate \
  --config /etc/caddy/Caddyfile --adapter caddyfile
sudo -n docker exec caddy-caddy-1 caddy reload \
  --config /etc/caddy/Caddyfile --adapter caddyfile</code></pre>

  <h3>4.5 Disable the old hector backup pipeline</h3>
  <pre><code>ssh tigo@hector
sudo -n systemctl disable --now office-backup.timer
sudo -n rm /etc/systemd/system/office-backup.{service,timer}
sudo -n rm /usr/local/bin/office-backup.sh
sudo -n systemctl daemon-reload</code></pre>

  <h2 id="phase-5">Phase 5 — Backup pipeline</h2>
  <p>
    A new daily backup runs on homework03, writing back to NFS. The
    hector timer triggers it over SSH.
  </p>

  <h3>5.1 office-backup.sh on homework03</h3>
  <pre><code>ssh homework03
sudo -n tee /usr/local/bin/office-backup.sh > /dev/null &lt;&lt;'EOF'
#!/usr/bin/env bash
# office-backup.sh — daily backup of Nextcloud AIO
# runs on homework03, writes to /srv/nc-files/backups (NFS → desslok)
set -euo pipefail

BACKUP_DIR="/srv/nc-files/backups"
STAMP="$(date -u +%Y%m%d)"
NAME="office-${STAMP}"

mkdir -p "${BACKUP_DIR}"

# 1. Postgres dump from the database container
docker exec nextcloud-aio-database \
  pg_dumpall -U nextcloud --no-owner --clean --if-exists \
  | gzip > "${BACKUP_DIR}/${NAME}-pgdump.sql.gz"

# 2. Tar the AIO container state (mastercontainer config + database-dump)
tar -C /usr/local/containers/nextcloudaio \
  -czf "${BACKUP_DIR}/${NAME}-aio-config.tar.gz" \
  nextcloud-aio-mastercontainer nextcloud-aio-database-dump

# 3. Tar the local nextcloud app volume (AIO-managed app code +
#    config — survives a fresh AIO install if we ever need to
#    restore from a corrupt mastercontainer state).
tar -C /usr/local/containers/nextcloudaio \
  -czf "${BACKUP_DIR}/${NAME}-aio-nextcloud-app.tar.gz" \
  nextcloud-aio-nextcloud

# 4. Tar user files (NFS root: admin/, race/, appdata_*/, etc.)
#    Exclude backups/ to avoid recursion.
tar -C /srv/nc-files \
  --exclude='backups' \
  -czf "${BACKUP_DIR}/${NAME}-ncdata.tar.gz" \
  .

# 5. Prune anything older than 14 days
find "${BACKUP_DIR}" -maxdepth 1 -type f -name 'office-*' -mtime +14 -delete

echo "OK: wrote ${NAME}-{pgdump.sql.gz,aio-config.tar.gz,aio-nextcloud-app.tar.gz,ncdata.tar.gz} to ${BACKUP_DIR}"
EOF

sudo -n chmod 755 /usr/local/bin/office-backup.sh
sudo -n chown root:root /usr/local/bin/office-backup.sh</code></pre>

  <h3>5.2 hector systemd unit (SSHes to homework03)</h3>
  <pre><code>ssh tigo@hector
sudo -n tee /etc/systemd/system/office-backup.service > /dev/null &lt;&lt;'EOF'
[Unit]
Description=Nextcloud AIO backup (homework03 -> desslok via NFS)
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=root
ExecStart=/usr/bin/ssh -o BatchMode=yes -o ConnectTimeout=30 \
    homework03 sudo /usr/local/bin/office-backup.sh
EOF

sudo -n tee /etc/systemd/system/office-backup.timer > /dev/null &lt;&lt;'EOF'
[Unit]
Description=Daily Nextcloud AIO backup timer

[Timer]
OnCalendar=*-*-* 03:30:00
RandomizedDelaySec=900
Persistent=true

[Install]
WantedBy=timers.target
EOF

sudo -n systemctl daemon-reload
sudo -n systemctl enable --now office-backup.timer
systemctl list-timers office-backup*
# Expect: NEXT 8h14min ... office-backup.timer office-backup.service</code></pre>

  <h3>5.3 Test run + verify</h3>
  <pre><code>ssh tigo@hector
sudo -n systemctl start office-backup.service
# Wait 10s, then check status
systemctl status office-backup.service | head -5
# Expect: Active: inactive (dead), Result: success

# Verify the files made it to desslok
ssh tigo@desslok ls -la /slab/container_storage/office/backups/
# Expect: office-20260810-pgdump.sql.gz (a few hundred KB)
#         office-20260810-aio-config.tar.gz (a few KB)
#         office-20260810-ncdata.tar.gz (a few hundred B, empty until you upload files)

# Spot-check the pgdump
zcat /slab/container_storage/office/backups/office-20260810-pgdump.sql.gz | \
  grep -cE '^CREATE TABLE'
# Expect: 155 (Nextcloud has ~155 oc_* tables)</code></pre>

</div>
</body>
</html>