Procedure

2.3 Create local bind targets

ssh homework03
# AIO install root
sudo -n mkdir -p /usr/local/containers/nextcloudaio

# Container bind targets (named volumes)
for sub in mastercontainer database database-dump redis apache nextcloud \
           collabora whiteboard notify-push imaginary talk fulltextsearch clamav; do
  sudo -n mkdir -p "/usr/local/containers/nextcloudaio/nextcloud-aio-$sub"
done

# /srv/nc-files is the NFS mount. AIO bind-mounts it directly into
# the nextcloud container at /mnt/ncdata via NEXTCLOUD_DATADIR.
# No intermediate /mnt/nc-data layer — see "Rewire 2026-08-11" note.

# Local backup stash (so the script can write the pgdump into NFS without recursion)
ls -la /usr/local/containers/nextcloudaio/

2.4 Pre-chown the bind targets

AIO's entrypoint scripts chown their bind target to the runtime UID. Doing it once explicitly avoids a startup warning:

ssh homework03
sudo -n chown -R 33:33   /usr/local/containers/nextcloudaio/nextcloud-aio-mastercontainer
sudo -n chown -R 33:33   /usr/local/containers/nextcloudaio/nextcloud-aio-apache
sudo -n chown -R 999:999 /usr/local/containers/nextcloudaio/nextcloud-aio-database
sudo -n chown -R 999:999 /usr/local/containers/nextcloudaio/nextcloud-aio-database-dump
sudo -n chown -R 999:999 /usr/local/containers/nextcloudaio/nextcloud-aio-redis
sudo -n chown -R root:root /usr/local/containers/nextcloudaio/nextcloud-aio-nextcloud
sudo -n chown -R 100:101 /usr/local/containers/nextcloudaio/nextcloud-aio-collabora

2.5 Rewire 2026-08-11 — single-mount architecture

After the initial deploy we discovered a sharp edge: when NEXTCLOUD_DATADIR is a different host path than the actual NFS mount (the original design used /srv/nc-files for NFS and /mnt/nc-data/nextcloud-data for AIO), a typo on either side silently creates an empty datadir inside the nextcloud container. Nextcloud then refuses to start with "Appdata directory is not present!" — but the empty datadir is real, so the NFS data is still there, just not mounted. That's the failure mode that took the office suite offline on 2026-08-11.

The rewire removes the intermediate /mnt/nc-data/nextcloud-data bind entirely:

• NFS export desslok:/slab/container_storage/office mounts at /srv/nc-files on homework03 (unchanged).
• NEXTCLOUD_DATADIR=/srv/nc-files in compose AND configuration.json's nextcloud_datadir field both point at the same path.
• AIO's containers.json template substitutes %NEXTCLOUD_DATADIR% with that path and creates a bind mount directly: host /srv/nc-files → container /mnt/ncdata.
• The mastercontainer no longer has the /srv/nc-files or /mnt/nc-data/nextcloud-data binds in its compose volumes: section.

Recovery if it ever breaks again:

# 1. Confirm what's in the NFS export
ssh desslok ls -la /slab/container_storage/office
# Expect: admin/  race/  appdata_*/  ...

# 2. Confirm the mount is healthy on homework03
ssh homework03 df -h /srv/nc-files
ssh homework03 ls -la /srv/nc-files
# Expect: same admin/, race/, ... as step 1

# 3. Check what AIO thinks the datadir is
ssh homework03 sudo cat \
  /var/lib/docker/volumes/nextcloud_aio_mastercontainer/_data/configuration.json \
  | jq -r .nextcloud_datadir
# Expect: "/srv/nc-files" — if not, fix with jq (see ~/.hermes
# creds or the rewire script notes in this repo's history)

# 4. Inspect what the running nextcloud container actually has bound
ssh homework03 sudo docker inspect nextcloud-aio-nextcloud \
  | jq -r '.[0].Mounts[] | "\(.Source) -> \(.Destination)"'
# Expect: "/srv/nc-files -> /mnt/ncdata" AND
#         "/var/lib/docker/volumes/nextcloud_aio_nextcloud/_data -> /var/www/html"
# If /mnt/ncdata is bound to something else, the container has stale config.

# 5. If the bind source is wrong, force AIO to re-spawn nextcloud:
ssh homework03 sudo docker rm -f nextcloud-aio-nextcloud
# Then trigger /api/docker/start from the admin UI (Apache must
# be stopped first; the nextcloud container does NOT auto-spawn
# on mastercontainer restart). See "Phase 6: Spawn lifecycle" below.

Phase 3 — AIO mastercontainer + setup wizard

Write the compose file, start the mastercontainer, and walk the setup wizard via the admin UI.

3.1 docker-compose.yaml

ssh homework03
sudo -n tee /usr/local/containers/nextcloudaio/docker-compose.yaml > /dev/null <<'EOF'
services:
  nextcloud-aio-mastercontainer:
    image: nextcloud/all-in-one:latest
    restart: always
    container_name: nextcloud-aio-mastercontainer
    network_mode: host
    environment:
      APACHE_PORT: "11000"
      APACHE_DISABLE_REWRITE_IP: "1"
      NEXTCLOUD_DATADIR: "/srv/nc-files"
      NEXTCLOUD_UPLOAD_LIMIT: "10G"
      NEXTCLOUD_MAX_TIME: "3600"
      AIO_DISABLE_BACKUP: "true"
      SKIP_DOMAIN_VALIDATION: "true"
      COLLABORA_ENABLED: "yes"
      ONLYOFFICE_ENABLED: "no"
      IMAGINARY_ENABLED: "no"
      TALK_ENABLED: "no"
      WHITEBOARD_ENABLED: "yes"
      FULLTEXTSEARCH_ENABLED: "no"
      CLAMAV_ENABLED: "no"
      NEXTCLOUD_DOMAIN: "office.rmf44.xyz"
      NEXTCLOUD_TRUSTED_CACERTS_DIR: "/usr/local/share/ca-certificates"
    volumes:
      - ./nextcloud-aio-mastercontainer:/container-volume
      - /var/run/docker.sock:/var/run/docker.sock:ro
      # NOTE: do NOT bind /srv/nc-files into the mastercontainer.
      # AIO bind-mounts it directly into the nextcloud container via
      # NEXTCLOUD_DATADIR. (Pre-rewire this entry also bound
      # /mnt/nc-data/nextcloud-data — that intermediate layer was
      # removed 2026-08-11.)
EOF

3.2 Start mastercontainer + pull the AIO passphrase

ssh homework03
cd /usr/local/containers/nextcloudaio
sudo -n docker compose up -d
sleep 10
sudo -n docker logs nextcloud-aio-mastercontainer 2>&1 | grep -E 'passphrase|AIO'
# Get the 12-word passphrase from the logs
sudo -n docker logs nextcloud-aio-mastercontainer 2>&1 | grep -oE '[a-z]+(?: [a-z]+){11}' | head -1

3.3 Walk the setup wizard

Open the admin UI on homework03 LAN IP :8443 (self-signed cert is fine — accept the warning):

ssh homework03
hostname -I | awk '{print $1}'
# 10.0.0.73
# Open https://10.0.0.73:8443 in browser

1. Paste the 12-word passphrase.
2. Enter office.rmf44.xyz as the desired Nextcloud domain.
3. Click "Start AIO setup" — this triggers mastercontainer to pull the other 7 containers and run the installation.
4. Wait ~10 minutes. The container list grows one by one. Status column cycles through "starting" → "running" → "healthy".
5. When all 8 are healthy, the admin UI shows "Open Nextcloud" — click it. Nextcloud loads at https://office.rmf44.xyz:11000 (LAN-side, before DNS cutover).
6. Log in as admin with the auto-generated password printed in the admin UI's "Nextcloud admin user" panel — save this. The user must change it on first login.
7. Verify Collabora: Files → + → New Document → Word Document. Document opens in the richdocuments iframe (no separate login prompt = working WOPI).
8. Verify Whiteboard: + → New Whiteboard. Whiteboard canvas loads.

3.4 Verify the configuration persisted

ssh homework03
sudo -n cat /usr/local/containers/nextcloudaio/nextcloud-aio-mastercontainer/configuration.json
# Expect: "officeSuite": "collabora", "isWhiteboardEnabled": true,
# "domain": "office.rmf44.xyz", "nextcloud_datadir": "/srv/nc-files"

Phase 4 — Public ingress + cutover

Make office.rmf44.xyz reachable via the public Caddy on hawker.

4.1 Confirm Cloudflare DNS

# Confirm office.rmf44.xyz A record points at hawker
dig office.rmf44.xyz +short
# 192.255.159.202

# If missing, add it via Cloudflare dashboard or:
curl -X POST https://api.cloudflare.com/.../zones/$ZONE/dns_records \
  -H "Authorization: Bearer $CF_API_TOKEN" \
  -d '{"type":"A","name":"office","content":"192.255.159.202","proxied":false}'

4.2 Add Caddy block on hawker

The first attempt used :80 as the upstream — that was the bug. Apache listens on :11000:

ssh tigo@hawker
# Edit /etc/caddy/Caddyfile
sudo -n sed -i '/^office.rmf44.xyz {/{
  N
  s|office.rmf44.xyz {\n\treverse_proxy 100.79.142.164:80|office.rmf44.xyz {\n\treverse_proxy 100.79.142.164:11000|
}' /etc/caddy/Caddyfile

# Validate + reload
sudo -n docker exec caddy-caddy-1 caddy validate \
  --config /etc/caddy/Caddyfile --adapter caddyfile
sudo -n docker exec caddy-caddy-1 caddy reload \
  --config /etc/caddy/Caddyfile --adapter caddyfile

4.3 Verify the cutover

curl -skI https://office.rmf44.xyz/
  # HTTP/2 302
  # location: /login
  curl -sk https://office.rmf44.xyz/login | grep -oE '<title>[^<]+</title>'
  # <title>Login - AIO</title>

  curl -sk https://office.rmf44.xyz/status.php
  # {"installed":true,"version":"34.0.2.1","...","maintenance":false}

  # Test login
  # 1. GET /login → grab requesttoken + cookies
  # 2. POST /login with user=admin + password + requesttoken
  # 3. Expect HTTP 303 → /apps/dashboard/

  # Verify the nextcloud container can see NFS user files
  ssh homework03 sudo docker exec nextcloud-aio-nextcloud \
    ls -la /mnt/ncdata/race/files/ | head
  # Expect: Documents/  Photos/  Templates/  ...

4.4 Tear down the old OnlyOffice

ssh tigo@hawker
cd /home/tigo/onlyoffice-stack 2>/dev/null || cd /opt/onlyoffice-stack
docker compose down -v
# Removes containers and anonymous volumes

# Remove the Caddy vhost block (if it's separate)
sudo -n python3 -c "
p = '/etc/caddy/Caddyfile'
with open(p) as f: s = f.read()
s = s.replace('\n\n# onlyoffice\nonlyoffice.rmf44.xyz {\n\treverse_proxy 127.0.0.1:9980\n}\n', '')
with open(p, 'w') as f: f.write(s)
"
sudo -n docker exec caddy-caddy-1 caddy validate \
  --config /etc/caddy/Caddyfile --adapter caddyfile
sudo -n docker exec caddy-caddy-1 caddy reload \
  --config /etc/caddy/Caddyfile --adapter caddyfile

4.5 Disable the old hector backup pipeline

ssh tigo@hector
sudo -n systemctl disable --now office-backup.timer
sudo -n rm /etc/systemd/system/office-backup.{service,timer}
sudo -n rm /usr/local/bin/office-backup.sh
sudo -n systemctl daemon-reload

Phase 5 — Backup pipeline

A new daily backup runs on homework03, writing back to NFS. The hector timer triggers it over SSH.

5.1 office-backup.sh on homework03

ssh homework03
sudo -n tee /usr/local/bin/office-backup.sh > /dev/null <<'EOF'
#!/usr/bin/env bash
# office-backup.sh — daily backup of Nextcloud AIO
# runs on homework03, writes to /srv/nc-files/backups (NFS → desslok)
set -euo pipefail

BACKUP_DIR="/srv/nc-files/backups"
STAMP="$(date -u +%Y%m%d)"
NAME="office-${STAMP}"

mkdir -p "${BACKUP_DIR}"

# 1. Postgres dump from the database container
docker exec nextcloud-aio-database \
  pg_dumpall -U nextcloud --no-owner --clean --if-exists \
  | gzip > "${BACKUP_DIR}/${NAME}-pgdump.sql.gz"

# 2. Tar the AIO container state (mastercontainer config + database-dump)
tar -C /usr/local/containers/nextcloudaio \
  -czf "${BACKUP_DIR}/${NAME}-aio-config.tar.gz" \
  nextcloud-aio-mastercontainer nextcloud-aio-database-dump

# 3. Tar the local nextcloud app volume (AIO-managed app code +
#    config — survives a fresh AIO install if we ever need to
#    restore from a corrupt mastercontainer state).
tar -C /usr/local/containers/nextcloudaio \
  -czf "${BACKUP_DIR}/${NAME}-aio-nextcloud-app.tar.gz" \
  nextcloud-aio-nextcloud

# 4. Tar user files (NFS root: admin/, race/, appdata_*/, etc.)
#    Exclude backups/ to avoid recursion.
tar -C /srv/nc-files \
  --exclude='backups' \
  -czf "${BACKUP_DIR}/${NAME}-ncdata.tar.gz" \
  .

# 5. Prune anything older than 14 days
find "${BACKUP_DIR}" -maxdepth 1 -type f -name 'office-*' -mtime +14 -delete

echo "OK: wrote ${NAME}-{pgdump.sql.gz,aio-config.tar.gz,aio-nextcloud-app.tar.gz,ncdata.tar.gz} to ${BACKUP_DIR}"
EOF

sudo -n chmod 755 /usr/local/bin/office-backup.sh
sudo -n chown root:root /usr/local/bin/office-backup.sh

5.2 hector systemd unit (SSHes to homework03)

ssh tigo@hector
sudo -n tee /etc/systemd/system/office-backup.service > /dev/null <<'EOF'
[Unit]
Description=Nextcloud AIO backup (homework03 -> desslok via NFS)
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=root
ExecStart=/usr/bin/ssh -o BatchMode=yes -o ConnectTimeout=30 \
    homework03 sudo /usr/local/bin/office-backup.sh
EOF

sudo -n tee /etc/systemd/system/office-backup.timer > /dev/null <<'EOF'
[Unit]
Description=Daily Nextcloud AIO backup timer

[Timer]
OnCalendar=*-*-* 03:30:00
RandomizedDelaySec=900
Persistent=true

[Install]
WantedBy=timers.target
EOF

sudo -n systemctl daemon-reload
sudo -n systemctl enable --now office-backup.timer
systemctl list-timers office-backup*
# Expect: NEXT 8h14min ... office-backup.timer office-backup.service

5.3 Test run + verify

ssh tigo@hector
sudo -n systemctl start office-backup.service
# Wait 10s, then check status
systemctl status office-backup.service | head -5
# Expect: Active: inactive (dead), Result: success

# Verify the files made it to desslok
ssh tigo@desslok ls -la /slab/container_storage/office/backups/
# Expect: office-20260810-pgdump.sql.gz (a few hundred KB)
#         office-20260810-aio-config.tar.gz (a few KB)
#         office-20260810-ncdata.tar.gz (a few hundred B, empty until you upload files)

# Spot-check the pgdump
zcat /slab/container_storage/office/backups/office-20260810-pgdump.sql.gz | \
  grep -cE '^CREATE TABLE'
# Expect: 155 (Nextcloud has ~155 oc_* tables)