ops-1 (Openship VPS)
The Backpack services box, what runs on it, and how it was built.
The first Backpack-owned server: Hostinger VPS KVM 2 (2 vCPU, 8 GB, 100 GB NVMe), Ubuntu 24.04, hostname ops-1.backpackops.com, IP 177.7.46.8, hPanel VPS id 1840425. Purpose per ADR 08: Openship trial substrate, running the mail server and per-client service spin-ups. Client static sites stay on Vercel.
Access
- SSH: key auth (local key ~/.ssh/backpack_ops on the workstation). Root password auth still enabled; rotate the root password when convenient since the initial one transited chat.
- Openship dashboard: localhost-only on the box (auth mode: none). Reach it through an SSH tunnel:
ssh -L 3001:localhost:3001 -L 4000:localhost:4000 root@177.7.46.8, then http://localhost:3001. Do not expose 3001/4000 publicly until Openship auth is configured. - Firewall (ufw): 22, 80, 443, and mail ports 25/587/465/143/993 open. API port 4000 is deliberately not allowed.
Stack
- Openship installed via npm (Node 22). The curl installer’s bun path corrupted its own entry script; the npm tarball is fine. CLI authed with a personal access token stored at /root/.openship-cli-token.
- The box is registered in Openship as server “ops-1” targeting 127.0.0.1. The CLI’s key-path SSH auth is broken (Invalid auth configuration); password auth works and is what the server entry uses.
- Known packaging bug: the npm dist omits dist/server/lua/*.lua; the files were fetched from the GitHub repo (packages/adapters/src/infra/lua/) into place. Re-apply after
openship updateif OpenResty installs fail with ENOENT site_logger.lua. - Mail stack: iRedMail via
openship mail setup, domain backpackops.com, hostname mail.backpackops.com.
Openship gotchas learned the hard way (npm dist v1.x)
- The npm dist omits two asset sets:
dist/server/lua/*.lua(OpenResty step) and the iRedMail engine. The engine must exist at the ABSOLUTE path/apps/email/engineon the control-plane host (sparse-cloneapps/email/enginefrom the GitHub repo). - Step 5 changes the hostname to
mail.<domain>, which makes the PGlite database lock look foreign after a service restart (“locked by a process on a different host”). Fix:rm /root/.openship/data.lock, restart openship. - A crashed setup leaves a “Setup already running on another replica or recently crashed” 409 for several minutes. Clear with
openship mail reset -y <id>plusopenship mail forget <id>, then run a freshmail setup. - The bundled iRedMail phones home and exits 255 as “out of date”. Neutralize the check in
pkgs/get_all.sh(replace the FETCH_CMD new_version line withtrue) in /root/iRedMail-engine and the /apps/email/engine source before step 9. - CLI
server add --auth-method keyis broken (Invalid auth configuration); use password auth. - The CLI needs a login even against the local API:
openship token create cli, thenopenship login --token ....
Mail server status (live 2026-07-19)
Postfix, Dovecot, Amavis, ClamAV, SpamAssassin, iRedAPD, and fail2ban run on the box; ports 25/587/465/143/993 listen. Let’s Encrypt cert on mail.backpackops.com with a renewal deploy hook reloading postfix and dovecot. First outbound send scored 10/10 on mail-tester: SPF pass, DKIM pass, SpamAssassin clean, no blocklists. SSL steps 12 and 13 of the wizard were completed manually (certbot standalone with OpenResty paused, certs symlinked over /etc/ssl/certs/iRedMail.crt and key).
Twenty CRM (live 2026-07-19)
Twenty runs at https://crm.backpackops.com as a Docker Compose stack in /root/twenty (server, worker, Postgres, Redis; secrets in /root/twenty/.env, chmod 600). Docker was installed through Openship (openship server install <id> -c docker). OpenResty vhost at conf/sites-enabled/crm.backpackops.com.conf proxies 127.0.0.1 with websocket upgrade; Let’s Encrypt cert with a deploy hook reloading OpenResty. First workspace signup is reserved for the owner (david@backpackops.com). Per ADR 12, DealFlow is built as the agent layer on Twenty’s GraphQL API rather than from scratch.
Listmonk (live 2026-07-19)
Listmonk runs at https://campaigns.backpackops.com (Docker Compose in /root/listmonk: app + Postgres 17, app bound to 127.0.0.1, OpenResty vhost + LE cert). SMTP is wired to the box’s own mail server (mail.backpackops.com, STARTTLS, postmaster auth), so campaigns send DKIM-signed immediately. For the 1.5M client, add an SES relay as a second SMTP block and a per-client instance per ADR 11. Version note: the settings API requires msg_retry_delay and congestion_backoff duration fields in each SMTP block or the app crash-loops on boot. The app container needs extra_hosts mapping mail.backpackops.com to host-gateway or it resolves the mail server to itself. A responsive base template (“Backpack Base”) is installed and set as default; per-client branded templates go in Campaigns, Templates.
Known issue queued: the server hosts backpackops.com as a local mail domain, so outbound mail TO @backpackops.com addresses is rejected (User unknown) instead of routing to Google. Fix: re-home the mail identity to a subdomain. Bounce processing hangs on the same task (bounce mailbox needs a subdomain whose MX points at this server); the SES path for the large client uses Listmonk’s native SES webhook instead.
Openship dashboard exposure
Auth upgraded from zero-auth to local login (openship system upgrade-to-auth); dashboard now public at https://ops.backpackops.com (OpenResty routes / to and /api to , websockets on). The SSH tunnel is no longer required.
Credentials
Openship (ops.backpackops.com) and Listmonk (campaigns.backpackops.com) currently use the owner’s temporary shared password (same as the console) at the owner’s request; change both on first login. The Openship dashboard’s public origin required two fixes, both must survive updates:
- A systemd drop-in (origins.conf) with OPENSHIP_EXTRA_TRUSTED_ORIGINS=https://ops.backpackops.com, or the API rejects logins with ORIGIN_REJECTED.
- The dashboard tells browsers to call the API at http://127.0.0.1:4000 (the CLI hardcodes OPENSHIP_LOCAL_API_URL when spawning the dashboard; a service env cannot override it). Symptom: “Could not reach the server. Is the API running?” on login. Fix lives in the OpenResty vhost: a sub_filter rewrites “http://127.0.0.1:4000” to “https://ops.backpackops.com” in proxied HTML (with proxy_set_header Accept-Encoding “”), which corrects the injected window.OPENSHIP_API_ORIGIN without patching vendor code. The iRedMail postmaster credential stays machine-generated in /root/credentials/postmaster (it authenticates Listmonk’s SMTP). Move everything into Infisical when it lands.
DNS and mail identity
- A records: mail.backpackops.com and ops-1.backpackops.com point at the box (Spaceship-managed zone).
- PTR: 177.7.46.8 resolves to mail.backpackops.com, set via the Hostinger API (POST /api/vps/v1/virtual-machines/1840425/ptr/1596701).
- Published: A/AAAA for mail, apex SPF merged to authorize BOTH Google and this server (v=spf1 include:_spf.google.com ip4.7.46.8 ip6:… ~all), DKIM selector “dkim”. The wizard’s prescribed MX change and DMARC p=quarantine were deliberately NOT applied.
- The backpackops.com apex MX stays on Google Workspace (that is the owner’s mailbox). The mail server is for fleet transactional sending; client sending domains get added per client. Never change the apex MX.
Openship 0.2.3 upgrade and the outbound relay (2026-07-22)
Openship on ops-1 is now 0.2.3, which splits the mail server into two responsibilities: the box hosts the receive side (Dovecot IMAP, your inboxes stay on your own hardware) while outbound can be relayed through Amazon SES or any SMTP provider for deliverability.
Upgrade notes:
- Pre-upgrade backup lives at
/root/backups/openship-pre-0.2.3-20260722.tar.gz(full.openshipstate) plusmail-state-pre-upgrade.json. Stop the service and cleardata.lockbefore any upgrade. - The mail registration survived the upgrade;
openship mail serversstill lists ops-1 with backpackops.com completed. If it ever drops,openship mail scan <serverId>thenopenship mail adopt <serverId>recovers it. - The 0.1.x packaging gaps are fixed in 0.2.3: lua files ship in dist, and the engine now stages to
/root/iRedMail-engineinstead of the absolute/apps/emailpath. - All daemons verified healthy post-upgrade: Postfix, Dovecot, Amavis, ClamAV, iRedAPD, fail2ban. Listmonk, Twenty, and the Hermes gateway were untouched.
Enabling the relay (once SES credentials exist): POST /api/mail/admin/<serverId>/relay (via openship api) with provider, host, port (587 or 465), username, password, and scope all or selected plus a domains list. It writes the Postfix sasl_passwd map and relayhost, so all non-local outbound flows through the provider while inboxes stay local. GET the same route to inspect, DELETE to disable. Currently unconfigured: waiting on an AWS SES account (SMTP credentials in a verified region).
Known issue that the relay does not fix: outbound to @backpackops.com addresses still fails because Postfix classifies the apex as a local domain and never relays it. That re-homing is tracked separately.
SES relay status (2026-07-22)
AWS account “backpack” (135330967912), region us-east-2, IAM user ses-backpack, credentials at /root/.aws/credentials on ops-1. Domain identity backpackops.com created; production access request filed via PutAccountDetails. /root/ses-relay-enable.sh derives the SMTP password from the stored secret and posts the Openship relay config; /root/ses-watch.sh (root crontab, every 30 min, log at /root/ses-watch.log) polls until DKIM verifies and production access is granted, then enables the relay, notifies David through Skippy, and removes itself. DKIM CNAMEs must be added manually in Spaceship DNS (no API access).
Update, same day: Spaceship exposes a DNS API and we now have credentials (root-only at /root/.spaceship.env on ops-1; headers X-API-Key / X-API-Secret against https://spaceship.dev/api/v1/dns/records/<domain>, PUT with {"force":true,"items":[...]} to upsert). The three SES DKIM CNAMEs were published through it and the identity is DKIM-verified. Remaining: AWS production access review; the ses-watch cron auto-enables the relay when it lands. backpackops.com DNS changes no longer need manual console work.
Relay live (2026-07-29) and the amavis TLS gotcha
AWS granted production access (50,000/day, 14/sec, us-east-2); the ses-watch cron auto-enabled the relay and removed itself as designed. One bug surfaced: Openship’s relay config sets smtp_tls_security_level=encrypt globally, which also hits Postfix’s internal re-injection hop into amavis on localhost — no TLS there, so ALL outbound deferred with “TLS is required, but was not offered by host 127.0.0.1”. Fix: add -o smtp_tls_security_level=none to the smtp-amavis service in /etc/postfix/master.cf (backup at master.cf.bak-ses), postfix reload, postqueue -f. Verified end to end after the fix: send → amavis CLEAN + DKIM-signed → SES 250 Ok (mailbox simulator). Listmonk needs no change; it rides the same Postfix.
Openship 0.6.1 upgrade (2026-08-07)
0.2.3 to 0.6.1 (the stability release) using the standard recipe: stop service, clear data.lock, tar backup (/root/backups/openship-pre-upgrade-20260807.tar.gz), npm install, restart. Everything survived without intervention this time: mail registration, all mail daemons, the SES relayhost with SASL, the smtp-amavis smtp_tls_security_level=none exception in master.cf, the relay state in Openship, dashboard, and all docker services. Verified end to end post-upgrade with a mailbox-simulator send (SES 250 Ok). New pre-upgrade habit worth keeping: snapshot postconf relayhost smtp_sasl_auth_enable smtp_tls_security_level and the master.cf amavis block to /root/backups/postfix-relay-pre.txt for a post-upgrade diff.