ADR 15: Shared client infrastructure
Share the building blocks, sign-in and domain tooling across client apps. Keep a database per client.
Status: proposed (2026-10-23). Plan only: nothing in the migration order is built yet. Each chunk ships on David’s go, one at a time. Reviewed at tier 3 (Sol 6.1 high); the findings and what changed are at the end.
Context
Backpack keeps building small apps for clients on backpackops.com subdomains. The Seattle King County NAACP forms app (Backpack-Ops/naacp-skc-intake, live 2026-10-22 at naacpskc.backpackops.com) is the first one that has every piece a client app needs, reviewed: WorkOS sign-in with an organization check, a plain email shell sent through Resend with idempotency keys, public-form guards under Postgres advisory locks, a sign-in lockout, and CSV export that cannot run formulas. Rain City, the needlepoint portal, the QuickBooks admin and the sites console each rebuilt some of this by hand, with a shared password in place of real accounts.
The question was whether to consolidate everything under one shared client app. The answer here is no: share the code, the sign-in provider and the domain tooling, and keep each client’s data and access separate (ADR 11).
Inventory (2026-10-23)
Every app on a backpackops.com name. All Vercel projects are on the withunify team (Unify, Pro) and deploy from GitHub on main. DNS for backpackops.com is at Spaceship (launch1/launch2.spaceship.net); mail is Google (MX smtp.google.com).
| Host | Repo | Vercel project | Data | Sign-in | Email sender | DNS |
|---|---|---|---|---|---|---|
| backpackops.com, www | Backpack-Ops/backpack-support-site | backpack-support-site | none (static Webflow export) | none | none | apex A 76.76.21.21, www CNAME cname.vercel-dns.com |
| concepts | Backpack-Ops/backpack-concepts | backpack-concepts | none (path gateway to each client’s concept project) | none | none | CNAME cname.vercel-dns.com |
| docs | Backpack-Ops/backpack-docs | backpack-docs | none (Blume static) | none | none | CNAME cname.vercel-dns.com |
| qb | Backpack-Ops/backpack-qb-admin | backpack-qb-admin | Vercel Blob, QuickBooks OAuth tokens encrypted | shared operator password, HMAC cookie | none | CNAME cname.vercel-dns.com |
| campaign-studio, studio | Backpack-Ops/campaign-studio | campaign-studio | Postgres (Neon via the Vercel integration), Prisma | WorkOS AuthKit app “Campaign Studio” (the environment default, deployed with an API key), staff org Backpack, workspace per org; password fallback | SES per client sending domain, through each client’s Listmonk engine (ADR 14) | CNAME 8fbc6e90adc07744.vercel-dns-016.com |
| iese | Backpack-Ops/iese-marketing | iese-marketing | none (static page) | none | none | CNAME cname.vercel-dns.com |
| javi | mrcoven94/javi-mech-forge | javi-mech-forge | none (Vite game) | none | none | CNAME cname.vercel-dns.com |
| naacpskc | Backpack-Ops/naacp-skc-intake | naacp-skc-intake | own Neon project naacp-skc-intake-db | WorkOS AuthKit app “Seattle King County NAACP”, keyless; org “Seattle King County NAACP” (externalId naacp-skc); access code with lockout when WorkOS is off | Resend, intake@notifications.backpackops.com | CNAME 858f78a031069247.vercel-dns-016.com plus _vercel TXT, written through the Spaceship API from ops-1. intake.naacpskc.org attached, waits on the branch’s CNAME at Squarespace |
| needlepoint | Backpack-Ops/needlepoint | needlepoint | Convex deployment | portal password, HMAC cookie; server calls Convex with a backend secret | Resend from Convex (REPORT_FROM_EMAIL) | CNAME f7f581267383f15e.vercel-dns-016.com |
| stitch | Backpack-Ops/needlepoint-site | needlepoint-site | none yet (signup to a Listmonk list, pending) | none | SES (MAIL_FROM), not yet wired | CNAME cname.vercel-dns.com |
| raincitymarket | Backpack-Ops/rain-city-needlepoint | rain-city-market | Vercel Blob (private), Google Sheet, Stripe, Shopify | staff passcode and scoped viewer tokens, HMAC cookies | Resend, reports@notifications.backpackops.com, with idempotency keys | CNAME cname.vercel-dns.com |
| sites | Backpack-Ops/site-editor | site-editor | GitHub repos (site repos, form-submissions data repo), Umami | editor password, cookie | none | CNAME cname.vercel-dns.com |
| magic | mrcoven94/tooth-fairy-live | tooth-fairy-live | Upstash Redis | none | none | CNAME 5134d0f3885e574c.vercel-dns-016.com |
Also on the zone, outside this list: fin and caps (Vercel), go (CNAME cname.dub.co), mail and ops-1 (A records to ops-1), campaigns, ops and crm (ops-1 services).
What the DNS shows:
- There is no wildcard record. Every name is an explicit CNAME.
- Most names point at the shared
cname.vercel-dns.com; campaign-studio, studio, naacpskc, needlepoint, magic and caps point at a project-specific*.vercel-dns-016.comtarget, which is what Vercel hands out for new domains now. - All 20 Vercel ownership values sit as separate TXT values on one name,
_vercel.backpackops.com. Vercel has asked for a TXT on every subdomain so far, which means it treats the apex as held by another Vercel scope. Removing a domain from a project and adding it again issues a new value, so stale values build up.
What the WorkOS account shows: one team (Backpack) with two projects. The Backpack project’s Production environment holds three AuthKit applications: Campaign Studio (the default application), Seattle King County NAACP (keyless) and Front Door. Campaign Studio and Front Door both run deployed with API keys. Organizations: Backpack (domain backpackops.com verified), Seattle King County NAACP and Fleet Canary. Civic Pulse has its own WorkOS project.
What WorkOS documents about applications in one environment: users and organizations are shared across all applications; hosted sign-in branding applies to the whole environment, not one application; each application has its own API keys; invitations sent from the dashboard always belong to the default application, while invitations sent through the API keep the context of the application whose key sent them.
Decision
1. A shared package: @backpack-ops/client-kit
A new public repo, Backpack-Ops/client-kit, holds the code client apps keep rebuilding. It is extracted from naacp-skc-intake as it stands (lib/auth.ts, lib/auth-config.ts, proxy.ts, lib/guard.ts, lib/db.ts, lib/email.ts, app/responses/export/csv.ts), with its tests, not rewritten.
| Module | What it does | Source in naacp-skc-intake |
|---|---|---|
auth |
getViewer({ orgId }) returning signed-out, no-access or ok, a sign-in route that scopes hosted sign-in to the client’s organization, an AuthKit proxy factory limited to the private paths, a config check that fails the build, and the access-code mode with a signed, expiring cookie |
lib/auth.ts, lib/auth-config.ts, proxy.ts, app/signin/route.ts |
email |
the plain table email shell (brand name, preheader, body, footer), esc(), send() to Resend with a 10 second timeout, an Idempotency-Key per message, and a failure log that never includes the message body |
lib/email.ts |
guards |
keyed source hash of the IP; per-source rate limit (the advisory lock in its own statement, then the count-and-insert statement, in one Read Committed transaction, so the insert’s snapshot is taken after the lock); client-id dedupe (UUID check plus a unique index); confirmation cap per address per day; sign-in lockout per connection and overall | lib/guard.ts, lib/db.ts, app/responses/actions.ts |
notify |
a sent flag stored with each submission, unsent alerts listed on the private page with a resend button, and a bounded retry inside Resend’s 24 hour idempotency window | team_email_sent in lib/db.ts, resend actions in app/responses/actions.ts |
csv |
csvCell() that prefixes = + - @ tab and carriage return with a quote, toCsv(), and a response helper with no-store |
app/responses/export/csv.ts |
Rules for the kit:
- It never holds data or opens a connection. Database helpers take the app’s own
sqlfunction, and the kit’s tables (kit_rate_events,kit_login_attempts,kit_confirmations) are created byensureKitSchema(sql)inside each client’s own database. - The sign-in mode is chosen, never inferred.
AUTH_MODEisworkosorcode. Inworkosmode a missing or short WorkOS setting refuses everyone and fails the build check; it never falls back to the access code. Once an app is on WorkOS, its access code and code cookies are deleted, not kept as a spare. With no mode set, the private pages refuse everyone. (This is the one deliberate change from naacp-skc-intake, which falls back to the code when WorkOS is incomplete.) - Pages check access themselves. The proxy only refreshes sessions; every private page, route and server action calls
getViewer(). - No client specifics. Field lists, email copy, branding and the data model stay in each app.
- Distribution: a pinned git tag (
github:Backpack-Ops/client-kit#v0.1.0), TypeScript source, consumed withtranspilePackages. The repo is public so Vercel builds need no extra token; it contains no secrets and no client names. Move to a registry only if the number of consumers makes tag bumps painful. - Tests: unit tests for every module, plus a concurrency test against a real Postgres where request B is made to wait on the lock while request A commits, proving B sees A’s row and the limit holds. Changes to
authorguardsare tier 2 reviews.
2. Sign-in: a client-only WorkOS project, one AuthKit application per product, one organization per client
A new WorkOS project, “Backpack Clients”, holds only client apps and client organizations. API keys belong to an application, but users and organizations are shared across the whole environment, so this plan treats any key in an environment as able to change any of its organizations’ members. Campaign Studio and Front Door are both deployed with keys in the Backpack project. A breach of either would let someone add themselves to a client’s organization and sign in to that client’s keyless app; separate databases would not stop it. In the new project no deployed app holds a key.
| One AuthKit app per product (chosen) | One shared “Backpack Clients” app | |
|---|---|---|
| Adding people | The welcome email links to that app’s own sign-in | Every link opens the one shared app, which then has to route people to their client’s app |
| Redirect URIs | Each app lists only its own hosts | One growing list; a mistake in it affects every client |
| Blast radius | An app can be paused or deleted alone | Every change touches every client |
| Keyless public client | Each deployed app runs with a client ID and cookie password only | Same, but one client ID across all apps |
| Branding | Environment-wide either way: the hosted sign-in page is neutral Backpack; each app’s own page before the hand-off carries the client’s brand | Same |
| Setup cost | One app per product, about five minutes, scriptable | None after the first |
Decided:
- Each client-facing product gets its own AuthKit application in the Backpack Clients Production environment, named for what the client sees (“Seattle King County NAACP”). Hosted sign-in uses neutral Backpack branding, because branding is per environment; a client that needs its own hosted page gets its own environment.
- Each client is one WorkOS organization, with
externalIdset to the registry client slug. Membership in that organization is the only grant of access. Roles come later, only when a client needs them. - Deployed client apps stay keyless. No
WORKOS_API_KEYin any client app’s Vercel environment, and the kit’s build check fails when one is set (AuthKit uses a key whenever it finds one). The one management key for the Backpack Clients environment lives in an operations-only Infisical folder (/ops/workos-clients), never in an app’s/apps/<project>/folder, so running or syncing an app’s secrets can never carry it. - People are added by exact address, not by invitation link.
ws client member add <client> <email>creates the user for that exact address and the membership, then sends a welcome email from the app (kitemail) that links to its sign-in, where they prove the address with an emailed code. WorkOS invitations are not used for client apps: an invitation can be accepted by another address on the same company domain, which would grant access to someone the client never named, and dashboard invitations open the default app. - Backpack staff get in by membership, not by bypass. David or a teammate who needs a client’s records is added to that client’s organization, visibly, and removed when done. Apps that hold client records never have a “staff see everything” path.
- Campaign Studio and Front Door stay in the Backpack project. Campaign Studio’s per-client workspaces are its own product. Front Door moving to its own project is a follow-up, no longer a client-safety issue once client apps have left.
3. Domains: scripted explicit subdomains, no wildcard
Chosen: ws domain add <name> <vercel-project>, one explicit name per app, the way every name was made so far, but scripted and safe to rerun.
- Record the intended binding (name, project, CNAME target, ownership value) in the run’s state file before changing anything. A rerun reads it and resumes through every step, registry included; a name that exists (on the zone or in the registry) and matches the recorded binding is carried on, a name that exists and does not match is refused. Infrastructure names are always refused (mail, ops-1, ops, crm, campaigns, notifications, go, fin, caps).
- Add the domain to the Vercel project through the API. Read back the project-specific CNAME target and, when Vercel asks for one, the ownership value for
_vercel.backpackops.com. - On ops-1 (
ssh -i ~/.ssh/backpack_ops root@177.7.46.8), a fixed script installed there (/root/bin/spaceship-dns) does the write, so the Spaceship credentials in/root/.spaceship.envnever leave the box:- snapshot the whole zone to
/root/backups/dns/backpackops.com-<timestamp>.jsonbefore any write; - write only the new records, additively (the CNAME and the one new
_vercelvalue), never a full replacement of a record set; - read the zone back and check that the new records are present and every record in the snapshot is still there, mail and the other
_vercelvalues included; a value that is already present counts as done, so a retry after a timed-out save passes; - send a custom User-Agent (Spaceship returns 403 to default client strings).
- snapshot the whole zone to
- Ask Vercel to verify, check the name at 1.1.1.1, and fetch the HTTPS page.
- Update projects.json and run
ws index.
ws domain remove deletes only the records that its add recorded: first the CNAME, then, after the record’s TTL has passed, the domain’s Vercel attachment and its own _vercel value. No name is left pointing at Vercel with no project behind it, and no other app’s values are touched.
Not chosen: a wildcard *.backpackops.com on a gateway project. What Vercel requires for a wildcard:
- Vercel must answer DNS challenges to issue and renew the wildcard certificate. That means either moving backpackops.com to Vercel nameservers (the zone also carries Google mail, SES and Resend DKIM, the ops-1 records and the dub.co link), or delegating
_acme-challengeto ns1/ns2.vercel-dns.com with NS records at Spaceship, enabling Vercel DNS on the domain, and adding a*CNAME. - Because Vercel treats the apex as held by another scope today, that ownership has to be settled first.
- A wildcard routes every unlisted name to one project. Reaching a client’s own project would then need a gateway that rewrites by host, which adds a hop, makes every app reason about forwarded hosts for cookies and WorkOS redirect URIs, and needs the client projects reachable without Vercel deployment protection. Any mistyped or retired name would also answer with a page.
At about one new client app a month, explicit names cost one script run each and keep the registry exact. If Backpack later runs one multi-tenant app with many client names, use a scoped wildcard such as *.app.backpackops.com with only _acme-challenge.app delegated, never the apex. Exact names keep working beside it, because an exact DNS record and an exact Vercel domain both win over a wildcard.
A client’s own domain (such as intake.naacpskc.org) is attached to the same project; the script prints the one CNAME the client’s DNS host needs. Backpack does not ask for client registrar logins.
4. Data: one database per client
- One Neon project per client app, named
<client>-<app>, in US West (Oregon), as naacp-skc-intake-db is. Two apps for one client may share a database only when they share one data model. A database never holds two clients. ws client db <client> <app>, in this order, each step recorded so a rerun resumes with the same project instead of making another: create the Neon project; runensureKitSchemaand the app’s migrations; connect and check the schema; store the connection strings in Infisical/apps/<project>/prd; only then write them into the Vercel project’s Production environment and redeploy. It never prints a connection string, and it refuses to replace a Production database setting that already exists; moving an app to a new database is a separate, deliberate step with a rollback to the old value.- Previews never see production data. Preview deployments get no database, or a Neon branch made schema-only (or from a separate synthetic parent) with its own credentials and synthetic fixtures. An ordinary Neon branch copies its parent’s rows, so the Vercel and Neon integration’s automatic preview branching stays off for client databases.
- Backups: point-in-time restore set to match the client’s sensitivity.
- Offboarding, in order: close the forms and stop writes; let pending work finish; take the final export and hand it over; wait for the client to confirm they have it; revoke memberships and delete the organization and AuthKit app; delete the app’s keys and secrets in Infisical and Vercel; remove the domain as in section 3; then delete the Neon project and any other store the app used (Blob, Convex).
Never shared across clients:
- a database, Neon project, schema, connection string, Vercel Blob store or Convex deployment;
- a WorkOS organization, or a code path that lets one client’s members, or a blanket staff role, read another client’s records;
- a WorkOS environment with any deployed app that holds an API key, or an app secrets folder that holds a WorkOS management key;
- per-app secrets:
SESSION_SECRET,WORKOS_COOKIE_PASSWORD, access codes,CRON_SECRET, database URLs, and Resend keys. Each client app gets its own sending-only Resend key restricted to its sending domain; the full-access key stays in Infisical for tooling; - client records in shared places: logs, error reports and analytics carry ids and counts, never form contents; review models and agents get synthetic fixtures only;
- by default for new apps with sensitive intake (discrimination complaints, health, money), the narrative itself in email: alert emails carry a reference, the category and a link to the signed-in page, because the shared Resend account keeps a copy of every message it sends. The NAACP team alert includes the person’s name and the full account today, because the branch works from email. Whether to trim it is the branch’s decision, raised with them in chunk 1, not changed quietly.
Shared on purpose: the kit’s code, the Backpack Clients WorkOS environment (one organization per client, no deployed keys), the Vercel team, the Resend account and the notifications.backpackops.com sender (sending-only keys stop one app reading another’s messages, but apps on one sending domain can still send as each other’s addresses; a client that needs that separation, or sends volume, gets its own sending domain, as in ADR 14), the Spaceship zone, and Infisical (one folder per app).
5. Migration order
One chunk at a time. Each ships to production with a live link before the next starts.
| # | Chunk | Live link when done |
|---|---|---|
| 0 | This ADR | docs.backpackops.com/decisions/shared-client-infrastructure |
| 1 | client-kit v0.1 extracted from naacp-skc-intake, and naacp-skc-intake switched to it. Behavior stays the same except the chosen sign-in mode; the NAACP Resend key becomes a sending-only key (tier 2 review: auth and personal data) | naacpskc.backpackops.com: complaint form, call form and /responses sign-in checked in production; the kit’s v0.1.0 tag |
| 2 | The Backpack Clients WorkOS project and ws client member add. Users and organizations do not carry across projects, so the NAACP move runs in order: build the new app, organization and members in the new project and check the member list against the old one; deploy all new settings at once (client ID, cookie password, organization ID, redirect URI); prove a member can open /responses and an outsider cannot; only then delete the old app, organization and memberships. The old settings stay saved in Infisical until that proof, so a redeploy with them is the rollback |
a real welcome email to david@backpackops.com that signs in to naacpskc.backpackops.com/responses, plus an outsider account refused |
| 3 | ws domain add/remove and the ops-1 DNS script, proven on a kit example app with synthetic data, including a forced retry and a remove |
kit-example.backpackops.com, plus the zone snapshots it wrote |
| 4 | ws client db, proven by provisioning the example app’s database, including a rerun that resumes |
kit-example.backpackops.com saving and listing a synthetic submission |
| 5 | Rain City Market staff dashboard moves from the shared passcode to a Rain City organization and AuthKit app; kit email replaces its send helper | raincitymarket.backpackops.com staff sign-in |
| 6 | Needlepoint portal moves from the portal password to the same Rain City organization. Convex is reachable from the internet, so first confirm every private Convex query, mutation and action checks the backend secret or a WorkOS token with the Rain City organization; test a signed-out caller, another organization’s token and the retired password | needlepoint.backpackops.com sign-in |
| 7 | Optional: QuickBooks admin and sites console move from shared passwords to the Backpack organization | qb.backpackops.com and sites.backpackops.com sign-in |
The next new client app starts on the kit, the domain script and the database script from day one, whatever chunk is in progress.
Not migrated: the static sites (apex, concepts, docs, iese, stitch, javi, magic) have no sign-in or data to share; they only use the domain script for future names. Campaign Studio keeps its own sign-in and workspace model.
Consequences
- A fix to a guard or the sign-in gate lands once in the kit and reaches each app with a tag bump, instead of being re-found per app.
- Clients sign in with their own accounts. Shared passwords go away app by app. The hosted sign-in page is neutral Backpack, not the client’s brand.
- A breach of one deployed client app exposes that client only: no WorkOS key, no full Resend key, no other client’s database URL, no shared secret.
- DNS changes become a reviewed, rerunnable script with a snapshot before every write, not hand edits.
- The kit becomes a dependency to maintain. Version tags and the example app keep it honest.
Follow-ups outside this plan
- Backpack’s Vercel projects live on the Unify team (withunify). Decide whether they move to a Backpack team before Scholar Fund winds down; it changes domain ownership checks.
- Front Door, a separate product with public sign-up, shares the Backpack WorkOS environment with Campaign Studio. Move it to its own WorkOS project, as Civic Pulse is.
- Move the Spaceship API credentials into Infisical
/opsso the domain script can run without a shell on ops-1.
Review (tier 3, 2026-10-23)
Sol 6.1 (high) critiqued the plan through the Codex CLI. The OpenCode Go panel was unavailable (weekly usage cap). Each finding, and what happened:
- Deployed WorkOS keys in the shared environment defeat client isolation (blocker). Accepted. Client apps move to a new Backpack Clients WorkOS project with no deployed keys (section 2, chunk 2).
- An incomplete WorkOS setup silently falls back to the access code (blocker). Accepted. The sign-in mode is now chosen, fails closed, and the code is deleted after the move (section 1).
- No boundary on the shared Resend account. Accepted. Sending-only key per app; full-access key in tooling only (section 4).
- The rate limit’s advisory lock and count in one statement would miss a concurrent insert. The reference code already takes the lock in its own statement, so the second statement’s snapshot follows the lock; the ADR’s wording was wrong. Wording fixed, and the concurrency test now forces the wait (section 1).
- Needlepoint’s move must protect Convex directly. Accepted (chunk 6).
- Branding is per environment, and API invitations keep their application. Confirmed in the WorkOS docs. Branding claims removed; invitations now go through the API with the app’s own key (section 2).
- A seeded Neon branch still holds production rows. Accepted. Schema-only branches or a synthetic parent; automatic preview branching off (section 4).
- Database provisioning could publish a half-made database or replace one on rerun. Accepted. Ordered, resumable, refuses to replace (section 4).
- The domain script cannot resume after a partial success, and the TXT write should be additive. Accepted. Recorded binding, additive writes, read-back of the whole snapshot (section 3).
- Offboarding could delete submissions that arrived after the export. Accepted. Writes stop first; deletion waits for the client’s confirmation; removal waits out the DNS TTL (sections 3 and 4).
- A failed alert email can go unnoticed. Accepted. The kit’s
notifymodule carries the reference app’s sent flag and resend button, plus a bounded retry (section 1).
A second Sol pass on the revision found no blockers. It judged findings 2 to 8, 10 and 11 closed, and 1 and 9 closed only with the extra changes below. It raised four more, all accepted:
- A WorkOS invitation can be accepted by another address on the same company domain. Client apps no longer use invitations; people are added by exact address with a welcome email (section 2).
- A management key in an app’s secrets folder could be deployed by mistake, and AuthKit would use it. The key lives in an operations-only folder, and the kit’s build check rejects any
WORKOS_API_KEY(section 2). - Moving NAACP between WorkOS projects had no cutover steps or real acceptance check. Chunk 2 now orders the move, proves member and outsider access, and keeps the old settings as the rollback.
- A domain run that stopped after the registry write could not resume. A matching recorded binding now resumes through the registry step (section 3).