ADR-015: Host every Bulk PDF agent on ARRNC API

Status

In-Review. This record requires explicit architecture approval before it may become Approved or merge. It does not authorize implementation, an ARRNC apply, a Kubernetes change, or production promotion.

This record supersedes only the hosting and final-storage deployment parts of ADR-012, especially Decision 2 and the statements that place staging or preview PDF agents in Kubernetes on PVCs. ADR-012 remains authoritative for the Bulk PDF domain, multipart, processing, normalization, and application-level storage-authority contracts unless this record says otherwise.

Context

The merged 2026-08-11 design split one image across two hosting models: production as a Docker application on the ARRNC API host, but staging and PR previews as Kubernetes workloads using PVC-backed storage. A later explicit owner decision corrected that split: production, staging, and dynamic PR preview agents are separate Docker containers on ARRNC API, with preview creation and teardown tied to the PR lifecycle. That correction was acknowledged but not merged into the authoritative documents.

The stale documents were subsequently followed to recover staging. As of 2026-08-30, read-only inspection showed a healthy Argo CD-managed staging Deployment with PDF-agent, ClamAV, and static server containers, a bound 20 Gi PVC, Service, and ingress. Operational approval of that recovery did not explicitly reverse the later ARRNC decision.

Current repository evidence also establishes:

• server-config already declares an exact, gated syrf-pdf-agent-production slot, a dedicated syrf-pdf-net, ClamAV, and a production-child CIFS mount;
• the ARRNC config service reconciles an exact container-name manifest and has no bounded dynamic preview-family contract yet;
• cluster-gitops enables the staging PDF agent and leaves production disabled;
• SyRF already validates canonical application roots production, staging, and previews/pr-N;
• the preserved, uncommitted Juniper hosting rewrite contains the corrected environment partition and lifecycle intent, but is mixed with unrelated work and is evidence only.

The mismatch is therefore documentary and architectural, not permission to change a live environment during this planning phase.

Decision

1. One hosting model

No Bulk PDF agent runs in GKE or another Kubernetes cluster. Production, staging, and each enabled PR preview run as separately named Docker containers on ARRNC API. Ordinary application services may remain in Kubernetes; only the PDF processing agent and its environment-owned final-storage path move.

The persistent slots are:

Environment	Container	Storage child
Production	syrf-pdf-agent-production	production
Staging	syrf-pdf-agent-staging	staging
Preview PR N	syrf-pdf-agent-pr-N	previews/pr-N

Names and roots are derived from a canonical positive, non-zero-padded PR number. Callers cannot supply arbitrary container names, host paths, networks, commands, or repositories.

2. Exact child-scoped storage

The physical Camarades share is partitioned into production, staging, and previews/pr-N children. Each ordinary agent sees only its exact child at /app/data/output and a private scratch directory at /app/data/scratch. It never sees the share root or a sibling environment.

Each child has:

• a distinct least-privilege SMB identity and server-side ACL;
• a direct child mount on ARRNC API, owned by the agent's pinned non-root UID/GID;
• an operator-created sentinel whose content binds environment, canonical child, and mount identity;
• fail-closed checks for mount source, sentinel, write capability, free space, and path identity;
• no CIFS credential in container environment, image, GitHub Actions, Kubernetes, or PR code.

Direct child mounting and child ACLs are activation prerequisites, not assumptions. If the SMB export cannot mount the exact child or enforce distinct identities, work stops for a separately reviewed storage decision. Mounting the common root and relying only on container bind paths is not an approved fallback.

Production backup/retention is unchanged by the staging migration, but Gate C must obtain an external owner attestation for the remote share: ARRNC's local host backup explicitly excludes the mounted PDF tree and is not evidence that the share is backed up. Staging is non-production and not promoted, but a delivered file remains while any job, Study, report, or other durable application reference can generate or consume its URL. The 30-day policy applies only to proven unreferenced/orphan test data after coordinated application-state retirement; a blanket age-delete is forbidden. Preview data is not backed up; teardown removes only the exact preview child after fencing, and an orphan reconciler must converge closure cleanup within 24 hours. Server-enforced quotas and ARRNC free-space alarms are required before enabling staging or previews. Preview concurrency is configurable from measured ARRNC CPU, memory, PID, I/O, disk, storage-quota, route/port, RabbitMQ, and scanner capacity; it is never hard-coded to one. Before each activation, one fleet-wide admission transaction reserves every required resource or queues the request without creating partial runtime, storage, broker, network, or route state. Before a queued request can reserve resources, the registrar atomically revalidates repository identity, open/enabled PR state, current head SHA, verified digest, and current authorized-maintainer approval. A stale or ineligible request is discarded without provisioning.

Dynamic preview storage has an implementation dependency on an audited university-storage-owner operation or an equivalently isolated pre-provisioned allocation pool. It creates or allocates the exact previews/pr-N child, distinct principal, ACL, quota, and server-side connection/bandwidth/IOPS limits without exposing that authority to GitHub or the preview container. The registrar reserves the allocation atomically with the other fleet resources and verifies the resulting audit contract before activation. Missing automation is work to deliver, not evidence that approved previews require a different hosting model; broad credentials and unbounded shared-SMB I/O remain prohibited fallbacks.

ARRNC root mounts the exact child and bind-mounts only that child into its disposable rootless preview container. SMB credentials, sibling mounts, and the common root never enter the container. Provisioning and teardown test source identity, no-follow behavior, sibling denial, mount loss, ordered container stop/unmount, and exact allocation retirement.

3. Declarative persistent slots and bounded dynamic previews

server-config is authoritative for host users, mounts, directories, networks, ClamAV, secrets scope, persistent slots, verification, and the ARRNC config-service policy.

Production and staging are explicit persistent manifest entries. Preview entries are controlled by a dedicated bounded family registrar built into the declarative config-service model. It accepts a canonical PR number and an immutable SyRF PDF-agent image digest with verified repository, commit, and PR provenance, then derives every other field. It may create, pause, reconcile, and retire only syrf-pdf-agent-pr-N records.

The registrar must reject:

• arbitrary image repositories, tags without resolved digests, commands, paths, ports, networks, environment names, and secret scopes;
• any pull request whose head repository is not exactly the SyRF base repository, including every fork; only explicitly approved same-repository SyRF heads may use this path;
• a request whose declared per-slot profile or requested concurrency exceeds a configured hard limit; transient exhaustion within valid limits queues instead of rejecting the request;
• a request whose PR is not open and explicitly enabled;
• any request that could expose production or staging storage or broker credentials.

An approved same-repository SyRF PR is a trusted preview workload, not an arbitrary hostile-code source. Its build job may publish only an image digest and attestation; it receives no ARRNC, storage, broker, or registrar credential. A protected, base-branch-controlled activation workflow performs the privileged request after re-verifying the open PR, fleet capacity, and an explicit authorized-maintainer approval record bound to the exact current head SHA and verified digest. Every synchronize event invalidates the record. The binding is rechecked immediately before initial registration and again immediately before dequeue-time reservation and provisioning. Activation uses a short-lived, preview-family-only identity. No job that checks out or executes branch code may hold that identity, and PR changes cannot replace the activation workflow or its pinned dependencies.

Before a preview digest reaches the host, protected activation must verify the registry manifest's compressed total and a signed build/verifier attestation of unpacked size against reviewed ceilings, then reserve Docker data-root headroom for pull, unpack, and rollback margin. Teardown removes preview-only image references and layers once no live or persistent slot uses them; reconciliation detects leaked layers and low headroom. If ownership cannot be proven, cleanup quarantines and alerts rather than pruning shared images.

The imperative arrnc-api-deploy gatekeeper remains a transport/execution boundary where needed, but it cannot become a second source of desired state. Any changes there must preserve the config-service's authorization, provenance, audit, and exact-record invariants.

4. Runtime parity

Release staging and production use the same SyRF PDF-agent image by immutable digest. A preview uses the same Dockerfile and runtime contract at the PR's own verified immutable digest. Environment differences are declarative values only.

Every slot has equivalent:

• ClamAV 1.4 contract, scan limits, readiness, signature freshness, and no externally published clamd port;
• environment-specific RabbitMQ virtual host, queue, user, and least-privilege permissions;
• CPU, memory, PID, restart, scratch, and disk limits;
• non-root execution, dropped capabilities, no privileged mode, no Docker socket, no host network, and a read-only root filesystem where the image permits it;
• mount/sentinel, clamd, broker, queue-owner, and free-space health checks;
• stop, drain, replay/idempotency, journal recovery, and compatible previous-digest rollback procedures. The first staging slot rolls back to stopped/absent until a prior digest is independently proven to honor paused startup without subscribing.

Each preview receives its own RabbitMQ identity and broker-enforced connection/channel, message-size, publish-rate, queue-length/bytes, and storage limits, plus shared-broker free-space alarms. Admission reserves broker and scanner capacity across all active previews before creating a slot. If aggregate limits cannot preserve persistent environments, requests queue until capacity is available or use a separately reviewed dedicated broker.

Each approved preview gets a disposable rootless Docker container, exact storage allocation, network namespace/policy, scanner boundary, resource reservation, and audit record. It never joins production or staging networks and receives no production or staging mount, secret, broker account, static-serving authority, Docker socket, privileged mode, host namespace, device, or host-administration capability. Production and staging remain separate persistent host containers.

ARRNC root enforces per-preview and aggregate CPU, memory, PID, block-I/O, disk, storage-quota, port/route, network, broker, and scanner ceilings. Container code cannot raise them. Bounded write/discard/fsync and simultaneous-preview load tests prove every reservation and acceptable production/staging latency and health. Failure to establish an exact slot leaves that request queued or failed closed without affecting other previews.

A separate Docker bridge is necessary but not sufficient. Root-controlled host egress policy denies preview traffic by default, blocks the host gateway, metadata endpoints, ARRNC/UoE internal CIDRs and persistent-environment endpoints, and permits only the exact preview scanner plus the minimum controlled DNS, presigned-S3 HTTPS, and preview RabbitMQ destinations. PR code cannot alter this policy. Activation tests both the allowlist and explicit denials. The root-controlled per-preview policy enforces ingress and egress rate and burst ceilings, including allowed presigned-S3 traffic. Simultaneous bounded-load tests prove aggregate reservations protect the ARRNC uplink and production/staging health.

The trusted ClamAV updater is outside the PR agent's network identity. It alone receives a separately constrained route to a pinned internal signature mirror, publishes a verified read-only signature set to the preview scanner, and proves refresh/failure behavior without opening agent egress.

Preview output is generated by code under test and can be malformed even when the approved source is trusted. It therefore remains non-public; IIS and every other public/static-serving route deny the preview child. A typed host-side inspect operation accepts only the registered PR, persisted job ID, expected derived relative output path, and idempotency key from a short-lived preview-family harness identity. Root resolves no-follow beneath the exact child and sends bounded bytes to an isolated trusted type/size validator and scanner. The harness receives only a redacted size/hash/scan result, never raw bytes or storage credentials; the capability is audit-bound and revoked before teardown. Public serving is out of scope until a separately reviewed trusted post-write promotion boundary exists.

Preview Bulk processing accepts only synthetic, non-sensitive fixtures supplied by the protected base-branch harness through a dedicated preview object prefix and broker path. That harness uses a protected synthetic identity/API path to create the matching Project, initiate the upload session, upload and acknowledge every multipart fixture part, and complete the session so the durable job has the exact environment, size, hash, and policy expected by the claim consumer. Only then may notifier publication proceed. Because object completion can race application persistence, a trusted notifier release gate durably holds the exact object/version event until it can prove the matching job is committed as Uploaded; release is idempotent and recovery reconciles held events without dropping or publishing them early. Human/user upload initiation is disabled for preview environments by a control outside PR code. No clinical, participant, production, or otherwise sensitive PDF may enter a PR-controlled agent or its preview child. If that enforcement cannot be proven, the preview Bulk capability stays disabled.

5. Exactly one queue owner

Two agents must never consume from the same environment queue during coexistence. The ARRNC staging container is first reconciled with consumption paused. Cutover requires:

1. close only new-session admission while allowing already accepted multipart sessions to sign, acknowledge, heartbeat, complete, cancel, or abandon;
2. prove active upload sessions quiescent, pause new notifier publication with a durable S3-object/outbox inventory, and require every accepted job terminal before the initial copy;
3. drain the Kubernetes consumer, switch public reads after the initial copy, then scale it to zero declaratively while retaining its PVC and rollback shape;
4. after that fence, perform and verify a final delta copy and prove no further PVC writes;
5. reconcile the notifier inventory and durable queue, enable only the ARRNC staging consumer, then resume notifier publication;
6. reopen new-session admission and run authenticated end-to-end proof.

Rollback reverses those steps only after the ARRNC consumer is fenced and its owned messages and claims are drained or safely recoverable. Queue depth alone is insufficient evidence.

6. Staging before retirement

ARRNC staging is implemented, applied, and proven before any Kubernetes resource is removed. Proof includes authenticated upload through the normal UI/API path, S3 notification, exact queue ownership, ClamAV clean and EICAR outcomes, child-scoped write and public read, CSV and Study effects, retry/idempotency, process and mount failure recovery, queue drain, observability, and stopped/absent plus retained-Kubernetes rollback. Previous-digest rollback is required only when a prior ARRNC digest has independently proven the paused, zero-subscription contract.

Only after that proof and explicit operational activation approval may cluster-gitops:

1. keep the old Deployment at zero and PVC retained through a soak;
2. capture a checksummed data manifest and a tested snapshot or equivalent restorable archive;
3. remove the Application, ingress, Service, Deployment, and chart ownership declaratively;
4. remove the PVC only after the retention evidence and rollback deadline pass.

Any rollback to a retained or restored Kubernetes claim first fences ARRNC, inventories all output written since the artifact was captured, and performs a collision-safe checksummed inverse transfer into that claim. The old static route or consumer is not enabled until every current stored reference resolves against the reconciled claim.

The live pod, Application, ingress, or PVC is never deleted imperatively or abruptly.

7. Preview lifecycle

An explicitly enabled, authorized same-repository PR creates its slot from a verified digest through the protected activation workflow. Readiness must prove the exact root, sentinel, network, ClamAV, RabbitMQ identity, and resource limits before any SyRF preview receives Bulk PDF configuration.

Closure, disablement, or staleness runs idempotent teardown:

1. disable new uploads and fence the consumer;
2. drain or quarantine the preview's queue and prove no active claims;
3. stop and remove the exact container and scanner;
4. remove the exact preview image reference and only layers proven unused by any other live or persistent slot;
5. revoke the exact RabbitMQ account/vhost and remove its secret material;
6. unmount the ordinary agent's exact child so PR code no longer has access;
7. give a root-controlled cleanup worker a one-use capability for only previews/pr-N, prove that its cleanup view is the expected remote child rather than a local mount underlay, then delete that child with no-follow path validation and remove the cleanup view;
8. revoke the exact output-inspection capability and remove the dynamic manifest record;
9. remove the disposable rootless container, writable runtime state, per-preview network, routes, ports, and only image layers proven unused by any other slot;
10. prove process/runtime/network/storage/credential absence, release only the corresponding proven-absent parts of the atomic fleet reservation, and publish durable absence evidence.

Capacity for any quarantined resource remains charged to its owning fleet reservation, or moves atomically to an explicit quarantine debit that counts against the same aggregate budget, until reclamation is proved. Containers, writable runtime state, identities, networks, and storage allocations are never reused across PRs without the reviewed reset, fencing, and identity-rotation contract.

If an exact-child cleanup capability or remote-mount proof is unavailable, quarantine the child and alert. Cleanup must never fall back to a common-share credential or delete the local directory under an unmounted path.

An hourly reconciler compares open, enabled PRs with registered slots. Cleanup must converge within 24 hours; failures alert and fail closed rather than broadening deletion authority.

8. Production is a separate promotion

This decision and the staging migration do not apply production configuration, deploy a production container, change production credentials, drain a production queue, enable the production feature flag, or migrate existing PDFs. Production stays untouched until:

• staging and preview evidence is accepted;
• a production promotion plan and diff are separately reviewed;
• rollback and operator coverage are current;
• the user explicitly approves production promotion.

9. Documentation authority

This ADR and its companion correction design and plan are authoritative for PDF-agent hosting. Older Kubernetes/PVC hosting statements are historical only and must carry a supersession notice. After the migration, documentation and chart/GitOps cleanup remove obsolete deployment guidance so it cannot be followed as a recovery runbook.

Approval gates

• Gate A — architecture approval: approve this planning PR after its checks and reviews are clean. No implementation begins before this approval.
• Gate B — staging activation approval: after implementation PRs are reviewed, dry-run evidence is attached, and ARRNC staging is ready with consumption paused, explicitly approve the drain/cutover and later Kubernetes retirement sequence.
• Gate C — production promotion approval: a distinct future approval after staging and preview acceptance. Gate A or B never implies Gate C.

Consequences

Positive:

• one deployment model and one storage shape across environments;
• staging and previews validate the same on-prem constraints as production;
• preview lifecycle and credentials become explicit, bounded, and auditable;
• Kubernetes retirement removes a known source of architectural drift.

Costs and risks:

• ARRNC needs measured multi-preview capacity, child-share ACLs, dynamic registration, cleanup, and atomic fleet admission;
• approved preview images run on a shared on-prem kernel, so dependency or account compromise and accidental destructive code remain residual risks bounded by rootless least privilege, environment isolation, immutable provenance, resource limits, audit, and teardown;
• staging cutover needs a short controlled upload freeze to preserve single-consumer ownership;
• restoring a retired Kubernetes agent is slower than restarting a container, so snapshot and inverse-copy rollback proof are load-bearing; a compatible previous ARRNC digest becomes an additional rollback option only after independent proof.

Rejected alternatives

• Keep staging/previews in Kubernetes: contradicts the corrected owner decision and preserves the hosting variation this record removes.
• Run staging on ARRNC but previews in Kubernetes: still leaves preview-only deployment and storage behavior untested for production.
• Mount the common share root and bind children: exposes more authority than an ordinary agent requires and turns a host-path convention into the primary isolation control.
• Allow arbitrary dynamic gatekeeper entries: broadens a deployment mechanism into an on-prem remote-code execution surface.
• Start ARRNC and Kubernetes consumers together for a soak: competing consumers make ownership, drain, retry, and rollback evidence ambiguous.
• Remove the Kubernetes Application/PVC as soon as ARRNC starts: makes recovery destructive before end-to-end proof and restorable data evidence exist.

References

• ARRNC hosting correction design
• Executable migration plan
• ADR-012