Operating a deployment

This page is for the operator of a running deployment of archmax automation. Getting one started is installation; every setting is in the environment variables and config references.

One container, its processes

The image’s entrypoint runs the whole platform in one container:

nginx, in the foreground, on PORT (default 8080): it serves the web app, passes /api/ to the API, streams responses unbuffered with a 300 s read timeout, and accepts request bodies up to 21 MiB, just above the API’s own 20 MiB upload limit;
the API, on the internal API_PORT (default 3001);
the dispatcher, in the background;
the container’s own Redis, when REDIS_URL is unset, with a password made at each start that only the API and the dispatcher receive.

When REDIS_URL is unset, the container also runs its own Redis (Valkey, on 127.0.0.1:6379), which serves this container’s API and dispatcher alone (see scaling dispatchers).

The API and the dispatcher start a worker for every turn and every call made through a connection; those are processes of their own too.

The entrypoint restarts the API, the dispatcher or its own Redis two seconds after it exits. nginx keeps the container up meanwhile, so a container that is running is not proof that the platform is: the image’s health check probes the API every 30 seconds, and a dispatcher that keeps exiting shows only as [entrypoint] dispatcher exited; restarting in 2s on the log.

Single-process mode

Given a command, the container runs only that command, without nginx and without the web app. This is how docker-compose.yml runs the API and the dispatcher as two services:

docker run --rm --env-file archmax.env ghcr.io/archmax-ai/automation:1.5.0 \
  pnpm --filter @archmax/dispatcher start
docker run --rm --env-file archmax.env -p 3001:3001 ghcr.io/archmax-ai/automation:1.5.0 \
  pnpm --filter @archmax/api start

In this mode PORT is the API’s own port (default 3001). Each process reads the whole environment and the config at boot, so the dispatcher needs the same env file and config as the API: the sign-in variables, the secret store’s (MASTER_KEY, or the AWS Secrets Manager store’s group), the S3 group, and APP_BASE_URL, since the dispatcher registers callback addresses while a trigger activates.

Scaling dispatchers

Every dispatcher process serves every queue: turns, design-agent turns, evaluations, provisioning, environment operations, the trigger queues, the park sweep and the data update sweep. More dispatcher processes, on more machines, against the same MongoDB and Redis, add capacity. A container that runs its own Redis serves alone: a second one’s dispatcher waits for the single-dispatcher lease the first one holds, and its /healthz fails meanwhile, so more than one container needs REDIS_URL naming a shared Redis. Once a minute the poll sweep enqueues one poll for each active polling trigger, and whichever dispatcher claims it runs it; one process runs ARCHMAX_TRIGGER_POLL_CONCURRENCY polls at once (4), so polling capacity is that times the dispatcher processes. Each poll runs in a call worker, so a dispatcher’s polls, callback runs and renewals together take at most ARCHMAX_WORKER_CALL_SLOTS (6) workers at once.

The processes share nothing on disk. S3 is the only durable store, and the scratch directory holds only what a git operation is using at the moment, under a per-agent Redis lock, so it needs neither a volume nor sharing.

Workers

Code the platform does not control runs in a worker: a process of its own that holds none of the platform’s secrets, opens no connection to MongoDB, Redis or S3, and reaches the platform’s data only through the API or dispatcher that started it. Each turn gets one (a workflow, evaluation or design-agent turn, with its tools, its scripts and its child sessions), and so does each call made through a connection outside a turn: a poll, a callback run, a trigger’s lifecycle, a dropdown’s options, a connection test, a remote mount listing, the catalog.

Each starts when its work is dispatched and exits when the work ends. Starting one adds 0.3 to 0.65 seconds to its work on a fast machine, and two to four times that on a cloud vCPU.

A process runs at most ARCHMAX_WORKER_TURN_SLOTS (3) turn workers and ARCHMAX_WORKER_CALL_SLOTS (6) call workers at once; further work waits, its job still claimed. Each worker has a memory limit and a wall-time limit (see the variables). One that exceeds a limit fails its turn or call with a message naming the limit, and the process goes on serving.

Inside the all-in-one container

All the container’s processes run as one user, so the image keeps them apart by other means:

Every program in the image is execute-only for that user, which makes every process non-dumpable: no process of the container’s user can read another’s environment or memory, PID 1’s included.
A worker runs the code the image compiled under the Node.js permission model: it reads the application’s packages and its own working directory, writes only that directory, and starts no process, thread or native addon. It is a second layer, not the boundary.
nginx runs as one process with an empty environment.
The container’s own Redis refuses a client without the password made at this start.

Run the container with --cap-drop ALL, or its equivalent on the host: an ECS task definition’s linuxParameters, or a pod’s securityContext, dropping ALL. The image needs nothing it removes, and with it a health check or a docker exec also starts non-dumpable. Never run it as root, which reads everything.

Worker hosts

A deployment serving more than one organization should run its workers on worker hosts: the same image with the worker-host command, on machines that can reach neither the data stores nor the cloud’s metadata and credentials. The API and the dispatcher dispatch every worker there when ARCHMAX_WORKER_HOST names one, authenticating each connection with ARCHMAX_WORKER_HOST_KEY; the host holds that key and nothing else of the platform’s, and refuses to start otherwise (see the variables). A host runs as many workers as its dispatchers send it, so its capacity is its machine’s: add hosts behind one address to add capacity. Its own log lines start with [worker-host]: each worker it starts and how it ended, and each connection it refused for not holding the key.

The lines to alert on

[worker] refused <operation> for <turn or call> (agent …, environment …): <what> — a worker asked for something its turn or call may not reach: another session’s checkpoints, a connection its workflow does not allow, a write to a read-only folder. One line per refusal, never with a value.
[worker] ended <worker> for <turn or call> (agent …, environment …): it … — a worker refused to start, exceeded a limit, lost its channel or was ended by a signal.

Reading the log

nginx, the API and the dispatcher write to the container’s one stdout. The entrypoint tags the API’s lines [api], the dispatcher’s [dispatcher], its Redis’s [redis] and its own [entrypoint]; a config entry left out at boot is a [config] line. What a boot does to the database is [data-updates]: one line per data update applied, the wait for another process applying them, a refusal to start, and the warning while ARCHMAX_ALLOW_NEWER_DATA is set (see Data updates at boot). nginx’s access log keeps failed requests only, because an open editor polls often enough to bury everything else. ACCESS_LOG changes that:

`ACCESS_LOG`	Access log
`errors` (default)	Only responses that are not 2xx or 3xx.
`all`	Every request, for debugging a proxy or a client.
`off`	None.

The secrets some addresses carry (the webhook, callback, AG-UI and MCP endpoint tokens, invitation tokens, the OAuth2 callback’s query) are replaced by [redacted] in the access log, in the request line and in the Referer. Every change to a credential writes one [credentials] line naming the change, the identifier, the environment and the member, never a value.

Health

GET /api/health answers {"status":"ok","version":"<release>"}, where version is the image’s APP_VERSION: the release number, unstable-<sha> for an unstable image, pr-<number> for a pull request image, or dev. It is answered before the database middleware, so it says the API process is up, not that MongoDB is reachable. Probe it after every change to the environment or the config; the container’s own health check calls it on 127.0.0.1:${API_PORT}.

GET /healthz on the public port is readiness, for a load balancer’s health check: it answers as /api/health does and, when the container runs its own Redis, only while this container’s dispatcher holds the single-dispatcher lease (recorded in MongoDB), answering 503 otherwise. With the API down, nginx answers it 502.

The config and the environment

Two documents, split by one rule. The config says what the platform offers: the design agent, the services, the Activepieces collections, the skill registries, the OAuth apps and the branding. Environment variables say where it runs: MongoDB, Redis, S3, the secret store, sign-in, ports, host directories. A host setting written in the config fails the boot as an unknown key, and an environment variable for a config setting is not read at all.

Both are read once at boot, so a change to either is a restart.
pnpm config:env --check resolves and validates a config exactly as a boot would; pnpm config:env prints it as the one line of JSON ARCHMAX_CONFIG holds, and --base64 encodes it for a host such as Railway, which rewrites ${{NAME}} inside a variable’s value.
A placeholder whose variable is unset or empty leaves out the entry holding it (a service, an OAuth app, a registry or the whole designAgent section), and the rest still boots. Production refuses to boot without designAgent.

A deployment signs people in one way, its sign-in mode, which ARCHMAX_SIGN_IN names:

Mode	`ARCHMAX_SIGN_IN`	How people sign in
OIDC sign-in	`oidc`	The web app signs in at an OIDC identity provider (Auth0, Cognito, Okta, Keycloak) with the code flow and PKCE, keeps its tokens in memory, and sends the access token to the API, which verifies it through the issuer’s discovery document. Works on any host.
Proxy sign-in: AWS ALB	`aws-alb`	An AWS Application Load Balancer signs people in before any request reaches the platform and forwards the identity it signs, which the API verifies. The web app holds no token. See AWS (ECS Fargate).
The dev user	`dev-user`, or unset	Every request is one fixed local user. Refused with `NODE_ENV=production`, which requires `ARCHMAX_SIGN_IN`.

With either mode, a route that changes something answers only the platform’s own pages: a request another site’s page sent (which, under proxy sign-in, a browser sends with the proxy’s session cookie) is refused with 403 before anything happens, and a route that takes JSON refuses a body of another type, such as a form.

A mode whose variables are incomplete is refused at boot, and so is a variable of a mode the deployment doesn’t name, such as an ARCHMAX_AWS_ALB_* variable left over beside ARCHMAX_SIGN_IN=oidc; the environment variables reference lists each mode’s. The web app reads the mode from GET /api/sign-in-settings when it loads, so the image carries no sign-in settings and one image serves every deployment. When the app cannot read the route, it shows archmax cannot be reached right now. with Try again, and nothing behind sign-in.

OIDC sign-in with Auth0. Set ARCHMAX_SIGN_IN=oidc, ARCHMAX_OIDC_ISSUER to the issuer of your Auth0 domain (https://<auth0-domain>/, with the trailing slash), ARCHMAX_OIDC_CLIENT_ID to the SPA application’s client id, and ARCHMAX_OIDC_AUDIENCE to the identifier of the Auth0 API for the platform. When people sign in with a password, also set ARCHMAX_AUTH0_DB_CONNECTION to the database connection’s name (Authentication → Database; Username-Password-Authentication unless renamed), which offers them Auth0’s change-password email. In the SPA application, list the app’s origin and https://<host>/silent-callback.html under Allowed Callback URLs, and the origin under Allowed Logout URLs; enable RP-initiated logout in Auth0, so signing out ends the Auth0 session too. After a reload the app restores the sign-in from a hidden frame that returns to silent-callback.html; serve Auth0 from a custom domain on the app’s own site (auth.example.com beside app.example.com), so browsers that block third-party cookies restore it too.

OAuth apps

An OAuth app lets owners connect Microsoft 365, Google Workspace or GitHub collections by choosing Connect and signing in; a collection no app lists cannot be signed in to. The repository’s docs/oauth-apps.md is the full setup guide. In short:

Every app registers one redirect URI, OAUTH2_REDIRECT_URL (https://<host>/api/oauth2/callback).
Each app is an entry in the config’s oauthApps list, naming its collections, with its client id and secret as placeholders for two variables per app.
pnpm oauth-apps:entra registers the Microsoft app through the Azure CLI, and pnpm oauth-apps:github the GitHub App through GitHub’s manifest page. Without --apply both only print what they would do; once they register the app they print its config entry. The Google client is made by hand in the Google Auth Platform.
Before customers sign in: publisher verification for Entra; brand verification for Google, plus a yearly security assessment for its restricted Drive and Gmail scopes. Until that assessment is done, run the lite Google collections on a second Google app, google-lite, with a consent screen of its own. It asks for no restricted scope, so it needs no assessment. Google verifies a consent screen, which every client registered beside it shares, so the lite app cannot share one with the full collections’ app.
To rotate a secret, create the new one, set the variable and restart: existing connections keep working. A new client id instead asks every connection signed in through the old app to reconnect.

A customer’s administrator may still have to act: grant admin consent per collection in their Microsoft directory, trust the client id in Google Workspace, or install the GitHub App on their organization.

Services

A service is a separate image that serves one platform collection’s tools over MCP, named in the config’s services list with its endpoint and the token it checks. Give every service its own token (16 characters or more, the same value set on the service), held in its own variable, ARCHMAX_SERVICE_<ID>_TOKEN in the example. The platform discovers each service when it warms its tool catalog; one that fails discovery is logged once, not offered, and tried again later, while the others are offered. A service URL may point at the private network.

Skill registries

A registry is a public git repository of skills/<slug>/SKILL.md bundles that owners browse and install from the Skills page. With no skillRegistries key the platform follows its own public registry; [] follows none; URLs must be https. Registries are cloned at startup into ARCHMAX_SKILL_REGISTRIES_DIR (/data/skill-registries in the image), in the background, and fast-forwarded on later boots. An unreachable registry logs a warning and serves its last clone. GET /api/skill-registries reports each one’s commit, skill count and last failure.

Branding

A deployment presents itself under the title, logo and colours of the config’s branding section, and archmax’s own wherever the section says nothing. A value it cannot use is left out with a line in the log rather than refusing the start, and the API logs a logo it cannot load. Branding a deployment walks through it.

Data updates at boot

Every MongoDB document carries the schema version of its collection’s shape. A release that changes what a collection stores ships a data update: the code that rewrites the stored documents to the new shape. The API and the dispatcher bring the database up to date when they start, before the API listens and before the dispatcher takes a job, so there is nothing to run by hand. A boot:

makes sure every collection has its schemaVersion_1 index (the first boot of a release that introduces it builds them);
reads which versions each collection holds, one index read per collection;
refuses to start on data it cannot read (below);
applies the pending data updates, oldest first, under a Redis lock. One process applies them; the other waits, logging [data-updates] waiting for another process to finish applying data updates every 30 seconds, and starts once nothing is left.

A boot with nothing to apply takes no lock and logs nothing. Every data update applied logs one line:

[data-updates] credentials v1 → v2 (SharePoint OAuth2 credentials gain the tenant `common` and the delegated access mode): 12 documents in 0.1s

A process killed in the middle of a data update leaves nothing broken: the next one to start continues with the documents not yet rewritten. While a new release is put into service the previous one keeps serving until the new one passes its health check, and documents it writes in that window carry the older version. The dispatcher’s data update sweep looks for them every five minutes and brings them forward.

When a boot refuses

A refusal is the process’s last output, starting [data-updates] Refusing to start. The entrypoint restarts the process two seconds later, so the same lines repeat until the cause is fixed.

The refusal says	It means	Do
`documents at version 6, above the version 5 this release knows`	An older release started against data a newer release has already updated: a rollback, or an image pinned to an older release.	Roll forward, restore the backup taken before the newer release, or roll back with the override.
`documents at version 1, below its baseline 2`	The database is older than any data update this release has.	Reset the database, or convert it with the one-off tool the release that needed it named.
`3 documents without a schemaVersion`	Something wrote documents around the platform, such as a script or a restore of foreign data.	Find and correct those documents; the platform does not guess a version.

Releasing a data update

The pull request of a release with a data update says so, and says whether the release before it can read what the data update writes. Before putting such a release into service:

back up the database;
if the previous release cannot read the result, stop it first, so the two never share the database, at the cost of a short outage.

A long data update may outlast the host’s health check. That is safe, because the next attempt continues it, but it can be kept off the boot by applying it beforehand with the release’s own tool, when the previous release can read the result:

pnpm data-updates            # each collection's versions and pending data updates; writes nothing
pnpm data-updates --apply    # applies them exactly as a boot does, waiting for a booting process

Run it from the repository at that release, with the deployment’s environment in the repository root’s .env (variables already set win), or with the release’s image:

docker run --rm --env-file archmax.env ghcr.io/archmax-ai/automation:1.5.0 \
  pnpm --filter @archmax/core exec tsx tools/data-updates.ts --apply

Rolling back past a data update

An older release refuses to start against documents a newer release’s data update rewrote. There are three ways back: restore the backup taken before the newer release, roll forward with a fix, or start the older release with ARCHMAX_ALLOW_NEWER_DATA=true. With the variable set, the older release logs one warning per collection holding newer documents, leaves those documents as they are, and otherwise starts as usual.

Set it only when the newer release’s pull request says its data update is readable by the release before it. Otherwise the older release writes fields in their old form into documents that carry the newer version, and the newer release never rewrites them again. Documents the older release inserts carry its own version, so the newer release brings them forward when it starts again. Remove the variable once rolled forward: every boot warns while it is set, and says so when nothing newer is left to allow.

Reconciling triggers after a release

The dispatcher does not reconcile trigger deployments at startup. A pass runs after a publish, a restore, a promotion, a credential change, a member’s Retry now, or a retry that is due. A release that changes how the desired set of trigger deployments is computed, the configuration fingerprint that decides whether one changed, or which trigger strategies are supported therefore reaches running environments only after every environment is reconciled, once that release is running (the pull request making such a change says so):

pnpm --filter @archmax/core exec tsx tools/reconcile-triggers.ts            # lists, writes nothing
pnpm --filter @archmax/core exec tsx tools/reconcile-triggers.ts --apply    # requests the passes

Run it from the repository at that release, with the deployment’s environment in the repository root’s .env (variables already set win); --agent <id> limits it to one agent. The image holds the same tool, so single-process mode runs it with the release’s own code:

docker run --rm --env-file archmax.env ghcr.io/archmax-ai/automation:1.5.0 \
  pnpm --filter @archmax/core exec tsx tools/reconcile-triggers.ts --apply