Docs menu

Operate

Staging and launch checks

Run a full CLI stack on testnets, smoke-test it on a schedule, try the CLI token, own-cloud servers and compliance there, then go to mainnet small.

Explain it

This page is for people taking their own CLI platform live: a practice setup on real machines with test money, the checks to run, and then going live with small amounts.

Do you need this page? Only if you run your own CLI platform. On clicloud.app this has been done for you. To set up a staging copy of your own app, use environments (see /docs/configuration).

Tests passing isn't the same as real money moving safely. This page is for operators taking their own CLI server live: a staging stack on real machines with testnets, the smoke test that checks it end to end, the CLI token on Robinhood Chain's testnet, checking own-cloud servers and compliance, and then mainnet with small amounts. Each stage ends with a check that must pass before the next.

The full walkthrough is deploy/STAGING.md in the repository. For what the platform consists of and every setting, see Running CLI yourself.

The testnet stack

Staging runs everything on testnets and everything else like production: two Ubuntu servers (a control plane and a worker), managed Postgres, an S3 bucket, its own AWS KMS keys, RPC endpoints and Resend for email.

  1. On the worker host, run sudo deploy/host-setup.sh, then run the worker with CLI_WORKER_SECRET.
  2. On the control plane, copy deploy/staging/env.staging.example to deploy/.env and fill in the blanks.
  3. Run npm run preflight. On a testnet it may warn, but it must not fail. Send the payout wallet it prints some testnet USDC and a little Base Sepolia ETH.
  4. Put your own sign-in in CLI_OPERATORS (that makes you an admin operator), run docker compose up -d and sign in at the console.

The env template

The template sets the stack up for testnets, with every newer feature switched on so staging exercises it:

GroupSettings
The stackCLI_PUBLIC_URL, CLI_APP_DOMAIN, CLI_RUNTIME=remote, CLI_WORKER_URL, CLI_WORKER_SECRET, CLI_SECRET_KEY, CLI_DATABASE_URL, CLI_S3_BUCKET, CLI_KMS_KEY_ID, CLI_OPERATORS
Money, testnets onlyCLI_NETWORK=eip155:84532 (Base Sepolia) and its USDC address, the Solana devnet network, the x402.org facilitator, CLI_PAYOUT_KMS_KEY_ID, and CLI_PAYOUT_REVIEW_OVER_USD=5
The CLI tokenThe CLI_TOKEN_* lines printed by the testnet deploy (below)
Growth and trustCLI_WELCOME_CREDIT_USD=2, CLI_WELCOME_CREDIT_WALLETS=true, CLI_PUBLIC_STATS=true, CLI_REQUIRE_OPERATOR_2FA=true, CLI_AUDIT_RETENTION_DAYS=400, CLI_BYOC_FEE_BPS=1500

Welcome credit for wallets is on because the smoke test signs in with a fresh wallet.

The smoke test

The smoke test drives a running CLI through its public API the way an agent would, and reports each step. Run it after every deploy to staging:

Terminal
npm run smoke -- --api https://staging.example.com

Set CLI_SMOKE_WALLET_KEY to a wallet holding Base Sepolia USDC, for when the project has no free credit. Without it, the test signs in with a random wallet that can't pay.

What it checks

First, checks that change nothing:

  1. health: /health answers.
  2. config: /v1/config lists the payment networks.
  3. skill file: /skill.md is the skill file.
  4. API reference: /v1/openapi.json lists its paths.
  5. error catalogue: /v1/errors lists the codes.
  6. signing key for credentials: /.well-known/jwks.json has a key.

Then, unless it's read-only:

  1. sign in with the wallet.
  2. create a project, noting any welcome credit.
  3. fund it over x402: skipped if the project already has at least $1 of credit; otherwise tops up 2 USDC.
  4. create a service from the small public image traefik/whoami:v1.10.
  5. deploy and go healthy: waits for the deploy, up to 10 minutes by default.
  6. the app answers: fetches the app's URL and fails on a 5xx.
  7. logs: reads the service's recent output.
  8. agent credential: makes a short-lived read-only agent key, gets the agent's credential with it, verifies it, and deletes the key (see /docs/agent-credentials).
  9. clean up: deletes the service.

With --rollback, after the logs it deploys a second time and rolls back to the first deploy (roll back). With --database, it adds a Postgres database, waits for it to run, and deletes it (add a database, delete the database).

If signing in, the project, funding, the service or the deploy fails, the rest of that chain is skipped. Logs, the credential and clean-up still run when there's a project or service to run them on, so a failed run doesn't leave a service behind.

Options

OptionWhat it does
--api <url>The CLI API to test (or set CLI_API_URL)
--read-onlyOnly the first six checks: nothing is created. Safe against production
--no-app-fetchSkip fetching the app, when apps aren't reachable from where the test runs
--rollbackAlso deploy again and roll back to the first deploy
--databaseAlso add a Postgres database and delete it
--image <image>, --port <port>Deploy another image instead of traefik/whoami:v1.10 on port 80
--timeout <seconds>How long to wait for the deploy (default 600)

It prints each step as ok, skip or FAIL, writes smoke-report.md and smoke-report.json where it runs, and exits non-zero if anything failed.

Run it on a schedule

The repository's staging smoke GitHub workflow runs the smoke test every six hours, and whenever you start it by hand. Add two repository secrets:

SecretValue
CLI_STAGING_APIYour staging API, such as https://staging.example.com
CLI_SMOKE_WALLET_KEYA wallet with Base Sepolia USDC

Each run adds the report to the run's summary and keeps it as an artifact named smoke-report.

The server also runs it on its own schedule: see The canary and the status page.

The canary and the status page

Every CLI server runs the smoke test against itself on a schedule, from the inside, and publishes the results as its status page.

Set up the canary project once: make a project for it, add credit (a few dollars a month covers it), and make an admin project key. Then set:

VariableMeaning
CLI_CANARY_KEYThe canary project's admin key
CLI_CANARY_PROJECTThe canary project's id
CLI_CANARY_EVERY_MINMinutes between runs (default 30; 0 turns it off)
CLI_CANARY_FETCH_APPfalse when deployed apps aren't reachable from the control plane
CLI_CANARY_IMAGE, CLI_CANARY_PORTDeploy another image instead of traefik/whoami:v1.10 on port 80

Each run deploys a test app, checks it answers, rolls it back, adds a Postgres database, reads logs, issues an agent credential, and deletes everything it made. It also checks both connectors: Claude's (/mcp) and ChatGPT's (/mcp/openai) publish their sign-in details, refuse a request without a token, list their tools and answer a read. Without a canary key it runs only the checks that change nothing. Only the leader runs it, one run at a time.

The status page is at /status on the API (for clicloud.app, https://api.clicloud.app/status, linked from the site's footer), and as JSON at GET /v1/platform/status. It shows each part (API and dashboard, deploys, databases, the Claude connector, the ChatGPT connector) with 90 days of daily results, and incidents. Parts this server has never checked are left off. It never shows check details or project ids.

Alerts: when a part fails twice in a row, every operator who signs in by email gets an email, and another when it passes again.

On the Operations page, Platform checks shows the latest result of each part with its detail and runs the canary on demand. Status page incidents posts an incident (title, impact, affected parts, first update) and its updates through to resolved. With the API: GET /v1/admin/platform, POST /v1/admin/platform/canary, POST /v1/admin/platform/incidents, POST /v1/admin/platform/incidents/:id/updates.

Restore drills

Every 7 days (CLI_DRILL_EVERY_DAYS; 0 turns it off), the leader downloads the newest backup of CLI's own database and decrypts it with the key provider, the same way a real restore would.

  • SQLite: opens the copy, runs an integrity check and reads the schema version and row counts.
  • Postgres: pg_restore --list must read the archive and find data for projects, services and the ledger. Set CLI_DRILL_DATABASE_URL to a scratch database and it restores into that too. Never point it at the live database.

A backup more than two days old fails the drill. Each drill is recorded as compliance evidence ("Automatic restore drill", alongside the drills people record by hand) and shows on the Operations page; a failure emails operators. Admins can run one now from Platform checks, or with POST /v1/admin/platform/drill.

Infrastructure drills

Some parts of CLI pass every test but depend on the real machines: Postgres failover, point-in-time recovery, restores of each database engine, sandbox isolation under gVisor, and paid routes and metrics through the edge router. The drills prove them on your staging server, the way a customer would use them:

Terminal
CLI_DRILL_KEY=<project key> npm run drills -- --api https://staging.example.com --project prj_…

Make a project for the drills, add a few dollars of credit, and give it an admin key with the read, deploy, manage and secrets scopes (the drills read their test data back with SQL). Each drill makes its own resources, named drill-…, and deletes them afterwards.

DrillWhat it proves
backup-postgres, backup-mysqlWrites a row, backs up, writes another, restores: only the first row is there
backup-redis, backup-mongoA backup and a restore both finish (their data isn't read back, so these report "check")
pitrPoint-in-time recovery restores Postgres to a moment between two writes (using the archive's own times, so clocks don't matter)
failoverA standby takes over: the data written before is there, and it takes new writes
sandboxThe sandbox can't reach the cloud metadata address (169.254.169.254), and runs under gVisor ("check" if gVisor isn't detected)
edgeAn unpaid call to a paid route gets a 402 with PAYMENT-REQUIRED from the router, a call paid from a stream gets through and is charged, and the router's request metrics see the traffic

Options: --only pitr,failover runs some of them, --no-app-fetch skips edge when deployed apps aren't reachable from where you run it, and --timeout <seconds> (default 900) is how long each wait may take. It prints each result, writes drills-report.md and exits non-zero if a drill failed.

Record the results on the server with --record and CLI_OPERATOR_TOKEN set to an operator's session token. They show on the Operations page under Platform checks → Infrastructure drills, and the restore drills (backup-*, pitr) are added to the compliance evidence. Run the drills after every infrastructure change, and before going live.

The launch check

Before submitting to Claude's or ChatGPT's directory, run npm run launch-check -- --api https://api.example.com --site https://example.com (add --reviewer-email and CLI_REVIEWER_PASSWORD to try reviewer sign-in). It checks that the sign-in metadata and both connectors' 401s name the API's host, the documentation links work, the legal pages are up and name your company, OpenAI's domain token is served, and the reviewer account works and has projects. It writes launch-report.md and exits non-zero on a failure.

Check the ChatGPT connector

scripts/check-connector.ts calls every connector tool against a real account and fails on any unclear error. Add --chatgpt to check /mcp/openai, with a token from a sign-in made for the ChatGPT connector.

The end-to-end check

After the smoke test passes, run the fuller end-to-end check with a funded testnet payer (CLI_E2E_PAYER_KEY), which also exercises databases, buckets, queues, tunnels and receipts. See Running CLI yourself. Then by hand: withdraw earnings from a test project and watch the payout confirm, and send USDC to a deposit address and watch it credit.

The CLI token on testnet

To try paying with the CLI token (see /docs/cli-token), deploy its contracts to Robinhood Chain's testnet with a throwaway key holding testnet ETH:

Terminal
cd contracts
RH_TESTNET_RPC_URL=… RH_TESTNET_CHAIN_ID=… RH_DEPLOYER_KEY=0x… npm run deploy:testnet

It deploys the token, the oracle, the credit contract and staking, all owned and run by that one key, and prints the CLI_TOKEN_* lines to paste into the staging .env. The deployer holds the whole test supply: send some to the wallets you test with.

There's no pool to read a price from on testnet, so push one yourself:

Terminal
TEST_PRICE_USD=0.02 npm run price:testnet

Run it from cron at least every two hours, or purchases pause on a stale price. The oracle refuses a push within a minute of the last one and moves at most 15% per push, so change the price in steps.

Check: send test CLI to a wallet, buy credit from the dashboard's Pay with CLI, and stake. The purchase shows as credit within a minute; staking credit appears within 15 minutes.

Check own-cloud servers on staging

Add a server with In your cloud: it connects out, start a machine in a separate cloud account with the cloud-init it gives you, and deploy a service to it (see /docs/own-cloud).

Check: the server shows online with no inbound port open except 80 and 443, the app answers on your app domain, and turning on Keep everything on our own servers makes a new service land there and refuses a sandbox.

Check compliance on staging

With CLI_REQUIRE_OPERATOR_2FA=true, set up a second factor as an operator first. Then give a second person the support role and check that they can see the abuse queue but get FORBIDDEN on the evidence export. Run one project access review and one operator review (see /docs/access-reviews), open and resolve a test security incident, record a restore drill after restoring a backup (decrypting it with npm run backup:decrypt), and download the evidence (see /docs/compliance).

Check: each one shows up in the export, including the role in operators, the incident in securityIncidents, and your second factor in mfaChanges.

Agents against staging

Before strangers' agents arrive, run npm run evals for the agents you plan to recommend, then point one at staging through the MCP server and have it deploy one of the examples. Every scenario should pass with no guardrail breaks.

Mainnet, small

Switch the same stack to real money:

  • CLI_NETWORK=eip155:8453 and mainnet USDC.
  • Leave CLI_FACILITATOR_URL unset, so CLI uses Coinbase's facilitator signed with the CDP key. The x402.org facilitator is refused on mainnet.
  • CLI_PAYOUT_REVIEW_OVER_USD=50, so every bigger payout waits for you.
  • Fund the payout wallet with only what you're prepared to lose.

npm run preflight must print Ready. with no failures. Keep running the smoke test against production with --read-only.

Check: a 1 USDC top-up, one paid route call, a 5 USDC withdrawal, a 5 USDC deposit and a card on-ramp purchase each show up once in the ledger, the statement and on chain. Restore last night's backup into a scratch database and count rows.

Open the doors after an external security review and legal advice, and raise the payout review threshold once a week of payouts has gone through cleanly.