Operate
Staging and launch checks
Run a full CLI stack on testnets, smoke-test it on a schedule, try the CLI token, own-cloud servers and compliance there, then go to mainnet small.
This page is for people taking their own CLI platform live: a practice setup on real machines with test money, the checks to run, and then going live with small amounts.
Do you need this page? Only if you run your own CLI platform. On clicloud.app this has been done for you. To set up a staging copy of your own app, use environments (see /docs/configuration).
Tests passing isn't the same as real money moving safely. This page is for operators taking their own CLI server live: a staging stack on real machines with testnets, the smoke test that checks it end to end, the CLI token on Robinhood Chain's testnet, checking own-cloud servers and compliance, and then mainnet with small amounts. Each stage ends with a check that must pass before the next.
The full walkthrough is deploy/STAGING.md in the repository. For what the platform consists of and every setting, see Running CLI yourself.
The testnet stack
Staging runs everything on testnets and everything else like production: two Ubuntu servers (a control plane and a worker), managed Postgres, an S3 bucket, its own AWS KMS keys, RPC endpoints and Resend for email.
- On the worker host, run
sudo deploy/host-setup.sh, then run the worker withCLI_WORKER_SECRET. - On the control plane, copy
deploy/staging/env.staging.exampletodeploy/.envand fill in the blanks. - Run
npm run preflight. On a testnet it may warn, but it must not fail. Send the payout wallet it prints some testnet USDC and a little Base Sepolia ETH. - Put your own sign-in in
CLI_OPERATORS(that makes you an admin operator), rundocker compose up -dand sign in at the console.
The env template
The template sets the stack up for testnets, with every newer feature switched on so staging exercises it:
| Group | Settings |
|---|---|
| The stack | CLI_PUBLIC_URL, CLI_APP_DOMAIN, CLI_RUNTIME=remote, CLI_WORKER_URL, CLI_WORKER_SECRET, CLI_SECRET_KEY, CLI_DATABASE_URL, CLI_S3_BUCKET, CLI_KMS_KEY_ID, CLI_OPERATORS |
| Money, testnets only | CLI_NETWORK=eip155:84532 (Base Sepolia) and its USDC address, the Solana devnet network, the x402.org facilitator, CLI_PAYOUT_KMS_KEY_ID, and CLI_PAYOUT_REVIEW_OVER_USD=5 |
| The CLI token | The CLI_TOKEN_* lines printed by the testnet deploy (below) |
| Growth and trust | CLI_WELCOME_CREDIT_USD=2, CLI_WELCOME_CREDIT_WALLETS=true, CLI_PUBLIC_STATS=true, CLI_REQUIRE_OPERATOR_2FA=true, CLI_AUDIT_RETENTION_DAYS=400, CLI_BYOC_FEE_BPS=1500 |
Welcome credit for wallets is on because the smoke test signs in with a fresh wallet.
The smoke test
The smoke test drives a running CLI through its public API the way an agent would, and reports each step. Run it after every deploy to staging:
npm run smoke -- --api https://staging.example.comSet CLI_SMOKE_WALLET_KEY to a wallet holding Base Sepolia USDC, for when the project has no free credit. Without it, the test signs in with a random wallet that can't pay.
What it checks
First, checks that change nothing:
- health:
/healthanswers. - config:
/v1/configlists the payment networks. - skill file:
/skill.mdis the skill file. - API reference:
/v1/openapi.jsonlists its paths. - error catalogue:
/v1/errorslists the codes. - signing key for credentials:
/.well-known/jwks.jsonhas a key.
Then, unless it's read-only:
- sign in with the wallet.
- create a project, noting any welcome credit.
- fund it over x402: skipped if the project already has at least $1 of credit; otherwise tops up 2 USDC.
- create a service from the small public image
traefik/whoami:v1.10. - deploy and go healthy: waits for the deploy, up to 10 minutes by default.
- the app answers: fetches the app's URL and fails on a 5xx.
- logs: reads the service's recent output.
- agent credential: makes a short-lived read-only agent key, gets the agent's credential with it, verifies it, and deletes the key (see /docs/agent-credentials).
- clean up: deletes the service.
With --rollback, after the logs it deploys a second time and rolls back to the first deploy (roll back). With --database, it adds a Postgres database, waits for it to run, and deletes it (add a database, delete the database).
If signing in, the project, funding, the service or the deploy fails, the rest of that chain is skipped. Logs, the credential and clean-up still run when there's a project or service to run them on, so a failed run doesn't leave a service behind.
Options
| Option | What it does |
|---|---|
--api <url> | The CLI API to test (or set CLI_API_URL) |
--read-only | Only the first six checks: nothing is created. Safe against production |
--no-app-fetch | Skip fetching the app, when apps aren't reachable from where the test runs |
--rollback | Also deploy again and roll back to the first deploy |
--database | Also add a Postgres database and delete it |
--image <image>, --port <port> | Deploy another image instead of traefik/whoami:v1.10 on port 80 |
--timeout <seconds> | How long to wait for the deploy (default 600) |
It prints each step as ok, skip or FAIL, writes smoke-report.md and smoke-report.json where it runs, and exits non-zero if anything failed.
Run it on a schedule
The repository's staging smoke GitHub workflow runs the smoke test every six hours, and whenever you start it by hand. Add two repository secrets:
| Secret | Value |
|---|---|
CLI_STAGING_API | Your staging API, such as https://staging.example.com |
CLI_SMOKE_WALLET_KEY | A wallet with Base Sepolia USDC |
Each run adds the report to the run's summary and keeps it as an artifact named smoke-report.
The server also runs it on its own schedule: see The canary and the status page.
The canary and the status page
Every CLI server runs the smoke test against itself on a schedule, from the inside, and publishes the results as its status page.
Set up the canary project once: make a project for it, add credit (a few dollars a month covers it), and make an admin project key. Then set:
| Variable | Meaning |
|---|---|
CLI_CANARY_KEY | The canary project's admin key |
CLI_CANARY_PROJECT | The canary project's id |
CLI_CANARY_EVERY_MIN | Minutes between runs (default 30; 0 turns it off) |
CLI_CANARY_FETCH_APP | false when deployed apps aren't reachable from the control plane |
CLI_CANARY_IMAGE, CLI_CANARY_PORT | Deploy another image instead of traefik/whoami:v1.10 on port 80 |
Each run deploys a test app, checks it answers, rolls it back, adds a Postgres database, reads logs, issues an agent credential, and deletes everything it made. It also checks both connectors: Claude's (/mcp) and ChatGPT's (/mcp/openai) publish their sign-in details, refuse a request without a token, list their tools and answer a read. Without a canary key it runs only the checks that change nothing. Only the leader runs it, one run at a time.
The status page is at /status on the API (for clicloud.app, https://api.clicloud.app/status, linked from the site's footer), and as JSON at GET /v1/platform/status. It shows each part (API and dashboard, deploys, databases, the Claude connector, the ChatGPT connector) with 90 days of daily results, and incidents. Parts this server has never checked are left off. It never shows check details or project ids.
Alerts: when a part fails twice in a row, every operator who signs in by email gets an email, and another when it passes again.
On the Operations page, Platform checks shows the latest result of each part with its detail and runs the canary on demand. Status page incidents posts an incident (title, impact, affected parts, first update) and its updates through to resolved. With the API: GET /v1/admin/platform, POST /v1/admin/platform/canary, POST /v1/admin/platform/incidents, POST /v1/admin/platform/incidents/:id/updates.
Restore drills
Every 7 days (CLI_DRILL_EVERY_DAYS; 0 turns it off), the leader downloads the newest backup of CLI's own database and decrypts it with the key provider, the same way a real restore would.
- SQLite: opens the copy, runs an integrity check and reads the schema version and row counts.
- Postgres:
pg_restore --listmust read the archive and find data for projects, services and the ledger. SetCLI_DRILL_DATABASE_URLto a scratch database and it restores into that too. Never point it at the live database.
A backup more than two days old fails the drill. Each drill is recorded as compliance evidence ("Automatic restore drill", alongside the drills people record by hand) and shows on the Operations page; a failure emails operators. Admins can run one now from Platform checks, or with POST /v1/admin/platform/drill.
Infrastructure drills
Some parts of CLI pass every test but depend on the real machines: Postgres failover, point-in-time recovery, restores of each database engine, sandbox isolation under gVisor, and paid routes and metrics through the edge router. The drills prove them on your staging server, the way a customer would use them:
CLI_DRILL_KEY=<project key> npm run drills -- --api https://staging.example.com --project prj_…Make a project for the drills, add a few dollars of credit, and give it an admin key with the read, deploy, manage and secrets scopes (the drills read their test data back with SQL). Each drill makes its own resources, named drill-…, and deletes them afterwards.
| Drill | What it proves |
|---|---|
backup-postgres, backup-mysql | Writes a row, backs up, writes another, restores: only the first row is there |
backup-redis, backup-mongo | A backup and a restore both finish (their data isn't read back, so these report "check") |
pitr | Point-in-time recovery restores Postgres to a moment between two writes (using the archive's own times, so clocks don't matter) |
failover | A standby takes over: the data written before is there, and it takes new writes |
sandbox | The sandbox can't reach the cloud metadata address (169.254.169.254), and runs under gVisor ("check" if gVisor isn't detected) |
edge | An unpaid call to a paid route gets a 402 with PAYMENT-REQUIRED from the router, a call paid from a stream gets through and is charged, and the router's request metrics see the traffic |
Options: --only pitr,failover runs some of them, --no-app-fetch skips edge when deployed apps aren't reachable from where you run it, and --timeout <seconds> (default 900) is how long each wait may take. It prints each result, writes drills-report.md and exits non-zero if a drill failed.
Record the results on the server with --record and CLI_OPERATOR_TOKEN set to an operator's session token. They show on the Operations page under Platform checks → Infrastructure drills, and the restore drills (backup-*, pitr) are added to the compliance evidence. Run the drills after every infrastructure change, and before going live.
The launch check
Before submitting to Claude's or ChatGPT's directory, run npm run launch-check -- --api https://api.example.com --site https://example.com (add --reviewer-email and CLI_REVIEWER_PASSWORD to try reviewer sign-in). It checks that the sign-in metadata and both connectors' 401s name the API's host, the documentation links work, the legal pages are up and name your company, OpenAI's domain token is served, and the reviewer account works and has projects. It writes launch-report.md and exits non-zero on a failure.
Check the ChatGPT connector
scripts/check-connector.ts calls every connector tool against a real account and fails on any unclear error. Add --chatgpt to check /mcp/openai, with a token from a sign-in made for the ChatGPT connector.
The end-to-end check
After the smoke test passes, run the fuller end-to-end check with a funded testnet payer (CLI_E2E_PAYER_KEY), which also exercises databases, buckets, queues, tunnels and receipts. See Running CLI yourself. Then by hand: withdraw earnings from a test project and watch the payout confirm, and send USDC to a deposit address and watch it credit.
The CLI token on testnet
To try paying with the CLI token (see /docs/cli-token), deploy its contracts to Robinhood Chain's testnet with a throwaway key holding testnet ETH:
cd contracts
RH_TESTNET_RPC_URL=… RH_TESTNET_CHAIN_ID=… RH_DEPLOYER_KEY=0x… npm run deploy:testnetIt deploys the token, the oracle, the credit contract and staking, all owned and run by that one key, and prints the CLI_TOKEN_* lines to paste into the staging .env. The deployer holds the whole test supply: send some to the wallets you test with.
There's no pool to read a price from on testnet, so push one yourself:
TEST_PRICE_USD=0.02 npm run price:testnetRun it from cron at least every two hours, or purchases pause on a stale price. The oracle refuses a push within a minute of the last one and moves at most 15% per push, so change the price in steps.
Check: send test CLI to a wallet, buy credit from the dashboard's Pay with CLI, and stake. The purchase shows as credit within a minute; staking credit appears within 15 minutes.
Check own-cloud servers on staging
Add a server with In your cloud: it connects out, start a machine in a separate cloud account with the cloud-init it gives you, and deploy a service to it (see /docs/own-cloud).
Check: the server shows online with no inbound port open except 80 and 443, the app answers on your app domain, and turning on Keep everything on our own servers makes a new service land there and refuses a sandbox.
Check compliance on staging
With CLI_REQUIRE_OPERATOR_2FA=true, set up a second factor as an operator first. Then give a second person the support role and check that they can see the abuse queue but get FORBIDDEN on the evidence export. Run one project access review and one operator review (see /docs/access-reviews), open and resolve a test security incident, record a restore drill after restoring a backup (decrypting it with npm run backup:decrypt), and download the evidence (see /docs/compliance).
Check: each one shows up in the export, including the role in operators, the incident in securityIncidents, and your second factor in mfaChanges.
Agents against staging
Before strangers' agents arrive, run npm run evals for the agents you plan to recommend, then point one at staging through the MCP server and have it deploy one of the examples. Every scenario should pass with no guardrail breaks.
Mainnet, small
Switch the same stack to real money:
CLI_NETWORK=eip155:8453and mainnet USDC.- Leave
CLI_FACILITATOR_URLunset, so CLI uses Coinbase's facilitator signed with the CDP key. The x402.org facilitator is refused on mainnet. CLI_PAYOUT_REVIEW_OVER_USD=50, so every bigger payout waits for you.- Fund the payout wallet with only what you're prepared to lose.
npm run preflight must print Ready. with no failures. Keep running the smoke test against production with --read-only.
Check: a 1 USDC top-up, one paid route call, a 5 USDC withdrawal, a 5 USDC deposit and a card on-ramp purchase each show up once in the ledger, the statement and on chain. Restore last night's backup into a scratch database and count rows.
Open the doors after an external security review and legal advice, and raise the payout review threshold once a week of payouts has gone through cleanly.