​
Hilt Pay API operator runbook

This page is for controlled private-alpha operators. It maps each supported CLI command to the incident it is meant to handle, the required confirmation, stop conditions, and rollback notes. The CLI is an authenticated client for staff-only API endpoints: 
python scripts/hilt_pay_api_ops_cli.py --help
Required environment: 
export HILT_ADMIN_API_BASE_URL=http://127.0.0.1:8000
export HILT_ADMIN_BEARER_TOKEN=staff-token
The CLI does not bypass API auth, run production migrations, charge cards, enable live plan checkout, activate non-Solana entitlements, or enable live Base/EVM/x402/USDT settlement. Non-Solana receipt creation remains staff-gated and requires explicit receipt-creation environment gates plus rail allowlisting. Operator-triggered dry-runs should include incident context when there is an active incident: 
python scripts/hilt_pay_api_ops_cli.py reconcile-stripe-api \
  --owner-id OWNER_UUID \
  --incident-id INC-123 \
  --operator-reason "Stripe subscription status mismatch"
Dashboard dry-runs on /hilt-pay-api provide the same incident id and reason fields. New reconciliation runs persist that context in access_reconciliation_runs.scope_json.operator_context.

​
Command map

CommandIncident scenarioFirst commandRequired confirmationStop conditionsRollback notes
list-webhook-eventsStripe billing webhook events are dead-lettered, stuck, or need audit review.python scripts/hilt_pay_api_ops_cli.py list-webhook-events --status dead_letter --limit 25None. Read-only.Stop if the operator cannot identify the Stripe event id or event type.None; read-only. Preserve output in the incident note.
inspect-webhook-eventA specific Stripe event needs failure context before replay.python scripts/hilt_pay_api_ops_cli.py inspect-webhook-event evt_123 --include-payloadNone. Read-only.Stop if payload id does not match the requested event id, owner metadata is missing, or event status is not dead_letter, received, or processing.None; read-only. Use redacted payload only in shared notes.
replay-webhook-eventA dead-lettered Stripe subscription event has corrected metadata/state and should feed the billing entitlement writer again.python scripts/hilt_pay_api_ops_cli.py replay-webhook-event evt_123 --reason "metadata fixed" --confirm-no-live-payment--confirm-no-live-payment is required.Stop if replay would create a payment, the payload is malformed, the event is not a dead letter, or owner/plan metadata is still missing.Replayed events only sync billing entitlement state. Roll back by running Stripe API reconciliation in dry-run and manually correcting the entitlement row through the approved writer path.
recover-stale-webhooksStripe events are stuck in received or processing after partial failure.python scripts/hilt_pay_api_ops_cli.py recover-stale-webhooks --reason "worker restart recovery" --min-age-seconds 900Apply mode requires --apply --confirm-no-live-payment.Stop if candidate age is below the incident threshold, if a worker may still be processing, or if more than the expected events appear.Dry-run first. Applied recovery replays stored events; if output is unexpected, stop automated retries and inspect each event id.
reconcile-stale-billingLocal Hilt Pay API billing entitlement rows are active/past_due past their period end.python scripts/hilt_pay_api_ops_cli.py reconcile-stale-billing --limit 50Apply mode requires --apply --confirm-expire-stale-billing.Stop if the stale row is under active support review, if period dates disagree with Stripe, or if the row is tied to a live payment dispute.Dry-run first. Applied mode expires local rows only; restore by syncing the verified Stripe or Hilt Pay API billing source back through the billing writer.
reconcile-stripe-apiLocal Stripe-backed plan entitlement diverges from Stripe subscription state.python scripts/hilt_pay_api_ops_cli.py reconcile-stripe-api --owner-id OWNER_UUID --limit 25Apply mode requires --apply --confirm-no-live-payment.Stop if Stripe metadata lacks hilt_owner_id or plan, if multiple current rows exist, or if Stripe fetch fails for the subscription.Dry-run first. Applied mode replaces local current entitlement state through the same writer; correct by replaying from Stripe after metadata is fixed.
list-stuck-idempotency-claimsA billing writer request may have crashed after claiming an idempotency key but before storing a response.python scripts/hilt_pay_api_ops_cli.py list-stuck-idempotency-claims --min-age-seconds 900 --limit 25None. Read-only.Stop if the writer may still be running, if the request hash cannot be matched, or if a side-effect row exists without a completed response.No release command exists in V0. Recovery requires incident review and an approved manual plan.
list-reconciliation-runsOperator needs persisted dry-run or apply history for an incident.python scripts/hilt_pay_api_ops_cli.py list-reconciliation-runs --scope stripe_api_billing_reconciliation --limit 25None. Read-only.Stop if the expected incident id or operator reason is absent from the run history.None; read-only. Preserve the run id in the incident note.
inspect-reconciliation-runOperator needs the exact summary and persisted operator context for one reconciliation run.python scripts/hilt_pay_api_ops_cli.py inspect-reconciliation-run RUN_UUIDNone. Read-only.Stop if the run owner, scope, or dry-run flag does not match the incident being reviewed.None; read-only.
rail-production-readinessOperator needs the current public-launch rail gate state.python scripts/hilt_pay_api_ops_cli.py rail-production-readinessNone. Read-only.Stop if any non-Solana registered/gated adapter is production-enabled before launch approval.None; read-only.
list-proof-contractsOperator needs verifier contract shape by rail.python scripts/hilt_pay_api_ops_cli.py list-proof-contracts --rail-id x402None. Read-only.Stop if the contract claims live verification, receipt mapping, or activation for non-Solana rails.None; read-only.
list-rail-proofsOperator needs recent proof backbone status across rails.python scripts/hilt_pay_api_ops_cli.py list-rail-proofs --rail-id base_usdc --limit 25None. Read-only.Stop if non-Solana proof state implies receipt creation or entitlement activation.None; read-only.
inspect-rail-proofOperator needs one proof’s redacted evidence and latest statuses.python scripts/hilt_pay_api_ops_cli.py inspect-rail-proof PROOF_UUIDNone. Read-only.Stop if owner, rail, proof type, or proof reference does not match the incident.None; read-only.
list-rail-proof-verification-attemptsOperator needs failed or repeated verifier attempts for one proof.python scripts/hilt_pay_api_ops_cli.py list-rail-proof-verification-attempts PROOF_UUIDNone. Read-only.Stop if attempts show owner mismatch, replay, unsupported rail, or unexpected live verifier use.None; read-only.
list-rail-proof-settlement-evidenceOperator needs settlement/finality evidence for one proof.python scripts/hilt_pay_api_ops_cli.py list-rail-proof-settlement-evidence PROOF_UUIDNone. Read-only.Stop if settlement is marked verified without a verified attempt or finality state.None; read-only.
dry-run-rail-proof-applyOperator needs to see whether current proof state would be blocked.python scripts/hilt_pay_api_ops_cli.py dry-run-rail-proof-apply PROOF_UUIDNone. Dry-run only.Stop if it reports would_activate_entitlement=true or would_apply=true for a non-Solana proof without the explicit receipt/apply gates and production rail approval.None; dry-run only.

​
Reconciliation run history

Use the staff endpoint or dashboard history section to inspect persisted dry-runs and reconciliation runs: 
GET /v1/admin/access/reconciliation-runs?limit=25
GET /v1/admin/access/reconciliation-runs?scope=stripe_api_billing_reconciliation
GET /v1/admin/access/reconciliation-runs/RUN_UUID
Each run includes scope, summary, dry_run, actor_type, actor_id, created_at, and optional operator_context.

​
Rail proof visibility

Use these commands when investigating x402, Base USDC, or EVM USDC proof evidence. They are read-only or dry-run only. 
python scripts/hilt_pay_api_ops_cli.py rail-production-readiness
python scripts/hilt_pay_api_ops_cli.py list-proof-contracts
python scripts/hilt_pay_api_ops_cli.py list-rail-proofs --limit 25
python scripts/hilt_pay_api_ops_cli.py inspect-rail-proof PROOF_UUID
python scripts/hilt_pay_api_ops_cli.py list-rail-proof-verification-attempts PROOF_UUID
python scripts/hilt_pay_api_ops_cli.py list-rail-proof-settlement-evidence PROOF_UUID
python scripts/hilt_pay_api_ops_cli.py dry-run-rail-proof-apply PROOF_UUID
Expected non-Solana proof lifecycle in private alpha:
  1. proof fingerprint exists
  2. verification attempt is verified, failed, pending, or requires_review
  3. settlement evidence is present only after verified evidence and finality
  4. receipt mapping remains blocked by default; gated non-Solana receipt creation is allowed only during controlled rehearsals or approved production rail launch
  5. activation check remains blocked for x402, base_usdc, and evm_usdc unless the explicit live-apply gate, rail allowlist, registry production state, and tuple-binding checks all pass
Incident triage:
  • Replay attempt: compare proof fingerprint, rail id, owner id, proof type, and verification attempts. Stop if a duplicate proof crosses owner boundaries.
  • Failed verification: inspect failure code and redacted evidence. Do not retry against live facilitator or live RPC from a public/user route.
  • Pending settlement: inspect finality state and confirmations. Do not mark settlement verified manually unless the offline verifier evidence already supports it.
  • Blocked receipt mapping: expected by default. Do not create a Hilt receipt by hand. Use only the staff-gated receipt creation path after verified settlement evidence and a fully bound payment/access product tuple.
  • Activation blocked: expected for non-Solana rails until approved launch gates are enabled. Do not activate an entitlement manually from proof evidence.
Operators must never manually set production_enabled=true, bypass merchant rail review, directly edit receipts, or activate live entitlements from non-Solana evidence. x402, base_usdc, and evm_usdc have live-shaped requirement adapters, but those adapters are not a production enablement approval.

​
Private-alpha release gates

Before production enablement, operators must complete and attach evidence for these gates:
  • Alert delivery drill: prove the selected alert backend delivers at least one Hilt Pay API private-alpha notification to the intended channel.
  • Kill-switch drill: prove a configured rail can be paused and that proof/session flows surface the blocked state without manual database edits.
  • Migration rehearsal: run migration 053 through 065 rehearsal on disposable and staging/scrubbed targets before any production migration window.
  • Solana end-to-end rehearsal: run the disposable DB test for live apply, mapped receipt evidence, billing evidence, revoke, audit, and rollback before any broader Solana controlled alpha.
  • Rate-limit and abuse posture: review proof submission, payment-session creation, webhook replay/recovery, and x402/API protected endpoints.
  • Rail provider/facilitator alerting: verify alert rules for proof verification failure, settlement/reorg, x402 facilitator failures, and unhealthy EVM provider evidence.
  • No production enablement: keep x402, base_usdc, evm_usdc, solana_usdt, erc20_usdt, and tron_usdt without production_enabled=true or non-Solana entitlement activation until the public launch gate is explicitly approved. Non-Solana receipt creation is rehearsal/launch-gated by its own environment flag and rail allowlist.

​
Production-shaped rehearsal

Before broad private-alpha operation, run the rehearsal against a disposable or staging-shaped database. This is read-only, but it still refuses to run unless the operator confirms the target is not production: 
export HILT_PAY_API_REHEARSAL_DATABASE_URL=postgresql://...
export HILT_PAY_API_REHEARSAL_CONFIRM_NON_PRODUCTION=1
export HILT_PAY_API_REHEARSAL_INCIDENT_ID=HILT-PAY-API-REHEARSAL-001
export HILT_PAY_API_REHEARSAL_OPERATOR_REASON="private-alpha production-shaped rehearsal"

python scripts/hilt_pay_api_private_alpha_rehearsal.py
The rehearsal checks:
  • migration preflight SQL in a read-only transaction
  • generated alert-rule integrity against the service-owned metric spec
  • operator CLI command availability for webhook, billing, idempotency, and reconciliation history flows
  • Solana controlled-alpha evidence markers for session rail, mapped receipt, entitlement activation, billing evidence, revoke, audit trail, alert rules, and transaction rollback
The report redacts database credentials and is safe to attach to an internal incident note. It does not run migrations, create payment sessions, enable non-Solana settlement, apply treasury wallets, or move money. For migration 053 through 065 rehearsal on a disposable database, use the migration preflight runner directly: 
export HILT_PAY_API_PREFLIGHT_DATABASE_URL=postgresql://localhost/..._disposable
export HILT_PAY_API_PREFLIGHT_CONFIRM_DISPOSABLE=1
export HILT_PAY_API_PREFLIGHT_APPLY_MIGRATIONS=1

python scripts/hilt_pay_api_migration_preflight.py
The runner refuses production-like DSNs, applies migrations only in explicit disposable rehearsal mode, then runs the preflight SQL in a read-only transaction. Stop if migration 058 proof tables, migration 059 activation-invariant checks, migration 060 USDT review scaffold checks, migration 061 receipt-mapping tuple checks, migration 062 platform rail/payment-requirement checks, migration 063 merchant fee collection checks, migration 064 Hilt Pay API pricing settings, or migration 065 agent bootstrap setup-intent checks fail, including indexes, uniqueness, FK shape, owner scoping, duplicate-risk checks, full activation binding, receipt/payment/access tuple binding, owner-isolation checks, control-plane table presence, merchant fee collection ledger presence, Base/EVM fee-collection readiness, Hilt Pay API Stripe price ID settings, Hilt Pay API plan config settings, setup-token storage, sandbox-key ownership, or any USDT scaffold rail appearing live. For the destructive disposable DB service-path rehearsal: 
export HILT_ACCESS_TEST_DATABASE_URL=postgresql://postgres:postgres@127.0.0.1:55439/postgres
export HILT_ACCESS_TEST_DATABASE_ALLOW_SCHEMA_DROP=1

python -m pytest tests/test_access_db_integration.py -q
This applies migrations 053 through 065 to a disposable schema and proves the Solana live-apply service path can activate a pending entitlement only from verified settlement and mapped receipt evidence. It also proves Hilt Pay API plan billing sync requires owner-bound payment and receipt evidence, receipt mappings bind to a single payment/access-product tuple, revocation writes an audit record, transaction rollback leaves the entitlement pending with no applied audit row, USDT child rails remain review-only scaffolds, platform rail/payment-requirement tables are present for the admin control plane, merchant-side fee collection tables/settings are present for the Base/EVM buyer-one-sign fee model, Hilt Pay API plan/Stripe price settings are present for admin-managed billing, and agent bootstrap setup intents are tracked as sandbox-first owner-approved records.

​
Scrubbed snapshot rehearsal

Use this when a production-shaped database snapshot has been restored into a non-production database and scrubbed. Do not run this against live production. 
export HILT_PAY_API_SNAPSHOT_REHEARSAL_DATABASE_URL=postgresql://...
export HILT_PAY_API_SNAPSHOT_REHEARSAL_CONFIRM_SCRUBBED=1

python scripts/hilt_pay_api_scrubbed_snapshot_rehearsal.py
The snapshot rehearsal adds value-redacting scrub checks before running the same preflight, alert, and CLI checks. It reports:
  • core Hilt table presence and row counts
  • sensitive-column inventory by table and column name only
  • scrub-risk counts for known email, settings, webhook signing secret, merchant settings, and Stripe payload locations
  • no customer emails, wallets, tokens, secrets, webhook payload values, or settings values
Treat any blocker as a hard stop. Regenerate or improve the snapshot scrub before repeating the rehearsal.

​
Stripe webhook dead letters

Use this when the internal Stripe webhook endpoint accepted a signed event but the billing entitlement writer rejected it or failed. Sequence:
  1. list dead letters with list-webhook-events --status dead_letter
  2. inspect one event with inspect-webhook-event EVENT_ID --include-payload
  3. fix the cause outside the replay path, such as missing owner metadata
  4. replay with replay-webhook-event EVENT_ID --reason "..." --confirm-no-live-payment
  5. run reconcile-stripe-api --owner-id OWNER_UUID in dry-run mode to confirm local state
Stop when the event payload is missing, owner metadata is missing, the subscription id is absent, or the replay would imply charging or creating a payment.

​
Stale webhook recovery

Use this when events are stuck in received or processing after a worker crash or partial failure. Start with dry-run: 
python scripts/hilt_pay_api_ops_cli.py recover-stale-webhooks \
  --reason "worker restart recovery" \
  --min-age-seconds 900 \
  --limit 25
Apply only after confirming no active worker still owns the event: 
python scripts/hilt_pay_api_ops_cli.py recover-stale-webhooks \
  --reason "worker restart recovery" \
  --min-age-seconds 900 \
  --limit 25 \
  --apply \
  --confirm-no-live-payment
Stop when the candidate count is unexpectedly high, the event is newer than the incident window, or the payload cannot be matched to the stored Stripe event id.

​
Stripe billing divergence

Use this when a local Hilt Pay API plan entitlement disagrees with Stripe subscription state. Start with dry-run: 
python scripts/hilt_pay_api_ops_cli.py reconcile-stripe-api \
  --owner-id OWNER_UUID \
  --limit 25
Apply only when Stripe is the verified source for that owner: 
python scripts/hilt_pay_api_ops_cli.py reconcile-stripe-api \
  --owner-id OWNER_UUID \
  --limit 25 \
  --apply \
  --confirm-no-live-payment
Stop when Stripe metadata is missing, multiple current local billing rows exist, or the issue is tied to an active dispute.

​
Stale billing rows

Use this when local plan entitlement rows remain active or past_due after current_period_end. Start with dry-run: 
python scripts/hilt_pay_api_ops_cli.py reconcile-stale-billing --owner-id OWNER_UUID
Apply only after verifying the row is not currently paid through Stripe or verified Hilt Pay API billing: 
python scripts/hilt_pay_api_ops_cli.py reconcile-stale-billing \
  --owner-id OWNER_UUID \
  --apply \
  --confirm-expire-stale-billing
Stop when a row is tied to an unresolved support ticket, a live dispute, or a Stripe subscription that still reports active entitlement.

​
Idempotency stuck claims

Use this when an operator sees idempotency_in_progress for a billing writer long after the original request should have completed. Read-only inspection: 
python scripts/hilt_pay_api_ops_cli.py list-stuck-idempotency-claims \
  --endpoint "POST /v1/admin/access/billing/stripe-sync" \
  --min-age-seconds 900
V0 intentionally does not provide a release/delete command. A future recovery endpoint must require dual approval and proof that no billing writer side effect is in flight. Stop when a billing entitlement row exists but the idempotency response is missing, when the request hash cannot be matched to the original request, or when a worker may still be running.

​
Future release endpoint design

The proposed release/apply endpoint is intentionally not enabled in V0: 
POST /v1/admin/access/idempotency/stuck-claims/{claim_id}/release
Required controls before implementation:
  • two-person approval with action billing_idempotency_stuck_claim_release
  • approval applied by a different admin than the requester
  • incident id and operator reason
  • side-effect proof showing whether the billing entitlement row was absent, completed, or reconciled
  • confirmation that no worker is still processing the claim
  • confirmation that no live payment, card charge, plan checkout, or on-chain settlement is created
  • audit record of the original claim fields, request hash, side-effect proof, and release result
Stop before release if the claim endpoint is not a billing writer endpoint, if the original request hash cannot be matched, if the side-effect row is ambiguous, or if recovery would depend on creating or charging a live payment.

​
Alert rules

The staff endpoint GET /v1/admin/access/observability/spec exposes the metric contract and structured alert rules. Operators should treat these as private-alpha definitions until wired into the selected production alert backend. Generated backend-ready private-alpha alert definitions live in: 
infra/alerts/hilt_pay_api_private_alpha_alert_rules.json
GCP Cloud Monitoring Terraform wiring can be generated from the same service-owned spec: 
python scripts/generate_hilt_pay_api_gcp_alerts.py
terraform -chdir=infra/gcp/terraform fmt
terraform -chdir=infra/gcp/terraform validate
The generated file is: 
infra/gcp/terraform/hilt_pay_api_alerts.generated.tf
The policies are disabled by default. Enabling them requires enable_hilt_pay_api_private_alpha_alerts = true, notification channel IDs, and explicit operator review. Do not apply alert Terraform from a worker room. Required alert classes:
  • pending entitlement age
  • webhook dead letters
  • rail disabled attempts
  • idempotency conflicts
  • owner isolation denials
  • entitlement check latency
  • setup readiness blocker spikes
  • rail required and no available rail spikes
  • billing entitlement ambiguity
  • wallet mismatch
  • approval queue age
  • pending rail review age
  • Stripe webhook dead letters
  • stale billing entitlements

​
Hard boundaries

Do not use these tools to:
  • create live Hilt plan checkout
  • enable Base, EVM, x402, or USDT live settlement
  • create non-Solana Hilt receipts outside the staff-gated receipt creation path and approved rail gates
  • activate live entitlements from x402, Base, or EVM evidence
  • bypass two-person approval for rail verification or treasury wallet changes
  • delete idempotency claims without an approved recovery design
  • run production migrations
  • make Hilt Pay API on-chain billing recurring automatically