Incident Response

Procedures that exist as real endpoints/artifacts — plus manual gaps. Do not blindly retry a refund after a timeout: the PG operation may already have succeeded. Confirm provider state and internal records first.

Recon runbook (integrity drift)

POST /api/v1/admin/reconciliation/run      # admin
GET  /api/v1/admin/reconciliation/reports  # list runs
GET  /api/v1/admin/reconciliation/report/:id

For each finding:

1. Read the finding type — the 7 checks are enumerated in Reconciliation.
2. auto_fixed items — stale inbox/outbox leases only; verify the follow-on effect (re-delivered? still failing?) and move on.
3. needs_review items — money/entitlement drift:
   • paid_no_entitlement → issue or attach the missing license, or refund.
   • unpaid_but_entitled → collect payment or revoke the license.
   • refunded_but_valid_entitlement → revoke the license (PATCH /licenses/:id {"status":"revoked"}).
   • subscription_expired_but_renewed → inspect the renewal order; restore or unwind.
   • duplicate_issuance_per_payment → keep one license, revoke the rest.
   • dead_letter/aged outbox → check OUTBOX_SINK_URL reachability, then let the lease-recovery path or a manual redrive handle it.
4. Record the decision — there is no auto-resolution workflow; findings stay needs_review until an operator acts (no SLA/owner — limitations §4.14–4.15).

Refund incident path

POST /api/v1/orders/:id/refund            # seller-owner
{ "amount_krw": 5000, "reason": "…" }     # amount optional → full remaining
GET  /api/v1/orders/:id/refund-attempts   # audit trail

• status:'succeeded' — done; ledger + license compensation applied.
• status:'failed' — read error_json (e.g. NO_PAYMENT_KEY, PAYMENT_KEY_INVALID, PG error); investigate the failure, verify with the PG whether a cancel occurred, and only then decide whether a new refund is permitted. Pre-recording alone does not guarantee safe retries.
• status:'needs_review' — the PG call's outcome was uncertain; the attempt was deliberately not marked refunded. Verify against the PG console, then compensate manually. There is no sweeper — this waits for you.

License compromise path

1. PATCH /api/v1/licenses/:id {"status":"revoked"} — effective on the next guarded call (live-row re-query).
2. GET /api/v1/security-events?lic_id= — look for replay (jti reuse), jkt anomalies, license_blocked bursts.
3. GET /api/v1/licenses/:id/bindings + /usage — enumerate bound devices and executions for the report.
4. If keys rotated as part of response: it is a hard cutover — in-flight entitlements die at 401 within 120 s; tell affected sellers to re-activate.

Payment rail incident

• Webhook storm / invalid signatures — bodies are never trusted; check TOSS webhook secret config before assuming an attack.
• 402 NOT_LIVE_PAYMENT surfacing — config drift: PAYMENT_LIVE_ENABLED or provider mismatch.
• 503 PAYMENT_PROVIDER_UNAVAILABLE — Portone is unimplemented; a request hit it anyway.

Escalation boundaries

• Real PG disputes/legal items are out of platform scope — external dependencies are tracked in limitations §2.
• Never run destructive ops (D1 reset, seed, rollback) without explicit authorization — charter-gated.

Financial-integrity incidents

Order of evidence: GET /api/v1/admin/financial-integrity/summary → exception detail (with event log) → recovery op lifecycle → audit_evidence rows → GET /api/v1/admin/audit/v2/verify (chain health).

Symptom	First moves
AUDIT_KEY_MISSING errors (500 on checkout/refund/settle/ops in prod)	NOUGAT_AUDIT_HMAC_KEY secret is not bound — audit v2 is failing closed by design. Restore the secret binding; do NOT downgrade the error. Non-prod emits nothing (no degraded rows).
ok:false on /admin/audit/v2/verify	first_mismatch names the first tampered/rewritten row; link_breaks counts broken explicit links; sequence_gaps counts burned sequences (crashed writer, not necessarily tampering). Preserve the DB — this is evidence, do not repair in place.
Run stays running / degraded	Check heartbeat_at; a stale run is reaped to failed by the next recon. degraded = some check ERROR/INCOMPLETE — partial coverage, not green.
≥3 consecutive failed/incomplete runs	alert:recon_failures evidence exists in audit_evidence (external paging is PLANNED) — treat as recon-outage incident.
Exception REOPENED after RESOLVED	The fingerprint re-appeared — the fix did not address the root cause; resume from TRIAGED, don't re-close.
Op FAILED during verify	Read verification_json.verification.note — it states the unmet expectation; fix the target state, then create a new op (FAILED is terminal).

Full surface: financial-integrity · reconciliation · refund · settlement · audit-evidence.