← Blog & Resources

Don't Panic When a FX Settlement Notification Goes Missing — Building an Institution-Grade FX Reconciliation System with OnyxClearing Webhooks

English

为什么换汇成功通知会丢 OnyxClearing(曜石承兑)的换汇流程是异步的:你的系统发起一笔机构级USDT换汇请求,OnyxClearing 在毫秒级定锚后锁定汇率,随后链上结算、合规USD账本入账,最后通过 webhook 通知你的服务。任何一环——网络抖动、服务重启、回调端点 5xx——都可能导致通知丢失或重复

Published October 21, 2026 · ThisKard team

Don't Panic When a FX Settlement Notification Goes Missing — Building an Institution-Grade FX Reconciliation System with OnyxClearing Webhooks

Why Settlement Notifications Get Lost

OnyxClearing's FX conversion flow is asynchronous: your system initiates an institutional-grade USDT conversion request, OnyxClearing locks in the rate within milliseconds of pricing, then settles on-chain, credits your compliant USD ledger, and finally notifies your service via webhook. Any point in this chain—a network blip, a service restart, a 5xx from your callback endpoint—can cause a notification to be lost or delivered twice.

For enterprise customers, this is no small matter. Large stablecoin settlements routinely run six or seven figures, and a single unmatched transaction means finance has to manually dig through block explorers and internal ledgers at month-end close. The right approach: treat webhooks as an accelerator, not the source of truth, and back them up with scheduled reconciliation.

Overall Architecture: Three Lines of Defense

  1. Webhook real-time layer: receive events immediately when a conversion settles, and update local order status
  2. Signature verification layer: prevent forged callbacks from contaminating your compliant USD ledger
  3. Active reconciliation layer: periodically call the query endpoints on api.thiskard.com to pull recent settlement records and catch any missed orders

For institutions on the progressive KYB track, we recommend tailoring reconciliation frequency by KYB tier: whitelisted-tier clients can run every 5 minutes, while new-channel clients run every 15 minutes. This is also part of a black-swan risk isolation strategy—when something goes wrong, you can throttle or freeze the reconciliation pipeline for one client segment without affecting other tracks.

Step-by-Step Workflow

Step 1: Register your webhook endpoint

Configure the callback URL in the console at thiskard.com/developers and subscribe to three event types: fx.settled, fx.failed, and fx.flagged.

Step 2: Receive and verify signatures

Every callback includes an X-Onyx-Signature header (HMAC-SHA256). Always verify it—otherwise anyone can push fake events to your system.

Step 3: Handle idempotency

Events may be delivered more than once. Use event.id as an idempotency key and simply return 200 for anything already processed.

Step 4: Scheduled reconciliation as a safety net

Here's a complete, runnable Python implementation:

import hmac, hashlib, time, requests

API_BASE = "https://api.thiskard.com"
WEBHOOK_SECRET = "whsec_xxx"
API_KEY = "sk_live_xxx"

def verify_signature(payload_body: bytes, signature: str) -> bool:
    expected = hmac.new(
        WEBHOOK_SECRET.encode(), payload_body, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, signature)

processed_event_ids = set()  # Use Redis/DB in production

def handle_webhook(payload: dict, signature: str, raw: bytes):
    if not verify_signature(raw, signature):
        return {"status": "rejected"}, 401

    if payload["id"] in processed_event_ids:
        return {"status": "duplicate"}, 200  # Idempotency: ack duplicate events immediately

    event_type = payload["type"]
    fx = payload["data"]
    order_id = fx["order_id"]

    if event_type == "fx.settled":
        # Compare against internally expected amount, then write to compliant USD ledger
        mark_settled(order_id, fx["usd_amount"], fx["tx_hash"])
    elif event_type in ("fx.failed", "fx.flagged"):
        flag_for_review(order_id, event_type)

    processed_event_ids.add(payload["id"])
    return {"status": "ok"}, 200

def reconcile(since_minutes: int = 15):
    """Active reconciliation: pull settled orders within the window and backfill missed notifications"""
    headers = {"Authorization": f"Bearer {API_KEY}"}
    params = {
        "settled_since": int(time.time()) - since_minutes * 60,
        "limit": 200,
    }
    resp = requests.get(
        f"{API_BASE}/v1/fx/orders", headers=headers,
        params=params, timeout=10,
    )
    resp.raise_for_status()

    for order in resp.json()["data"]:
        local = get_local_order(order["order_id"])
        if local is None:
            continue  # Not initiated by this system, skip
        if local["status"] != "settled" and order["status"] == "settled":
            mark_settled(
                order["order_id"],
                order["usd_amount"], order["tx_hash"],
            )
            alert_ops(f"Reconciliation backfill: {order['order_id']}")

def mark_settled(order_id, usd_amount, tx_hash):
    # TODO: update status in a DB transaction + write audit log
    pass

def flag_for_review(order_id, event_type): pass
def get_local_order(order_id): return {"status": "pending"}
def alert_ops(msg): print("[ALERT]", msg)

if __name__ == "__main__":
    # Scheduled every 5 minutes via celery beat / cron
    reconcile(since_minutes=10)

Step 5: Amount verification and alerting

When backfilling orders through reconciliation, always cross-check three things: the locked rate, the USD credit amount, and the on-chain tx_hash. If the amount deviates from your local expectation beyond your slippage tolerance (we recommend configuring this at the basis-point level), do not auto-post it to the ledger—route it to manual review. This is exactly the boundary design in institutional risk-control hedging workflows: the system surfaces problems, humans make the final call.

Error Handling and Best Practices

  • Return 200 from callbacks quickly. Push heavy business logic to an async queue—otherwise OnyxClearing will treat the delivery as failed and retry, making the backlog worse
  • Guard against retry storms: OnyxClearing retries 5xx responses with exponential backoff. If your endpoint stays down, degrade to "reconciliation-only mode" first
  • Overlapping time windows: deliberately make the reconciliation query window larger than the schedule interval (e.g., a 10-minute window on a 5-minute schedule) to avoid missing orders at boundaries; the idempotency key makes duplicate processing harmless
  • Key isolation: store the webhook secret and API key separately, and use constant-time comparison for verification (see hmac.compare_digest above)
  • Audit trails: write all reconciliation backfills to immutable logs, and retain them for at least 180 days to meet progressive KYB audit requirements
  • Sandbox first: before going to production, simulate three scenarios in the sandbox at thiskard.com/developers—lost events, duplicate deliveries, and amount anomalies

Conclusion

Webhooks will get lost—but your ledger can't. Treat notifications as "hints" rather than "facts," back them up with active reconciliation, prevent forgery with signature verification, and prevent duplicates with idempotent design. Your FX pipeline will then meet institution-grade reliability standards—no matter how large the transaction, your compliant USD ledger stays accurate to the cent.

Ready to start building? Visit thiskard.com/developers for API docs, SDKs, and a sandbox environment.