Skip to main content

Runbook: Crypto Payment Incidents

This runbook covers the manual procedures for crypto payment edge cases that require human intervention. Automated handling covers the happy path; the scenarios below are ones that fall out of automation.

Partial Payment (partially_paid)​

What happened​

The customer sent less crypto than the invoice required. This can happen because:

  • The customer deducted network/gas fees from the sent amount.
  • Exchange rate moved between invoice creation and payment.
  • The customer fat-fingered the amount.

Signals​

  • Outbox kind: nowpayments_partial_payment
  • Ledgerline admin notification email (sent via outbox email_send)
  • No order confirmation has been sent to the customer

Decision tree​

  1. Check the shortfall. Pull the outbox message payload (paymentId, paidAmount, expectedAmount, payCurrency). Calculate shortfall: expectedAmount - paidAmount.

  2. If shortfall < $1.00 USD equivalent → this is almost certainly a rounding/fee artifact. Manually mark the order paid in AdminJS (open the order, set status to paid_pending_fulfillment). Log the action in the Decisions log.

  3. If shortfall >= $1.00 → contact the customer. Options:

    • Ask them to send the remaining amount to the same address (NOWPayments will accept it if the invoice is still open).
    • Issue a partial refund and cancel the order.
    • Issue store credit for the paid amount.

  4. If the invoice has expired → the payment window is closed. Contact the customer and offer a new invoice or refund via their original payment method.

Actions​

  • Mark order paid (AdminJS): Orders → find order by orderRef → change status.
  • Issue refund: NOWPayments does not support automated refunds. Contact NOWPayments support or manually transfer the customer's crypto back from the NOWPayments dashboard.

Failed / Expired Payment​

What happened​

Payment was not received before the invoice expired, or NOWPayments explicitly marked it failed.

Signals​

  • Outbox kind: nowpayments_mark_failed
  • Order remains in payment_pending status.

Procedure​

  1. Confirm the order is in payment_pending status (AdminJS or DB).
  2. Check if the customer should be notified — the outbox nowpayments_mark_failed handler should have sent a notification. If not, send one manually.
  3. The inventory was not released at this point. Once the order status transitions to failed, inventory should be released. Confirm this happened by checking inventory levels.
  4. If the customer contacts support wanting to retry, create a new invoice via the store checkout flow.

Webhook Not Being Received​

Symptoms​

Payment shows as completed on NOWPayments dashboard but Ledgerline did not update the order.

Checklist​

  1. Verify IPN URL is set in NOWPayments Dashboard → IPN Settings → Callback URL. Should be: https://<your-render-url>/api/nowpayments/webhook
  2. Check Render logs for POST /api/nowpayments/webhook entries around the time of the payment.
  3. Check for 401 responses: These mean the HMAC-SHA512 signature failed. Verify NOWPAYMENTS_IPN_SECRET in Render matches the IPN secret in the NOWPayments dashboard exactly (no trailing spaces).
  4. Check for 500 responses: Could mean NOWPAYMENTS_IPN_SECRET is not set (and IS_PRODUCTION=true), or the outbox enqueue failed.
  5. Replay from NOWPayments: The NOWPayments dashboard lets you re-send IPN notifications. Use this to trigger a replay once the issue is resolved.

Manual payment marking​

If the webhook cannot be replayed or is taking too long:

  1. In AdminJS, find the order by orderRef (the order_id from the NOWPayments payment).
  2. Verify the payment on the NOWPayments dashboard (payment_id, actually_paid vs pay_amount).
  3. Manually set order status to paid_pending_fulfillment.
  4. Log the manual action and the NOWPayments payment_id in the order notes.

Signature Verification Failures (401 flood)​

Symptoms​

Render logs show a flood of nowpayments-webhook: rejected — invalid HMAC-SHA512 signature at 401.

Possible causes​

  1. IPN secret mismatch — the most common cause. Compare NOWPAYMENTS_IPN_SECRET in Render with the current value in NOWPayments Dashboard.
  2. NOWPayments secret rotation — if someone regenerated the IPN secret in the dashboard, update Render immediately.
  3. Scan / probe traffic — attackers probing the webhook endpoint. The rate limiter (default 120 req/min) will throttle these.

Resolution​

  1. Regenerate the IPN secret in NOWPayments if compromised.
  2. Update NOWPAYMENTS_IPN_SECRET in Render and redeploy.
  3. Verify IPN delivery resumes by checking Render logs.

Currency No Longer Available​

Symptoms​

listAvailableCurrencies() returns a shorter list than expected. A coin the tenant was using is no longer in the response.

Cause​

NOWPayments occasionally suspends coins for maintenance or regulatory reasons.

Procedure​

  1. Verify on the NOWPayments status page or dashboard.
  2. If the coin is temporarily suspended: the existing invoices may still be payable once the coin is re-enabled. Monitor and re-enable in Ledgerline config when available.
  3. If the coin is permanently removed: update tenant configuration to remove it from the supported payment methods list. Notify affected customers.
  4. The currency cache TTL is 24 h. Once the coin comes back, wait up to 24 h or manually clear the Valkey key nowpayments:currencies:v1 to force a refresh.

Clearing the currency cache​

# Via Render shell or local dev with VALKEY_URL set:
redis-cli -u $VALKEY_URL DEL nowpayments:currencies:v1
Last updated on by renovate[bot]