Webhooks (Asynchronous Webhooks)

1. Concept Overview & Systemic Problem

Traditional system integration through constant polling (Polling / Short Polling) suffers from two mutually exclusive problems:

1. High Latency: If the client checks for changes every 60 seconds, data is delivered with an average delay of 30 seconds.
2. Massive Wastage of Computational Resources: 99% of HTTP requests return an empty response 304 Not Modified or [], overloading the database and network stack.

Webhooks (HTTP Callbacks) shift the paradigm from Pull to Push. Instead of asking, "Are there new data?", the consumer registers their URL (Webhook Listener). Once an event occurs in the provider's system (successful transaction in Stripe, new commit in GitHub, incoming message in Telegram), the provider sends an HTTP POST request with a JSON payload directly to the subscriber's server.

Traditional Polling (Resource Wastage):
Client  ---> GET /events?since=... ---> Server (Empty: [])
Client  ---> GET /events?since=... ---> Server (Empty: [])
Client  ---> GET /events?since=... ---> Server (New event!)

Event-Driven Webhook (Instant Push):
Event Provider (Stripe / GitHub / Telegram)
       |
       |  Event: charge.succeeded (Event Payload + HMAC signature)
       |  HTTP POST https://api.mysite.com/webhooks/stripe
       v
Consumer (Signature Validation -> 200 OK -> Background Queue RabbitMQ/Redis)

2. Architectural Taxonomy & Mental Model

The webhook lifecycle relies on three architectural pillars:

1. Cryptographic Authentication (HMAC Signature):
   • The provider generates a cryptographic hash of the request body (raw body) using a shared secret key (Webhook Secret), adding it to the header (e.g., Stripe-Signature or X-Hub-Signature-256).
   • The consumer computes the same hash and validates it, eliminating the risk of request forgery by an attacker.
2. Idempotency and Deduplication (Idempotency Engine):
   • Each event has a globally unique ID (evt_xxxxxxxxxx).
   • The consumer atomically checks for the ID in the cache/DB (e.g., via INSERT ... ON CONFLICT DO NOTHING or Redis SET key NX EX 86400).
3. Guaranteed Delivery and Retry Policy (Retry & Dead-Letter Queue):
   • If the consumer's server is unavailable, the provider retries requests on an exponential scale (Exponential Backoff with jitter): after 5 minutes, 30 minutes, 2 hours, 12 hours.
   • Events that fail to process after the maximum number of attempts are sent to a Dead-Letter Queue (DLQ) for manual analysis by engineers.

3. Technical Pipeline & Internal Mechanics

Production Handler with HMAC Verification (Node.js / Express / TypeScript)

[!IMPORTANT] To verify the signature, the raw request body (raw buffer) must be obtained, not the parsed req.body object; otherwise, the hash will not match due to whitespace discrepancies and JSON key sorting.

import express, { Request, Response } from "express";
import crypto from "crypto";

const app = express();

// Getting raw buffer for webhook endpoints
app.post(
  "/api/v1/stripe-webhook",
  express.raw({ type: "application/json" }),
  async (req: Request, res: Response) => {
    const signature = req.headers["stripe-signature"] as string;
    const secret = process.env.STRIPE_WEBHOOK_SECRET!;

    if (!signature) {
      return res.status(400).send("Missing signature header");
    }

    // 1. Cryptographic signature validation via HMAC SHA-256
    const computedHmac = crypto
      .createHmac("sha256", secret)
      .update(req.body)
      .digest("hex");

    // Safe comparison in constant time
    const isValid = crypto.timingSafeEqual(
      Buffer.from(signature),
      Buffer.from(computedHmac)
    );

    if (!isValid) {
      return res.status(401).send("Invalid signature");
    }

    const payload = JSON.parse(req.body.toString());
    const eventId = payload.id;

    // 2. Idempotent deduplication check
    const isNew = await redis.set(`webhook:processed:${eventId}`, "1", "NX", "EX", 86400);
    if (!isNew) {
      // Event already processed — return 200 OK without re-execution
      return res.status(200).json({ status: "already_processed" });
    }

    // 3. Sending heavy work to BullMQ queue
    await eventQueue.add("process_payment", payload);

    // 4. Instant acknowledgment of receipt
    return res.status(200).json({ received: true });
  }
);

4. Production Engineering Scenarios

01. Payment Gateway for AI SaaS (Stripe / Paddle / LiqPay)

A user subscribes to the Pro plan. The payment system generates the event invoice.payment_succeeded. The webhook instantly updates the user's status in the database, resets token limits in Redis, and sends a welcome email, allowing the user to continue working without reloading the page.

02. Triggering an Autonomous CI/CD Pipeline (GitHub Webhooks)

An engineer performs a git push to the main branch. GitHub sends a payload with the commit hash to the Coolify or ArgoCD server. The server verifies the secret, clones the modified code, runs unit tests in an isolated Docker container, and updates the production service using a Blue/Green strategy.

03. Asynchronous Collection of Inference Results from Heavy AI Models

When invoking video generation models (Runway, Kling) or training LoRA, the request takes several minutes. Instead of keeping an open HTTP connection, the client provides a webhook_url. Once the GPU cluster completes the generation, it sends a webhook with the final S3 link to the generated media.

5. Pitfalls, Common Mistakes & Security

1. Parsing JSON Before Signature Verification: If the standard middleware express.json() runs before signature verification, the serializer may alter the field order, escape Unicode characters, or format numbers, leading to a mismatch in the HMAC checksum and an Invalid Signature error.
2. Synchronous Execution Blocking (Webhook Deadlock): Performing heavy transactions or long HTTP requests to external APIs within the webhook handler will inevitably lead to a timeout on the provider's side. Return 200 OK immediately after signature validation and enqueue the work.
3. Lack of Timestamp Validation (Replay Attacks): An attacker who intercepts a legitimate signed webhook request may resend it after an hour. Providers include a timestamp in the signature. Ensure that the difference between the request time and the server's system clock does not exceed an acceptable window (e.g., 5 minutes).