Webhooks, Layer by Layer: From “Something Happened” to GitHub and CircleCI on the Wire
From the sentence you’d use in a design review down to TLS, signatures, and retries—then GitHub firing CircleCI and CircleCI talking back to GitHub, as two different kinds of integration, grounded in a workshop you can code along with.
Aymen Chikeb
4/25/2026
Two systems that do not share a database still need to move in step. One of them holds the truth about what just happened; the other needs to react. A webhook is the narrow bridge between them: when the truth changes, the holder opens an HTTPS connection to a URL you chose earlier and leaves a structured message on your doorstep.
This article walks that bridge from the outside in—like peeling an onion—until we reach something concrete: GitHub nudging CircleCI to run a pipeline, then CircleCI updating GitHub so the commit shows green or red. Same product surface, two different directions on the wire.
The roles, trade-offs, and reference shapes below mirror my workshop Webhook – Event Architecture — registry, signer, dispatcher, idempotency exercises, and the diagrams in diagram/. If you prefer learning by typing the seams yourself, start there.
Layer 1: The surface
Here is the whole idea in one breath: an event occurs somewhere authoritative; that system POSTs a payload to your HTTP endpoint. You pre-registered the URL. You did not ask in a loop; the producer chose the moment.
That already sketches a division of labor. Time belongs to the producer—it knows when state changed. Space belongs to you—you own the URL, the process, the scaling model. The catchphrase people use—“don’t call us, we’ll call you”—is not marketing; it is a literal description of who initiates the TCP session.
Contrast that with the alternative you still see in older integrations: your worker wakes up on a cron, calls someone’s API, compares timestamps or cursors, and usually learns nothing changed. Cost scales with wall clock, not with business events. Webhooks invert that: cost tracks change. When nothing happens upstream, your servers stay quiet.
Layer 2: The product contract
Under the friendly URL hides a bilateral agreement.
The producer tends to guarantee delivery attempts, a stable schema (or versioned evolution), and often retries with backoff when you return 5xx or time out. It rarely guarantees exactly once. Assume at least once and sleep well.
The subscriber guarantees a reachable endpoint, honest status codes—2xx when the message is accepted for processing, not when every downstream saga finished—and idempotent handling, because the same logical event can land twice. If the platform signs deliveries, you verify before you trust the JSON.
Why not “just use the public API”? An API is a language: query, mutate, paginate, negotiate auth. A webhook is a single tense—past: “this already happened.” It is deliberately narrow. The workshop README states the architectural payoff the same way: notify external subscribers without teaching your payment or VCS core about every partner’s stack or headcount—open subscription instead of a growing list of imports inside domain code (workshop README).
Layer 3: The architecture inside your system
When you ship the platform that emits webhooks, that narrow channel deserves its own module boundary. The workshop names the pieces explicitly:
| Role | Responsibility |
|---|---|
| WebhookRegistry | eventType → [(subscriberUrl, secret)]: lifecycle of subscriptions |
| WebhookSigner | e.g. HMAC-SHA256(secret, rawBody) so recipients can prove it was you |
| WebhookDispatcher | Resolve subscribers, sign, POST, record outcomes, apply retry policy |
| WebhookEvent | eventId, eventType, createdAt, data: something you can log, replay, and dedupe |
| WebhookSubscriber | A row in that registry: who cares about which event, with what secret |
| WebhookHandler (consumer side) | supports(eventType) + handle(event): small, testable units |
That table is the Observer pattern with a WAN between subject and observer: the registry is the observer list; dispatch(event) is notifyObservers(); each HTTPS endpoint is an observer that happens to live in someone else’s VPC (workshop README).
Fan-out, cryptography, and retry semantics are cross-cutting. They rot quickly if they leak into PaymentCaptured or RepositoryPushReceived. The workshop’s “why it exists” list is the smell test: partners you do not control, partners you have not met yet, outages that should not become code edits, and trust at the boundary. That list belongs in one place.
Layer 4: The protocol on the wire
Strip the names and you are left with mechanics anyone with curl can reason about.
- Verb: Modern integrations overwhelmingly use POST with a JSON body. (You may still see GET-with-query legacy hooks; treat them as exceptions.)
- Body: The event, serialized. For HMAC verification, the bytes on the wire are canonical. Re-serializing from a parsed object is a classic way to verify yourself into
401hell. - Headers: Delivery id, event type, signature (on GitHub,
X-Hub-Signature-256), sometimes a timestamp for anti-replay. - Response: Return 2xx when you have taken custody of the message: validated, persisted, or enqueued. Heavy work belongs behind that line, not inside the request thread that the producer’s delivery worker is blocking on.
GitHub’s implementation is a clean reference: rich JSON, explicit delivery identifiers you can key idempotency on, and cryptographic proof tied to the shared secret you configured when you created the hook.
Layer 5: Production reality
This is the layer where diagrams meet pager noise.
Duplicates follow inevitably from retries. The workshop is blunt about it: treat eventId (or the vendor’s delivery id) as part of your domain logic, not as logging trivia (consequences).
Order is not a global invariant across subscribers or even across consecutive deliveries to one subscriber. If ordering matters, encode it in the payload—monotonic counters, vector clocks, or domain-specific version fields—not in your hope that the network preserves arrival order.
Latency cuts both ways. A thirty-second handler makes their delivery agent wait thirty seconds. Production dispatchers acknowledge fast and hand off to a queue; a pool of workers performs outbound HTTP. The workshop README warns the same: do not let a slow consumer pin your dispatcher thread.
Secrets age. Rotation is a joint maneuver: issue new material, dual-verify during overlap, retire old MAC keys. Signing is worthless if the secret lives in a Slack channel.
Queues—SQS, Kafka, NATS, whatever fits your estate—sit at this layer so often that “webhook endpoint” and “message producer” become the same small, boring function: verify, enqueue, return 200.
Layer 6: GitHub and CircleCI end to end
Most confusion I see in interviews is not “what is a webhook?” It is forgetting that CI uses two different channels to complete one user-visible story.
One timeline, two channels
GitHub → CircleCI: the hook fires the work
You push; GitHub must tell CircleCI that the graph changed and a pipeline may be warranted. GitHub plays producer; CircleCI is subscriber. The vehicle is an HTTP POST carrying a signed JSON document describing the repository event you subscribed to.
Read that carefully: it is an event, not a dump of your entire monorepo. Authentication is shared secret + HMAC; ignore unsigned bodies. Idempotency matters because redeliveries and overlapping events (multiple hooks, retried deliveries) are normal, not accidents.
The workshop files that under “CI triggers are webhooks: the host POSTs JSON to CircleCI / Jenkins / your ingress” (Where you meet webhooks in the wild — workshop README).
CircleCI → GitHub: the API paints the outcome
When the job finishes, you want a check on the commit. That path is not “another webhook from GitHub into CircleCI.” It is CircleCI, holding a token, calling GitHub’s REST or GraphQL API to create or update commit status or a check run.
Different direction, different auth model, different failure semantics: the build may have succeeded while the status call failed—you now have a visibility incident, not a compile incident. Retry and monitor that leg independently.
GitHub may then emit new webhook events (status, check_run) to other systems watching your repo. Fan-out continues; the pattern repeats.
The README names the split in one line worth memorizing: push notification (webhook) versus status reporting (REST callback) (same section in the workshop README).
Where to go next
- Code it:
architectures/webhook
—
context/for scenarios,before/versusafter/for the polling smell,diagram/for figures,implementation/if you want empty seams to fill in. Read the vendor truth: GitHub’s webhook documentation and delivery headers; CircleCI’s docs on how projects connect to GitHub. Nothing beats reconciling marketing screenshots with what actually hits your ingress logs.
Build seriously: Re-read the workshop consequences until idempotency, async fan-out, and secret rotation feel like a checklist, not a blog paragraph.
Webhooks are not a special protocol. They are HTTP used with discipline: a producer that initiates, a subscriber that verifies and admits work, and operations that assume duplication, delay, and partial failure are normal. Once you see GitHub and CircleCI as two arrows—inbound hook to start, outbound API to report—the same diagram applies everywhere else Stripe, Twilio, or your own billing service shows up.
