What are Webhooks May 2021 · Ian

If you are in technology you've likely heard of the term Webhooks. They have been growing in popularity (see the google trends image above) and are the most common way for web applications to passively speak to each other. Most, if not all of you, have probably been displayed data that was delivered by a Webhook behind the scenes.

While Webhooks have become more popular, I find that they are still not well understood. My goal with this post is to help with that. I am going to breakdown the specifics of Webhooks, the different parties involved, the pieces that go into Webhooks, why use webhooks and high level how they work. With a better understanding, more developers can see their value and implement their own solution.

What are Webhooks?

Webhooks are a way of sending data passively from one web application to another using HTTP. This is routinely done via a callback when some event occurs in a web service and it wants to notify other applications of the event. These other applications that want to be notified are referred to as subscribers or consumers. In order to receive webhook notifications, they need to provide a web URL where the webhook callback will be sent. Once an event occurs, the application will send out a webhook request via a POST HTTP call to the predetermined subscriber URL. Once the subscriber receives the webhook it would parse the data and do something with it internally.

Parties in a Webhook

There are two main parties in a the webhook ecosystem, let's go over them.

Provider

The Webhook Provider is the web application sending out webhook messages when some event occurs in their system. These events are defined by the Provider and need to be programatically triggered within their code (like below).

def send_order_created_event
  Webhook.send_event(:order, webhook_payload, webhook_endpoint_url)
end

Subscriber

The Webhook Subscriber is the entity that has subscribed to receive updates for an event on the Provider's application. In order to receive those updates, the Subscriber needs to give the Provider a URL where those Webhook messages can be received. The URL on the Subscriber's application is setup to receive messages asynchronously. The subscriber endpoint must always return a 200 response code or else the provider will have to assume that the message was not successfully sent.

Parts of a Webhook

The structure of Webhooks aren't overly complicated but are worth repeating as the terminology is non-standard.

Endpoint

Endpoints are the records that tell the Provider where to send the webhook messages. They will contain a URL that points to an endpoint on the Subscriber's web application. The Subscriber needs to build that endpoint into their software and it needs to be able to receive POST requests asynchronously. These endpoints need to always be on and should return a 200 response code no matter what.

Event

Events are the trigger that dictates when data should be sent to the subscribers. Examples of these are a user gets created or an order gets shipped. Whenever an event gets triggered, the Provider's web application will take the data that changed, package it up into a JSON payload and send it out to the Subscribers that have subscribed to the respective event. The Endpoint URL is how the Providers know where to send the data. Subscribers can choose what events they want to be updated on. When they receive the Webhook message they will consume it and do something with it.

Message

The Webhook Message is the payload of JSON data that gets sent over HTTP from the Provider to the Subscriber when an event gets triggered. The structure of these payloads should be consistent so the subscriber knows exactly how to consume it. This is what the payload looks like for HostedHooks (see below). Note that we send over the event type with it so the Subscriber knows which event got triggered and can dynamically do the right thing with the data.

{
    "type":"user.created",
    "version":"1.0",
    "created":"2021-04-14T21:07:06.072-04:00",
    "data": {
        "id":123123123,
        "note":"this is a test",
        "other_id":1231231123
    }
}

Why use Webhooks?

As software becomes more popular and businesses turn digital, applications need to speak with each other and share data. There are multiple ways to accomplish this, but Webhooks are one of the most efficient and easier ways to do it.

An alternative strategy is something called Polling, where the subscriber application will repeatedly (every x seconds/minutes/hours) make requests to the provider web application checking to see if any data has changed. This is obviously very inefficient as requests will be made when no data has changed.

For this reason, a push strategy (callbacks), like Webhooks are the more favorable and efficient strategy to sharing data between web applications.

Examples of Webhooks

Webhooks are especially helpful for platforms that want to programatically notify their users of when the state (data) of a record changes. One good example of this is when someone buys something on a Shopify e-commerce store. There are 3rd party applications (manufacturer, shippers, etc) that are associated with the store to help it operate and they need to be updated of certain events.

Let's say a customer buys a product on a Shopify store, that one purchase could trigger 100s of events between the order changing it's status (created, accepted, cancelled), shipments updating (airbills, tracking, delivery updates), payment changes (charges, refunds, chargedbacks) among many others. All of those events have data tied to them that has changed and should be sent to other web applications so they can fulfill some operational goal. The best way to accomplish this is using Webhooks.

That is just one example, here are some more:

• New Message Notifications
• Shipment Updates
• Background Process Completed
• Order State Changes
• Custom Events
• Billing State Changes

How do I build Webhooks?

Webhook architecture is not trivial, there is a significant amount of work that that goes into building a reliable and scalable solution. It's important your solution can reliably store events and asynchronously publish them to your subscribers. When dealing with independent web applications as your consumers, you will inevitably have to deal with downtime. It's important that your solution can handle those issues with error handling and retries. Another piece often overlooked is the experience that your Subscribers have when subscribing and integrating with your Webhook solution. Is it self-serve, can they retry sending events, can they monitor the logs, etc. A good webhook solution is a platform and should be built with all parties considered.

You could absolutely build all of this yourself (many have), but if you wanted to save time and money along with the longer term maintenance costs, you could use a Webhooks as a Service platform like HostedHooks. In a future post we'll dive into what we've built at HostedHooks so you can asses the Build vs Buy debate.

Illustration Credit

The images used in this article were created using Visual Creator Kit - Simple Figures Ver. and in collaboration with Bryan from Startup Illustrated where he visualizes start-up advice, quotes, and concepts.