Company
TeamResearch & Development
PublicationEngineering Blog

Infrastructure·14 min read

Introducing Transmitter

How OuterSignal delivers millions of merchant actions a day to an unreliable and unpredictable world.

Mario Hernandez Founding Engineer, Head of Backend & Infrastructure LinkedIn GitHub
Share

At OuterSignal, everybody is somebody. We research the customer behind every order, and merchants turn what we learn into action. When a shopper becomes a VIP, a merchant might tag them in Shopify, add them to a Klaviyo flow, sync them to a Meta audience, post an alert in Slack, update their HubSpot contact, or call their own webhook. Every one of those is a request to a system we don't control, and at any moment, some of those systems are failing.

Our first delivery system dealt with that the simplest way there is. It read every action from Kafka, in order, and sent it. When a send failed, it waited for the failure to clear and tried again. That design is easy to build and easy to reason about, and it works, right up until the thing at the front of the line can't be sent.

Here's what that looked like on September 21. A merchant's webhook started rejecting our requests with a 401. Nothing unusual: credentials expire, keys get rotated. But that one rejected record sat at the front of a Kafka partition, and our delivery system had exactly one answer to a record it couldn't send: wait. Before that partition moved again, it held up 20,028 records from up to 74 merchants who had done nothing wrong.

Figure 1 / One bad credentialIllustrative, not to scale. Play it, or drag the timeline

Before: one Kafka partition

Every merchant's records wait in one line, in order.

After: a virtual queue per credential

Each merchant's credential gets its own line.

0.0s

Each dot is a delivery for one merchant. Red dots belong to the merchant whose credential the provider started rejecting. On the left, everyone waits. On the right, only that merchant does.

One broken credential shouldn't be everyone's problem. That idea is why we built Transmitter, the system that now delivers every one of those requests. It's a Go service, running as four instances, that reads from Kafka and keeps its jobs in Postgres. Today it sends about 7 million updates a day across 25 kinds of action and 13 providers. This post covers why we built it, how it works, and what we learned when it met production.

Aren't queues supposed to be reliable?

Delivering one request is easy. Delivering millions of them, to 13 providers, for every merchant on the platform, means that at any moment something is broken. A provider has an outage or starts throttling us. A merchant revokes an API key. A webhook starts answering every request with an error. A provider changes an endpoint without warning. None of that is our fault, but how we respond decides whether one merchant's broken webhook is that merchant's problem or everyone's.

September 21 wasn't a freak accident. It was the design working as built. Kafka is excellent at moving events, but an ordered partition can't skip the record at its front, so our first system waited. When a record hit a rate limit or a retryable error, the system paused the whole partition for up to five minutes. A record rejected for bad credentials could hold it for up to six hours. Most credentials shared partitions, so every record behind the stuck one waited with it. In one week in September, the system paused partitions 82,180 times.

There was a second problem. A few merchants generate most of the load, in bursts: recomputing a segment, replaying an action, re-scoring every customer against new personas. Those bulk jobs shared a line with the live changes that matter most, like a shopper's new order. One merchant's re-run produced 406,366 sends that took about three hours, and their live changes waited behind every one.

And we were slow on purpose. Our rate limits sat far below what providers allow. We sent Omnisend 15 requests a minute where it allows 400, so 10,000 updates took 11 hours. Its batch endpoint could have sent them in under a minute.

We needed a system with three properties:

  1. A failure stops only itself. A broken credential, a misconfigured action, or a failing provider must never delay another merchant's delivery.
  2. Live never waits for bulk. A shopper's own change never sits behind a recompute, whatever bulk work is running.
  3. We get the most out of every provider's limits, including its batch APIs.

The queue we actually want

Look closely at every failure above, and they share a shape: work for one credential waited behind work for another. A provider rate-limits a credential, a merchant revokes a credential, a webhook fails for a credential. The unit of failure is the credential, so the unit of isolation has to be the credential too.

The ideal design, then, is a queue for every credential every merchant connects. One stuck credential would stall only its own queue, and everything else would keep moving.

Built with real queues, that's a nightmare to run. Every merchant who connects a tool creates a new queue, and every disconnect leaves one to clean up. Kafka partitions are fixed when a topic is created and aren't meant to come and go as merchants connect tools. And the people on call would have to answer "why is this one queue stuck?" across a fleet of queues that changes every day.

Queue systems that create queues on demand solve only half the problem. We also need four things a queue doesn't do:

  • Replace, don't append. When a shopper's segment changes three times while a provider is down, only the last change matters. A queue delivers all three, in order. We need the newest to replace the rest.
  • Priority within a credential. A shopper's live change must jump ahead of the same credential's bulk backlog. A queue serves in arrival order.
  • Duplicates blocked forever. The same action must never be sent twice, even when it arrives again days later. Queues that deduplicate do it only for a short window.
  • Answers for the merchant. "How many updates are waiting, and when did the last one succeed?" A queue can tell you its depth, but not which merchant's work is stuck or why.

Each of those is a query against rows. What we wanted was the behavior of a queue per credential with the power of a database: virtual queues.

Figure 2 / How an action travelsIllustrative, not to scale. Hover or tap a stage
Stage
Follow a credential

Illustrative: four instances, as in production. Eight partitions and three credentials stand in for many. Arrows show where an action goes next. Labels are not our schema.

The database as a queue

Transmitter builds those virtual queues in Postgres. Kafka still carries every action to Transmitter, but once an action arrives, it becomes a row in a delivery ledger, and the row is the job. A credential's queue is simply its rows.

That one decision does a lot of work:

  • A queue per credential costs nothing. A new credential is a new value in a column, not new infrastructure.
  • Duplicates stop at the door. Each row has a delivery key. A second copy of the same action finds its row already there and writes nothing.
  • A crash loses nothing. A job's state lives in the row, not in a process's memory.

Every one of those guarantees is a transaction. Queuing a job and blocking its duplicate is one insert. Replacing a stale state write with a newer one is one update. Claiming a job and proving you still own it happen together, or not at all. A queue would give us each of these as a separate step, with a gap between them where things go wrong.

"Don't use Postgres as a queue" is good advice when many workers fight over the same rows. Ours never do. Every action's Kafka key is its credential, so all of a credential's work lands on one partition, and Kafka gives each partition to exactly one instance. Only that owner claims the partition's rows. Inside an instance, each credential has its own live lane and bulk lane, and each lane reads only its own queued rows, oldest first. A parked credential's rows wait for a retry time that hasn't come yet, so its lane finds nothing to claim, and the lanes beside it keep sending.

Figure 3 / The ledger as virtual queuesStep through it, or use the arrow keys

CredentialDelivery keyActionValueStateClaimed by

Partition owner epoch:

Simplified. Each row is one job in the delivery ledger, and a credential's rows are its queue. Labels are illustrative, not our schema.

One owner per partition works only if handing a partition over is safe. Kafka moves partitions between instances during deploys and failures, and when ownership moves, the old owner might still be mid-send. So every ownership change raises an epoch number, and every claim and settle checks it in the same transaction. A write from a stale owner changes nothing. We call this fencing, and it's why a partition can move in under a second without a job being sent twice.

Two kinds of jobs

A job in Transmitter is one action for one shopper: tag this customer in Shopify, post this Slack alert, send this webhook event. Jobs come in two kinds, and they need opposite handling.

A state write says what a provider should hold: a tag, a profile field, membership in a list or an audience. Sending it twice does no harm, and only the latest value matters. If a shopper's segment changes three times while a provider is down, we send only the last one.

A one-off send happens once: an alert, an email, a ticket, a webhook event. A second copy is a duplicate the receiver sees. These go out at most once, and every webhook request carries an idempotency key so the receiver can tell a retry from a new event.

Treating the two differently is what lets Transmitter be aggressive about retries without ever spamming a merchant's Slack channel.

Figure 4 / Replace, don't appendStep through it, or use the arrow keys

A plain queue

Every change is appended and sent in order.

The delivery ledger

The newest state write replaces the older one.

Illustrative: one shopper's segment changes three times while the provider is down. State writes collapse to the newest value. One-off sends, such as alerts and webhook events, are never collapsed, and each goes out at most once.

Spending each credential's budget

Every action is labeled where it starts. A change from a shopper's own activity goes to the live lane. Recomputes, backfills, replays and re-scores go to the bulk lane. The service that produces the change sets the label, so a bulk job can't sneak into the live lane by accident. And if live work itself floods, the excess moves to the bulk lane, so even a burst of live changes can't starve the line behind it.

One instance owns a credential's live and bulk lanes together, so it decides in one place how to spend that credential's budget. Every provider gets a profile: its rate windows, how many requests it allows in flight, and the batch sizes its API supports. Transmitter paces each credential against that profile and honors a provider's Retry-After header. Where a provider has a batch API, we use it: Shopify and Meta audiences take up to 10,000 rows per job, Emarsys 1,000 contacts per call, and HubSpot 100 contacts per upsert.

When things break

When a provider rejects a credential, Transmitter parks that credential's lane. Its jobs stay queued, the lane retries on a growing backoff, and every other credential keeps sending. This is September 21 again, with the opposite ending. If the rejection lasts an hour, the merchant gets an email saying what stopped, why, and how to fix it.

Transmitter also tells an outage apart from a contract change. An outage is temporary, and an identical retry will eventually succeed. A contract change is permanent: the provider changed something, and the same request will fail forever. On October 15, Klaviyo changed an endpoint we depended on. Within seven minutes, Transmitter had stopped retrying across 143 merchants, instead of hammering a call that could never succeed.

The merchant sees all of it. Every action in the app shows its own delivery status: delivering or failing, why, since when, how many updates are waiting, and when it last succeeded. Transmitter's senders write each credential's health the moment it changes, and the app reads that one place directly, not a history spread across logs.

Figure 5 / A credential's lifecycleSelect a state or a numbered transition, or use the arrow keys

Red: the credential is failing. Dashed: an action, not a state.

Each credential moves through these states on its own. While one credential is parked, retrying or halted, every other credential keeps sending.

In production

Before Transmitter touched production, we load-tested it against a copy of the production ledger. Every pass bar was set before the run, and every check had to fail on a build we knew was broken. Then we moved providers onto Transmitter one at a time, and ten days after the first commit, the old system was deleted.

Transmitter is built for 2,000 actions a second, about 172 million a day, and our load tests check it every day as the code changes.

Real traffic is already climbing toward it. On November 5, Transmitter delivered 20.6 million updates in a single day, peaking at 1,043 a second. That's about three times what the old system delivered in a week. That day used 12% of daily capacity, and its busiest second hit about half the design rate.

Figure 6 / Daily loadMeasured, October 8 to November 13

Updates delivered per day

Every delivery on Transmitter, after the old system was deleted on October 7.

Busiest day vs. design capacity

20.6M

updates on November 5, peaking at 1,043 a second

Daily volume20.6M of 172.8M

0172.8M a day

Peak rate1,043 of 2,000

02,000 a second

Transmitter is built for 2,000 actions a second, about 172.8 million a day. Its busiest day used of the daily capacity. At its peak that day, it ran at of the designed rate.

Show the data as a table
Day (UTC)Updates delivered

Source: Transmitter's delivery counter (deliveries with outcome "sent"), summed per UTC day. Capacity is the design target, checked by daily load tests.

What we got wrong

Our first version of parking had a deadline. If a credential stayed rejected for 24 hours, Transmitter dropped its queued jobs, on the theory that they were stale. That theory was wrong. A merchant who fixes a key on day two still expects their customers to be updated, and a state write is never really stale: it's just late. We removed the deadline. Today a parked credential's jobs wait as long as it takes.

We also found a failure mode the design hadn't anticipated. A merchant can lose access to OuterSignal while their jobs are still queued, and those jobs shouldn't go out. Now every job checks that the merchant's account is active before any provider call. When the merchant comes back, the work they missed is replayed.

Something is always broken

Every provider Transmitter talks to fails sometimes, and on most days at least one of them is failing right now. Transmitter doesn't try to prevent that. It decides who a failure belongs to: the one credential that caused it, and nobody else.

The idea carries over to any system that does work against services it doesn't control. Give each thing that can fail its own line. Keep the lines somewhere you can query. And tell an outage, which a retry will eventually get past, apart from a contract change, which no retry will.

Two problems come next. The first is scale. Our busiest day used 12% of daily capacity, but its busiest second already hit half the design rate, and we're about to onboard dozens, and eventually hundreds, of the largest commerce brands. Do we partition delivery, and the database under it, by merchant to spread the load, or is there more to get out of a single database? We're load-testing the next version of the delivery ledger to find out.

The second is retention. Old delivery records pile up in the ledger long after anyone needs them there. We're building a system that moves them to cold storage automatically, so the database holds only the work that still matters.

If those sound like problems you'd want to own, we're hiring backend and infrastructure engineers.

Share this post
CompanyOuterSignal
TeamResearch & Development
PublicationResearch BlogComing soon
PublicationEngineering BlogAll posts →