# Designing a notification system for reliable delivery

> Design a notification system that handles retries, user preferences, quiet hours, provider failures, and delivery state without turning queues into a bottleneck.

- URL: https://ferdousazad.com/blog/notification-system-reliable-delivery
- Author: Ferdous Azad (Founder @ DataXgen AI)
- Published: 2026-11-13
- Updated: 2026-11-13
- Tags: distributed-systems, cloud-infrastructure, devops, security

## 1. Introduction

A notification system is easy to sketch and surprisingly easy to get wrong. The hard part isn’t sending an email or push message. It’s deciding what should be sent, keeping the user’s preferences intact, and recovering when a queue or provider fails without sending duplicates.

I’d design it as a stateful delivery workflow. The durable in-app notification is the source of truth; email, SMS, and push are delivery attempts that can be retried, delayed, or dropped according to policy. This distinction makes failures easier to reason about and gives product teams room to add channels without changing the event-producing services.

![Microsoft Push Notification Service](/uploads/2026/11/notification-server-ed8e521d.jpg)

*Image: Oshwah, [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0/), via [Wikimedia Commons](https://commons.wikimedia.org/w/index.php?curid=143041812)*

## Problem statement and requirements

Assume a product with a web app and mobile apps. Product services emit events such as `invoice.overdue`, `comment.mentioned`, or `shipment.delayed`. The system turns eligible events into notifications and delivers them through in-app, email, push, or SMS.

The numbers below are assumptions for design discussion, not a forecast. Assume 10 million registered users, 5 million notification events per day, and a burst rate around 20 times the average. That is roughly 58 events per second on average and about 1,200 per second during a burst. A single campaign to a large audience can create a bigger short-lived fanout than the normal event stream.

### Functional requirements

- Accept events from product services without making those services wait for channel providers.
- Apply user preferences, channel eligibility, and quiet hours.
- Store in-app notifications and expose APIs to list, read, and dismiss them.
- Send eligible notifications through one or more channels.
- Retry transient failures, record final outcomes, and prevent duplicate effects where possible.
- Support scheduled delivery and bulk announcements without letting either overwhelm transactional traffic.

### Non-functional requirements

- A product event should not disappear because a downstream provider is unavailable.
- Duplicate events and retries should not create duplicate inbox entries.
- Delivery should be observable by event, user, channel, and provider.
- User data and notification content should be protected, with access limited to the services that need it.
- Notification latency should be appropriate to the event. A security alert and a weekly digest don’t need the same delivery policy.

I wouldn’t promise exactly-once delivery across external providers. A timeout can happen after a provider accepts a request but before the system receives its response. The practical goal is idempotent processing inside our system, plus duplicate reduction where the provider supports idempotency keys.

## High-level architecture

```mermaid
flowchart LR
    A["Product services"] --> B["Transactional outbox"]
    B --> C["Event stream"]
    C --> D["Event validator and deduplicator"]
    D --> E["Preference and policy service"]
    E --> F["Notification planner"]
    F --> G["In-app notification store"]
    F --> H["Channel queues"]
    H --> I["Email workers"]
    H --> J["Push workers"]
    H --> K["SMS workers"]
    I --> L["External providers"]
    J --> L
    K --> L
    L --> M["Provider callbacks"]
    M --> N["Delivery status store"]
    G --> O["Web and mobile clients"]
    N --> P["Metrics and operations"]
```

The transactional outbox closes a common reliability gap. A product service writes its business change and an event record in the same database transaction. A separate publisher moves outbox records to the event stream. Without that step, the service can commit a change and crash before publishing its event.

The event stream decouples producers from notification processing. Consumers validate the event, deduplicate it, resolve policy, and create delivery work. Channel workers run independently so an email provider outage doesn’t block push notifications.

## Data model

Keep the business event, the user-facing notification, and the channel attempt as separate records. They have different lifetimes and different retry behavior.

| Record | Important fields | Purpose |
|---|---|---|
| Event | `event_id`, `type`, `created_at`, `payload_ref` | Deduplicate producer events |
| Notification | `notification_id`, `user_id`, `event_id`, `status`, `created_at` | Durable inbox item |
| Delivery | `notification_id`, `channel`, `status`, `attempt_count` | Track each channel independently |
| Preference | `user_id`, `category`, `channel`, `enabled` | User and category policy |
| Attempt | `delivery_id`, `provider_id`, `result`, `attempted_at` | Audit and troubleshoot sends |

Use a producer-supplied `event_id` as the idempotency key. A uniqueness constraint on the event and recipient combination can prevent duplicate inbox rows if a consumer replays an event. For notifications that legitimately fan out to multiple users, the recipient is part of the key.

A notification’s state should not double as its delivery state. For example, an inbox item can exist even when push delivery fails. Store channel outcomes separately, and let the client’s read state belong to the inbox record.

Keep large or sensitive payloads out of the event stream when possible. Events should carry the identifiers and data needed to plan delivery, or a reference to data that an authorized worker can fetch. This reduces accidental exposure in logs and replay tools.

## Key components

### Event intake and deduplication

Validate event type, schema version, required fields, and producer identity before processing. Reject malformed events to a dead-letter path with enough metadata to debug them, but avoid copying sensitive message content into the error record.

Deduplication should be durable. An in-memory cache can reduce repeated database checks, but it cannot be the correctness boundary because it expires and disappears on restart. Use a uniqueness constraint or durable idempotency store.

### Policy and preference evaluation

Preferences often include category-level controls, channel opt-outs, quiet hours, locale, and device tokens. Resolve policy close to notification planning, then store the decision or policy version used. Otherwise, a later preference change can make it difficult to explain why an old message was sent.

Define precedence explicitly. A reasonable policy is that legal or security-critical messages can follow a separate product-defined rule, while marketing messages require consent and honor opt-outs. The exact exceptions are a product and compliance decision, not a queue setting.

Quiet hours need a timezone. If a user has no valid timezone, choose and document a fallback rather than silently applying server time. Scheduled messages should be re-evaluated before dispatch so a user who opts out after scheduling does not receive a stale send.

### Notification planning and channel workers

The planner creates the durable inbox item and one delivery record per eligible channel. Workers consume channel-specific queues, render templates, call providers, and record results. Separate queues make it possible to set different concurrency limits and retry policies for email, push, and SMS.

Retry transient errors with backoff and jitter. Don’t retry permanent failures such as an invalid address indefinitely. Use a maximum attempt policy and a dead-letter queue, then expose an operator workflow for inspection or replay.

Provider callbacks can improve status accuracy, but they are not a substitute for local records. Verify callback signatures, make callback handling idempotent, and correlate each callback with the provider’s message identifier.

> [!WARNING]
> A provider timeout does not prove that a message was not sent. Before retrying, check whether the provider supports idempotency keys or status lookup. Otherwise, a retry may deliver a duplicate.

## Scaling & bottlenecks

At the assumed average of about 58 events per second, the event intake is not the main architectural challenge. Bursts, campaigns, and provider limits matter more. A single campaign to a million recipients creates a million delivery decisions even if the campaign itself is one event.

For large fanout, split audience selection from delivery. Produce recipient batches, then enqueue work at a controlled rate. Don’t place a million-recipient list in one queue message: it is hard to retry partially, can exceed message-size limits, and creates an awkward failure boundary.

Partition durable inbox data by user identifier so a user’s list and read-state operations remain localized. The exact database and partition strategy depend on query patterns and retention needs. Keep the API’s common query simple, typically “latest notifications for this user,” and avoid cross-user scans on the request path.

Apply backpressure at channel workers. If a provider allows only a limited request rate, the worker pool should respect that limit rather than pushing the excess into provider errors. Queue age is often more useful than queue depth alone: a growing backlog tells you whether a channel is falling behind and how stale delivery has become.

A notification API should page through results with a stable cursor, not fetch the user’s whole history. Set retention and archival rules for old delivery attempts; otherwise, operational records grow forever. Keep unread inbox items according to product needs, but avoid treating every provider attempt as permanent data.

## Trade-offs

| Choice | Benefit | Cost |
|---|---|---|
| Durable inbox first | Reliable in-app state | Extra write and storage |
| One queue per channel | Isolates failures and limits | More queue operations |
| Store rendered content | Stable audit and replay | More sensitive data at rest |
| Render at send time | Easier template updates | Output may differ on retry |
| Fan out on write | Fast recipient reads | Expensive large broadcasts |
| Fan out on read | Cheap broadcast creation | More work on client requests |

For most product notifications, I’d store the rendered or versioned content needed to explain what was sent. If content includes rapidly changing data, store a template version and the required immutable inputs. Rendering a retry with a newly changed template can otherwise produce a message different from the original attempt.

Fanout-on-write is a good default for individual events because it keeps inbox reads simple. For a broad announcement, a hybrid can work: store the campaign once and materialize per-user inbox entries in batches, or let the read path combine campaign and personal notifications. The second option saves writes but complicates pagination, unread counts, and deletion behavior.

## Interview tips

Start by clarifying what “delivered” means. Does it mean persisted in the inbox, accepted by a provider, or confirmed as displayed? Those are distinct states, and conflating them leads to misleading reliability claims.

Then ask about event volume, burst shape, channels, ordering, preferences, and campaign size. State assumptions before drawing boxes. At the assumed scale here, a queue and worker pool are enough as concepts; don’t add a fleet of specialized services until a requirement justifies the operational cost.

Walk through a failure. The provider accepts a push, the worker times out, and the queue redelivers the job. Explain where the idempotency key comes from, what state is recorded, and what happens if the provider cannot deduplicate. This tests the real design more than a diagram with many components.

> [!TIP]
> Treat in-app persistence and external delivery as separate outcomes. That one modeling choice makes retries, user-visible history, and incident investigation much easier to explain.

## What I’d do

I’d start with a transactional outbox, a durable event stream, an inbox store, and separate workers for each channel. I’d add preference checks before enqueueing, idempotency at event and recipient boundaries, and provider-aware rate limits. For a small product, this can be implemented with a modest number of services; the important part is preserving the boundaries, not adopting every infrastructure component at once.

The next design decision should be the delivery contract: define what counts as accepted, sent, delivered, and read. Once those states are clear, the schema, metrics, retry rules, and user-facing behavior become much easier to settle.

## Key takeaways

- Model notifications as durable user state plus independent channel delivery attempts.
- Use a transactional outbox and durable idempotency checks to handle retries and replays.
- Expect provider timeouts to create uncertainty; exactly-once external delivery is not a safe promise.
- Isolate channel queues, apply backpressure, and batch large fanouts.
- Define delivery states and user preference rules before choosing infrastructure.