System Design: Building a Real-Time Notification Service at Scale

The Problem Space

Every modern application needs notifications. A user receives an email when someone comments on their post. A dashboard updates in real time when a deployment completes. A phone buzzes when a payment is received.

At small scale, this is simple: listen for an event, send a message. At scale — millions of users, thousands of events per second — the problem becomes genuinely interesting.

Requirements

Before designing anything, pin down the requirements:

Functional:

• Support multiple channels: push, email, SMS, in-app, webhook
• Allow users to configure notification preferences
• Support templated messages with variable substitution
• Guarantee delivery (at-least-once semantics)

Non-functional:

• Handle 10,000+ events per second
• Deliver notifications within 500ms for real-time channels
• Support 50M+ users
• 99.99% availability

High-Level Architecture

The system breaks into five distinct layers:

Event Sources → Ingestion → Processing → Delivery → Tracking

Each layer is independently scalable and connected through message queues.

1. Event Ingestion

Events arrive from many sources: application servers, cron jobs, third-party webhooks. An ingestion API normalizes these into a standard event format:

interface NotificationEvent {
  id: string;              // Idempotency key
  type: string;            // "comment.created", "payment.received"
  userId: string;
  payload: Record<string, unknown>;
  timestamp: number;
}

The ingestion layer validates the event, deduplicates using the idempotency key, and publishes to a durable message queue (Kafka, SQS, or similar).

2. Preference Resolution

Not every event should generate a notification. The preference resolver:

1. Looks up the user's notification preferences from a fast cache (Redis)
2. Determines which channels are enabled for this event type
3. Checks rate limits and quiet hours
4. Fans out one message per channel

This is a critical optimization point. If a user has disabled email notifications for comments, we should never reach the email delivery layer.

3. Template Engine

Each notification type has templates for each channel:

{
  "comment.created": {
    "push": {
      "title": "{{authorName}} commented on your post",
      "body": "{{commentPreview}}"
    },
    "email": {
      "subject": "New comment on \"{{postTitle}}\"",
      "templateId": "comment-notification-v2"
    }
  }
}

The template engine resolves variables, handles localization, and produces channel-specific payloads.

4. Delivery Layer

Each channel has its own delivery service:

• Push: Firebase Cloud Messaging or APNs, with device token management
• Email: SES, SendGrid, or Postmark with DKIM/SPF
• SMS: Twilio or SNS with carrier routing
• In-app: WebSocket connections via a connection manager
• Webhook: HTTP POST with exponential backoff retry

Each delivery service is a separate consumer group on the message queue, scaling independently based on channel volume.

5. Tracking and Analytics

Every delivery attempt is logged:

interface DeliveryLog {
  notificationId: string;
  channel: string;
  status: "sent" | "delivered" | "failed" | "bounced";
  timestamp: number;
  latencyMs: number;
  vendorResponse?: string;
}

This feeds dashboards, alerting, and retry logic.

The Real-Time Challenge

For in-app notifications, users expect instant delivery. This requires persistent connections:

WebSocket Connection Manager:

• Each server maintains WebSocket connections to thousands of clients
• A distributed pub/sub layer (Redis Pub/Sub or NATS) routes messages to the correct server
• Connection state is tracked in a shared registry

User connects → Register in connection registry
Event arrives → Look up user's server → Publish to that server's channel
Server receives → Push to WebSocket

The connection registry maps user IDs to server instances. When a notification arrives, the system looks up which server holds the user's connection and routes accordingly.

Handling Failures

At scale, everything fails. The system needs:

Retry with exponential backoff:

const retryDelays = [1000, 5000, 30000, 300000, 3600000];
 
async function deliverWithRetry(
  notification: Notification,
  attempt: number = 0
): Promise<void> {
  try {
    await deliver(notification);
  } catch (error) {
    if (attempt < retryDelays.length) {
      await scheduleRetry(notification, retryDelays[attempt]);
    } else {
      await moveToDeadLetterQueue(notification);
    }
  }
}

Dead letter queues for messages that exhaust retries, with dashboards for manual inspection and replay.

Circuit breakers per delivery vendor. If Twilio is down, stop sending SMS and queue messages rather than exhausting application threads.

Data Storage Choices

Different parts of the system need different storage:

Key Takeaways

1. Separate ingestion from delivery — different scaling characteristics
2. Fan out at the preference layer — avoid wasted work downstream
3. Each channel is its own microservice — independent scaling and failure isolation
4. Everything goes through a queue — resilience over synchronous calls
5. Idempotency keys everywhere — at-least-once delivery means duplicates will happen; make processing idempotent

The notification system is deceptively simple on the surface but touches nearly every distributed systems challenge: ordering, exactly-once semantics, connection management, multi-tenancy, and graceful degradation. Getting it right means thinking carefully about where failures can occur and designing explicit strategies for each.