Designing a Notification System at Scale with Spring Boot and Kafka

Designing a Notification System at Scale with Spring Boot and Kafka

A notification system often starts with a simple requirement: send an email, text message, or push notification when something happens in an application. For a small system, calling an external provider directly may be enough. As traffic grows, however, notification delivery becomes a distributed systems problem.

A production notification platform needs to handle traffic spikes, provider failures, duplicate events, retries, delivery tracking, and multiple notification channels without slowing down the application that generated the notification.

This article explores how a scalable notification system can be designed using Spring Boot and Kafka, with a focus on asynchronous processing, fault tolerance, idempotency, retries, and horizontal scaling.

Defining the Requirements

Before choosing technologies, it is useful to define what the system is expected to do.
At a functional level, the notification system should support multiple channels, such as:

• Email
• SMS
• Push notifications

It should also be able to accept notification requests from multiple business services. An order service may send an order confirmation, a payment service may send a payment receipt, and an authentication service may send a one-time password or security alert.
The non-functional requirements are more important once traffic increases. The system should be highly available, fault tolerant, horizontally scalable, and able to absorb sudden bursts of traffic.

A failure in an email or SMS provider should not cause the original business transaction to fail.

For example, if an order has already been placed successfully, an unavailable email provider should not prevent the order API from responding to the customer.

Why Asynchronous Processing Matters

One possible implementation is for the order service to call the notification provider directly.

Order Service
     |
     v
Email Provider

This approach is straightforward, but it creates tight coupling between the business service and the notification provider.

If the email provider takes several seconds to respond, the order request may also be delayed.

If the provider becomes unavailable, the order service must decide how to handle that failure.

A more resilient approach is to separate notification creation from notification delivery.

Order Service
     |
     v
Kafka
     |
     v
Notification Worker
     |
     v
Email Provider

Instead of sending the notification directly, the order service publishes an event to Kafka. The notification service consumes the event and handles delivery independently.

This allows the original request to complete without waiting for the external provider.

It also provides an important buffering mechanism. If thousands of notification events arrive in a short period, Kafka can retain them while consumers process the backlog at a sustainable rate.

Designing the Notification Event

The event should contain enough information for the notification service to determine what needs to be sent.
A simplified Java model might look like this:

public class NotificationEvent {

    private String eventId;
    private String userId;
    private NotificationType type;
    private String recipient;
    private String templateId;
    private Map<String, String> data;

}

The notification type can be represented as an enum:

public enum NotificationType {
    EMAIL,
    SMS,
    PUSH
}

The eventId is especially important because it can later be used to detect duplicate processing.

The templateId allows the system to separate the event itself from the final message content. Instead of publishing complete email HTML or SMS text, the event can reference a template and provide the dynamic values required to populate it.

Choosing a Kafka Topic Strategy

There are several ways to organize Kafka topics for notifications.

One option is to use a shared topic:

Notification-events

Each message contains its notification type, and a notification service routes the event internally to the appropriate channel.
Another option is to create separate topics:

email-notifications
sms-notifications
Push-notifications

A shared topic keeps producer logic simple and provides a centralized stream of notification activity.

Separate topics provide greater isolation. Email, SMS, and push workloads may have very different traffic patterns and provider limitations, so independent topics can make it easier to scale and configure each channel separately.

There is no single correct choice. The better design depends on traffic patterns, operational requirements, and how independently each channel needs to evolve.

Producer configuration matters here too. On a past Kafka upgrade, our team consolidated from one producer thread per connection down to a single shared producer thread for the entire server, which cut overhead significantly as traffic grew. (Updating Microservices with Netty 5, Kafka 3, and React: Whirlpool Revisited covers the full upgrade.)

Processing Notifications with Spring Boot

Spring Boot integrates naturally with Kafka through Spring for Apache Kafka.

A consumer can listen for notification events:

@KafkaListener(
    topics = "notification-events",
    groupId = "notification-service"
)
public void consume(NotificationEvent event) {
    notificationService.process(event);
}
The service can route the notification based on its type:
public void process(NotificationEvent event) {
    switch (event.getType()) {
        case EMAIL ->
            emailService.send(event);
        case SMS ->
            smsService.send(event);
        case PUSH ->
            pushService.send(event);
    }
}

In a larger system, each channel would likely have its own service or worker implementation so it can be deployed and scaled independently.

If you’re setting up a service like this from scratch, our guide on building microservices with Spring Boot walks through the broader lifecycle, from defining bounded contexts through testing and deployment.

Handling Failures with Retries

External providers are not always available. An SMS provider may return a temporary error. An email service may time out. A push provider may throttle requests.

Immediately marking every temporary failure as permanently failed would result in unnecessary message loss. Retries are therefore an important part of the design.

A typical retry sequence might look like:

Attempt 1
   |
 Failure
   |
Wait 5 seconds
   |
Attempt 2
   |
 Failure
   |
Wait 30 seconds
   |
Attempt 3

Instead of retrying continuously, the system should use increasing delays between attempts. This pattern is commonly referred to as exponential backoff.

Backoff reduces the chance of overwhelming a provider that is already under load.

It is also important to distinguish between retryable and non-retryable failures. A network timeout may be worth retrying, while an invalid phone number usually is not.

Dead-Letter Queues

Retries need a stopping point.

If a notification continues to fail after the configured number of attempts, it should be moved somewhere that allows the system to continue processing other events.

A dead-letter queue, or DLQ, provides this mechanism.

Notification-dlq

Messages in the DLQ can later be inspected, replayed, or investigated by an operations team.

A DLQ prevents repeatedly failing records from consuming retries indefinitely and helps keep the normal processing pipeline moving.

Preventing Duplicate Notifications

Message brokers commonly use delivery models where a message may occasionally be delivered more than once.

That means consumers should be designed to handle duplicate events safely.

Without duplicate protection, the same order event could result in multiple confirmation emails being sent to the customer.

An event identifier can be used to make notification processing idempotent.

Before sending a message, the service checks whether the event has already been processed.

However, this does not completely eliminate duplicates. If the external provider successfully sends the notification but the application crashes before recording the event as processed, Kafka may redeliver the same event. When supported, provider-side idempotency keys can help reduce duplicate sends. Otherwise, exactly-once delivery to an external provider is difficult to guarantee.

Conceptually:

Receive event
     |
     v
Has eventId already been processed?
     |
   Yes ----> Ignore
     |
    No
     |
     v
Send notification
     |
     v
Store eventId

Idempotency is particularly important when retries are involved because the application may not always know whether a provider successfully processed a request before a timeout occurred.

Tracking Notification Status

Persisting notification state is useful for both operational visibility and customer support.

A notification record might include:

id
event_id
user_id
type
recipient
status
retry_count
created_at
sent_at
Possible statuses include:
PENDING
PROCESSING
SENT
FAILED

This information makes it possible to answer questions such as:

• Was a notification sent?
• How many attempts were made?
• Which provider handled it?
• When did the failure occur?
• Can the notification be retried?

It also provides data for monitoring delivery success rates over time.

Scaling with Kafka Consumer Groups

One of the main benefits of Kafka is the ability to distribute work across multiple consumers.

For example:

                 Kafka Topic
                       |
                 Consumer Group
                       |
          ---------------------------
          |            |            |
          v            v            v
Worker 1     Worker 2     Worker 3

As the number of notification events increases, additional workers can be added to the consumer group.

Kafka partitions determine how much parallel processing is possible within a consumer group. If a topic has six partitions, up to six consumers in the same group can actively process those partitions at one time. Adding more than six consumers to that group would not increase active parallelism unless the topic also had more partitions.

This means partition count should be considered early when estimating future throughput requirements.

Rate Limiting External Providers

Scaling internal workers does not necessarily mean external providers can handle unlimited traffic.

An SMS provider may permit only a specific number of requests per second. Increasing the number of consumers without controlling outbound traffic could cause the provider to start rejecting requests.

Rate limiting can be applied before calling the external provider.

Common approaches include token bucket algorithms, shared counters stored in Redis, or application-level libraries such as Resilience4j.

The appropriate limit should be based on the provider’s contract and the throughput characteristics of each notification channel.

Making Notification Failures and Delays Visible

A scalable notification system also needs to be observable.

Useful metrics include:

• Notifications successfully delivered
• Failed notifications
• Retry counts
• Kafka consumer lag
• Dead-letter queue size
• Provider response time
• Notification processing latency

Spring Boot Actuator and Micrometer can expose application metrics, which can then be collected by monitoring systems such as Prometheus and visualized using Grafana.

Consumer lag is particularly useful because it shows whether notification workers are keeping up with the number of incoming events.

How the Pieces Fit Together in a Production Notification Architecture

A simplified architecture can be represented as:

          Business Services
  Order / Payment / User Services
                      |
                      v
                   Kafka
                      |
          ---------------------------
          |            |            |
          v            v            v
     Email Worker   SMS Worker   Push Worker
          |            |            |
          v            v            v
     Email Provider SMS Provider Push Provider

                   Notification   Workers
                  |                                    |
                  v                                   v
          Notification DLQ       Notification Database

The important design principle is separation of responsibility.

Business services decide that a notification needs to exist. Kafka provides asynchronous communication and buffering. Notification workers handle channel-specific delivery. External providers perform the final delivery, while persistence and monitoring provide operational visibility.

Key Takeaways for Building a Scalable Notification System

Sending a notification is easy. Sending notifications reliably at scale requires considerably more design.

Moving notification delivery behind an asynchronous messaging layer prevents external providers from slowing down business operations. Kafka provides buffering and horizontal scalability, while retries and dead-letter queues help the system recover from failures.

Idempotency protects users from duplicate notifications, rate limiting prevents external services from being overwhelmed, and observability makes it possible to understand how the system behaves in production.

The result is a notification architecture that can grow from a relatively small application into a system capable of supporting high-volume email, SMS, and push workloads without tightly coupling notification delivery to the services that generate those events.