Design a Tiered Messaging Platform
Design a billion-message platform where critical OTPs cannot be starved by bulk campaigns: tier isolation, providers, retries, scheduling, and failure handling.
Page content
An authentication service sends an OTP at the same moment Marketing launches a campaign to 50 million users.
Both messages enter the same platform, but they do not have the same value:
OTP useful for seconds; blocks a login
Marketing message useful for hours; may wait
If bulk traffic fills the same queues, workers, or provider quota used by the OTP, calling one message “P0” does not help. Priority is meaningful only when the system reserves the resources needed to enforce it.
The central question is:
How do we deliver more than a billion SMS, email, and push messages per day while preventing low-priority traffic from starving critical messages?
We will start with requirements, estimate the traffic, and first examine the simple shared-queue design. Its failure will lead us to tier isolation.
1. Clarify the problem
The channels are:
SMS
Email
Push through FCM/APNs
Before designing, I would clarify latency, delivery guarantees, scheduling, tracking, providers, geography, and ordering. For this walkthrough, assume:
Tier SLAs P0 <1s, P1 <10s, P2 <5m, P3 <1h
Traffic 1B messages/day, 10× peak
Delivery durable acceptance, at-least-once processing
Scheduling P2 and P3 only
Tracking full lifecycle, query API, client webhooks
Tenants trusted internal services with quotas and recipient controls
Providers multiple providers selected dynamically
Regions start single-region, then evolve globally
Ordering optional key; P0 may overtake lower tiers
The latency SLA ends when a healthy provider accepts the request. The platform cannot promise that a carrier or powered-off phone will deliver an SMS within one second.
2. Requirements
Functional requirements
- Internal services submit SMS, email, and push messages.
- Every message carries a P0–P3 tier.
- P2/P3 may be scheduled.
- Messages use versioned templates and variables.
- Clients can query delivery status and receive signed webhooks.
- Retryable provider failures are retried.
- Multiple providers are supported per channel.
- Operators can inspect and replay operational DLQs.
- Tenant quotas, consent, quiet hours, and recipient limits are enforced.
Tier contracts
P0 Critical OTP, payment/security alert p99 < 1 second
P1 Important booking/order confirmation p99 < 10 seconds
P2 Normal receipt/account update p99 < 5 minutes
P3 Best effort marketing campaign within 1 hour
Non-functional requirements
- Acknowledged messages must be durable.
- P3 must not consume capacity reserved for P0.
- Processing is at least once.
- External duplicates remain possible after an ambiguous provider outcome.
- Each tier scales and degrades independently.
- Operational status is queryable for 30 days.
- Sensitive recipient and message data is protected.
3. Scale estimation
Messages/day 1 billion
Average size 2 KB including metadata
Peak multiplier 10×
Throughput
Average
= 1B / 86,400
≈ 11,600 messages/second
Peak
≈ 116,000 messages/second
Assume:
| Tier | Share | Average/sec | Peak/sec |
|---|---|---|---|
| P0 | 1% | 116 | 1,160 |
| P1 | 9% | 1,040 | 10,400 |
| P2 | 30% | 3,480 | 34,800 |
| P3 | 60% | 6,960 | 69,600 |
P0 is small and strict. P3 dominates volume and often arrives as a bursty campaign. That mismatch drives the isolation strategy.
Channel capacity
Assume push 60%, email 30%, SMS 10%:
Push peak ~69,600/sec
Email peak ~34,800/sec
SMS peak ~11,600/sec
Provider contracts need this capacity plus failover headroom. Provider B is not a real SMS backup if it cannot absorb Provider A’s traffic during an outage.
Storage and queue capacity
1B × 2 KB
≈ 2 TB/day of ingress payload
One day in Kafka with replication factor 3 is roughly 6 TB before overhead. Operational state at 1 KB/message is about 1 TB/day or 30 TB for 30 days before replication.
Historical events should move to object storage and an analytical database rather than remain in the serving store.
4. APIs
Submit a message
Request:
POST /v1/messages
Authorization: Bearer <service-token>
Idempotency-Key: order-981-confirmed
{
"channel": "SMS",
"tier": "P1",
"recipient": "+919876543210",
"template_id": "order-confirmed",
"template_version": 7,
"variables": {
"order_id": "981",
"eta": "30 minutes"
},
"expires_at": "2026-08-15T10:10:00Z",
"ordering_key": "order:981",
"status_webhook": "https://orders.internal/message-status"
}
Response:
HTTP 202 Accepted
{
"message_id": "msg_7f31",
"status": "QUEUED",
"accepted_at": "2026-08-15T10:00:00Z"
}
Return 202 after durable acceptance, not after contacting a provider.
Other APIs
GET /v1/messages/{message_id}
POST /v1/templates
GET /v1/templates/{template_id}/versions/{version}
POST /v1/campaigns
Bulk campaigns upload a recipient manifest to object storage. A campaign expander emits individual P3 messages at a controlled rate instead of receiving millions of recipients in one request.
5. Access patterns and data model
The system must:
- Fetch current state by
message_id. - deduplicate
(tenant_id, idempotency_key). - List recent messages for a tenant.
- Fetch attempts for one message.
- correlate a provider callback by
provider_message_id. - find scheduled work due in a time range.
- retrieve an immutable template version.
Message
Message
-------
message_id
tenant_id
channel
tier
recipient
template_id
template_version
variables_reference
status
scheduled_at
expires_at
ordering_key
created_at
updated_at
Partition by hash(message_id). Maintain a tenant-history view partitioned by (tenant_id, date_bucket) and ordered by creation time.
Idempotency record
Key (tenant_id, idempotency_key)
Value request_hash, message_id, expires_at
The same key and payload return the original result. Reusing the key for different content returns 409.
Delivery attempt
Partition key message_id
Sort key attempt_number
provider_id
provider_message_id
outcome
normalized_error
started_at
completed_at
Index provider_message_id for webhook correlation.
Template
Partition key (tenant_id, template_id)
Sort key version
Template versions are immutable so a queued message renders exactly what was approved at acceptance.
Scheduled message
Partition key (time_bucket, shard)
Sort key (scheduled_at, message_id)
Schedulers query only current buckets, not every future message.
Data ownership
Operational truth Message and current status
Attempt history Provider calls and outcomes
Queue Durable work and short-term replay
Redis state Rate limits, provider health, configuration cache
Historical archive Object storage / analytical database
6. Why one priority queue fails
The simplest design is:
Messaging API
↓
One priority queue
↓
One worker fleet
↓
Providers
If P0 keeps arriving, workers always choose P0 and P3 starves.
If P3 adds 20 million records, it may consume broker disk, network, worker memory, and provider quota before P0 reaches the head.
Logical priority cannot isolate shared physical bottlenecks.
7. Tier isolation
Use separate queues and workers:
P0 → dedicated queue → dedicated workers
P1 → dedicated queue → dedicated workers
P2 → dedicated queue → dedicated workers
P3 → dedicated queue → dedicated workers
This gives independent lag, retention, scaling, and failure domains.
Reserve capacity:
P0 20%
P1 25%
P2 25%
P3 30%
Unused high-tier capacity may be borrowed downward. Lower tiers cannot consume the reservation needed by higher tiers.
Apply this rule to every bottleneck:
- Kafka throughput and storage;
- worker concurrency;
- database writes;
- network connections;
- rate-limit tokens;
- provider credentials and quotas.
Separate topics do not protect P0 if all tiers still share one exhausted provider account.
Start with logical isolation and Kafka quotas. Move P0 to dedicated brokers, worker nodes, network pools, and provider accounts when its SLA is contractual or shared infrastructure creates noisy-neighbor incidents.
8. Kafka design
Use topics per (channel, tier):
send.sms.p0 send.email.p0 send.push.p0
send.sms.p1 send.email.p1 send.push.p1
...
send.sms.p3 send.email.p3 send.push.p3
Twelve send topics allow different retention, partitions, consumers, and alerts.
Default partition key:
hash(message_id)
For optional ordering:
hash(tenant_id + ordering_key)
This preserves order within one channel and tier. There is no ordering across tiers because P0 is allowed to overtake P2.
Consumer groups are independently scalable:
sms-p0-senders
sms-p3-senders
email-p1-senders
...
Kafka fits because we need high throughput, durable buffering, replay, partition order, independent consumers, and measurable lag.
Managed queues such as SQS/Pub/Sub may be preferable when operational simplicity matters more than replay. RabbitMQ has rich routing but priority queues can starve and are harder at this retention/throughput. Redis Streams is useful at smaller scale but weaker as the primary billion-message durable log.
9. High-level architecture
Internal services
│
▼
┌─────────────────┐
│ Messaging API │
│ auth, policy, │
│ idempotency │
└────────┬────────┘
▼
┌─────────────────┐
│ Message Store │
│ + Outbox / CDC │
└────────┬────────┘
▼
Tier Router
┌────┼────┐
▼ ▼ ▼
P0 topics ... P3 topics
│ │
▼ ▼
Dedicated worker pools
│
▼
┌─────────────────┐
│ Provider Router │
│ health, quota, │
│ tier, cost │
└───┬────┬────┬───┘
▼ ▼ ▼
SMS Email Push providers
│
▼
Provider webhooks → Delivery Tracker → Client webhooks
Submission
- Authenticate the service and determine its tenant.
- Validate tier policy, channel, template, recipient, and TTL.
- Apply tenant admission limits.
- Atomically write message, idempotency record, and outbox event.
- Return
202. - CDC/outbox routes the message to its channel/tier topic.
Sending
- A tier worker consumes the message.
- It checks expiry, consent, quiet hours, and recipient limits.
- It renders the immutable template.
- It reserves provider quota.
- Provider Router selects an adapter.
- The adapter sends with a stable idempotency key where supported.
- The worker persists the attempt before committing its Kafka offset.
10. Durable acceptance
This is unsafe:
write Message
↓
Kafka publish fails
↓
message never sends
Atomically write:
Message
Idempotency record
Outbox event
An outbox relay or CDC publishes later.
At 120K/sec, one relational outbox is a bottleneck. Use sharded relational stores, a distributed KV with conditional writes and streams, or another partitioned operational store with CDC.
11. Worker architecture
Separate fleets by channel and tier:
SMS P0 warm capacity, reliability-first
SMS P3 cost-first, aggressively throttled
Email P0 reserved provider accounts
Email P3 bulk batching
Push P0 reserved APNs/FCM capacity
Worker flow:
Consume
↓
Acquire processing lease
↓
Check expiry and policy
↓
Render template
↓
Reserve quota
↓
Select provider
↓
Send
↓
Persist attempt
↓
Commit offset
Scale by oldest message age, lag, ingress, provider latency, and available quota—not CPU alone. Adding workers while a provider quota is exhausted only creates more throttling.
Keep P0 workers warm. Waiting for autoscaling after backlog appears may already violate a one-second SLA.
12. Provider abstraction and routing
Use channel-specific adapters:
SmsProvider.send()
EmailProvider.send()
PushProvider.send()
Each adapter normalizes provider payloads, responses, errors, idempotency, and webhooks while preserving channel-specific capabilities.
Provider Router considers:
channel and tier
recipient country/carrier
regional/data-residency rules
health and latency
remaining quota
historical delivery success
cost
tenant/sender identity
Routing policy:
P0 best reliability, reserved quota, rapid failover
P1 reliability with moderate cost awareness
P2 balance cost and performance
P3 cheapest healthy provider, strict throttle
Circuit breaker flow:
Provider A failures: 5% → 10% → 30%
↓
Open circuit
↓
Route eligible traffic to B
↓
Half-open probes
↓
Close after sustained recovery
A timeout does not prove failure. Provider A may have sent the SMS and lost its response. Query A or retry with its idempotency key before failing over; otherwise accept the tier-specific duplicate risk.
13. Hierarchical rate limiting
Limits form a hierarchy:
Platform
└── Region
└── Channel
└── Provider account
└── Tier reservation
└── Tenant
└── Recipient
Examples:
Tenant 100K/sec
Provider A SMS 50K/sec
P0 reservation 10K/sec
Recipient OTP 5 / 10 minutes
Marketing daily cap + quiet hours
Redis token buckets provide low-latency atomic coordination. At high scale, provider keys become hot, so grant workers short token leases that they spend locally.
If Redis fails, P0 may use conservative local reserved limits. P2/P3 pause. Recipient marketing limits fail closed.
14. Retry strategy
Retryable:
- timeout or transport failure;
- provider
5xx; - temporary provider failure;
429withRetry-After.
Non-retryable:
- invalid address or device token;
- missing template variable;
- prohibited sender;
- blocked or opted-out recipient.
Use exponential backoff with jitter and per-tier policies:
P0 alternate provider, then 1s, 3s, 10s within TTL
P1 2s, 10s, 30s, 2m
P2 30s, 2m, 10m, 30m
P3 minutes/hours within campaign window
Do not sleep inside a Kafka consumer. Persist RETRY_PENDING with next_attempt_at, then use delayed retry buckets/scheduler to re-enter the send topic.
Cap retry concurrency separately so retries cannot starve new P0 traffic.
15. Idempotency and external duplicates
Critical failure:
Worker sends SMS
↓
Worker crashes before storing success
↓
Kafka redelivers
↓
SMS may send twice
At-least-once processing means a handler may run more than once. Kafka exactly-once cannot undo an external SMS.
Use:
provider_idempotency_key = message_id + logical_attempt
If supported, provider replay returns the original result. Otherwise a timeout is ambiguous. Persist UNKNOWN, reconcile by provider request id if possible, and apply a duplicate policy:
P0 OTP duplicate may be safer than loss
P3 marketing suppress duplicate when uncertain
Never claim exactly-once external delivery without provider support.
16. Delivery tracking
CREATED
↓
SCHEDULED or QUEUED
↓
PROCESSING
↓
PROVIDER_ACCEPTED
↓
SENT
↓
DELIVERED
Branches:
RETRY_PENDING
FAILED
EXPIRED
SUPPRESSED
Provider callbacks may duplicate or arrive out of order. Deduplicate by (provider_id, provider_event_id) and prevent state regression:
DELIVERED cannot regress to SENT
FAILED cannot overwrite DELIVERED
Client webhook delivery uses a signed event id, retries, circuit breaker, and its own DLQ. The query API remains authoritative.
17. Scheduling
Do not put tomorrow’s message in Kafka and block a partition.
Store P2/P3 future work in time buckets:
Partition key (minute/hour bucket, shard)
Sort key (scheduled_at, message_id)
Schedulers:
- read current and near-future buckets;
- claim due rows with leases;
- verify cancellation and expiry;
- publish to P2/P3 send topics;
- mark dispatch complete.
Duplicate publication after a scheduler crash is safe because downstream handling is idempotent.
Large campaigns keep recipient manifests in object storage and expand near send time at a controlled rate.
18. Dead-letter queues
Expected business outcomes are not DLQs:
invalid recipient → FAILED
opted out → SUPPRESSED
expired OTP → EXPIRED
DLQs contain operationally unprocessable work:
- unknown schema;
- poison payload;
- repeated worker crash;
- renderer invariant violation;
- exhausted retries caused by unexplained system failure.
Use channel/tier-aware DLQs and provide redacted inspection, attempt history, audited replay, and replay eligibility.
19. Backpressure
Normal P3 peak is ~70K/sec. A 100× spike is 7M/sec.
Ten minutes would create:
7M × 600
= 4.2B messages
≈ 8.4 TB payload
≈ 25 TB replicated
The platform cannot accept unlimited bulk work.
For P3:
- enforce campaign quotas;
- require manifests;
- throttle expansion;
- return
429or delay the campaign; - preserve P0/P1 reservations.
A 100× P0 spike reaches ~116K/sec. Isolation protects it from P3, but provider capacity is still finite. Use emergency quota and failover, then reject beyond the contracted capacity envelope. Priority cannot manufacture carrier capacity.
When lag grows, identify the true bottleneck. More workers do not fix provider quota, slow storage, poison retries, or insufficient Kafka partitions.
20. Storage by responsibility
| Responsibility | Suitable storage | Why |
|---|---|---|
| Operational message state | Distributed KV/wide-column or sharded SQL | Key lookup, conditional transitions, TTL |
| Templates/configuration | SQL | Low volume, constraints, approval/audit |
| Scheduled work | Time-bucketed KV/wide-column | Due-time range scans |
| Rate limits/provider health | Redis | Low-latency ephemeral atomic state |
| Queue/replay | Kafka | Durable ordered streams |
| Historical events | Object storage/Parquet | Cheap long retention |
| Operational analytics | ClickHouse/Druid | High-volume aggregations |
Kafka transports events; Redis accelerates decisions. Neither is the operational source of message truth.
21. Consistency and ordering
Strong or conditional consistency is required for:
- idempotency reservation;
- message plus outbox creation;
- message status transition;
- immutable template version;
- marketing opt-out checks;
- provider quota when contractual limits must not be exceeded.
Eventual consistency is acceptable for:
- provider health scores;
- aggregate dashboards;
- client webhooks;
- provider delivery receipts;
- cost analytics.
Optional ordering keys preserve order only within one channel and tier. Strict sequencing means a retrying message blocks later messages, so it is opt-in rather than default.
22. Failure scenarios
Kafka unavailable
Outbox rows accumulate. API acceptance continues only within a bounded safety window; stop accepting P0 before its SLA becomes impossible. Recovery replays outbox rows. Duplicate publication is safe.
Redis unavailable
P0 uses conservative local reserved quota if safe. P2/P3 pause. Rebuild ephemeral provider health and token state after recovery.
One provider down
Circuit opens. P0 uses reserved failover capacity. P3 may wait. Ambiguous timed-out sends may duplicate.
All providers for a channel down
Messages remain retryable until TTL, then become FAILED or EXPIRED. Do not retry forever.
Worker crashes before sending
Kafka redelivery safely retries.
Worker crashes after sending
Provider idempotency or reconciliation prevents duplication where possible. Otherwise external delivery is ambiguous.
Duplicate webhook
Webhook dedup and monotonic transitions make it harmless.
Provider throttles
Honor Retry-After, reduce route weight, preserve higher-tier reservations. Additional workers would worsen the incident.
P3 rises 100×
Admission control and campaign throttling protect P0/P1. Rejected bulk traffic is explicit, never silently dropped after acceptance.
P0 rises 100×
Use emergency providers and warm capacity. Reject beyond the provider capacity envelope rather than promise an impossible SLA.
Operational database unavailable
Stop accepting. Existing sends should pause if outcomes cannot be recorded safely.
Region fails
Promote one fenced region. Replayed ambiguous sends may duplicate, but durable accepted messages remain within the replication RPO.
Consumer lag grows
Alert on oldest age by tier. Determine whether workers, provider quota, partitions, retries, or storage is limiting before scaling.
23. Multi-region evolution
Start in one region across availability zones. Later assign each message an immutable home region:
Global routing
├── US region: US tenants/providers
└── EU region: EU tenants/providers
Benefits:
- recipient/provider locality;
- data residency;
- smaller failure domains.
Replicate durable state and Kafka asynchronously for disaster recovery. Do not actively process one message in two regions.
Use a fenced regional epoch. The DR region processes only after acquiring a newer epoch, preventing both regions from sending the same campaign during a partition.
P0 may justify synchronous metadata replication and warm provider capacity. P3 can tolerate a larger recovery delay.
24. Observability
Queue and tier
- ingress by tenant/channel/tier;
- consumer lag;
- oldest message age;
- retry age;
- scheduler lateness;
- DLQ rate.
Delivery and provider
- acceptance-to-provider latency;
- provider acceptance and delivery rates;
- retry/suppression/expiry rate;
- provider p99 latency, timeouts,
5xx, and429; - quota remaining and circuit state.
Alert directly on tier SLOs:
P0 p99 provider acceptance > 1 second
P1 oldest age > 10 seconds
P2 oldest age > 5 minutes
P3 backlog threatens campaign window or retention
Oldest age is more meaningful than queue depth.
Trace by message, tenant, attempt, provider, tier, channel, and region. Redact recipient and content.
25. Security and abuse
- Authenticate callers with workload identity, mTLS, or OAuth client credentials.
- Authorize tenants by templates, channels, sender ids, and allowed tiers.
- Prevent Marketing Service from labeling every campaign P0.
- Isolate tenant status queries and encrypt sensitive payloads.
- Keep provider credentials in a rotating secrets manager.
- Enforce opt-outs, quiet hours, OTP frequency limits, and daily campaign caps.
- Audit sensitive templates and sends.
If callers may self-declare all traffic critical, the tier model collapses.
26. Major trade-offs
Separate queues versus one queue
Separate queues provide isolation and independent scaling but add operational complexity and may leave capacity idle. Strict P0 requirements justify that cost.
Kafka versus managed queues
Kafka offers replay and stream control. Managed queues reduce operations. The essential decision is separate durable tier lanes, not Kafka by name.
Shared versus dedicated P0 infrastructure
Shared systems use resources efficiently. Dedicated systems reduce blast radius. Begin logically isolated and move to physical isolation when incidents or contracts justify it.
Fast failover versus duplicate risk
Rapid provider failover improves delivery probability after a timeout but can duplicate a message whose first send actually succeeded.
Ordering versus latency
Strict ordering creates head-of-line blocking during retry. Keep it optional and scoped.
27. Final architecture
INTERNAL SERVICES
│
▼
┌────────────────────────┐
│ Messaging API │
│ auth, policy, quota, │
│ idempotency, validate │
└───────────┬────────────┘
▼
┌────────────────────────┐
│ Message Store + Outbox │
└───────────┬────────────┘
│ CDC
▼
┌────────────┐
│ Tier Router│
└─────┬──────┘
┌──────────────────┼──────────────────┐
▼ ▼ ▼
P0 channel topics P1/P2 topics P3 topics
reserved capacity isolated workers throttled bulk
│ │ │
└──────────────────┼──────────────────┘
▼
┌─────────────────────┐
│ Policy + Rate Limits│
│ Provider Router │
└───┬────────┬────────┘
▼ ▼
SMS / Email / FCM / APNs
│
▼
Provider Webhooks
│
Delivery Tracker
┌─────┴─────┐
▼ ▼
Client webhooks Analytics
The mental model is:
P0 has a one-second SLA
↓
P3 can arrive in campaigns
↓
Shared resources create starvation
↓
Reserve capacity at every bottleneck
↓
Allow borrowing only downward
↓
Use admission control when capacity is exhausted
28. Interview-ready summary
Top 10 decisions
- Define tier SLAs as provider-acceptance targets.
- Separate queues and workers by channel and tier.
- Reserve broker, worker, limiter, and provider capacity.
- Permit only downward borrowing.
- Atomically persist message, idempotency, and outbox.
- Use at-least-once processing with honest duplicate semantics.
- Route providers by reliability, quota, region, tier, and cost.
- Schedule with time buckets, not sleeping consumers.
- Separate business failures from operational DLQs.
- Give each message one fenced home region.
Top failure scenarios
- Kafka outage: durable outbox buffers within a safety window.
- Redis outage: P0 degrades conservatively; P2/P3 pause.
- Provider outage: circuit breaker and reserved failover.
- All providers down: retry until TTL, then fail.
- Crash after send: external result may duplicate.
- Duplicate webhook: dedupe and monotonic status.
- Provider throttling: reduce traffic; more workers do not help.
- P3 burst: campaign admission protects critical tiers.
- P0 burst: emergency capacity, then explicit rejection.
- Region loss: fenced failover prevents dual processing.
Senior-level differentiators
- Reserve capacity at providers, not only Kafka.
- Control which tenants may use P0.
- Explain ambiguous provider timeouts.
- Never claim exactly-once external delivery without provider support.
- Treat retries as traffic with their own quota.
- Alert on oldest message age, not only lag count.
- Keep strict ordering opt-in.
- Stop bulk expansion before queue storage becomes the incident.
Likely interviewer follow-ups
- How do you calculate partition counts?
- How does downward borrowing work?
- What if P0 exceeds provider capacity?
- How do you reconcile an ambiguous SMS timeout?
- Why Kafka instead of SQS?
- How do campaigns respect recipient time zones?
- What happens if the outbox grows faster than Kafka recovery?
- How do you migrate a tenant between regions?
- How are distributed provider tokens implemented?
- What prevents Marketing from using P0?
A 1–2 minute verbal answer
I would support SMS, email, and push across P0 through P3. At one billion messages per day, traffic averages about 11,600 per second and peaks around 116,000. P3 is most of the volume, while P0 is small but has a one-second provider-acceptance SLA.
The central decision is tier isolation. I would use separate channel-and-tier topics and independently scaled workers. P0 receives reserved broker, worker, network, limiter, and provider capacity. Lower tiers may borrow unused capacity but cannot consume its reservation. If shared brokers or provider accounts still create noisy neighbors, I would physically isolate P0.
The API enforces identity, tier policy, quotas, templates, and idempotency, then atomically writes the message and outbox before returning 202. Workers check TTL, consent, recipient limits, and provider quota before a reliability-, health-, region-, tier-, and cost-aware router selects a provider.
Processing is at least once. A crash after an external send can duplicate unless the provider supports idempotency or reconciliation. Provider callbacks are deduplicated into a monotonic status machine. Scheduling uses time-bucketed storage; retries re-enter through delayed buckets; poison operational failures go to inspectable DLQs. The key SLIs are oldest message age by tier, provider-acceptance latency, retry rate, and delivery success.
For the broader interview framework around this problem, see the System Design Interview Complete Guide.
