Design a Customer-Facing Search System

An interviewer types two queries:

hotels in Goa
cheap hotels near Baga Beach with pool

The first could match thousands of properties. The second mixes free text, a place, a price intent, and an amenity. Even after we identify good hotels, the room shown as available may sell out before checkout.

Before drawing boxes, I would ask the questions that change the design:

• What are we searching: hotels, rooms, destinations, or all travel products?
• Is this keyword search, semantic search, or both?
• Which filters, sorts, facets, and geo queries are required?
• Must displayed price and availability be exact?
• Is personalization in scope, and how much user data may the request path load?
• How fresh must catalog, price, and inventory changes be?
• How many searchable documents, searches, and catalog updates do we expect?
• What latency and availability count at the Search API boundary versus the user’s device?
• How deep may users paginate?
• Do we begin in one region, and what is the regional failure expectation?

If the interviewer provides no further constraints, I would state the following assumptions explicitly:

Domain                    Customer-facing hotel/property search
Searchable properties     100 million documents
Search traffic            100 million searches/day
Average QPS               about 1,157
Initial peak estimate      10× average, about 11,600 QPS
Capacity posture           extra headroom for bursts, skew, and failures
Catalog update peak       about 10,000 updates/second
Core retrieval            lexical full-text + filters + geo
Semantic/vector search    later extension, not core
Personalization           one compact user profile for reranking
Price and availability    cached/indicative in results; revalidate at booking
Page size                 20
Deep pagination           cursor/search_after, capped near 1,000 results
Search availability       approximately 99.99%, with degraded responses
Initial deployment        single region
Later deployment          independent regional read clusters, async replication

The scope includes query understanding, retrieval, ranking, result enrichment, autocomplete, index maintenance, analytics, and serving reliability. Booking, payment, supplier connectivity internals, and recommendation homepages are outside the core design. The Search API may call pricing and availability in batches, but those services own their facts.

1. Establish the search mental model

Search is not one database query. It is a narrowing pipeline:

Query
  → Query Understanding
  → Retrieval / Candidate Generation
  → Ranking
  → Personalization
  → Enrichment
  → Final Response

For “cheap hotels near Baga Beach with pool,” a plausible request becomes:

100,000,000 indexed properties
        ↓ text, geo, amenity, policy filters
about 1,000 retrieval candidates
        ↓ deterministic or learned ranking
top 50 rank/rerank candidates
        ↓ price and availability enrichment
overfetch around sold-out properties
        ↓
return 20 results

Retrieval answers, “Which documents might satisfy the query?” Ranking answers, “In what order should eligible candidates appear?” These are different jobs.

Running an expensive model over 100 million documents would require 100 million feature vectors and model evaluations per request. At peak traffic, that is impossible and unnecessary. An inverted index and geo/filter structures cheaply remove almost everything first. The ranker spends its budget only on a bounded candidate set.

We will return to this distinction throughout the design. It prevents a common interview mistake: calling every scoring step “search” and hiding where the real cost occurs.

2. Requirements, targets, retention, and edge cases

The core product should:

1. Search hotel names, descriptions, landmarks, neighborhoods, and destinations.
2. Filter by dates, guests, amenities, star rating, property type, policies, and price bands.
3. Search around a point or named place and sort by relevance, price, rating, or distance.
4. Return facets such as amenity and star-rating counts.
5. Personalize ranking for a known user without excluding valid results.
6. Show indicative price and availability, then validate both during booking.
7. Provide autocomplete, spelling help, and useful zero-result recovery.
8. paginate consistently enough for browsing without offering arbitrary deep scans.
9. Record impressions, clicks, refinements, and bookings for measurement and ranking.

Non-functional targets

An interviewer may propose:

p50 < 100 ms
p95 < 300 ms
p99 < 500 ms
availability ≈ 99.99%

Those targets are useful only after defining the boundary. A warm, unpersonalized Search API response in the same region may fit below 100 ms. A cold request that loads a profile, fans out across shards, and enriches live prices is unlikely to do so reliably. Mobile DNS, TLS, radio wake-up, and cross-region distance also make end-to-end latency higher than service latency.

I would therefore negotiate:

Search service p50             target near 100 ms for warm requests
Search service p95             about 300 ms including bounded enrichment
Search service p99             about 500 ms, with optional work shed
User-perceived mobile latency  measured separately by geography/network
Autocomplete p95               around 50–100 ms service-side
Catalog freshness              usually seconds; bounded by pipeline lag
Price/inventory freshness      timestamped and explicitly non-authoritative
Availability                   about 99.99% for a useful search response

We should benchmark before committing. Shard fanout, index shape, filter cardinality, result cache hit rate, model cost, and supplier latency can move these values substantially.

Consistency and retention

Different facts need different guarantees:

Hotel catalog DB             durable source of truth
Search document              eventually consistent derived projection
Price/inventory result       cached observation with timestamp
Booking validation           authoritative at transaction time
User profile                 compact, eventually updated feature snapshot
Experiment assignment        stable for an experiment
Analytics                    durable, delayed by minutes or hours

Catalog entities live while active and retain deletion/audit history according to business policy. Kafka topics might retain days to weeks, long enough for replay and operational recovery. Search analytics can remain in a data lake for months or years under privacy policy; raw user identifiers should be minimized or pseudonymized. Query-result cache entries live seconds or minutes, autocomplete popularity data hours, and price/inventory cache entries only as long as the product can honestly label their freshness.

Important edge cases include:

• “Goa” could mean the state, a city label, or a property containing Goa in its name.
• “Cheap” depends on dates, occupancy, currency, market, and current inventory.
• Baga Beach has a geo point, but “near” has no universal radius.
• A hotel is renamed, moved, suspended, or deleted while an index update is delayed.
• A query contains misspellings, mixed languages, emoji, or no text at all.
• A user combines filters that legitimately produce zero results.
• Price changes between the result page and booking.
• One shard is slow while the others return.
• A viral event makes “hotels near stadium” a hot query and geo area.
• A crawler requests costly facets, huge radii, and deep pages.
• Results change between page one and page two.

3. Estimate the workload before choosing storage

Round numbers reveal the shape of the system; they are not a hardware quote.

Search traffic

100,000,000 searches/day / 86,400
≈ 1,157 average QPS

10× stated peak
≈ 11,600 QPS

Ten times average is only an initial estimate. Travel demand is geographically and temporally skewed. A sale, weather event, holiday announcement, or partner campaign can concentrate requests on one destination and cache key. Capacity planning should test perhaps 15,000–25,000 QPS at the chosen latency target, then preserve headroom for one availability zone or several data nodes being unavailable. The exact number comes from traffic history and failure drills, not from multiplying by a fashionable constant.

At 20 results and perhaps 8–15 KB compressed per response, peak egress could be on the order of:

11,600 requests/second × 8–15 KB
≈ 93–174 MB/second before protocol overhead

Images should be URLs served from a CDN, not bytes returned by Search.

Index storage

Suppose the source hotel object is 5–10 KB, but the normalized searchable document averages roughly 4 KB of stored source and indexed fields. The inverted index, doc values for sorting/facets, geo structures, and segment overhead might make the on-disk primary index roughly 1.5–2.5 times that amount.

100M documents × 4 KB                    ≈ 400 GB raw projection
Primary search index at 1.5–2.5×         ≈ 600 GB–1 TB
One replica                              ≈ 1.2–2 TB
Operational disk headroom, merges,
growth, and watermarks                   perhaps 2–4 TB provisioned

This range is deliberately broad. Analyzer choices, duplicated text, nested objects, high-cardinality doc values, deleted-document churn, and compression dominate the result. Build a representative index and measure it.

If healthy data nodes should remain below roughly 60–70% disk to leave room for merges and recovery, a cluster might begin with dozens rather than hundreds of nodes, depending on node disk, CPU, memory, and required query concurrency. Node count cannot be derived from bytes alone: a 2 TB index may still need more nodes for 12K QPS and tail latency.

Shards and replicas

If benchmarked shard sizes land around 20–50 GB, 600 GB–1 TB of primary data suggests tens of primary shards. That is a starting range, not a prescription. We must load-test:

• latency as fanout grows;
• indexing and segment-merge pressure;
• shard recovery time;
• heap per shard;
• cache locality;
• concurrent queries per node; and
• behavior with a node or zone missing.

One replica doubles stored bytes, improves read capacity, and permits a shard copy to survive one node failure. Allocate primary and replica copies across failure domains. Additional replicas can add read throughput but also amplify indexing and storage cost.

Update throughput

At 10,000 catalog updates/second peak:

10,000 updates/second × perhaps 2–5 KB/event
≈ 20–50 MB/second entering the indexing path

OpenSearch writes are not tiny row updates. Each update creates new segment work and later merges; frequent partial updates may still read or rewrite a document internally. Batch with the bulk API, tune refresh intervals, and benchmark sustained indexing while representative search traffic runs. A cluster that handles either 12K search QPS or 10K updates/sec separately may fail when both happen together.

Caches

If the result cache stores 500,000 hot normalized queries at 10 KB each:

500,000 × 10 KB ≈ 5 GB payload

With key, allocator, replication, and metadata overhead, provision materially more. But hit rate matters more than the arithmetic. Dates, occupancy, locale, currency, filters, sort, experiments, and personalization fragment the key space. A 50 GB cache with a 5% hit rate may be less valuable than a 5 GB autocomplete cache with a 90% hit rate.

4. Define the API and query model

The public API should expose product concepts, not the search engine’s raw DSL.

Search

Request:

POST /v1/hotels/search
Authorization: Bearer <optional-user-token>
Content-Type: application/json

{
  "query": "cheap hotels near Baga Beach with pool",
  "stay": {
    "check_in": "2026-12-18",
    "check_out": "2026-12-21",
    "rooms": 1,
    "adults": 2,
    "children": 0
  },
  "filters": {
    "amenities": ["pool"],
    "star_rating": { "min": 3 },
    "refundable": true
  },
  "geo": {
    "center": { "lat": 15.5553, "lon": 73.7517 },
    "radius_km": 8
  },
  "sort": "RELEVANCE",
  "locale": "en-IN",
  "currency": "INR",
  "page_size": 20,
  "cursor": null
}

Response:

HTTP 200 OK

{
  "request_id": "srch_7fa2",
  "query": {
    "original": "cheap hotels near Baga Beach with pool",
    "normalized": "cheap hotels near baga beach with pool",
    "interpreted_location": "Baga Beach, Goa",
    "applied_intents": ["BUDGET_SENSITIVE", "AMENITY_POOL"]
  },
  "results": [
    {
      "property_id": "prop_184",
      "name": "Sea Palm Baga",
      "distance_km": 0.9,
      "rating": 4.3,
      "price": {
        "amount": 6200,
        "currency": "INR",
        "as_of": "2026-08-16T04:28:10Z",
        "indicative": true
      },
      "availability": {
        "status": "LIKELY_AVAILABLE",
        "as_of": "2026-08-16T04:28:08Z"
      }
    }
  ],
  "facets": {
    "star_rating": { "3": 412, "4": 188, "5": 45 }
  },
  "relaxations": [],
  "next_cursor": "<opaque-signed-cursor>"
}

The response says “indicative” instead of implying that search owns a bookable price.

Autocomplete

Request:

GET /v1/search/suggestions?prefix=bag&locale=en-IN&limit=8

Response:

HTTP 200 OK

{
  "suggestions": [
    { "text": "Baga Beach, Goa", "type": "LANDMARK", "entity_id": "poi_baga" },
    { "text": "Baga, Goa", "type": "LOCALITY", "entity_id": "loc_baga" },
    { "text": "Baga Beach hotels with pool", "type": "QUERY" }
  ]
}

Impression and click events

Request:

POST /v1/search/events
Content-Type: application/json

{
  "event_id": "evt_a12",
  "request_id": "srch_7fa2",
  "event_type": "CLICK",
  "property_id": "prop_184",
  "position": 3,
  "occurred_at": "2026-08-16T04:28:15Z"
}

Response:

HTTP 202 Accepted

Events carry an id so ingestion can deduplicate retries. They are not synchronous updates to the online search index.

Internal query model

After understanding, the service works with a typed request:

SearchQuery
-----------
original_text
normalized_tokens
destination_entity_id
geo_shape_or_radius
must_filters
preference_signals
sort
stay_dates
occupancy
locale
currency
experiment_id
profile_version
page_size
cursor

must_filters are contractual constraints such as “pool” when explicitly selected in the UI. Preference signals such as “cheap” can influence ranking without inventing a hard cutoff unless the product has defined one.

The cursor is opaque, signed, and versioned. It can encode the point-in-time identifier, normalized-query hash, sort values of the last result, tie-breaking property_id, and expiry. The server rejects a cursor reused with different filters rather than producing undefined pages.

5. Model the searchable hotel document

The source catalog may be normalized across hotel, address, amenity, policy, and media tables. Search needs a denormalized document aligned with read patterns:

HotelSearchDocument
-------------------
property_id                 keyword; filter, tie-breaker
name                        text + keyword subfield
description                 text
brand_name                  text + keyword
destination_names           text + keyword
landmark_names              text
amenity_names               text for matching
amenity_ids                 keyword; filter/facet
property_type               keyword; filter/facet
star_rating                 numeric; filter/sort/facet
guest_rating                numeric; filter/sort
review_count                numeric; ranking
location                    geo_point
country_code                keyword
city_id / locality_id       keyword; filter/routing hint
languages                   keyword
refundable_hint             boolean; filter hint
price_band_hint             keyword/numeric; retrieval/ranking hint
availability_hint           boolean/timestamp; retrieval hint
popularity_score            numeric; ranking feature
quality_score               numeric; ranking feature
thumbnail_url               stored-only
display_address             stored-only
catalog_version             numeric; external version
updated_at                  date; operations/freshness

The mapping type is a correctness decision:

• Text fields are analyzed into tokens. A search for “sea view” can match a description containing those words.
• Keyword fields preserve exact values. amenity_ids=POOL and country_code=IN should not be tokenized.
• Numeric/date fields support ranges, sorting, and aggregations.
• Geo fields support radius, bounding-box, and distance operations.
• Stored-only fields are returned but do not need to be searched or sorted.
• Dynamic fields such as live room prices and room inventory usually do not belong as authoritative values in this document.

name often needs both representations:

name              analyzed text for "sea palm"
name.keyword      exact value for deduplication or controlled sort

“Exact text” is not the same as a text mapping. Exact matching, filtering, grouping, and stable sorting generally use keyword; linguistic matching uses text.

Data ownership remains explicit:

Catalog DB                 source of truth for hotel metadata
Inventory system           source of truth for sellable room inventory
Pricing system             source of truth for current offers and total price
OpenSearch document        eventually consistent searchable projection
Redis/profile store        compact derived personalization features
Result cache               disposable response acceleration
Kafka                      durable transport/replay, not business truth

6. Let the access pattern choose storage

The dominant read is not “get hotel by id.” It is:

match several analyzed text fields
AND exact amenity/policy filters
AND numeric/date constraints
AND geo radius or distance
THEN calculate relevance and facets
THEN return global top-K

The system also needs high update throughput, horizontal read scaling, typo tolerance, language analyzers, and predictable top-K retrieval.

Start with the simplest possible implementation: the catalog’s relational database.

SELECT h.*
FROM hotels h
JOIN hotel_amenities a ON a.hotel_id = h.id
WHERE (
  h.name ILIKE '%baga%'
  OR h.description ILIKE '%baga%'
)
AND a.amenity = 'pool'
AND h.star_rating >= 3
ORDER BY h.rating DESC
LIMIT 20;

This may work for an early catalog. It fails gradually:

• leading-wildcard LIKE scans are expensive;
• joins multiply rows and complicate facets;
• linguistic relevance is weak;
• typo tolerance and stemming require specialized extensions;
• range, geo, relevance, and aggregation work compete on one query;
• a normalized catalog schema is not shaped for low-latency denormalized reads;
• 100 million rows and 12K peak QPS put search load next to transactional work; and
• independent search scaling and index rebuilds become awkward.

PostgreSQL full-text and geo extensions can carry a smaller product surprisingly far. The decision is not “SQL cannot search.” It is that this workload benefits from a dedicated, independently scalable, derived search index.

That establishes the need for OpenSearch or Elasticsearch.

7. What the search engine contributes

An inverted index maps terms to documents:

pool    → [hotel 4, hotel 19, hotel 184, ...]
baga    → [hotel 7, hotel 184, hotel 910, ...]
beach   → [hotel 2, hotel 7, hotel 184, ...]

Intersecting or combining posting lists is much cheaper than scanning 100 million documents.

Before indexing, an analyzer tokenizes and normalizes text. It may lowercase, normalize accents, remove selected stop words, or stem words. Analyzer design is language- and field-specific. Applying aggressive English stemming to hotel names or Indian place names can damage exact identity matches, so names often use a conservative analyzer plus additional subfields.

BM25 is a reasonable lexical relevance baseline. It rewards term matches while accounting for term rarity and document length. Field boosts can make a name match stronger than the same word buried in a description.

Exact filters use keyword/numeric structures and cached bitsets where appropriate. They narrow eligible documents without treating “has pool” as fuzzy relevance. Facets use aggregations over matching documents. Geo queries use indexed spatial structures to find points in a radius or shape and calculate distance. Fuzziness can recover edit-distance typos, but broad fuzzy matching on every token is expensive and can reduce precision.

OpenSearch is still not the source of truth. It may be seconds behind, can be rebuilt, and may return a hotel whose current inventory is zero. That is acceptable only because later stages and booking validation define the consistency boundary honestly.

8. Understand the query before retrieving

Consider:

cheap hotels near Baga Beach with pool

A query-understanding layer might produce:

category             HOTEL
destination entity   poi_baga_beach
geo center           (15.5553, 73.7517)
radius               product default, perhaps 5–10 km
required amenity     POOL
price intent         BUDGET_SENSITIVE preference
remaining text       hotels

Use the least complicated mechanism that gives reliable behavior:

1. Normalize Unicode, casing, punctuation, locale, and common abbreviations.
2. Use destination and landmark dictionaries to resolve known entities.
3. Apply patterns for dates, occupancy, star ratings, amenities, and “near.”
4. Use entity extraction or a small classifier when rules become brittle.
5. Preserve confidence and fall back to lexical search when understanding is uncertain.

Rules are deterministic, cheap, inspectable, and easy to guard. Dictionaries give controlled destination identity. ML helps with varied language but needs training data, evaluation, versioning, and a fallback.

An LLM can parse long natural-language requests and unseen phrasing, but it is not automatically the best hot-path parser. It adds variable latency, per-request cost, nondeterminism, prompt/model version operations, and the risk of inventing filters. If used, constrain it to a typed schema, validate every output, cache safe interpretations, impose a short deadline, and fall back to rules plus lexical retrieval. A smaller trained model may be faster and easier to evaluate for this narrow task.

Spelling and ambiguity

For “Bega Beech,” use:

• known destination aliases;
• token-level edit distance;
• query-log evidence;
• phonetic/transliteration variants where the locale needs them; and
• confidence thresholds.

Do not silently rewrite a hotel brand with low confidence. The response can say, “Showing results for Baga Beach” and offer the original query.

If “Goa” resolves to several entities, rank by entity type, market, locale, and popularity, or ask the user to disambiguate. Query understanding should produce evidence, not pretend certainty.

9. Distributed retrieval and global top-K

The index is divided into primary shards, each with replica copies. A coordinating node handles a request:

Search API
    │ structured query
    ▼
Coordinator
    ├──► shard copy 0 ── local filters + BM25 + geo ── top-K
    ├──► shard copy 1 ── local filters + BM25 + geo ── top-K
    ├──► shard copy 2 ── local filters + BM25 + geo ── top-K
    └──► ...
              │
              ▼
        global score merge
              │
              ▼
       bounded candidate set

Each shard searches its local documents, applies mandatory filters and geo constraints, scores lexical matches, and returns its local top candidates. The coordinator merges those sorted lists into a global top-K. A later fetch phase obtains stored fields for the winners.

If there are 40 shards and we need 1,000 candidates globally, each shard need not return every match, but it must return enough local candidates to avoid missing global winners. Engine query semantics and reranking design determine the window. Larger windows improve recall at the cost of CPU, memory, and network.

Filters and geo should run during candidate generation, not after ranking millions of ineligible documents. A hard pool filter can remove non-pool properties early. “Cheap,” unless the UI defines an exact price ceiling, should remain a ranking preference because authoritative prices depend on the stay.

Shard design is a benchmarked trade-off

Too few large shards make recovery, relocation, and single-shard query work slow. Too many small shards increase coordinator fanout, heap overhead, scheduling, and tail latency. Growth also matters: adding nodes does not automatically split existing shards.

Allocate replicas across zones and let the engine route queries to healthy copies. Rebalancing after a node failure consumes disk and network; throttle it so recovery does not destroy foreground latency. Use watermarks and capacity headroom before disks approach full.

Sharding by city sounds attractive because travel queries are local, but it is not universally safe:

• Goa or Paris can become hot while other city shards sit idle.
• “Beach hotels in western India” crosses many cities.
• properties near boundaries and multi-destination queries become awkward.
• large cities need splitting while tiny cities create shard proliferation.

Hash-based document distribution balances writes and storage but fans each query across many shards. Destination-aware routing reduces fanout and improves cache locality, but requires a routing directory, supports multi-route queries, and needs protection for hot destinations. A hybrid can group destinations into balanced routing partitions and send broad queries to more partitions. Choose only after measuring query geography and skew; there is no universal hotel-search shard key.

10. Rank after retrieval

Retrieval has produced perhaps 1,000 candidates. Begin ranking with an explainable weighted score:

score =
  w1 × lexical_relevance
  + w2 × guest_rating_quality
  + w3 × price_attractiveness_hint
  + w4 × distance_quality
  + w5 × popularity
  + w6 × availability_confidence

Normalize features before combining them. A rating from 1–5, distance in kilometers, and popularity count have incompatible ranges. Apply business guardrails separately: suppress suspended properties, cap excessive sponsored influence, preserve diversity, and avoid allowing one noisy feature to dominate.

This deterministic ranker is a good first production system because teams can inspect why hotel 184 outranks hotel 910.

Learning to rank

Once impression and outcome data are trustworthy, a learning-to-rank model can combine:

• lexical and field-match features;
• destination and geo distance;
• rating, review volume, and quality;
• indicative price competitiveness;
• recent availability rate;
• property popularity and conversion;
• query-property historical interactions;
• device, locale, market, and stay context; and
• compact user preferences.

CTR and conversion labels are biased by previous ranking: results near the top receive more exposure. Training must account for position and selection bias through randomized exploration, propensity weighting, interleaving, or carefully designed experiments. Otherwise the model learns that “whatever we already placed first is best.”

Offline features such as long-window popularity, property quality, and learned embeddings can be computed in batch or streaming pipelines and written to the index/feature store. Online features such as current stay dates, distance, request locale, and a compact profile are calculated during the request. Do not fetch ten feature services: missing one should not stall search.

An expensive model reranks perhaps the best 50–200 candidates, never all 100 million documents. The lexical engine remains a high-recall candidate generator; the ML layer spends bounded CPU on precision.

Experiments and guardrails

Assign a user or stable anonymous device consistently to an experiment so pagination and repeat visits do not alternate models. Include the experiment/model version in cache and cursor semantics where it changes ordering.

Evaluate offline NDCG or recall, then online CTR and booking conversion with guardrails:

search latency and errors
zero-result rate
price/availability mismatch
cancellation or support rate
property and geographic diversity
revenue alongside user value

A ranking win that raises clicks by showing unavailable bargains is not a product win.

11. Personalize with one compact lookup

Suppose Alice often books quiet boutique hotels near beaches, while Dev prefers low-priced family properties with pools. The same retrieved set can be reranked differently:

Alice: boutique affinity + beach affinity + higher quality preference
Dev:   budget sensitivity + family/pool affinity

At request time, load one compact profile from Redis or a low-latency feature store:

UserSearchProfile
-----------------
user_id_hash
budget_band
preferred_property_types
amenity_affinities
quality_preference
destination_affinities
profile_version
updated_at

Use it only to rerank eligible candidates. Do not let personalization erase explicit filters or make the system unable to find an exact hotel name.

Anonymous and cold-start users receive the non-personalized ranker using query, market, and aggregate popularity. Give the profile lookup a short timeout; on timeout, continue without it. This preserves availability and prevents Redis from becoming a mandatory dependency.

Profiles should contain only features needed for ranking, have deletion and retention controls, and avoid sensitive inferences. Users need appropriate transparency and controls. Raw browsing history does not belong in every search request or log.

12. Treat price and availability as a separate consistency problem

This is where a plausible architecture becomes a travel-search architecture.

Hotel name, location, star rating, and amenities change relatively slowly. Room inventory and stay-specific prices can change by the second and depend on:

property + room type + check-in/out + occupancy
+ cancellation terms + meal plan + currency + promotion

Putting every combination into the main property document would explode document size and write rate. Updating the search index synchronously for every sold room would also create merge pressure and still not make the result transactionally bookable.

Keep a careful boundary:

Search index
  static catalog + coarse price band + recent availability hint
  useful for retrieval and ranking, eventually consistent

Pricing service
  current offer calculations and cached supplier rates

Availability service
  current inventory view

Booking service
  authoritative revalidation and reservation

The request should reduce first, then enrich:

1,000 retrieved candidates
        ↓ rank cheaply
50–80 properties selected for enrichment
        ├──► batch pricing request
        └──► batch availability request
                 run in parallel
        ↓
remove/label unavailable results, rerank if needed
        ↓
return 20

Pricing and availability calls run concurrently, so their contribution is approximately the slower branch plus merge overhead, not the sum of both latencies.

Use batch APIs:

getPrices(property_ids[50], stay, occupancy, currency)
getAvailability(property_ids[50], stay, occupancy)

Batching avoids 100 network round trips and lets downstream services optimize supplier/cache access. Split oversized batches internally and bound concurrency.

Deadlines and honest degradation

Pass a request deadline downstream. If pricing is slow, return cached prices with as_of and “price may have changed,” or omit price for affected results. If availability is unavailable, show “availability unknown—check rooms” rather than “available.” Product policy decides whether unknown results are included.

Overfetch because enrichment may remove candidates:

need 20 displayed
enrich 50
12 sold out
return best 20 of remaining 38

If too many disappear, fetch a bounded second candidate window only if the deadline allows. Do not create an unbounded retrieve-enrich loop.

Strong consistency belongs at booking:

Search result     indicative offer, eventual/cached
Hotel detail      fresher quote, still may expire
Booking submit    authoritative price + inventory validation
Reservation       atomic/leased inventory decision

The booking API returns a changed-price confirmation or sold-out response when reality differs. Search availability can be 99.99% even when a supplier is down because it returns a useful degraded result; it must not claim a room is reserved.

13. Cache at several boundaries, with careful keys

No single cache solves search.

CDN                  public destination/landing responses only
Search API cache     normalized non-personalized result skeletons
Engine query cache   repeated filters/segments, engine-managed
Autocomplete cache   hot prefixes
Profile cache        one compact user feature object
Price cache          stay/occupancy-specific observations
Availability cache   short-lived inventory observations

A correct search-result key includes fields that change meaning:

normalized query
+ normalized filters
+ geo/radius
+ sort
+ locale
+ currency
+ dates
+ occupancy
+ page/cursor where allowed
+ experiment/model version

Omitting occupancy from a price-bearing cache key can show a one-person quote to a family. Normalize unordered filters so pool,wifi and wifi,pool share a key.

CDNs cannot safely cache arbitrary personalized authenticated responses as public objects. They are useful for public destination pages, anonymous common searches with controlled variation, and static assets. Personalized search generally caches a common retrieval skeleton, then performs user-specific reranking and enrichment.

Use short TTLs based on data semantics. Explicit catalog invalidation helps hot changes, but global query-result invalidation for every property update is expensive; short TTL plus versioning is often the accepted trade-off. Price and availability TTLs are shorter and carry timestamps.

For hot misses, request coalescing lets one request recompute while others wait briefly or receive stale data. Add TTL jitter to prevent simultaneous expiry. Stale-while-revalidate can keep a viral query available while refreshing in the background.

If Redis fails, bypass result/profile caches with strict admission control, skip personalization, and protect OpenSearch and enrichment services. Do not retry Redis repeatedly inside one request. A cache failure should reduce speed or freshness, not multiply load until the primary systems fail.

14. Pagination without deep offset scans

from + size is easy for shallow pages:

from=980, size=20

Each shard may need to collect and sort up to 1,000 local hits so the coordinator can discard the first 980 globally. Cost grows with depth and shard count. Results also shift as documents update.

Use a stable sort:

(ranking_score DESC, property_id ASC)

For price sort:

(indicative_price ASC, property_id ASC)

The unique property id breaks ties. Open a short-lived point-in-time snapshot when stable paging matters, then use search_after with the last result’s sort values. The cursor can contain:

version
PIT id
query/filter hash
sort mode
last score or sort value
last property_id
expiry
signature

The client cannot edit an opaque signed cursor to inject engine parameters. A point-in-time view stabilizes indexed documents, but live price and availability can still change between pages; the UI must tolerate that.

Cap traversal around 1,000 results. Most customers refine before that point, and unrestricted export is not the purpose of the endpoint. Analytics or partner export needs a separate asynchronous API.

15. Autocomplete and zero-result recovery

Autocomplete has a tighter budget and a different data shape than full search. It should not execute the entire main search on every keystroke.

Build a small suggestion index containing:

• destinations, neighborhoods, landmarks, brands, and hotel names;
• normalized prefix forms and locale variants;
• popularity and recent trend scores;
• typo/transliteration aliases; and
• optional curated query suggestions.

Keystroke "bag"
      ↓
Autocomplete Service
      ├── hot-prefix Redis cache
      └── prefix/suggestion index
      ↓
8 suggestions in tens of milliseconds

Debounce on the client and cancel superseded requests. Rate-limit automation. Hot prefixes such as “go” or “new” benefit greatly from cache.

For a misspelled full query, spell correction can retry a corrected form when confidence is high and disclose the rewrite. If zero results remain, relax constraints in a controlled sequence:

exact phrase → normal term match
tight geo radius → slightly wider radius
optional preference → lower weight

Never silently relax safety or contractual filters such as accessibility requirements, explicit maximum price, required room capacity, pet policy, or “pool” when the user selected it as mandatory. The response should say what changed: “No matches within 2 km; showing properties within 5 km.” Query relaxation is product behavior, not a hidden engine trick.

16. Build the index without unsafe dual writes

The Catalog Service already commits hotel changes to its database. This is unsafe:

write Catalog DB
write OpenSearch
return success

Either write can succeed alone, and making the user request wait for OpenSearch couples catalog availability to a derived index.

Use a transactional outbox or CDC:

Catalog writer transaction
    ├── update hotel tables
    └── append outbox row / commit-log position
                    │
                    ▼
             CDC / outbox relay
                    │
                    ▼
                  Kafka
                    │
                    ▼
          idempotent Search Indexer
                    │ bulk API
                    ▼
                OpenSearch

Kafka has concrete purposes here:

1. durably buffers a 10K/sec burst while indexers catch up;
2. decouples catalog transactions from search-engine health;
3. allows replay when rebuilding an index;
4. lets search, cache invalidation, analytics, and other consumers move independently; and
5. exposes consumer offsets and lag for operations.

The source transaction’s outbox id or CDC offset proves a committed change was captured. Kafka consumer-group offsets show how far workers have read. The indexed catalog_version and oldest lagging event show whether OpenSearch has applied it. No one metric alone proves end-to-end freshness.

Ordering, retries, and idempotency

Partition catalog events by property_id so updates for one hotel remain ordered while different hotels process in parallel. There is no need for global catalog order.

Use an external version or compare version in indexing:

hotel 184 version 42 arrives → index version 42
hotel 184 version 41 retries → reject as stale
hotel 184 version 42 repeats → same final state

This makes at-least-once delivery safe. A delete emits a tombstone with a version; retaining tombstones long enough prevents a delayed older update from resurrecting the hotel.

Retry transient bulk failures with exponential backoff and jitter:

short wait → retry
longer randomized wait → retry
bounded attempt/time budget exceeded → quarantine/DLQ

Jitter keeps thousands of workers from retrying simultaneously. Permanent mapping or malformed-document failures go to a quarantine/DLQ with property id, version, error, and redacted payload reference. Alert and provide replay after correction. A DLQ is not a place to forget data.

Backpressure starts at indexers: reduce bulk concurrency when OpenSearch rejects writes, pause partitions if necessary, and let Kafka retain the backlog. Use bounded in-memory queues so a slow cluster does not crash workers.

If Kafka is unavailable, Catalog commits still include durable outbox rows. Relays stop, outbox age grows, and search becomes stale. Apply storage limits and operational escalation; if the outbox threatens the source database, degrade nonessential catalog writes or spool through an approved durable alternative. Do not discard acknowledged changes or synchronously switch to direct OpenSearch dual writes.

Zero-downtime rebuild and mapping migration

Analyzer and incompatible mapping changes require a new index:

index_v1 ← current read alias

create index_v2 with new mapping
        ↓
backfill a consistent catalog snapshot into v2
        ↓
consume changes after snapshot position
        ↓
catch v2 up to live Kafka offset
        ↓
validate counts, samples, recall, freshness, and latency
        ↓
atomically switch read alias v1 → v2
        ↓
monitor; switch back to v1 if needed
        ↓
retire v1 after rollback window

For hundreds of millions of documents, throttle backfill separately from live changes. One approach writes live events to both versions while the backfill runs; another records the snapshot offset and lets v2 consume from there after bulk load. In either case, version checks prevent an old backfill document from overwriting a newer live update.

Validation must go beyond document count. Compare sampled fields, deleted documents, destination distributions, golden-query relevance, zero-result rate, and performance. Alias switching is quick; building trustworthy bytes is the long operation.

17. Keep analytics off the indexing control path

Search responses emit impression events; clients or the API emit clicks, filter changes, sorts, zero-result queries, and booking conversions:

Search/Client events
        ↓
Analytics ingestion
        ↓
Kafka analytics topics
        ↓
Data lake / warehouse
        ↓
quality dashboards + feature pipelines + model training
        ↓
versioned offline features/models

Use separate topics or at least separate consumer groups, quotas, and retention from catalog indexing. A click storm must not delay hotel deletion or catalog freshness. Analytics can tolerate more lag and often has different schemas, privacy controls, and retention.

Join impressions to clicks and bookings using request/session identifiers with controlled retention. Record queries, filters, sorts, positions, model/experiment version, zero results, and freshness labels. Minimize user identity and sensitive free text. Aggregate or delete according to regional privacy obligations.

Feedback reaches serving through reviewed, versioned features and model deployments—not by letting every click mutate a property’s online index score immediately. That separation prevents noisy feedback loops and makes rollback possible.

18. Budget the critical path

A realistic warm p95 budget might be:

Gateway, auth, normalization                         15 ms
Query understanding                                 20 ms
Profile lookup (parallel, optional)                  15 ms
OpenSearch distributed retrieval                    70 ms
Rank/rerank                                          25 ms
Pricing batch             ┐
                          ├ run in parallel          90 ms max branch
Availability batch        ┘
Merge, serialize, network inside service             25 ms
Contingency / queueing                                  40 ms
-----------------------------------------------------------
Illustrative service-side p95                       ~285 ms

The profile lookup can start after identity and basic normalization while query understanding proceeds. Pricing and availability begin together after ranking chooses enrichment candidates. Their elapsed cost is approximately max(pricing, availability), not their sum.

This table is not produced by adding each dependency’s p99. Component p99s occur on different requests and may be correlated by shared overload; blindly summing them is neither a valid end-to-end p99 nor a capacity model. Measure distributed traces and model queueing under load.

Warm cache hits may bring p50 near or below 100 ms, especially without personalization or live enrichment. Cold, personalized, enriched requests—particularly those crossing a region or reaching a cold supplier cache—may not. For p99 near 500 ms, optional components need deadlines:

profile late        → generic ranking
ML ranker late      → deterministic score
price late          → cached/omitted price with timestamp
availability late   → unknown label
one shard late      → partial response only under explicit policy

The client separately measures DNS, connection setup, device rendering, and network time. Service SLOs and user-perceived SLOs should be shown on the same dashboard but not conflated.

19. Design graceful degradation before failure

The hierarchy should be deliberate:

full personalized + ML-ranked + fresh enrichment
        ↓
generic deterministic ranking + cached enrichment
        ↓
lexical/filter/geo results + stale labels
        ↓
cached popular result skeleton
        ↓
honest temporary-unavailable response

A timeout stops waiting after a deadline. A circuit breaker observes repeated timeout/failure outcomes:

calls fail repeatedly
      ↓
breaker opens
      ↓
new calls fail fast or use fallback
      ↓ after cooldown
a few probe calls are allowed
      ↓
healthy probes close breaker; failed probes reopen it

This prevents every Search API worker from waiting on a known-sick dependency. Bulkheads—separate thread/connection pools and concurrency budgets—keep pricing saturation from consuming capacity needed for retrieval. Admission control rejects excess or expensive work before it fills the system. Bounded queues turn overload into controlled errors instead of unbounded latency and memory exhaustion.

Search-engine and shard failures

If all of OpenSearch is unavailable, serve a short-lived stale result cache for common queries if policy permits, omit personalized reordering that depends on missing candidates, and return a clear temporary error for misses. Recovery restores healthy nodes or fails over to a regional cluster, then warms critical caches. Search documents are derived and can be replayed; committed catalog data is not lost. Admission control prevents fallback SQL scans from taking down the catalog DB.

If one shard copy fails, OpenSearch should route to its replica. If a shard has no available copy or times out, the coordinator may fail the request by default. For a discovery product, we may return explicitly marked partial results only when policy and metrics permit; silently presenting incomplete geography can be misleading. Repair/relocate the shard from a replica or rebuild. Isolate slow shards with per-shard timeouts and avoid retrying the full fanout repeatedly.

When the cluster is overloaded, reject low-priority or high-complexity queries, reduce expensive facets and fuzzy expansion, cap concurrency, serve stale cache, and scale only when disk/CPU balance permits. Users receive simpler results or 429/503 rather than indefinite spinners. Bounded queues and load shedding stop queueing delay from turning into a total outage.

Cache and optional ranker failures

If Redis is unavailable, skip profile personalization and result-cache reads, and use the deterministic ranker. Search still works but latency and OpenSearch load rise. Apply tighter admission control and request coalescing in process; do not send retry storms to Redis. Cache contents may be lost and safely refill.

If the result cache alone fails, bypass it and protect OpenSearch with concurrency limits. Viral requests may receive local short-TTL cached responses. No source data is lost.

If the ML ranker fails or times out, use the deterministic weighted score already calculated. The user sees less-tailored ordering, not an error. Keep model execution in a separate pool so a model leak cannot exhaust Search API workers.

Price and availability failures

If pricing is slow, stop at its deadline and return recent cached prices with timestamps or omit price. The user can still discover properties; booking revalidates. The breaker limits calls while pricing recovers.

If availability is unavailable, mark inventory unknown or show only safely cached observations according to product policy. Never convert unknown to “available.” Overfetch cannot solve a complete availability outage, so avoid repeatedly asking for more candidates.

If both are degraded, return catalog results with an honest “Check rooms and current price” action. Search remains available because enrichment is not allowed to cascade into retrieval.

Indexing and freshness failures

If Kafka is unavailable, outbox rows accumulate and the index grows stale. Catalog changes remain durable. Users may briefly see old names or suspended inventory hints; sensitive removals may require a separate tightly controlled emergency denylist at read time. Relays resume from the durable position after recovery. Monitor oldest outbox age and source-DB space.

If an index worker crashes, Kafka reassigns its partitions after the consumer timeout. Uncommitted events replay, external versions make duplicates harmless, and consumer lag temporarily rises. Bound retries so one poison document moves to quarantine rather than blocking a whole partition forever.

If the index is stale while the pipeline appears healthy, compare source versions, outbox positions, Kafka offsets, and indexed versions. Reads can disclose limited freshness only where useful; operations may switch to a previously healthy index or replay missing ranges. The catalog DB still holds truth.

Skew, abuse, and regional failures

A viral query creates a hot result key, hot destination, and possibly hot shards. Serve cached common skeletons, coalesce refreshes, pre-warm known campaigns, replicate cache reads, and reserve capacity for nonviral traffic. One hot destination should not consume all query workers.

A malicious query may request a 1,000 km radius, wildcard-like terms, many high-cardinality facets, script sorts, and page 1,000. The API allows only approved filters and sorts, computes a query-complexity cost, caps geo radius/facets/page depth, rate-limits by IP/account/device, and rejects over-budget requests before OpenSearch. Raw DSL never crosses the public boundary.

During a geographic region failure, global routing sends new requests to another regional search cluster. Users may see a slightly stale index and cold caches, and latency may rise. No synchronous call returns to the failed region. Capacity reservations and failover admission control prevent the survivor from collapsing. Async replication resumes from offsets; regional caches refill. Catalog writes follow their own authoritative failover policy rather than search inventing one.

20. Evolve from one region to regional reads

Start in one region because it is easier to operate and enough to validate relevance:

Users → one regional Search API → one OpenSearch cluster
                              → regional caches/enrichment clients

As geography, latency, residency, and disaster-recovery needs grow, build independent serving cells:

                    Global traffic manager
                         ┌─────┴─────┐
                         ▼           ▼
                    Region IN    Region EU
                    Search API   Search API
                    OpenSearch   OpenSearch
                    Redis        Redis
                         ▲           ▲
                         └── async replicated catalog events

Geo routing sends users to a nearby healthy region. Each region performs retrieval, ranking, caching, and enrichment locally. Do not make a synchronous OpenSearch call across regions; a distant or failed region would become part of every request’s critical path.

Catalog changes flow through asynchronously replicated Kafka topics, a global event log, or regional relays with durable positions. Each regional indexer applies the same versioned events to its own index. Regions may be seconds apart, which is acceptable for catalog projection but must be measured as regional staleness.

Keep data residency and locality explicit. User profiles may remain in their legal geography; when unavailable elsewhere, use generic ranking. A global control plane distributes mappings, synonym versions, experiment configuration, schema compatibility, and traffic policy, but the serving data plane should continue with last-known-good configuration if the control plane is down.

On regional failover, the destination region needs spare search and enrichment capacity. It may temporarily disable personalization, expensive facets, or cold deep pages. When the failed region returns, replay events to its recorded offset, validate freshness, warm caches, and shift traffic gradually rather than sending 100% immediately.

The accepted trade-off is eventual regional catalog consistency in exchange for low latency and failure independence.

21. Security, privacy, and abuse controls

Public hotel discovery may work anonymously, but authenticated sessions are needed for saved preferences, member prices, and profile use. The gateway validates tokens; internal services use workload identity and encrypted connections.

If the platform hosts multiple brands, partners, or private inventories, enforce tenant/market authorization before retrieval and again before returning restricted offers. Never depend on stale index membership as the only authorization check.

Layer defenses:

• WAF rules for known bots and attacks;
• token-bucket limits by IP, account, device, and API key;
• separate budgets for autocomplete and full search;
• query-complexity limits for fuzzy terms, facets, radius, and candidate windows;
• allowlisted filters and sorts instead of raw OpenSearch DSL;
• cursor signatures, expiries, and pagination caps;
• bot challenges or partner quotas where appropriate; and
• bulkheads so abusive anonymous traffic cannot consume member/booking capacity.

Encrypt data in transit and at rest. Keep PII out of indexed hotel documents, Kafka keys, traces, and general logs. Minimize profile features, support deletion, and audit administrative access.

Treat query text as untrusted. Parameterize engine requests, escape log fields, protect dashboards from log injection, and avoid logging sensitive queries or authorization headers. Search syntax should be product-defined; users should not be able to submit scripts, regexes, or arbitrary fields.

22. Observe relevance, systems, and outcomes together

At the Search API, measure:

• QPS, p50/p95/p99 latency, timeout and error rates by region;
• zero-result and partial-result rates;
• query relaxations, spelling rewrites, and facet usage;
• candidate counts at retrieval, rerank, enrichment, and return;
• result-cache hit rate and stale-response rate; and
• end-to-end client latency by geography and network.

At OpenSearch, measure:

• query and fetch latency by shard/index;
• CPU, JVM heap/GC, disk utilization, and segment merge time;
• thread-pool queue/rejections and circuit-breaking events;
• shard count, unassigned shards, relocation/recovery time;
• indexing throughput, refresh time, and replica lag; and
• hot shards, slow queries, and cache efficiency.

For Redis and enrichment:

• cache hit rate, evictions, memory, hot keys, and command latency;
• profile timeout/fallback rate;
• pricing/availability batch latency, errors, breaker state, and freshness age;
• fraction of candidates removed as sold out; and
• indicative-to-booking price/availability mismatch.

For indexing:

• outbox backlog and oldest unpublished row;
• Kafka producer errors, consumer lag, and oldest event age;
• documents indexed/sec, bulk rejection rate, retries, and DLQ growth;
• source-version-to-index-version staleness; and
• index rebuild progress and validation differences.

For ranking and business outcomes:

• model latency/errors and deterministic fallback rate;
• NDCG/recall on judged query sets;
• impressions, clicks, reformulations, CTR, and booking conversion;
• position bias diagnostics;
• diversity and fairness guardrails; and
• cancellation/support outcomes caused by stale offers.

Distributed tracing follows a request through understanding, shard retrieval, ranker, profile, pricing, and availability with candidate counts and deadlines. Avoid putting raw sensitive query/profile data into spans.

Alert on outcomes: rising user-visible latency, zero results, stale regional indexes, booking mismatch, unavailable shard copies, or enrichment fallback—not only CPU. A green Kafka consumer lag does not prove documents are searchable, so use synthetic “write catalog update → retrieve expected document” probes.

23. Important trade-offs and accepted costs

OpenSearch versus SQL

SQL is operationally simpler and authoritative. Specialized full-text extensions are a valid early choice. At 100 million documents with mixed lexical, geo, filter, facet, typo, and top-K needs, a dedicated index gives better retrieval structures and independent scaling. Its failure is staleness or unavailability of a derived projection; the accepted cost is a CDC/Kafka pipeline, rebuilds, and eventual consistency.

Offset versus search_after

Offset is easy and permits page numbers, but deep pages force every shard to collect discarded results and moving data causes skips. Point-in-time plus search_after keeps work bounded and ordering more stable. Its cost is opaque state, cursor expiry, and no arbitrary jump to page 37. Customer browsing fits the latter, with a practical 1,000-result cap.

Synchronous versus asynchronous enrichment

Synchronous pricing/availability improves visible freshness but adds dependencies and tail latency. Fully asynchronous enrichment makes search fast but can show stale offers. We synchronously call bounded batch services after narrowing, under deadlines, and fall back to timestamped cache. Booking remains authoritative.

Cache versus fresh reads

Caching absorbs hot queries and dependency latency but fragments across dates, occupancy, locale, experiments, and personalization. Stale results are expected. Use semantic TTLs and visible timestamps; never let a cache claim a booking guarantee.

Rules versus ML ranking

Rules are explainable, deterministic, and reliable for launch. They plateau as interactions become complex. ML can improve ordering but introduces biased labels, model operations, latency, and fallback requirements. Begin with rules, collect clean impressions, then rerank a bounded set with ML behind experiments.

One versus separate event paths

A single Kafka cluster can reduce infrastructure, but catalog indexing and analytics have different criticality, schemas, traffic bursts, and retention. At minimum isolate topics, quotas, and consumer groups; at larger scale use separate clusters if analytics can threaten catalog freshness. The accepted cost is more operational surface.

Strong versus eventual consistency

Making every search result strongly consistent with inventory would couple 12K QPS discovery traffic to many transactional systems and still race with another buyer. We accept eventual catalog and indicative offer data, then require strong authoritative validation during booking.

Single region versus active-active regional reads

One region is simpler and avoids replication lag. It adds distance and a larger failure domain. Independent regional read clusters lower latency and survive regional loss, but duplicate index/cache cost and can serve slightly different catalogs. Search tolerates that trade because its projection is derived; booking needs a stricter ownership model.

24. Final architecture

Only now do we have enough constraints to justify the full design:

                                    REQUEST PATH

 Web / Mobile
      │
      ▼
 CDN / WAF ── public cache only
      │
      ▼
 API Gateway ── auth, rate limit, complexity budget
      │
      ▼
 Search Service
      ├────► Query Understanding ── rules/dictionaries/model fallback
      │
      ├────► Result Cache
      │
      ├────► OpenSearch Coordinator
      │            ├──► shard copies: text + filters + geo
      │            └──► global top-K candidates
      │
      ├────► Profile Store (one optional lookup)
      │
      ├────► Ranker (deterministic, then optional ML rerank)
      │
      ├────► Pricing Batch Service ─┐
      │                             ├── parallel enrichment
      └────► Availability Batch ────┘
                     │
                     ▼
              final merge + cursor
                     │
                     ▼
                  Response


                                    INDEXING PATH

 Catalog Service
      │ one transaction
      ▼
 Catalog DB + Outbox/CDC
      │ durable change
      ▼
 Kafka ───────────────► other independent consumers
      │ property_id order
      ▼
 Search Indexers ── bulk, retry, version, tombstone, DLQ
      │
      ▼
 OpenSearch write alias ──► index_v1 / index_v2 migration


                                    LEARNING PATH

 Search impressions / clicks / filters / zero results / bookings
      │
      ▼
 Analytics Kafka ──► Data Lake / Warehouse
                         │
                         ▼
                features + model training
                         │ reviewed/versioned deploy
                         ▼
                  Ranker / Profile Store

Every request-path arrow has a bounded purpose:

1. CDN/WAF absorbs safe public traffic and abuse; it does not cache arbitrary user responses.
2. The gateway authenticates, limits, and rejects expensive query shapes.
3. Query Understanding converts language into typed intent with confidence and fallback.
4. OpenSearch retrieves a high-recall candidate set using lexical, exact, geo, and facet structures.
5. One profile lookup supplies optional user features; failure means generic ranking.
6. The ranker orders a bounded set, with deterministic scoring always available.
7. Pricing and availability enrich reduced candidates in parallel under deadlines.
8. Final merge overfetches around sold-out results, labels freshness, signs the cursor, and returns 20.

Every indexing arrow preserves recoverability:

1. Catalog and outbox commit together, so an acknowledged source update cannot miss capture.
2. Kafka durably buffers, orders per property, and supports replay.
3. Indexers batch, retry transient errors, quarantine permanent ones, and apply external versions.
4. OpenSearch remains a replaceable projection behind an alias.

The learning path is separate so a click spike cannot contaminate or delay catalog indexing. Only validated, versioned features and models return to online serving.

25. Search request data flow

For “cheap hotels near Baga Beach with pool”:

1. Validate request, rate limit, assign deadline/experiment
2. Normalize text, dates, occupancy, locale, and filters
3. Resolve Baga Beach; classify "cheap"; require POOL
4. Build text + keyword filter + geo OpenSearch query
5. Fan out to shard copies; merge local top-K into ~1,000 candidates
6. Apply deterministic ranking; optional ML/profile rerank on top 50–200
7. Select and overfetch ~50 properties
8. Fetch price and availability batches in parallel
9. Remove or label unavailable/unknown entries under product policy
10. Return 20 results, facets, disclosed relaxations, and signed cursor
11. Emit impression analytics asynchronously

The sequence repeats the governing mental model:

understand → retrieve broadly → rank narrowly
→ personalize optionally → enrich dynamically → respond honestly

26. Indexing document flow

For a hotel adding a pool:

1. Catalog transaction updates amenity records and outbox version 42
2. CDC/outbox relay publishes PropertyChanged(184, version=42)
3. Kafka retains the event, partitioned by property_id
4. Search Indexer consumes and loads/builds the denormalized document
5. Bulk write applies external version 42
6. OpenSearch refresh makes the document searchable
7. Consumer offset advances after successful handling
8. Freshness monitor verifies source/index version and end-to-end retrieval

If step 5 fails transiently, retry with jitter. If the document violates a mapping, quarantine it and alert. If the worker dies before committing its offset, version 42 replays safely. If the hotel is deleted at version 43, a tombstone ensures a delayed version 42 cannot restore it.

27. Interview-ready recap

Scale calculations

Searches/day              100M
Average search QPS        ~1,157
Initial 10× peak          ~11,600 QPS
Planned/tested capacity   higher for skew, failures, and campaigns

Documents                 100M
Raw search projection     ~400 GB at an assumed 4 KB/document
Primary index             ~600 GB–1 TB illustrative range
One replica               ~1.2–2 TB
Provisioned disk          more for merges, growth, recovery, watermarks

Catalog update peak       ~10,000/second
Update ingress            perhaps 20–50 MB/second before indexing overhead
Hot result cache example  ~5 GB payload for 500K × 10 KB entries

All storage, shard, node, and latency numbers require representative benchmarking under simultaneous query, update, merge, and failure load.

Latency budget

The illustrative service p95 is about 285 ms: roughly 15 ms gateway/normalization, 20 ms understanding, 70 ms distributed retrieval, 25 ms rank/rerank, 90 ms for the slower of parallel pricing/availability, 25 ms merge/serialization, and 40 ms contingency. Profile loading is optional and overlapped. Warm p50 near 100 ms is plausible; cold personalized enriched requests may exceed it. p99 near 500 ms depends on deadlines and graceful fallback.

Top 10 decisions

1. Keep retrieval separate from ranking and bound every candidate window.
2. Use OpenSearch as a derived index, never hotel, price, or inventory truth.
3. Normalize query language into typed intent with confidence and fallback.
4. Retrieve lexically with filters and geo; add semantic retrieval only after core quality is measured.
5. Start with deterministic ranking, then ML-rerank a small candidate set.
6. Load at most one compact personalization profile and make it optional.
7. Batch price and availability in parallel after narrowing; validate at booking.
8. Use point-in-time plus search_after, signed cursors, and a depth cap.
9. Feed versioned, idempotent indexers through outbox/CDC and Kafka.
10. Begin single-region, then run independent regional search clusters with async replication.

Top 10 failure scenarios

1. OpenSearch unavailable: stale common cache or clear error; never scan the catalog DB.
2. Shard failure: use a replica; explicitly govern partial results and repair from healthy copies.
3. Cluster overload: shed facets/fuzziness and expensive traffic using bounded queues and admission control.
4. Redis failure: bypass caches, skip profile personalization, protect OpenSearch from miss amplification.
5. Ranker failure: deterministic ranking preserves useful results.
6. Pricing/availability failure: timestamped stale or unknown labels; booking remains authoritative.
7. Kafka/indexer failure: outbox and retained offsets preserve changes; idempotent replay catches up.
8. Stale index: measure source-to-index version, use emergency deny controls where necessary, and replay/rebuild.
9. Viral or malicious query: coalesce hot keys and enforce rate plus complexity budgets.
10. Region failure: geo-route to an independent cluster, accept temporary staleness, and avoid synchronous cross-region calls.

Most important trade-offs

The accepted design spends operational complexity on a dedicated derived index because SQL is not the best serving shape for mixed lexical, facet, filter, geo, and top-K retrieval at this scale. It accepts eventual catalog and regional consistency because strong correctness belongs in catalog and booking systems. It accepts stale, labeled enrichment to preserve search availability. It gives up page-number jumps for bounded search_after pagination, and begins with rules so ML quality gains can be measured rather than assumed.

Senior/Staff-level insights

• Latency targets are meaningless until the API versus user-perceived boundary is named.
• Candidate generation and ranking have different recall, precision, and cost goals.
• Search capacity is often CPU/latency-bound before it is disk-bound.
• Shard count and routing are empirical workload decisions, not formulas.
• Price and availability freshness must be modeled separately from static catalog search.
• “Available” is a business claim; unknown must not be converted to true.
• At-least-once delivery is safe when document versions and tombstones make writes idempotent.
• Reindexing is a normal operation and needs alias rollback, offset capture, and semantic validation.
• Circuit breakers work only with deadlines, bounded retries, bulkheads, and a useful fallback.
• Ranking analytics are biased by prior exposure; more clicks are not automatically better relevance.
• Regional search can be active-active because it is derived; transactional booking ownership is a different design.
• Outcome metrics such as booking mismatch and source-to-search freshness are stronger than infrastructure-only health.

Likely interviewer follow-ups

• How would you measure retrieval recall before introducing a neural model?
• How do you choose shard count and candidate window size?
• Would you route queries by destination, and how would you handle broad queries?
• How does search_after remain stable when scores or prices change?
• What exactly happens when one shard times out?
• How do you prevent a delayed update from resurrecting a deleted hotel?
• How would you index 100 million documents without overwhelming live search?
• How do member-only prices affect caching and authorization?
• How would you train an LTR model without learning position bias?
• When would semantic/vector retrieval be worth its cost?
• How do you handle multi-language place names and transliteration?
• What SLO applies to catalog freshness in each region?

Adversarial challenge questions

• Your p50 target is under 100 ms; which features do you remove first when cold enrichment takes 150 ms?
• A hotel is legally suspended, but Kafka is down. How do you hide it immediately without making a second source of truth?
• Goa becomes 40% of global traffic overnight. Which shard/routing choice fails first?
• Redis fails during a viral campaign. How do you stop cache misses from destroying OpenSearch?
• The ML model improves CTR but raises booking-price mismatch. Do you launch it?
• Pricing returns in 40 ms and availability in 250 ms. Why does your request not take 290 ms?
• A point-in-time cursor expires between pages. What does the API tell the client?
• A backfill event version 17 arrives after live version 22. Which layer rejects it?
• The regional index is 20 minutes stale after failover. Is 99.99% “availability” still honest?
• Why not put every price and room combination into OpenSearch and remove enrichment?

28. A 1–2 minute verbal answer

I would first scope this to hotel discovery over 100 million properties, with lexical text, exact filters, geo, facets, one optional personalization profile, and indicative price and availability. At 100 million searches per day we average about 1,157 QPS; 10× gives 11.6K peak, but I would benchmark and reserve more capacity for destination skew and failures. The core pipeline is query understanding, retrieval, ranking, personalization, enrichment, and response.

Query understanding resolves “cheap hotels near Baga Beach with pool” into a destination, geo radius, mandatory pool filter, and budget preference. OpenSearch is a derived index that uses an inverted index, keyword/numeric filters, geo structures, and distributed shard top-K to reduce 100 million documents to around 1,000 candidates. A deterministic ranker, then optionally an ML model with one compact profile, reranks only the best 50–200. We enrich around 50 properties through parallel batch pricing and availability calls, overfetch around sold-out results, and return 20 with freshness timestamps. Booking revalidates price and inventory authoritatively.

Catalog changes commit with an outbox or CDC position, flow through Kafka partitioned by property id, and are applied by idempotent versioned indexers. Kafka buffers bursts and supports replay. Mapping changes build index_v2 from a snapshot, catch up changes, validate, and switch an alias with rollback. Search uses signed point-in-time/search_after cursors and caps deep traversal.

The service degrades in layers: skip profile personalization, fall back from ML to deterministic ranking, use timestamped cached enrichment, shed facets and expensive queries, and serve safe stale query cache before returning an error. I would start in one region, then deploy independent regional OpenSearch and cache clusters fed asynchronously, with no synchronous cross-region search calls. The key metrics are user latency and zero results, shard health, source-to-index freshness, enrichment fallback and mismatch, and booking outcomes—not just QPS.

The compact mental model is:

Search cheaply narrows.
Ranking carefully orders.
Enrichment cautiously updates.
Booking authoritatively decides.

For the broader interview framework around this problem, see the System Design Interview Complete Guide.