Search Fan-Out & Partial Results Architecture

The core fan-out problem is not parallelism itself. It is preventing slow or unhealthy providers from owning the entire search SLA.

Production scenario

Eight providers are called. Five return below 500 ms, two around 1.5 seconds and one frequently times out. The traveler should see useful results within one second.

Architecture flow

Request
 -> eligible providers
 -> bounded parallel calls
 -> collect successes
 -> first useful threshold
 -> partial response
 -> late arrivals
 -> final merge

Bounded concurrency

Calling 30 providers at once can exhaust connection pools and upstream limits. Concurrency should be bounded by provider, region and request priority.

Early-return criteria

“First provider returned” is not enough. Define a useful threshold such as minimum mapped hotels, provider diversity, offer coverage and maximum initial latency.

Partial/final state

Make response state explicit so the client knows sorting or prices may change when late results arrive.

Trade-offs

Early response improves perceived latency but can change the result set later. Waiting for all providers is more deterministic but exposes users to tail latency.

Failure modes

Watch for late ranking jumps, duplicate merges, broken cancellation propagation, orphaned work, retry storms and analytics that mix partial with final results.

Cache considerations

Provider-level caching and stale-while-revalidate can reduce tail latency without caching the whole merged response.

Observability

Track time-to-first-useful-result, time-to-final-result, completed-provider count, partial-to-final delta, cancellations, orphan tasks and tail latency.

Alternatives

If supplier count is small and latency stable, partial-result complexity may not be justified. It becomes valuable when tail latency dominates.

Production checklist

Define total latency budget, provider timeouts, bounded concurrency, cancellation propagation, partial-state contract, late-result deduplication, deterministic ranking and separate partial/final analytics.