Treasure Hunt Engine: The Moment the Documentation Stopped Telling the Truth

An SRE team uncovered critical performance issues with their Treasure Hunt Engine, where the UI froze and irrelevant results were returned, contradicting existing documentation. Investigation revealed the engine used an undocumented two-stage retrieval process, involving an approximate nearest neighbor (ANN) filter and a GPU reranker, with the ANN stage causing unexpected latency spikes.