In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspective

This research paper explores Retrieval-Augmented Generation (RAG) through the lens of in-context optimization. It demonstrates that a single linear self-attention layer can execute a gradient-descent step on a unified linearized RAG objective, revealing an exact regime where retrieval-augmented prediction and in-context optimization align.