Semantic Chunking — AI articles, news & research

RESEARCHarXiv CS.CL·16d ago

Query-Adaptive Semantic Chunking for Retrieval-Augmented Generation: A Dynamic Strategy with Contextual Window Expansion

This paper introduces Query-Adaptive Semantic Chunking (QASC), a dynamic strategy for Retrieval-Augmented Generation (RAG) systems that integrates user queries into document segmentation. QASC employs cosine similarity scoring, contextual window expansion, and chunk-level score aggregation to optimize context retrieval, addressing limitations of fixed chunking methods.

RAG Natural Language Processing Information Retrieval Semantic Chunking

Day 4 - Chunking continued - RAG

This content discusses different chunking methodologies for RAG, explaining Semantic Chunking which adds sentences to a chunk based on a relevance factor. It also mentions Embedding Chunking and the need for trial and error to determine the best method for a given dataset.