How to solve semantic chunking fragmentation when building RAG over complex PDF tables and technical documentation? Use hierarchical, table‑aware chunking combined with a hybrid dense‑sparse index and a post‑retrieval merge that re‑assembles fragmented rows.
1. Extract raw layout – Use pdfplumber (v0.11) or PyMuPDF with page.get_text("dict") to capture bounding boxes for tables, figures, and headings.
2. Table‑aware tokenization – Apply langchain.text_splitter.RecursiveCharacterTextSplitter with custom separators ["\n\n", "\t", " "] and chunk_size=512, chunk_overlap=64. Add a pre‑processor that detects a table cell grid and forces each row to stay within a single chunk.
```python
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
separators=["\n\n", "\t", " "],
chunk_size=512,
chunk_overlap=64,
length_function=lambda t: len(t.split())
)
chunks = splitter.split_documents(docs)
```
3. Embedding generation – Use InstructorEmbedding (v2) for dense vectors and BM25Encoder from rank_bm25 for sparse tokens. Store both in the same vector DB.
4. Hybrid index creation – In Pinecone (v3) set metric="cosine" and enable metadata_config for sparse vectors. Example:
```python
index = pinecone.Index("rag-hybrid")
index.upsert(vectors=[(id, dense_vec, {"sparse": sparse_vec}) for id, dense_vec, sparse_vec in zip(ids, dense, sparse)])
```
5. Retrieval query – Compute dense query embedding and sparse BM25 weights, then issue a hybrid search with top_k=10, alpha=0.7 (dense weight) and beta=0.3 (sparse weight). Filter results with similarity>0.78 and bm25_score>1.2.
6. Post‑retrieval merge – Group chunks by table_id metadata, re‑order rows using their original row_index, and concatenate until the LLM token limit (≈4k) is reached.
Tool comparison
| Tool | Dense model | Sparse support | Hybrid API |
|------|-------------|----------------|-----------|
| Pinecone | InstructorEmbedding | BM25Encoder | alpha/beta params |
| Milvus | OpenAI text-embedding-3-large | sparse_vectors plugin | search_params |
| Weaviate | text2vec‑openai | bm25 module | HybridSearch |
These steps keep table semantics intact, avoid fragmented rows, and deliver high‑recall retrieval for complex PDFs.