Semantic Chunking
Обновлено 26 августа 2024
Splits the text based on semantic similarity.
Taken from Greg Kamradt's wonderful notebook: https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/5_Levels_Of_Text_Splitting.ipynb
All credit to him.
At a high level, this splits into sentences, then groups into groups of 3 sentences, and then merges one that are similar in the embedding space.
Install Dependencies
!pip install --quiet langchain_experimental langchain_openai
Load Example Data
# This is a long document we can split up.
with open("../../state_of_the_union.txt") as f:
state_of_the_union = f.read()
Create Text Splitter
from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai.embeddings import OpenAIEmbeddings
text_splitter = SemanticChunker(OpenAIEmbeddings())
Split Text
docs = text_splitter.create_documents([state_of_the_union])
print(docs[0].page_content)