SentenceSplitter
Defined in: packages/core/src/node-parser/sentence-splitter.ts:23
Parse text with a preference for complete sentences.
Extends
Section titled “Extends”SentenceSplitter<
Options>(nodes,options?):TextNode<Metadata>[]
Defined in: packages/core/src/node-parser/sentence-splitter.ts:23
Parse text with a preference for complete sentences.
Type Parameters
Section titled “Type Parameters”Options
Section titled “Options”Options extends Record<string, unknown>
Parameters
Section titled “Parameters”options?
Section titled “options?”Options
Returns
Section titled “Returns”Constructors
Section titled “Constructors”Constructor
Section titled “Constructor”new SentenceSplitter(
params?):SentenceSplitter
Defined in: packages/core/src/node-parser/sentence-splitter.ts:56
Parameters
Section titled “Parameters”params?
Section titled “params?”Partial<SentenceSplitterParams> & SplitterParams & object
Returns
Section titled “Returns”SentenceSplitter
Overrides
Section titled “Overrides”MetadataAwareTextSplitter.constructor
Properties
Section titled “Properties”includeMetadata
Section titled “includeMetadata”includeMetadata:
boolean=true
Defined in: packages/core/src/node-parser/base.ts:17
Inherited from
Section titled “Inherited from”MetadataAwareTextSplitter.includeMetadata
includePrevNextRel
Section titled “includePrevNextRel”includePrevNextRel:
boolean=true
Defined in: packages/core/src/node-parser/base.ts:18
Inherited from
Section titled “Inherited from”MetadataAwareTextSplitter.includePrevNextRel
chunkSize
Section titled “chunkSize”chunkSize:
number=1024
Defined in: packages/core/src/node-parser/sentence-splitter.ts:27
The chunk size for each chunk. If no tokenSizer is set, this is the number of characters in the chunk. If a tokenSizer is set, this is the number of tokens in the chunk.
chunkOverlap
Section titled “chunkOverlap”chunkOverlap:
number=200
Defined in: packages/core/src/node-parser/sentence-splitter.ts:31
The chunk overlap of each chunk when splitting. If no tokenSizer is set, this is the number of characters in the overlap. If a tokenSizer is set, this is the number of tokens in the overlap.
separator
Section titled “separator”separator:
string=" "
Defined in: packages/core/src/node-parser/sentence-splitter.ts:35
Default separator for splitting into words
paragraphSeparator
Section titled “paragraphSeparator”paragraphSeparator:
string="\n\n\n"
Defined in: packages/core/src/node-parser/sentence-splitter.ts:39
Separator between paragraphs.
secondaryChunkingRegex
Section titled “secondaryChunkingRegex”secondaryChunkingRegex:
string="[^,.;。？！]+[,.;。？！]?"
Defined in: packages/core/src/node-parser/sentence-splitter.ts:43
Backup regex for splitting into sentences.
extraAbbreviations
Section titled “extraAbbreviations”extraAbbreviations:
string[] |undefined=[]
Defined in: packages/core/src/node-parser/sentence-splitter.ts:48
Extra abbreviations to consider while splitting into sentences. For example, for contracts, you may want to consider “LLC.” as an important abbreviation
id:
string
Defined in: packages/core/src/schema/type.ts:22
Inherited from
Section titled “Inherited from”Methods
Section titled “Methods”postProcessParsedNodes()
Section titled “postProcessParsedNodes()”
protectedpostProcessParsedNodes(nodes,parentDocMap):TextNode<Metadata>[]
Defined in: packages/core/src/node-parser/base.ts:27
Parameters
Section titled “Parameters”parentDocMap
Section titled “parentDocMap”Map<string, TextNode<Metadata>>
Returns
Section titled “Returns”Inherited from
Section titled “Inherited from”MetadataAwareTextSplitter.postProcessParsedNodes
getNodesFromDocuments()
Section titled “getNodesFromDocuments()”Defined in: packages/core/src/node-parser/base.ts:84
Parameters
Section titled “Parameters”documents
Section titled “documents”Returns
Section titled “Returns”Inherited from
Section titled “Inherited from”MetadataAwareTextSplitter.getNodesFromDocuments
splitTexts()
Section titled “splitTexts()”splitTexts(
texts):string[]
Defined in: packages/core/src/node-parser/base.ts:126
Parameters
Section titled “Parameters”string[]
Returns
Section titled “Returns”string[]
Inherited from
Section titled “Inherited from”MetadataAwareTextSplitter.splitTexts
splitTextsMetadataAware()
Section titled “splitTextsMetadataAware()”splitTextsMetadataAware(
texts,metadata):string[]
Defined in: packages/core/src/node-parser/base.ts:142
Parameters
Section titled “Parameters”string[]
metadata
Section titled “metadata”string[]
Returns
Section titled “Returns”string[]
Inherited from
Section titled “Inherited from”MetadataAwareTextSplitter.splitTextsMetadataAware
getMetadataString()
Section titled “getMetadataString()”
protectedgetMetadataString(node):string
Defined in: packages/core/src/node-parser/base.ts:151
Parameters
Section titled “Parameters”Returns
Section titled “Returns”string
Inherited from
Section titled “Inherited from”MetadataAwareTextSplitter.getMetadataString
parseNodes()
Section titled “parseNodes()”Defined in: packages/core/src/node-parser/base.ts:161
Parameters
Section titled “Parameters”Returns
Section titled “Returns”Inherited from
Section titled “Inherited from”MetadataAwareTextSplitter.parseNodes
splitTextMetadataAware()
Section titled “splitTextMetadataAware()”splitTextMetadataAware(
text,metadata):string[]
Defined in: packages/core/src/node-parser/sentence-splitter.ts:82
Parameters
Section titled “Parameters”string
metadata
Section titled “metadata”string
Returns
Section titled “Returns”string[]
Overrides
Section titled “Overrides”MetadataAwareTextSplitter.splitTextMetadataAware
splitText()
Section titled “splitText()”splitText(
text):string[]
Defined in: packages/core/src/node-parser/sentence-splitter.ts:97
Parameters
Section titled “Parameters”string
Returns
Section titled “Returns”string[]
Overrides
Section titled “Overrides”MetadataAwareTextSplitter.splitText
_splitText()
Section titled “_splitText()”_splitText(
text,chunkSize):string[]
Defined in: packages/core/src/node-parser/sentence-splitter.ts:101
Parameters
Section titled “Parameters”string
chunkSize
Section titled “chunkSize”number
Returns
Section titled “Returns”string[]