tokenizer
pl: tokenizers
Related Words
parsersegmentertokenlexiconvocabularystringcharacterwordmorphemephonemesyntaxsemanticsgrammarparsingsegmentationsplittingdelimiterwhitespacepunctuationregexregular expressionalgorithmpreprocessingnormalizationstemminglemmatizationcorpustextlanguagelinguisticsnatural language processingmachine learningdeep learningneural networkembeddingvectortensorencodingdecodingdictionaryindexmappinghashstreambuffercompilerinterpreterlexerlexical analysistokenizationpatternmatchsplitjoinconcatenateslicearraylistsequenceunicodeasciicharsetautomatonstate machine