Uses of Interface
opennlp.tools.tokenize.Tokenizer
-
Packages that use Tokenizer Package Description opennlp.tools.cmdline.parser opennlp.tools.formats.brat opennlp.tools.formats.muc opennlp.tools.tokenize Contains classes related to finding token or words in a string.opennlp.tools.util.featuregen This package contains classes for generating sequence features. -
-
Uses of Tokenizer in opennlp.tools.cmdline.parser
Methods in opennlp.tools.cmdline.parser with parameters of type Tokenizer Modifier and Type Method Description static Parse[]ParserTool. parseLine(String line, Parser parser, Tokenizer tokenizer, int numParses) -
Uses of Tokenizer in opennlp.tools.formats.brat
Constructors in opennlp.tools.formats.brat with parameters of type Tokenizer Constructor Description BratDocumentParser(SentenceDetector sentenceDetector, Tokenizer tokenizer)BratDocumentParser(SentenceDetector sentenceDetector, Tokenizer tokenizer, Set<String> nameTypes)BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples)Creates a newBratNameSampleStream.BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples, Set<String> nameTypes)Creates a newBratNameSampleStream. -
Uses of Tokenizer in opennlp.tools.formats.muc
Constructors in opennlp.tools.formats.muc with parameters of type Tokenizer Constructor Description MucNameContentHandler(Tokenizer tokenizer, List<NameSample> storedSamples)MucNameSampleStream(Tokenizer tokenizer, ObjectStream<String> samples) -
Uses of Tokenizer in opennlp.tools.tokenize
Classes in opennlp.tools.tokenize that implement Tokenizer Modifier and Type Class Description classSimpleTokenizerPerforms tokenization using character classes.classTokenizerMEA Tokenizer for converting raw text into separated tokens.classWhitespaceTokenizerThis tokenizer uses white spaces to tokenize the input text.classWordpieceTokenizerA WordPiece tokenizer.Constructors in opennlp.tools.tokenize with parameters of type Tokenizer Constructor Description TokenizerEvaluator(Tokenizer tokenizer, TokenizerEvaluationMonitor... listeners)Initializes the current instance with the givenTokenizer.TokenizerStream(Tokenizer tokenizer, ObjectStream<String> input) -
Uses of Tokenizer in opennlp.tools.util.featuregen
Constructors in opennlp.tools.util.featuregen with parameters of type Tokenizer Constructor Description TokenPatternFeatureGenerator(Tokenizer supportTokenizer)Initializes a new instance.
-