Example usage for org.deeplearning4j.text.tokenization.tokenizer Tokenizer setTokenPreProcessor

Introduction

In this page you can find the example usage for org.deeplearning4j.text.tokenization.tokenizer Tokenizer setTokenPreProcessor.

Prototype

void setTokenPreProcessor(TokenPreProcess tokenPreProcessor);

Source Link

Document

Set the token pre process

Usage

From source file:com.gmo.isto.dlwork.tools.JapaneseTokenizerFactory.java

License:Apache License

@Override
public Tokenizer create(String toTokenize) {
    Tokenizer t = new JapaneseTokenizer(toTokenize);
    t.setTokenPreProcessor(tokenPreProcess);
    return t;/*w  w w.  j  ava 2s  . c o m*/
}

---

From source file:com.gmo.isto.dlwork.tools.JapaneseTokenizerFactory.java

License:Apache License

@Override
public Tokenizer create(InputStream toTokenize) {
    Tokenizer t = new JapaneseStreamTokenizer(toTokenize);
    t.setTokenPreProcessor(tokenPreProcess);
    return t;//www  .  j  av  a  2  s .  co m
}

---