Example usage for org.apache.lucene.analysis.cn.smart HMMChineseTokenizer HMMChineseTokenizer

Introduction

In this page you can find the example usage for org.apache.lucene.analysis.cn.smart HMMChineseTokenizer HMMChineseTokenizer.

Prototype

public HMMChineseTokenizer(AttributeFactory factory) 

Source Link

Document

Creates a new HMMChineseTokenizer, supplying the AttributeFactory

Usage

From source file:cn.tung.javacn.pinyin.SimpleChineseAnalyzer.java

License:Apache License

@Override
public TokenStreamComponents createComponents(String fieldName, Reader reader) {
    final Tokenizer tokenizer = new HMMChineseTokenizer(reader);
    TokenStream result = new PorterStemFilter(tokenizer);
    if (!stopWords.isEmpty()) {
        result = new StopFilter(result, stopWords);
    }//from   w  w  w.j a v  a 2s  . c om
    return new TokenStreamComponents(tokenizer, result);
}

---