Example usage for org.apache.lucene.analysis.cjk CJKAnalyzer tokenStream

Introduction

In this page you can find the example usage for org.apache.lucene.analysis.cjk CJKAnalyzer tokenStream.

Prototype

public final TokenStream tokenStream(final String fieldName, final Reader reader) 

Source Link

Document

Returns a TokenStream suitable for fieldName, tokenizing the contents of reader.

Usage

From source file:jp.mwsoft.cjkanalyzers.CJKAnalyzerNoSplitKatakana.java

License:Apache License

public static void main(String[] args) throws Exception {

    Set<String> stopWords = new HashSet<String>();
    stopWords.add("??");
    stopWords.add("??");

    java.io.StringReader reader = new java.io.StringReader("??????");

    CJKAnalyzer analyzer = new CJKAnalyzer(Version.LUCENE_35);
    TokenStream stream = analyzer.tokenStream("test", reader);

    for (int i = 0; i < 10; i++) {
        stream.incrementToken();/*w w  w .  ja va2  s .  co  m*/
        System.out.println(stream);
    }
}

---