Example usage for org.apache.lucene.analysis.util CharTokenizer CharTokenizer

Introduction

In this page you can find the example usage for org.apache.lucene.analysis.util CharTokenizer CharTokenizer.

Prototype

public CharTokenizer(AttributeFactory factory, int maxTokenLen) 

Source Link

Document

Creates a new CharTokenizer instance

Usage

From source file:com.mathworks.xzheng.analysis.keyword.SimpleKeywordAnalyzer.java

License:Apache License

@Override
protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
    Tokenizer tokenizer = new CharTokenizer(Version.LUCENE_46, reader) {
        @Override// www . ja v  a2s.  co  m
        protected boolean isTokenChar(int c) {
            // TODO Auto-generated method stub
            return true;
        }
    };
    return new TokenStreamComponents(tokenizer);
}

---