Example usage for org.apache.lucene.analysis.pl PolishAnalyzer tokenStream

Introduction

In this page you can find the example usage for org.apache.lucene.analysis.pl PolishAnalyzer tokenStream.

Prototype

public final TokenStream tokenStream(final String fieldName, final Reader reader) 

Source Link

Document

Returns a TokenStream suitable for fieldName, tokenizing the contents of reader.

Usage

From source file:org.omegat.tokenizer.LucenePolishTokenizer.java

License:Open Source License

@SuppressWarnings("resource")
@Override/*  w  w w  . j av  a  2s .  c  o m*/
protected TokenStream getTokenStream(final String strOrig, final boolean stemsAllowed,
        final boolean stopWordsAllowed) throws IOException {
    if (stemsAllowed) {
        CharArraySet stopWords = stopWordsAllowed ? PolishAnalyzer.getDefaultStopSet() : CharArraySet.EMPTY_SET;
        PolishAnalyzer analyzer = new PolishAnalyzer(stopWords);
        return analyzer.tokenStream("", new StringReader(strOrig));
    } else {
        return getStandardTokenStream(strOrig);
    }
}

---