Example usage for org.apache.lucene.analysis.standard StandardTokenizer end

Introduction

In this page you can find the example usage for org.apache.lucene.analysis.standard StandardTokenizer end.

Prototype

@Override
    public final void end() throws IOException 

Source Link

Usage

From source file:com.github.jiloc.USTweetsAnalyzer.Analyzer_Index.java

/**
 * Take in input a string and tokenize it into an ArrayList of strings(tokens) which is returned 
 * @param text - a string that has to be splited 
 * @return an ArrayList of strings //from w  ww .j  a v  a2s .  c o m
 * @throws IOException 
 */
public ArrayList<String> tokenizeText(String text) throws IOException {
    StringReader reader = new StringReader(text);
    StandardTokenizer tokenizer = new StandardTokenizer(Version.LUCENE_41, reader);
    CharTermAttribute charTermAttrib = tokenizer.getAttribute(CharTermAttribute.class);
    tokenizer.reset();
    ArrayList<String> tokens = new ArrayList<String>();

    while (tokenizer.incrementToken()) {
        tokens.add(charTermAttrib.toString());

    }
    tokenizer.end();
    tokenizer.close();
    //  System.out.println("tokenizetext: "+tokens.toString());
    return tokens;

}

---