Example usage for org.apache.lucene.analysis.wikipedia WikipediaTokenizerFactory create

Introduction

In this page you can find the example usage for org.apache.lucene.analysis.wikipedia WikipediaTokenizerFactory create.

Prototype

@Override
    public WikipediaTokenizer create(AttributeFactory factory) 

Source Link

Usage

From source file:org.apache.solr.analysis.TestWikipediaTokenizerFactory.java

License:Apache License

public void testTokenizer() throws IOException {
    Reader reader = new StringReader("This is a [[Category:foo]]");
    WikipediaTokenizerFactory factory = new WikipediaTokenizerFactory();
    Tokenizer tokenizer = factory.create(reader);
    assertTokenStreamContents(tokenizer, new String[] { "This", "is", "a", "foo" }, new int[] { 0, 5, 8, 21 },
            new int[] { 4, 7, 9, 24 },
            new String[] { "<ALPHANUM>", "<ALPHANUM>", "<ALPHANUM>", WikipediaTokenizer.CATEGORY },
            new int[] { 1, 1, 1, 1, });
}

---