Example usage for org.apache.lucene.analysis.wikipedia WikipediaTokenizer CATEGORY

Introduction

In this page you can find the example usage for org.apache.lucene.analysis.wikipedia WikipediaTokenizer CATEGORY.

Prototype

String CATEGORY

To view the source code for org.apache.lucene.analysis.wikipedia WikipediaTokenizer CATEGORY.

Click Source Link

Usage

From source file:org.apache.solr.analysis.TestWikipediaTokenizerFactory.java

License:Apache License

public void testTokenizer() throws IOException {
    Reader reader = new StringReader("This is a [[Category:foo]]");
    WikipediaTokenizerFactory factory = new WikipediaTokenizerFactory();
    Tokenizer tokenizer = factory.create(reader);
    assertTokenStreamContents(tokenizer, new String[] { "This", "is", "a", "foo" }, new int[] { 0, 5, 8, 21 },
            new int[] { 4, 7, 9, 24 },
            new String[] { "<ALPHANUM>", "<ALPHANUM>", "<ALPHANUM>", WikipediaTokenizer.CATEGORY },
            new int[] { 1, 1, 1, 1, });
}

---