Example usage for org.apache.lucene.analysis.standard StandardTokenizer ALPHANUM

Introduction

In this page you can find the example usage for org.apache.lucene.analysis.standard StandardTokenizer ALPHANUM.

Prototype

int ALPHANUM

To view the source code for org.apache.lucene.analysis.standard StandardTokenizer ALPHANUM.

Click Source Link

Document

Alpha/numeric token type

Usage

From source file:org.sindice.siren.analysis.TestTupleTokenizer.java

License:Apache License

@Test
public void testLanguage() throws Exception {
    this.assertTokenizesTo(_t, "\"test\"@en", new String[] { "test", "en" },
            new String[] { "<ALPHANUM>", "<LANGUAGE>" });
    this.assertTokenizesTo(_t, "\"toto@titi.fr \"@fr", new String[] { "toto", "titi.fr", "fr" },
            new String[] { StandardTokenizer.TOKEN_TYPES[StandardTokenizer.ALPHANUM],
                    StandardTokenizer.TOKEN_TYPES[StandardTokenizer.ALPHANUM],
                    TupleTokenizer.TOKEN_TYPES[TupleTokenizer.LANGUAGE] });
}

---