Example usage for weka.core.tokenizers WordTokenizer subclass-usage

Introduction

In this page you can find the example usage for weka.core.tokenizers WordTokenizer subclass-usage.

Usage

From source file com.reactivetechnologies.analytics.lucene.InstanceTokenizer.java

class InstanceTokenizer extends WordTokenizer {

    /**
     * Converts String attributes into a set of attributes representing word occurrence information from the text contained in the strings. 
     * The set of words (attributes) is determined by the first batch filtered (typically training data). Uses a Lucene analyzer to tokenize
     * the string. NOTE: The text string should either be the first or last attribute