Java String Tokenize conservativeTokenize(String text)

Description

Conservatively normalize a string while tokenizing it

License

Open Source License

Declaration

public static List<String> conservativeTokenize(String text) 

Method Source Code

//package com.java2s;
//License from project: Open Source License 

import java.util.Arrays;

import java.util.List;

public class Main {
    /** Conservatively normalize a string while tokenizing it */
    public static List<String> conservativeTokenize(String text) {
        String[] token_arr = text.toLowerCase()
                .split("[ \t~`@#$%^&\\*\\(\\)_\\+-=\\{\\}\\[\\]:\";'<>\\?,./\\|\\\\]+");
        return Arrays.asList(token_arr);
    }/*from  w  w w  .j a v  a2s . c o m*/
}

Related

1. countCommonTokens(String string1, String string2)
2. escapedTokens(String s, char separator)
3. extractTokens(String strStartToken, String strEndToken, String strExpression)
4. extractTokens(String text, String delim)