edu.iu.cnets.klatsch.parser
Class TokenStream

java.lang.Object
  extended by edu.iu.cnets.klatsch.parser.TokenStream

public class TokenStream
extends java.lang.Object

This class encapsulates a buffer and makes it possible to read a stream of tokens from it, with one-token lookahead for the parser. The class is public, but only Parser really needs to interact with it.


Nested Class Summary
private static class TokenStream.State
           
 
Field Summary
(package private)  java.lang.String buffer
          the buffer we're converting
(package private)  int offset
          the current offset within the buffer
(package private)  Token peek
          a buffered token to peek at, for ease of parsing
 
Constructor Summary
TokenStream(java.lang.String buffer)
          Creates a new token stream based on the given string data.
 
Method Summary
(package private)  boolean atTerminator()
          Determines if the stream is at an expression terminator (EOF)
(package private)  boolean eof()
          Determines if the stream is at its end.
(package private)  Token get()
          Returns the next token in the stream.
(package private)  Token peek()
          Returns the next token in the stream without actually advancing the pointer within the buffer.
private  Token readToken()
          Reads the next token from the buffer and returns it.
(package private)  Token require(Token.Type type)
          Reads the next token in the stream, which must match the given type to avoid causing an error.
(package private)  Token requireKeyword(java.lang.String keyword)
          Reads the next token in the stream, which must be a keyword of the given type to avoid causing an error.
(package private)  void reset()
          Resets the internal state of the token stream so that subsequent reads will start over from the beginning.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

buffer

java.lang.String buffer
the buffer we're converting


offset

int offset
the current offset within the buffer


peek

Token peek
a buffered token to peek at, for ease of parsing

Constructor Detail

TokenStream

public TokenStream(java.lang.String buffer)
Creates a new token stream based on the given string data.

Parameters:
buffer - the string to tokenize
Method Detail

atTerminator

boolean atTerminator()
               throws TokenException
Determines if the stream is at an expression terminator (EOF)

Returns:
true if we've read to the end of a statement
Throws:
TokenException

eof

boolean eof()
      throws TokenException
Determines if the stream is at its end.

Returns:
true if EOF and false otherwise
Throws:
TokenException

get

Token get()
    throws TokenException
Returns the next token in the stream. Use peek instead for a read with no side-effects.

Returns:
the next token
Throws:
TokenException - if an error was encountered

peek

Token peek()
     throws TokenException
Returns the next token in the stream without actually advancing the pointer within the buffer. Use get() instead for a read with side-effects.

Returns:
the next token
Throws:
TokenException - if an error was encountered

requireKeyword

Token requireKeyword(java.lang.String keyword)
               throws TokenException
Reads the next token in the stream, which must be a keyword of the given type to avoid causing an error.

Parameters:
keyword - the type of keyword required
Returns:
the next token
Throws:
TokenException - if an error or incorrect type was encountered

require

Token require(Token.Type type)
        throws TokenException
Reads the next token in the stream, which must match the given type to avoid causing an error.

Parameters:
type - the type required
Returns:
the next token
Throws:
TokenException - if an error or incorrect type was encountered

reset

void reset()
Resets the internal state of the token stream so that subsequent reads will start over from the beginning.


readToken

private Token readToken()
                 throws TokenException
Reads the next token from the buffer and returns it. If no more tokens are available, this will be the EOF token.

Returns:
the next token
Throws:
TokenException - if an error was encountered