Package org.antlr.runtime
Class CommonTokenStream
- java.lang.Object
-
- org.antlr.runtime.BufferedTokenStream
-
- org.antlr.runtime.CommonTokenStream
-
- All Implemented Interfaces:
IntStream
,TokenStream
- Direct Known Subclasses:
TokenRewriteStream
public class CommonTokenStream extends BufferedTokenStream
The most common stream of tokens where every token is buffered up and tokens are filtered for a certain channel (the parser will only see these tokens). Even though it buffers all of the tokens, this token stream pulls tokens from the tokens source on demand. In other words, until you ask for a token using consume(), LT(), etc. the stream does not pull from the lexer. The only difference between this stream and BufferedTokenStream superclass is that this stream knows how to ignore off channel tokens. There may be a performance advantage to using the superclass if you don't pass whitespace and comments etc. to the parser on a hidden channel (i.e., you set $channel instead of calling skip() in lexer rules.)- See Also:
UnbufferedTokenStream
,BufferedTokenStream
-
-
Field Summary
Fields Modifier and Type Field Description protected int
channel
Skip tokens on any channel but this one; this is how we skip whitespace...-
Fields inherited from class org.antlr.runtime.BufferedTokenStream
lastMarker, p, range, tokens, tokenSource
-
-
Constructor Summary
Constructors Constructor Description CommonTokenStream()
CommonTokenStream(TokenSource tokenSource)
CommonTokenStream(TokenSource tokenSource, int channel)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
consume()
Always leave p on an on-channel token.int
getNumberOfOnChannelTokens()
Count EOF just once.protected Token
LB(int k)
Token
LT(int k)
Get Token at current input pointer + i ahead where i=1 is next Token.void
reset()
void
setTokenSource(TokenSource tokenSource)
Reset this token stream by setting its token source.protected void
setup()
protected int
skipOffTokenChannels(int i)
Given a starting index, return the index of the first on-channel token.protected int
skipOffTokenChannelsReverse(int i)
-
-
-
Constructor Detail
-
CommonTokenStream
public CommonTokenStream()
-
CommonTokenStream
public CommonTokenStream(TokenSource tokenSource)
-
CommonTokenStream
public CommonTokenStream(TokenSource tokenSource, int channel)
-
-
Method Detail
-
consume
public void consume()
Always leave p on an on-channel token.- Specified by:
consume
in interfaceIntStream
- Overrides:
consume
in classBufferedTokenStream
-
LB
protected Token LB(int k)
- Overrides:
LB
in classBufferedTokenStream
-
LT
public Token LT(int k)
Description copied from interface:TokenStream
Get Token at current input pointer + i ahead where i=1 is next Token. i<0 indicates tokens in the past. So -1 is previous token and -2 is two tokens ago. LT(0) is undefined. For i>=n, return Token.EOFToken. Return null for LT(0) and any index that results in an absolute address that is negative.- Specified by:
LT
in interfaceTokenStream
- Overrides:
LT
in classBufferedTokenStream
-
skipOffTokenChannels
protected int skipOffTokenChannels(int i)
Given a starting index, return the index of the first on-channel token.
-
skipOffTokenChannelsReverse
protected int skipOffTokenChannelsReverse(int i)
-
reset
public void reset()
- Overrides:
reset
in classBufferedTokenStream
-
setup
protected void setup()
- Overrides:
setup
in classBufferedTokenStream
-
getNumberOfOnChannelTokens
public int getNumberOfOnChannelTokens()
Count EOF just once.
-
setTokenSource
public void setTokenSource(TokenSource tokenSource)
Reset this token stream by setting its token source.- Overrides:
setTokenSource
in classBufferedTokenStream
-
-