Package org.antlr.runtime
Class Lexer
java.lang.Object
org.antlr.runtime.BaseRecognizer
org.antlr.runtime.Lexer
- All Implemented Interfaces:
TokenSource
A lexer is recognizer that draws input symbols from a character stream.
lexer grammars result in a subclass of this object. A Lexer object
uses simplified match() and error recovery mechanisms in the interest
of speed.
-
Field Summary
FieldsModifier and TypeFieldDescriptionprotected CharStream
Where is the lexer drawing characters from?Fields inherited from class org.antlr.runtime.BaseRecognizer
DEFAULT_TOKEN_CHANNEL, HIDDEN, INITIAL_FOLLOW_STACK_SIZE, MEMO_RULE_FAILED, MEMO_RULE_UNKNOWN, NEXT_TOKEN_RULE_NAME, state
-
Constructor Summary
ConstructorsConstructorDescriptionLexer()
Lexer
(CharStream input) Lexer
(CharStream input, RecognizerSharedState state) -
Method Summary
Modifier and TypeMethodDescriptionemit()
The standard method called to automatically emit a token at the outermost lexical rule.void
Currently does not support multiple emits per nextToken invocation for efficiency reasons.getCharErrorDisplay
(int c) int
What is the index of the current character of lookahead?int
Returns the EOF token (default), if you need to return a custom token instead override this method.getErrorMessage
(RecognitionException e, String[] tokenNames) What error message should be generated for the various exception types? Not very object-oriented code, but I like having all error message generation within one method rather than spread among all of the exception classes.int
getLine()
Where are you getting tokens from? normally the implication will simply ask lexers input stream.getText()
Return the text matched so far for the current token or any text override.void
match
(int c) void
void
matchAny()
void
matchRange
(int a, int b) abstract void
mTokens()
This is the lexer entry point that sets instance var 'token'Return a token from this source; i.e., match a token on the char stream.void
Lexers can normally match any char in it's vocabulary after matching a token, so do the easy thing and just kill a character and hope it all works out.void
Report a recognition problem.void
reset()
reset the parser's state; subclasses must rewinds the input streamvoid
setCharStream
(CharStream input) Set the char stream and reset the lexervoid
Set the complete text of this token; it wipes any previous changes to the text.void
skip()
Instruct the lexer to skip creating a token for current lexer rule and look for another token.void
void
Methods inherited from class org.antlr.runtime.BaseRecognizer
alreadyParsedRule, beginResync, combineFollows, computeContextSensitiveRuleFOLLOW, computeErrorRecoverySet, consumeUntil, consumeUntil, displayRecognitionError, emitErrorMessage, endResync, failed, getBacktrackingLevel, getCurrentInputSymbol, getErrorHeader, getGrammarFileName, getMissingSymbol, getNumberOfSyntaxErrors, getRuleInvocationStack, getRuleInvocationStack, getRuleMemoization, getRuleMemoizationCacheSize, getTokenErrorDisplay, getTokenNames, match, matchAny, memoize, mismatchIsMissingToken, mismatchIsUnwantedToken, pushFollow, recover, recoverFromMismatchedSet, recoverFromMismatchedToken, setBacktrackingLevel, toStrings, traceIn, traceOut
-
Field Details
-
input
Where is the lexer drawing characters from?
-
-
Constructor Details
-
Lexer
public Lexer() -
Lexer
-
-
Method Details
-
reset
public void reset()Description copied from class:BaseRecognizer
reset the parser's state; subclasses must rewinds the input stream- Overrides:
reset
in classBaseRecognizer
-
nextToken
Return a token from this source; i.e., match a token on the char stream.- Specified by:
nextToken
in interfaceTokenSource
-
getEOFToken
Returns the EOF token (default), if you need to return a custom token instead override this method. -
skip
public void skip()Instruct the lexer to skip creating a token for current lexer rule and look for another token. nextToken() knows to keep looking when a lexer rule finishes with token set to SKIP_TOKEN. Recall that if token==null at end of any token rule, it creates one for you and emits it. -
mTokens
This is the lexer entry point that sets instance var 'token'- Throws:
RecognitionException
-
setCharStream
Set the char stream and reset the lexer -
getCharStream
-
getSourceName
Description copied from interface:TokenSource
Where are you getting tokens from? normally the implication will simply ask lexers input stream.- Specified by:
getSourceName
in interfaceTokenSource
- Specified by:
getSourceName
in classBaseRecognizer
-
emit
Currently does not support multiple emits per nextToken invocation for efficiency reasons. Subclass and override this method and nextToken (to push tokens into a list and pull from that list rather than a single variable as this implementation does). -
emit
The standard method called to automatically emit a token at the outermost lexical rule. The token object should point into the char buffer start..stop. If there is a text override in 'text', use that to set the token's text. Override this method to emit custom Token objects. If you are building trees, then you should also override Parser or TreeParser.getMissingSymbol(). -
match
- Throws:
MismatchedTokenException
-
matchAny
public void matchAny() -
match
- Throws:
MismatchedTokenException
-
matchRange
- Throws:
MismatchedRangeException
-
getLine
public int getLine() -
getCharPositionInLine
public int getCharPositionInLine() -
getCharIndex
public int getCharIndex()What is the index of the current character of lookahead? -
getText
Return the text matched so far for the current token or any text override. -
setText
Set the complete text of this token; it wipes any previous changes to the text. -
reportError
Description copied from class:BaseRecognizer
Report a recognition problem. This method sets errorRecovery to indicate the parser is recovering not parsing. Once in recovery mode, no errors are generated. To get out of recovery mode, the parser must successfully match a token (after a resync). So it will go: 1. error occurs 2. enter recovery mode, report error 3. consume until token found in resynch set 4. try to resume parsing 5. next match() will reset errorRecovery mode If you override, make sure to update syntaxErrors if you care about that.- Overrides:
reportError
in classBaseRecognizer
-
getErrorMessage
Description copied from class:BaseRecognizer
What error message should be generated for the various exception types? Not very object-oriented code, but I like having all error message generation within one method rather than spread among all of the exception classes. This also makes it much easier for the exception handling because the exception classes do not have to have pointers back to this object to access utility routines and so on. Also, changing the message for an exception type would be difficult because you would have to subclassing exception, but then somehow get ANTLR to make those kinds of exception objects instead of the default. This looks weird, but trust me--it makes the most sense in terms of flexibility. For grammar debugging, you will want to override this to add more information such as the stack frame with getRuleInvocationStack(e, this.getClass().getName()) and, for no viable alts, the decision description and state etc... Override this to change the message generated for one or more exception types.- Overrides:
getErrorMessage
in classBaseRecognizer
-
getCharErrorDisplay
-
recover
Lexers can normally match any char in it's vocabulary after matching a token, so do the easy thing and just kill a character and hope it all works out. You can instead use the rule invocation stack to do sophisticated error recovery if you are in a fragment rule. -
traceIn
-
traceOut
-