21.5 Interface to the lexical analyzer

The lexical analyzer function that is passed to the parser is expected to be a function of zero arguments that returns two values each time it is called. The first value is the next token on the input and the second value is the semantic value corresponding to that token. If there is no more input, then the lexical analyzer may return either the token :eoi or nil.

For example:

(defparser my-parser   
   ...)
 
(defun my-lexer (stream)   
   .. read next token from stream .. 
  (values token value))
(defun my-symbol-to-string (symbol) 
   .. returns a string ..)
(defun my-parse-stream (stream)  
  (let ((lexer #'(lambda () (my-lexer stream))))  
    (my-parser lexer #'my-symbol-to-string)))

Note that during error correction, the parser may push extra tokens onto the input, in which case they are given the semantic value nil. The semantic actions should therefore be capable of dealing with this situation. Manipulation of the input (for example pushing extra tokens) is done within the parser generator and the lexical analyzer need not concern itself with this.