m3tk/src/syn/M3CLex.i3


*************************************************************************
                      Copyright (C) Olivetti 1989                        
                          All Rights reserved                            
                                                                         
 Use and copy of this software and preparation of derivative works based 
 upon this software are permitted to any person, provided this same      
 copyright notice and the following Olivetti warranty disclaimer are      
 included in any copy of the software or any modification thereof or     
 derivative work therefrom made by any person.                           
                                                                         
 This software is made available AS IS and Olivetti disclaims all        
 warranties with respect to this software, whether expressed or implied  
 under any law, including all implied warranties of merchantibility and  
 fitness for any purpose. In no event shall Olivetti be liable for any   
 damages whatsoever resulting from loss of use, data or profits or       
 otherwise arising out of or in connection with the use or performance   
 of this software.                                                       
*************************************************************************

 Copyright (C) 1993, Digital Equipment Corporation           
 All rights reserved.                                        
 See the file COPYRIGHT for a full description.              

INTERFACE M3CLex;

IMPORT Rd;
IMPORT M3AST_LX;
IMPORT M3CHash, M3CToken, M3CReservedWord, M3CSrcPos;
Lexer for Modula 3

TYPE
  T <: Public;
  Public = OBJECT
  METHODS
    init(rd: Rd.T;
         identifiers: M3CReservedWord.Table;
         literals: M3CHash.Table;
         callBack: CallBack): T;
    current(): M3CToken.T;
    next(): M3CToken.T RAISES {Rd.Failure};
    position(): M3CSrcPos.T;
    literal(): Literal_rep;
    identifier(): Symbol_rep;
    disable();
    disabled(): BOOLEAN;
    reset(pos := M3CSrcPos.Null; s: Rd.T := NIL);
    currentTokenToText(): TEXT;
  END;

  CallBack = OBJECT
  METHODS
    badChar(ch: CHAR);
    comment(t: TEXT);
    pragma(t: TEXT);
    whiteSpace(t: TEXT);
  END;
A CallBack object is used by the lexer to notify the caller of tokens that the lexer cannot handle. badChar(ch) will be called for any illegal character. comment will be called for each comment, with the comment text in c , pragma will be called for each pragma with the pragma text in p, and whiteSpace will be called with the white space between tokens.

Symbol_rep = M3AST_LX.Symbol_rep;      (* Represents an identifier *)
  Literal_rep = M3AST_LX.Literal_rep;    (* Represents a numeric, character *)
                                         (* or text literal *)

REVEAL
  Symbol_rep <: M3CHash.Id;              (* Identifiers and literals are *)
  Literal_rep <: M3CHash.Id;             (* both stored in hash tables *)
New(T).init: creates a new lexer. The lexer will read from the given reader'rd'. Any identifiers found will be put into the identifiers hash table. Note that this table already contains all the reserved words; hence only one hash lookup/entry operation is needed for a reserved word or identifier. Any literals found will be put in the literals hash table. The appropriate callBack method will be called when a bad character, comment, pragma or whitespace is found.

current returns the current token. next advances to the next token and then returns the (new) current token. position returns the current position of the lexer encoded as an M3CSrcPos.T.

literal returns a handle for the current literal. Literals are represented by a hash-id for a TEXT that is created by the lexer. The texts are distinguished as follows:

\begin(itemize} \item Valid numeric literals. The text starts with a digit and ends with a hex digit. \item Valid text literals. The text starts and ends with double quote character. \item Valid character literals. The text starts and ends with single quote character. \end{itemize}

In the case of an error in a literal, for example, a based number with one of its digits is out of range, or a text literal with a mising closing quote, the call will return an [|it invalid literal} value. Invalid literals always have at least one character and their first character can be used to distinguish the literal type (numerics start with a digit, texts with double quote, chars with single quote). They are distinguished by the last character being inappropriate, that is not a (hex) digit, single quote or double quote, respectively. If NOT current() IN M3CToken.T.Literals the result of literal is undefined.

If current() = M3CToken.T.Identifier, then identifier returns a unique representative for the identifier text, otherwise the result is undefined.

disable disables the lexer; any call of next will return M3CToken.T.Void and the position will not advance.

disabled returns TRUE if and only if the lexer is disabled.

reset resets the lexer. It sets the current symbol to M3CToken.T.Void and enables the lexer it is disabled. If pos is not M3CSrcPos.Null the lexer position is set to pos. If s is not NIL sets the lexer stream to be s and sets the lexer position to be pos if pos # M3CSrcPos.Null or line 1 offset 0 otherwise.

The TokenToText procedure returns a TEXT that describes the token argument. The currentTokenToText methods returns a text describing the current token; this may give more information than TokenToText because it incorporates identifier names or literal values if the token is in those classes.

PROCEDURE TokenToText(token: M3CToken.T): TEXT;

END M3CLex.