Lexer.hpp

A general-purpose, fast lexer.

Build a lexer that can convert input strings or streams into a series of provided tokens.

Use AddToken(name, regex) to provide the relevant tokens. ‘name’ is the unique name for this token. ‘regex’ is the regular expression that describes this token. It will return a unique ID associated with this token that will be used in output Token objects later. You may provide additional arguments to indicate of the lexeme should be saved in a token, whether the token should be saved at all (unsaved token types are simply left out of the output), and a more detailed description.

IgnoreToken(name, regex) uses the same arguments, but is used for tokens that should be skipped over during lexical analysis

Names and IDs can be recovered later using GetTokenID(name) and GetTokenName(id).

Either strings or streams can be converted into their associated Tokens. Tokenize(in) returns the entire series of tokens as a TokenStream. TokenizeNext(in, &line) return the next token only, and updates the line number Process(in) returns the next token (even if it’s normally ignored); does not track line

Finally, GetLexeme() returns the lexeme from the most recent token found.

Note

Status: BETA

Defines

INCLUDE_EMP_COMPILER_LEXER_HPP_GUARD

Typedefs

using Lexer = Lexer_Base<255>

Functions

static bool AtEOF(std::istream &is)

static bool AtLineStart(std::istream &is)

template<int MAX_ID> class Lexer_Base

#include <Lexer.hpp>

A lexer with a set of token types (and associated regular expressions) MAX_ID determines the maximum number of allowed lexical rules.

Public Functions

inline size_t GetNumTokens() const: How many types of tokens can be identified in this Lexer?

void Reset(): Reset the entire lexer to its start state.

int AddToken(String name, String regex, bool save_lexeme = true, bool save_token = true, String desc = ""): Add a new token, specified by a name and the regex used to identify it. Note that token ids count down with highest IDs having priority.

int IgnoreToken(String name, String regex, String desc = ""): Add a token to ignore, specified by a name and the regex used to identify it.

int GetTokenID(String name) const: Get the ID associated with a token type (you provide the token name)

const TokenType &GetTokenType(int id) const: Get the full information about a token (you provide the id)

String GetTokenName(int id) const: Get the name associated with a token type (you provide the ID)

bool GetSaveToken(int id) const: Identify if a token should be saved.

void Generate() const: Create the NFA that will identify the current set of tokens in a sequence.

Token ToToken(std::string_view in_str) const: Shortcut to just get a single token.

Token TokenizeNext(std::string_view in_str, size_t &cur_line, size_t &start_pos, bool keep_all = false) const: Perform a single step in the Tokenize process, updating line number and start position as we go.

Token TokenizeNext(std::istream &in_stream, size_t &cur_line, bool keep_all = false) const: Perform a single step in the Tokenize process, updating line number as we go.

inline Token TokenizeNext(std::istream &in_stream) const: Perform a single step in the Tokenize process.

TokenStream Tokenize(std::string_view str, String name = "in_view", bool keep_all = false) const: Turn an input string into a vector of tokens.

TokenStream Tokenize(std::istream &is, String name = "in_stream", bool keep_all = false) const: Turn an input stream of text into a vector of tokens.

TokenStream Tokenize(const vector<String> &str_v, String name = "in_string vector", bool keep_all = false) const: Turn a vector of strings into a vector of tokens.

inline TokenStream TokenizeFile(String filename) const: Load from a file and tokenize it.

inline const String &GetLexeme() const: Get the lexeme associated with the last token identified.

void Print(std::ostream &os = std::cout) const: Print the full information about this lexer (for debugging)

void DebugString(String test_string) const: Try out the lexer on a string and demonstrate how it’s tokenized.

void DebugToken(int token_id) const

void WriteCPP(CPPFile &file, String object_name = "Lexer", String DFA_name = "DFA", String token_name = "Token", bool save_lexeme = true, bool save_line_id = true, bool save_col_id = true) const

Private Functions

Token Process(std::string_view in_str, size_t &start_pos) const: Get the next token found in an input string. Do so by examining one character at a time. Keep going as long as there is a chance of a valid lexeme (since we need to choose the longest one available), saving best option so far. Once no more other valid lexemes are possible, stop and return the best we’ve found so far.

Token Process(std::istream &is) const: Shortcut to process a stream rather than a string.

Private Members

vector<TokenType> token_set = {}: List of all active tokens types.

std::map<String, int> token_map = {}: Map of token names to id.

int cur_token_id = MAX_ID : ID for next new token (higher has priority)

mutable bool generate_lexer = false: Do we need to regenerate the lexer?

mutable DFA lexer_dfa = {}: Table driven lexer implementation.

mutable String lexeme = {}: Current state of lexeme being generated.

Private Static Functions

static inline const TokenType &ERROR_TOKEN()

Private Static Attributes

static constexpr int ERROR_ID = -1: Code for unknown token ID.