tomlf_de_lexer Module

Provides tokenization for TOML documents.

The lexer provides a way to turn a stream of characters into tokens which are further processed by the parser and turned into actual TOML data structures. In the current structure no knowledge about the character stream is required in the parser to generate the data structures.

The validity of all tokens can be guaranteed by the lexer, however syntax errors and semantic errors are not detected until the parser is run. Identification of invalid tokens and recovery of the tokenization is done on a best effort basis.

To avoid overflows in the parser due to deeply nested but unclosed groups, the lexer will always tokenize a complete group to verify it is closed properly. Unclosed groups will lead to the first token of the group getting invalidated, to allow reporting in the parsing phase.

Derived Types

type, public, extends(abstract_lexer) ::  toml_lexer

Tokenizer for TOML documents.

Components

Type	Visibility	Attributes		Name		Initial
integer,	public		::	buffer	=	0	Index in the buffer queue
character(kind=tfc, len=:),	public,	allocatable	::	chunk			Current source chunk, for convenience stored as character array rather than string
type(toml_context),	public		::	context			Douple-ended queue for buffering tokens
character(len=:),	public,	allocatable	::	filename			Name of the source file, used for error reporting
integer,	public		::	pos	=	0	Current internal position in the source chunk
type(stack_item),	public,	allocatable	::	stack(:)			Stack of scopes, used to identify the current state of the lexer
integer,	public		::	top	=	0	Last scope of the lexer

Type-Bound Procedures

generic, public :: extract => extract_string, extract_integer, extract_float, extract_bool, extract_datetime	Extract a token
procedure, public :: extract_bool	Extract a boolean from a token
procedure, public :: extract_datetime	Extract a timestamp from a token
procedure, public :: extract_float	Extract a float from a token
procedure, public :: extract_integer	Extract an integer from a token
procedure, public :: extract_string	Extract a string from a token
procedure, public :: get_info	Get information about source
procedure, public :: next	Obtain the next token

Subroutines

public subroutine new_lexer_from_file(lexer, filename, error)

Create a new instance of a lexer by reading from a file

Arguments

Type	Intent	Optional	Attributes		Name
type(toml_lexer),	intent(out)			::	lexer	Instance of the lexer
character(len=*),	intent(in)			::	filename	Name of the file to read from
type(toml_error),	intent(out),		allocatable	::	error	Error code

public subroutine new_lexer_from_string(lexer, string)

Create a new instance of a lexer by reading from a string.

Arguments

Type	Intent	Optional	Attributes		Name
type(toml_lexer),	intent(out)			::	lexer	Instance of the lexer
character(kind=tfc, len=*),	intent(in)			::	string	String to read from

public subroutine new_lexer_from_unit(lexer, io, error)

Create a new instance of a lexer by reading from a unit.

Read more…

Arguments

Type	Intent	Optional	Attributes		Name
type(toml_lexer),	intent(out)			::	lexer	Instance of the lexer
integer,	intent(in)			::	io	Unit to read from
type(toml_error),	intent(out),		allocatable	::	error	Error code