LuminousSimpleScanner Class Reference

A largely automated scanner. More...

Inheritance diagram for LuminousSimpleScanner:

Collaboration diagram for LuminousSimpleScanner:

Public Member Functions
	main ()
	the method responsible for tokenization
Public Member Functions inherited from LuminousScanner
	__construct ($src=null)
	constructor
	add_filter ($arg1, $arg2, $arg3=null)
	Add an individual token filter.
	add_identifier_mapping ($name, $matches)
	Adds an identifier mapping which is later analysed by map_identifier_filter.
	add_stream_filter ($arg1, $arg2=null)
	Adds a stream filter.
	highlight ($src)
	Public convenience function for setting the string and highlighting it.
	init ()
	Set up the scanner immediately prior to tokenization.
	map_identifier_filter ($token)
	Identifier mapping filter.
	nestable_token ($token_name, $open, $close)
	Handles tokens that may nest inside themselves.
	pop ()
	Pops the top element of the stack, and returns it.
	push ($state)
	Pushes some data onto the stack.
	record ($string, $type, $pre_escaped=false)
	Records a string as a given token type.
	record_range ($from, $to, $type)
	Helper function to record a range of the string.
	remove_filter ($name)
	Removes the individual filter(s) with the given name.
	remove_stream_filter ($name)
	Removes the stream filter(s) with the given name.
	skip_whitespace ()
	Skips whitespace, and records it as a null token.
	start ()
	Flushes the token stream.
	state ()
	Gets the top element on $state_ or null if it is empty.
	tagged ()
	Returns the XML representation of the token stream.
	token_array ()
	Gets the token array.
Public Member Functions inherited from Scanner
	add_pattern ($name, $pattern)
	Allows the caller to add a predefined named pattern.
	bol ()
	Beginning of line?
	check ($pattern)
	Non-consuming lookahead.
	eol ()
	End of line?
	eos ()
	End of string?
	get ($n=1)
	Consume a given number of bytes.
	get_next ($patterns)
	Look for the next occurrence of a set of patterns.
	get_next_named ($patterns)
	Find the index of the next occurrence of a named pattern.
	get_next_strpos ($patterns)
	Look for the next occurrence of a set of substrings.
	index ($pattern)
	Find the index of the next occurrence of a pattern.
	match ()
	Get the result of the most recent match operation.
	match_group ($g=0)
	Get a group from the most recent match operation.
	match_groups ()
	Get the match groups of the most recent match operation.
	match_pos ()
	Get the position (offset) of the most recent match.
	next_match ($consume_and_log=true)
	Automation function: returns the next occurrence of any known patterns.
	peek ($n=1)
	Lookahead into the string a given number of bytes.
	pos ($new_pos=null)
	Getter and setter for the current position (string pointer).
	pos_shift ($offset)
	Moves the string pointer by a given offset.
	remove_pattern ($name)
	Allows the caller to remove a named pattern.
	reset ()
	Reset the scanner.
	rest ()
	Gets the remaining string.
	scan ($pattern)
	Scans at the current pointer.
	scan_until ($pattern)
	Scans until the start of a pattern.
	string ($s=null)
	Getter and setter for the source string.
	terminate ()
	Ends scanning of a string.
	unscan ()
	Revert the most recent scanning operation.

Protected Attributes
	$overrides = array()
	Overrides array.
Protected Attributes inherited from LuminousScanner
	$case_sensitive = true
	Whether or not the language is case sensitive.
	$filters = array()
	Individual token filters.
	$ident_map = array()
	A map of identifiers and their corresponding token names.
	$rule_tag_map = array()
	Rule remappings.
	$state_ = array()
	State stack.
	$stream_filters = array()
	Token stream filters.
	$tokens = array()
	The token stream.
	$user_defs
	Identifier remappings based on definitions identified in the source code.

Additional Inherited Members
Static Public Member Functions inherited from LuminousScanner
static	guess_language ($src, $info)
	Language guessing.
Public Attributes inherited from LuminousScanner
	$version = 'master'
	scanner version.
Protected Member Functions inherited from LuminousScanner
	rule_mapper_filter ($tokens)
	Rule re-mapper filter.
	user_def_filter ($token)
	Filter to highlight identifiers whose definitions are in the source.

Detailed Description

A largely automated scanner.

LuminousSimpleScanner implements a main() method and observes the patterns added with Scanner::add_pattern()

An overrides array allows the caller to override the handling of any token. If an override is set for a token, the override is called when that token is reached and the caller should consume it. If the callback fails to advance the string pointer, an Exception is thrown.

Member Function Documentation

LuminousSimpleScanner::main

(

)

the method responsible for tokenization

The main method is fully responsible for tokenizing the string stored in string() at the time of its call. By the time main returns, it should have consumed the whole of the string and populated the token array.

Reimplemented from LuminousScanner.

Reimplemented in LuminousStatefulScanner.

Member Data Documentation

LuminousSimpleScanner::$overrides = array()

protected

Overrides array.

A map of TOKEN_NAME => callback.

The callbacks are fired by main() when the TOKEN_NAME rule matches. The callback receives the match_groups array, but the scanner is unscan()ed before the callback is fired, so that the pos() is directly in front of the match. The callback is responsible for consuming the token appropriately.

---

The documentation for this class was generated from the following file:

• src/core/scanner.class.php