USGS Astrogeology Isis API

This class tokenizes a stream. That is, it will take a stream, from a file or a string, and break the contents of the stream into keyword-value pairs. Examples of such are PDS labels, the standard Isis command line, and VICAR labels. Note that this does not validate the stream to ensure it is of PDS- or VICAR-type. It simply creates a list of keyword-value pairs which can be parsed by another object. The ruleset for tokenizing is straightforward. Consider SPACECRAFT=MARS_GLOBAL_SURVEYOR. The keyword would be SPACECRAFT and the value would be MARS_GLOBAL_SURVEYOR. Other valid examples include: LINES=5, FOCAL_LENGTH=12.4, INSTRUMENT="CAMERA_A", LIST=(0,1,5), and DOGS=("LAB","PUG","BULL"). The later examples, are considered arrays and therefore, will have multiple values associated with the keyword. Comments are allowed in the stream and are indicated by either "#" or "/ *" as the first character on the line.

Definition at line 77 of file PvlTokenizer.h.


Public Member Functions
	PvlTokenizer ()
	Constructs a Tokenizer with an empty token list.
	~PvlTokenizer ()
	Destroys the Tokenizer object and token list.
void	Load (std::istream &stream, const std::string &terminator="END")
	Loads the Token list from a stream.
void	Clear ()
	Empties the token list.
std::vector< Isis::PvlToken > &	GetTokenList ()
Protected Member Functions
std::string	ReadComment (std::istream &stream)
	Reads and returns a comment from the stream.
std::string	ReadToken (std::istream &stream)
	Reads and returns a token from the stream.
bool	SkipWhiteSpace (std::istream &stream)
	Skips over whitespace so long as it is not inside quotes.
std::string	ReadToSingleQuote (std::istream &stream)
std::string	ReadToDoubleQuote (std::istream &stream)
std::string	ReadToParen (std::istream &stream)
std::string	ReadToBrace (std::istream &stream)
void	ParseCommaList (Isis::PvlToken &t, const std::string &cl)
	This routine parses a string containing a comma separated list.
void	ValidateCharacter (int c)
	Make sure a character is valid printable (non-control) character.
Protected Attributes
std::vector< Isis::PvlToken >	tokens
	The array of Tokens parse out of the stream.

Constructor & Destructor Documentation

Isis::PvlTokenizer::PvlTokenizer ( )

Constructs a Tokenizer with an empty token list.

Definition at line 32 of file PvlTokenizer.cpp.

References Clear().

Isis::PvlTokenizer::~PvlTokenizer ( )

Destroys the Tokenizer object and token list.

Definition at line 37 of file PvlTokenizer.cpp.

References Clear().

Member Function Documentation

void Isis::PvlTokenizer::Clear ( )

Empties the token list.

Definition at line 42 of file PvlTokenizer.cpp.

References tokens.

Referenced by PvlTokenizer(), and ~PvlTokenizer().

void Isis::PvlTokenizer::Load	(	std::istream &	stream,
		const std::string &	terminator = `"END"`
	)

Loads the Token list from a stream.

The loading will be terminated upon reaching either 1) end-of-stream, or 2) a programmer specified terminator string

Parameters:

	stream	The input stream to tokenize
	terminator	If the tokenizer see's this string as a token in the input stream it will cease tokenizing. Defaults to "END"

Exceptions:

Isis::iException::Parse

Definition at line 58 of file PvlTokenizer.cpp.

References _FILEINFO_, Isis::PvlToken::AddValue(), c, e, Isis::PvlToken::GetKey(), Isis::PvlToken::GetKeyUpper(), Isis::Message::KeywordValueBad(), Isis::iException::Message(), ParseCommaList(), ReadComment(), ReadToBrace(), ReadToDoubleQuote(), ReadToken(), ReadToParen(), ReadToSingleQuote(), SkipWhiteSpace(), tokens, Isis::iString::UpCase(), and ValidateCharacter().

Referenced by Isis::operator>>(), and Isis::ProcessImportVicar::SetVicarFile().

void Isis::PvlTokenizer::ParseCommaList	(	Isis::PvlToken &	t,
		const std::string &	cl
	)			`[protected]`

This routine parses a string containing a comma separated list.

Each of the items in the list is stored as a value in the Token.

Parameters:

	t	Token to load the comma separated list
	cl	iString containing comma separated list

Definition at line 459 of file PvlTokenizer.cpp.

References Isis::PvlToken::AddValue(), c, ReadToBrace(), ReadToDoubleQuote(), ReadToParen(), ReadToSingleQuote(), and SkipWhiteSpace().

Referenced by Load().

string Isis::PvlTokenizer::ReadComment ( std::istream & stream ) [protected]

Reads and returns a comment from the stream.

Parameters:

stream

Input stream to read from

Returns:: string

Definition at line 216 of file PvlTokenizer.cpp.

References c, and ValidateCharacter().

Referenced by Load().

string Isis::PvlTokenizer::ReadToken ( std::istream & stream ) [protected]

Reads and returns a token from the stream.

A token is delimited by either whitespace or an equal sign. In the case of whitespace the token will be considered valueless. That is, there will be no value in the value side of the token (e.g., KEYWORD=).

Parameters:

stream

Input stream to read from