![[attach.png]](../idldoc-resources/attach.png)
templating/
mgfftokenizer__define.pro
Parses a file given a pattern of delimiters and produces tokens one at a time.
Class description for mgfftokenizer
Properties
- string_array init
- pattern init
Routines
mgfftokenizer::restorePos, mementoRestores the tokenizer to the state/location it was in when the given memento was produced.
result = mgfftokenizer::savePos()Saves the current state/location of the tokenizer in a memento structure.
result = mgfftokenizer::getCurrentLine( [number=long])Returns the current line of the tokenized file.
result = mgfftokenizer::next( [pre_delim=string] [, post_delim=string] [, /newline])Returns the next token of the file.
result = mgfftokenizer::done()Returns whether there are any more tokens in the file.
mgfftokenizer::resetResets the tokenizer to the beginning of the tokenized file.
mgfftokenizer::cleanupFrees resources.
result = mgfftokenizer::init(filename [, pattern=string] [, /string_array])Creates a tokenizer for a given file with a given pattern.
mgfftokenizer__defineDefine instance variables.
Routine details
top source mgfftokenizer::restorePos
mgfftokenizer::restorePos, memento
Restores the tokenizer to the state/location it was in when the given memento was produced.
Parameters
- memento in required type=structure
memento produced by save_pos method
top source mgfftokenizer::savePos
result = mgfftokenizer::savePos()
Saves the current state/location of the tokenizer in a memento structure.
Return value
structure
top source mgfftokenizer::getCurrentLine
result = mgfftokenizer::getCurrentLine( [number=long])
Returns the current line of the tokenized file.
Return value
string
Keywords
- number out optional type=long
line number of returned line
top source mgfftokenizer::next
result = mgfftokenizer::next( [pre_delim=string] [, post_delim=string] [, /newline])
Returns the next token of the file.
Return value
string
Keywords
- pre_delim out optional type=string
delimiter before the returned token
- post_delim out optional type=string
delimiter after the returned token
- newline out optional type=boolean
true if token is first on a new line
top source mgfftokenizer::done
result = mgfftokenizer::done()
Returns whether there are any more tokens in the file. Parses a new line of the file if necessary.
Return value
1B if no more tokens or 0B otherwise
top source mgfftokenizer::reset
mgfftokenizer::reset
Resets the tokenizer to the beginning of the tokenized file.
top source mgfftokenizer::init
result = mgfftokenizer::init(filename [, pattern=string] [, /string_array])
Creates a tokenizer for a given file with a given pattern. Creating the tokenizer opens the file.
Return value
1 if successful, 0 otherwise
Parameters
- filename in required type=string/strarr
filename of the file to be tokenized or (if /STRING_ARRAY) string array of text to be tokenized
Keywords
- pattern in optional type=string default=space
regular expression (as in STRPSLIT) to split the text of the file into tokens
- string_array in optional type=boolean
set to indicate that filename argument is a string array to be tokenized instead of a filename
File attributes
| Modification date: | Mon Nov 29 18:32:28 2010 |
| Lines: | 293 |
| Docformat: | rst rst |
Contact me if you have enhancement requests or bug fixes.