Parsing and the Token StreamA line of Smojo code or a Smojo script in a text file is represented as a token stream - a sequence of strings delimited by spaces. This token stream is then interpreted token by token. Each token represents a word or number or string. Quite often, your word needs to access tokens ahead of the token stream. You can do this using the
|Used in compilation mode. It takes the following token of the enclosing word. The token is stored as a string and it replaces the
|Takes one argument, a character telling the word when to stop.|
|Converts the following token into a character. Some useful characters are \n (carriage return) \r (linefeed) and \t (tab).|
|A character representing a space|
|Takes an XT as an argument. If the XT returns 0, this input will be ignored. If the XT returns an item, say r and 1, the parser will then process the item r. If -1 is returned, the parser will pass this input.|
|Takes an XT as an argument. This removes the parser which has been included previously.|
-PARSER, we are able create some words so that we can read in inputs of special forms and process them into a useful and valid information. A simple example is shown in Example 3 where we now can read fractions.
: coming-up "What comes after?" . cr next-token . ; coming-up Me! What comes after? Me! okExample 2
CHOMPreads the entire line to the carriage return and put the result on the stack.
: chomp [char] \n parse ; chomp Hello World! . Hello World!Example 3 The line comment word
\reads the entire line to the carriage return. The result is dropped then from the stack.
: \ [char] \n parse drop ;Example 4
: fraction? ( "s" -- f ) [char] / indexofc 0 > ; : to-number ( "a/b" -- d ) "/" split drop int swap int swap /. ; : fractions ( "s" -- 0 | r 1 | -1 ) dup fraction? -> to-number 1 | otherwise drop -1 |. ; 1/2 Error: Unknown word 1/2 ' fractions +parser 1/2 1/2 +. . cr 1.0 ok ' fractions -parser 4/5 Error: Unkown word 4/5
What are other words in Smojo which uses parsing?
next-token ( \ etc.
Question 2Write the stack comment word
: ( [char] ) parse drop ; immediate
In Example 3, what happens if we do the following? Why does it work even though 5 and 2 are not fractions?
' fractions +parser
5 2 + .
' fractions -parser
When the token is not a fraction,
fractionswill return -1 and the parser will pass the input.