Parsing and the Token Stream

A line of Smojo code or a Smojo script in a text file is represented as a token stream - a sequence of strings delimited by spaces. This token stream is then interpreted token by token.
	   Each token represents a word or number or string. Quite often, your word needs to access tokens ahead of the token stream. You can do this using the 

Word Action
Used in compilation mode. It takes the following token of the enclosing word. The token is stored as a string and it replaces the
Takes one argument, a character telling the word when to stop.
Converts the following token into a character. Some useful characters are \n (carriage return) \r (linefeed) and \t (tab).
A character representing a space
Takes an XT as an argument. If the XT returns 0, this input will be ignored. If the XT returns an item, say r and 1, the parser will then process the item r. If -1 is returned, the parser will pass this input.
Takes an XT as an argument. This removes the parser which has been included previously.

Parsing is the process of analysing a string of symbols. A parser takes input data and checks for correct syntax in the process. 
	   Using a 
, we are able create some words so that we can read in inputs of special forms and process them into a useful and valid information. A simple example is shown in Example 3 where we now can read fractions.


Example 1
	: coming-up
    		"What comes after?" . cr
    		next-token .

	coming-up Me!

	What comes after?
Example 2

reads the entire line to the carriage return and put the result on the stack.

	: chomp [char] \n parse ;
	chomp Hello World!
	Hello World!
Example 3

The line comment word 
reads the entire line to the carriage return. The result is dropped then from the stack.

	: \ [char] \n parse drop ;
Example 4
	: fraction? ( "s" -- f ) 
		[char] / indexofc 0 > ;

	: to-number ( "a/b" -- d )
		"/" split drop
		int swap int swap /. ;
	: fractions ( "s" -- 0 | r 1 | -1 ) 
		dup fraction? -> to-number 1 |
		otherwise	 drop -1 |.
	Error: Unknown word 1/2

	' fractions +parser
	1/2 1/2 +. . cr
	1.0 ok

	' fractions -parser
	Error: Unkown word 4/5


Question 1

What are other words in Smojo which uses parsing?

Question 2

Write the stack comment word 
using parsing.

Question 3

In Example 3, what happens if we do the following? Why does it work even though 5 and 2 are not fractions?

	' fractions +parser
	5 2 + .
	' fractions -parser

Next: Decompilation