Parsing and the Token Stream

A line of Smojo code or a Smojo script in a text file is represented as a token stream - a sequence of strings delimited by spaces. This token stream is then interpreted token by token.
	   Each token represents a word or number or string. Quite often, your word needs to access tokens ahead of the token stream. You can do this using the 
PARSE
 word.

		
	


  	

    	Word
    	Action
  	

  	
	

    	
NEXT-TOKEN

    	Used in compilation mode. It takes the following token of the enclosing word. The token is stored as a string and it replaces the 
NEXT-TOKEN

			
	


	

    	
PARSE

    	Takes one argument, a character telling the word when to stop.
	
	

  	
	

    	
[CHAR]

    	Converts the following token into a character. Some useful characters are \n (carriage return) \r (linefeed) and \t (tab).
	
	

  
	

    	
BL

    	A character representing a space
	
	

  
	

    	
+PARSER

    	Takes an XT as an argument. If the XT returns 0, this input will be ignored. If the XT returns an item, say r and 1, the parser will then process the item r. If -1 is returned, the parser will pass this input.  
	
	


	

    	
-PARSER

    	Takes an XT as an argument. This removes the parser which has been included previously.
	
	


	


	
Parsing is the process of analysing a string of symbols. A parser takes input data and checks for correct syntax in the process. 
	   Using a 
+PARSER
 and 
-PARSER
, we are able create some words so that we can read in inputs of special forms and process them into a useful and valid information. 
	   A simple example is shown in Example 3 where we now can read fractions.
 

	
Examples

	Example 1
	
	: coming-up
    		"What comes after?" . cr
    		next-token .
	;

	coming-up Me!

	What comes after?
	Me!
	ok
	



	Example 2	
	
CHOMP
 reads the entire line to the carriage return and put the result on the stack.

	
	: chomp [char] \n parse ;
	
	chomp Hello World!
	.
	Hello World!
	
	
			
	
	Example 3	
	
The line comment word 
\
 . 
\
 reads the entire line to the carriage return. The result is dropped then from the stack.

	
	: \ [char] \n parse drop ;
	
	
	
	
	Example 4	
	
	: fraction? ( "s" -- f ) 
		[char] / indexofc 0 > ;

	: to-number ( "a/b" -- d )
		"/" split drop
		int swap int swap /. ;
	
	: fractions ( "s" -- 0 | r 1 | -1 ) 
		dup fraction? -> to-number 1 |
		otherwise	 drop -1 |.
	;
	
	1/2
	Error: Unknown word 1/2

	' fractions +parser
	
	1/2 1/2 +. . cr
	1.0 ok

	' fractions -parser
		
	4/5
	Error: Unkown word 4/5
	
		


	
Quiz

	
Question 1

	
What are other words in Smojo which uses parsing?

	
	
	


	
	
	
Question 2

	
Write the stack comment word 
(
 using parsing.

	

	
	


	
	
	
Question 3

	
In Example 3, what happens if we do the following? Why does it work even though 5 and 2 are not fractions? 
 	
	' fractions +parser

	
	5 2 + .

	
	' fractions -parser


	
	


	
 
Next: Decompilation