The Interpreter, Chapter 2
The continuing story of one man’s quest to write an interpreter.
Jackson knew that all great languages needed string handling. So he began to add support for strings. What he did not know was the difficulty that strings would bring. Unlike anything he had created before, they required that the lexical analyzer turn a new eye on the input given. It must understand when text is enclosed in quotes. If not, it is read as usual; but if so, it must be treated as a single block of text. The analyzer had to be smarter.
The next great challenge was to implement arrays. In standard convention, they are much like parentheses: a pair of brackets, with some number of commas in between. Jackson found that a very similar definition could be used for both brackets and parentheses. But he changed the way arrays worked to be cleaner, to better fit the calculator-like syntax of his language. It was more Lisp-like: something like (array 1 2 3) instead of [1, 2, 3]. It required no special new syntax. And it allowed for a native implementation of both arrays and linked lists.
Yes, Jackson wanted linked lists. So many languages have either arrays or linked lists, but Jackson wanted both. To keep things simple, he defined a similar syntax for both: lists were defined to be written as (list 1 2 3).
There was some trouble with the implementation of strings, especially when putting strings inside parentheses. The nesting was confusing the interpreter and creating some very unusual bugs. But then the most amazing thing happened. Jackson saw one little line of code, the line that was reassessing the array of tokens, and saw a way to fix it. He had been trying to improve that line for weeks, but at this moment he had a sudden insight. He realized a way to vastly improve the line, making it much simpler. And, lo and behold, it worked perfectly.