Tokenizer

A Tokenizer tells the Spreadsheet how to identify formulas and string tokens in the user entered value of cell.

Spreadsheet comes with a built-in tokenizer in rowsncolumns/calculator module. Developers can pass a custom tokenizer, especially if you want to additional highlighting capabilities.