TS Tokenizer

Tokenized: Returns plain tokens, one per line.

Tagged: Returns tokens with their tags. Returns tokens along with their tags. The tags are not part-of-speech labels; they indicate each token’s functional role.

Lines: Preserves the original line structure and returns tokenized lines as lists.

Tagged Lines: Returns tokenized lines as lists of tuples in the form (token, tag). (token, tag).