When we're lexing certain types (strings, symbols, lists, etc.) we have string content associated with the tokens. More...
Data Fields | |
| pm_buffer_t | buffer |
| The buffer that we're using to keep track of the string content. | |
| const uint8_t * | cursor |
| The cursor into the source string that points to how far we have currently copied into the buffer. | |
Detailed Description
When we're lexing certain types (strings, symbols, lists, etc.) we have string content associated with the tokens.
For example:
"foo"
In this case, the string content is foo. Since there is no escaping, there's no need to track additional information and the token can be returned as normal. However, if we have escape sequences:
"foo\n"
then the bytes in the string are "f", "o", "o", "\", "n", but we want to
provide our consumers with the string content "f", "o", "o", "
". In these cases, when we find the first escape sequence, we initialize a pm_buffer_t to keep track of the string content. Then in the parser, it will automatically attach the string content to the node that it belongs to.
Field Documentation
◆ buffer
| pm_buffer_t pm_token_buffer_t::buffer |
The buffer that we're using to keep track of the string content.
It will only be initialized if we receive an escape sequence.
The documentation for this struct was generated from the following file:
- src/prism.c
