diff options
| author | Ulf Magnusson <ulfalizer@gmail.com> | 2015-06-16 16:10:55 +0200 |
|---|---|---|
| committer | Ulf Magnusson <ulfalizer@gmail.com> | 2015-06-16 17:29:03 +0200 |
| commit | b7b5474954e3af31fd4da222a8d84795c917c136 (patch) | |
| tree | 7f14184f4ef13698288d81eeb5a1f00663c4f021 /tests | |
| parent | eedfb571d08347c5c6bf35afa87bc7538686455e (diff) | |
Clean up and optimize tokenization some more.
- Get rid of the weird strip/reindex dance and keep 's' intact
throughout. This also fixes _tokenization_error() output, as it would
previously report the stripped string.
- Strip trailing whitespace in the regexes. This makes the stripping
loop very cheap.
- Shorten a comment that might've been more confusing than helpful.
Diffstat (limited to 'tests')
0 files changed, 0 insertions, 0 deletions