Skip to content

Keep normalized whitespace in sync after filtering - #934

Open
JunaidAhamed-7777 wants to merge 1 commit into
andialbrecht:masterfrom
JunaidAhamed-7777:fix-whitespace-normalized-value
Open

JunaidAhamed-7777 wants to merge 1 commit into
andialbrecht:masterfrom
JunaidAhamed-7777:fix-whitespace-normalized-value

Conversation

@JunaidAhamed-7777

@JunaidAhamed-7777 JunaidAhamed-7777 commented Oct 13, 2026 •

Copy link
Copy Markdown

Fixes #547.

StripWhitespaceFilter changes whitespace token values but leaves their cached normalized values unchanged. A tab collapsed to a space consequently fails token.match(token.ttype, ' '), and regex matching also sees the original text.

Update normalized when this filter changes the whitespace value. Regression tests cover leading/repeated whitespace, parentheses, and identifier lists, checking both literal and regex matching. All three cases fail before the fix.

  • Tests: python -m pytest tests -q --tb=short --basetemp=../tmp-sqlparse on Python 3.14: 509 passed, 2 xfailed, 1 xpassed.
  • Ruff: unavailable locally; CI pending.
  • Regression tests included.
  • No public API or documentation change needed.

git diff --check passes.

Baseline comparison: unchanged base 60cdc649726bf1bc4f1b336050560b336da715ec reports 506 passed, 2 xfailed, 1 xpassed. Both base and final code unexpectedly pass tests/test_regressions.py::test_issue484_comments_and_newlines; the three additional passing cases are the new whitespace regressions.

AI-assisted using GPT-6; human review follows draft submission.

@JunaidAhamed-7777
JunaidAhamed-7777 marked this pull request as ready for review October 13, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Token's attributes value and normalized may become inconsistent

1 participant