Currently, unexpected tokens in the parser are shown as the text
found, plus the internal token name, including the notorious
"unexpected '::' (T_PAAMAYIM_NEKUDOTAYIM)".
This commit replaces that with a more user-friendly format, with
two main types of token:
* Tokens which always represent the same text are shown like
'unexpected token "::"' and 'expected "::"'
* Tokens which have variable text are given a user-friendly
name, and show like 'unexpected identifier "foo"', and
'expected identifer'.
A few tokens have special cases:
* unexpected token """ -> unexpected double-quote mark
* unexpected quoted string "'foo'" -> unexpected single-quoted
string "foo"
* unexpected quoted string ""foo"" -> unexpected double-quoted
string "foo"
* unexpected illegal character "_" -> unexpected character 0xNN
(where _ is almost certainly a control character, and NN is the
hexadecimal value of the byte)
The \ token has a special case in the implementation just to stop
bison making a mess of escaping it and it coming out as \\
Don't include a trailing newline in T_COMMENT tokens, instead leave
it for a following T_WHITESPACE token. The newline does not belong
to the comment logically, and this makes for an ugly special case,
as other tokens do not include trailing newlines.
Whitespace-sensitive tooling will want to either forward or backward
emulate this change.
Closes GH-5182.
This is a bit tricky: In this cases we have "namespace as", which
means that we will only recognize "namespace" as an identifier when
the lookahead token is already at the "as". This means that
zend_lex_tstring picks up the wrong identifier.
We solve this by actually assigning the identifier as the semantic
value on the parser stack -- as in almost all cases we will not
actually need the identifier, this is just an (offset, size)
reference, not a copy of the string.
Additionally, we need to teach the lexer feedback mechanism used
by tokenizer TOKEN_PARSE mode to apply feedback to something
other than the very last token. To that purpose we pass through
the token text and check the tokens in reverse order to find the
right one.
Closes GH-5668.
RFC: https://wiki.php.net/rfc/throw_expression
This has an open issue with temporaries that are live at the time
of the throw being leaked. Landing this now for easier testing and
will revert if we cannot resolve the issue.
Closes GH-5279.
Aside from a few very specific syntax errors for which detailed exceptions are
thrown, generally PHP just emits the default error messages generated by bison on syntax
error. These messages are very uninformative; they just say "Unexpected ... at line ...".
This is most problematic with constructs which can span an arbitrary number of lines, such
as blocks of code delimited by { }, 'if' conditions delimited by ( ), and so on. If a closing
delimiter is missed, the block will run for the entire remainder of the source file (which
could be thousands of lines), and then at the end, a parse error will be thrown with the
dreaded words: "Unexpected end of file".
Therefore, track the positions of opening and closing delimiters and ensure that they match
up correctly. If any mismatch or missing delimiter is detected, immediately throw a parse
error which points the user to the offending line. This is best done in the *lexer* and not
in the parser.
Thanks to Nikita Popov and George Peter Banyard for suggesting improvements.
Fixes bug #79368.
Closes GH-5364.
To clean up the mess here a bit, check for invalid octal digits
with an explicit loop instead of mixing this into the string to
number conversion.
Also clean up some type usage.