One funny thing: while the original author used Unicode 0xFFFD (generic
replacement character) for invalid bytes in CP1251 and CP1252, for CP1254
they used 0xFFFE, which is not a valid Unicode codepoint at all, but is a
reversed byte-order mark. Probably this was by mistake.
Anyways,
- Fixed identify filter, which was completely wrong.
- Don't convert Unicode 0xFFFE to a random (but valid) CP1254 byte.
- When converting CP1254 to CP1254, don't pass invalid bytes through silently.
- Identify filter was as wrong as wrong can be.
- Invalid CP1251 byte 0x98 was converted to Unicode 0xFFFD (generic
replacement character), rather than respecting `mb_substitute_character`.
- Unicode 0xFFFD was converted to some random CP1251 byte.
- When converting CP1251 to CP1251, don't pass invalid bytes through silently.
The implementation is based on the upstream PMurHash. The following
variants are implemented
- murmur3a, 32-bit hash
- murmur3c, 128-bit hash for x86
- murmur3f, 128-bit hash for x64
The custom seed support is not targeted by this implementation. It will
need a major change to the API, so then custom arguments can be passed
through `hash_init`. For now, the starting hash is always zero.
Fixes bug #68109, closes#6059
Signed-off-by: Anatol Belski <ab@php.net>
Co-Developed-by: Michael Wallner <mike@php.net>
Signed-off-by: Michael Wallner <mike@php.net>
Also remove a bogus test (bug62545.phpt) which wrongly assumed that all invalid
characters in CP1251 and CP1252 should map to Unicode 0xFFFD (REPLACEMENT
CHARACTER).
mbstring has an interface to specify what invalid characters should be
replaced with; it's called `mb_substitute_character`. If a user wants to see
the Unicode 'replacement character', they can specify that using
`mb_substitute_character`. But if they specify something else, we should
follow that.
It's a bit surprising how much was broken here.
- Identify filter was utterly and completely wrong.
- Instead of handling invalid CP1252 bytes as specified by
`mb_substitute_character`, it would convert them to Unicode 0xFFFD
(generic replacement character).
- When converting ISO-8859-1 to CP1252, invalid ISO-8859-1 bytes would
be passed through silently.
- Unicode codepoints from 0x80-0x9F were converted to CP1252 bytes 0x80-0x9F,
which is wrong.
- Unicode codepoint 0xFFFD was converted to CP1252 0x9F, which is very wrong.
Also clean up some unneeded code, and make the conversion table consistent with
others by using zero as a 'invalid' marker, rather than 0xFFFD.
While fixing bugs in mbstring, one of my new test cases failed with a strange
error message stating: 'Warning: Undefined array key 1...', when clearly the
array key had been set properly.
GDB'd that sucker and found that JIT'd PHP code was calling directly into
`zend_hash_add_new` (which was not converting the numeric string key to an
integer properly). But where was that code coming from? I examined the disasm,
looked up symbols to figure out where call instructions were going, then grepped
the codebase for those function names. It soon became clear that the disasm I
was looking at was compiled from `zend_jit_fetch_dim_w_helper`.
This testing mode executes the test multiple times in the same
process (but in different requests). It is primarily intended to
catch tracing JIT bugs, but also catches state leaks across
requests.
Closes GH-6365.
JIT ignores that the `zend_write` callback is overwritten, so we define
our own callback and caller.
We also fix the "inconsistent DLL binding" warnings on Windows, by
introducing `PHP_ZEND_TEST_API`.
Closes GH-6391.
Dropping the dtor arg args[3] rather than using STR_COPY: Since
PHP 8, we no longer support separation in call_user_function(),
so we also don't need to worry about things like arguments being
replaced with references.
I don't like the previous behaviour where the bytes to hash change
every time the code changes, that may make it difficult to compare
hash() performance changes over time.
Use a fixed number instead, and allow passing an override for a
different length.
Closes GH-6386.
[ci skip]
We cannot simply switch to use_result here, because the fetch_row
methods in get_result mode and in use_result/store_result mode
are different: In one case it accepts a statement, in the other
a return value zval. Thus, doing a switch to use_result results
in a segfault when trying to fetch a row.
Actually supporting get_result with cursors would require adding
cursor support in mysqlnd_result, not just mysqlnd_ps. That would
be a significant amount of effort and, given the age of the issue,
does not appear to be particularly likely to happen soon.
As such, we simply generate an error when using get_result()
with cursors, which is much better than causing a segfault.
Instead, parameter binding needs to be used.
The EOF flag also gets set on error, so we always end up ignoring
errors here.
However, we should only check errors for unbuffered results. For
buffered results, this function is guaranteed not to error, and
querying the errno may return an unrelated error.
Alex Dowad