mirror of
https://github.com/doctrine/orm.git
synced 2026-03-24 06:52:09 +01:00
Upgrading from 2.16.3 to 2.17.0 breaks serialization when using symfony/var-exporter #7245
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @alcohol on GitHub (Nov 16, 2023).
BC Break Report
Could not find anything in release notes so far that predicted this would break. Was also not yet able to determine which, if any, of the changes listed cause this behaviour. Will update this issue as I get further along.
Stacktrace is as follows (omitted a few lines at the bottom for brevity):
Summary
See stacktrace above.
Previous behavior
No exception was thrown.
Current behavior
Exception is thrown.
How to reproduce
Hard to tell, might be related to our code, might not be? Still debugging.
Update(s)
It seems that turning on lazy ghosts resolves the issue. However, that does not seem to be the default configuration in our symfony application. Is this the right way to deprecate something?Turning it back off does not break things. Looking more likely to be cache related, making this a non-issue. Will try a new deployment with some more rigorous cache purging.Sigh. Works on development environment now, broken on production still. More debugging needed.
@kerbert101 commented on GitHub (Nov 16, 2023):
2a9390dseems to be causing this@itarcontact commented on GitHub (Nov 16, 2023):
Hi. I have the same error. Reverting to 2.16.3 solves the problem.
@alcohol commented on GitHub (Nov 16, 2023):
For me it was resolved by purging our cache more explicitly.
@kerbert101 commented on GitHub (Nov 16, 2023):
I can fix the issue by replacing the code that returns the properties to be serialized.
In
AbstractSqlExecutorin the __sleep method:@itarcontact commented on GitHub (Nov 16, 2023):
@alcohol Which version do you finally have?
@alcohol commented on GitHub (Nov 16, 2023):
I take it back, development release works (but runs in development mode). Production broken again. Manually purging caches did not resolve it there. Diving back into what the cause is.
@alcohol commented on GitHub (Nov 16, 2023):
Weird, locally and on development we use
On production we use
Switching to
filelocally, installing2.16.3and then upgrading to2.17.0breaks things. But removing the cache directories solves it. We perform the same step during a production deployment though. So I'm confused why that does not resolve it. Maybe fpm file cache related? Going to see if a stop, cache purge, start - solves it.Update
Still broken on production. Working fine locally and on development. Investigating further..
@KhorneHoly commented on GitHub (Nov 16, 2023):
Upgrade to 2.17.0 also broke our code, downgrading to 2.16.3 now.
@RobinDev commented on GitHub (Nov 16, 2023):
Same here in prod : purging the cache offers me a few request working, then same error. Investigating too.
@PhilETaylor commented on GitHub (Nov 16, 2023):
This is what initially fixed it for me when I mentioned it here https://github.com/doctrine/orm/pull/11027#issuecomment-1793871887 2 weeks ago.
By "more explicitly", I mean I had to physically flush the Redis cache manually by flushing the whole redis db - for some reason the cli commands just did not clear the cache "enough".
Ive been running the new code in production for 2 weeks now with no issues since that day 2 weeks ago https://github.com/doctrine/orm/pull/11048
Is Redis the clue here? 🤷♂️
@alcohol commented on GitHub (Nov 16, 2023):
I would prefer not to clear our entire Redis cache.. but it is worth investigating perhaps.
@greg0ire commented on GitHub (Nov 16, 2023):
Clear the query cache (not the result cache, just the query cache) should be sufficient. What happened is I made the serialization format compatible with 3.0.x, so now all mentions of
_sqlStatementsin the cache should be replaced withsqlStatements, but 2.17.0 should still be able to deserialize an old version of the cache, thanks toe8afa9f80c/lib/Doctrine/ORM/Query/Exec/AbstractSqlExecutor.php (L92-L94)My assumption when writing this piece of code is that you cannot have both properties null at the same time after deserialization.
@alcohol commented on GitHub (Nov 16, 2023):
That is a part of our release workflow though; so that is not sufficient unfortunately 😞
@itarcontact commented on GitHub (Nov 16, 2023):
@PhilETaylor probably yes. We also use redis and cannot do flush it in the prod environment. This is why we have reverted to 2.16.3.
@alcohol commented on GitHub (Nov 16, 2023):
Is
nullthe correct value to check though? What about usingisset()instead?@greg0ire commented on GitHub (Nov 16, 2023):
I think so yes
@greg0ire commented on GitHub (Nov 16, 2023):
If anyone can reproduce this with a test, it would help a lot. The test could be contributed here: https://github.com/doctrine/orm/blob/2.17.x/tests/Doctrine/Tests/ORM/Functional/ParserResultSerializationTest.php
@alcohol commented on GitHub (Nov 16, 2023):
I'm at a loss then kind of.
Our own code where the issue starts is:
This is a custom repository class that extends
Gedmo\Tree\Entity\Repository\NestedTreeRepositoryand implementsDoctrine\Bundle\DoctrineBundle\Repository\ServiceEntityRepositoryInterfaceThe entity it manages implements
Knp\DoctrineBehaviors\Contract\Entity\TranslatableInterfaceand usesKnp\DoctrineBehaviors\Model\Translatable\TranslatableTrait, hence the translations being left joined in the builder.From there on onwards though it is all Doctrine vendor code.
It even explicitly calls
executeIgnoreQueryCacheso I don't think the query cache is related to this.@greg0ire commented on GitHub (Nov 16, 2023):
I would be very surprised if this is unrelated to my change wrt the query cache.
Here is what a cache entry should look like:
e8afa9f80c/tests/Doctrine/Tests/ORM/Functional/ParserResults/single_select_2_17_0.txt (L1)Can somebody produce a cache entry that produces a broken
SingleSelectExecutorobject?@kerbert101 commented on GitHub (Nov 16, 2023):
It seems like the
PhpFilesAdapterused in (our) production environment cannot serialize the properties returned by__sleepinAbstractSqlExecutor, which will result in null values in the cache. Changing the code inAbstractSqlExecutorto return the properties without the \0*\0 prefix seems to fix the issue. I'm not sure if this will cause other issues, but this seems to fix it:@kerbert101 commented on GitHub (Nov 16, 2023):
With the code above:
The faulty code does not return anything, meaning the sqlStatements key is not being filled at all.
@greg0ire commented on GitHub (Nov 16, 2023):
@kerbert101 what's weird is that the piece of code you're modifying is the one that governs serialization. Its goal is to ensure
_sqlStatementsis not serialized (since we havesqlStatements). Why are you switching toget_object_vars(), BTW?@kerbert101 commented on GitHub (Nov 16, 2023):
@greg0ire Yes, I'm aware of the fact that it governs what to serialize. I'm switching to
get_object_varsto get rid of the \0x\0 prefixes, which I thought was causing the issue.As you can see in the grep result, my modification will still ensure sqlStatements is serialized, not _sqlStatements.
Anyways, i'm not saying this is the fix, but this is what I see
@greg0ire commented on GitHub (Nov 16, 2023):
Ok… I'm trying to simplify, but both seem to work the same:
So why did I use
(array) $this? Well,get_object_vars()does not seem to work with inheritance: https://3v4l.org/ce10i@kerbert101 commented on GitHub (Nov 16, 2023):
The
PhpFilesAdapterusesVarExporter.The result of
VarExporter::exportinPhpFilesAdapterwith my change:Result of unmodified code:
Notice the missing
sqlStatementskey.You can check this by dumping the result of
VarExporter::exportinPhpFilesAdapteron line 227@greg0ire commented on GitHub (Nov 16, 2023):
Ah, so no
serialize()here maybe? I think you're on to something :)@greg0ire commented on GitHub (Nov 16, 2023):
I see that in another class, @derrabus implemented 2 methods instead of one for unserialization:
e8afa9f80c/lib/Doctrine/ORM/Query/ParserResult.php (L134-L140)Maybe
VarExportercalls one, andunserializecalls the other?@alcohol commented on GitHub (Nov 16, 2023):
I could not find any mention of
SingleSelectExecutorin our Redis database. I did find it however inside many cached files on disk invar/cache/prod/pools/system, indeed making use of thePhpFilesAdapterformat.Related config
and
cache.systemis either aPhpFilesAdapteror a chained adapter using bothApcuAdapterandPhpFilesAdapter, seecf8a997114/src/Symfony/Component/Cache/Adapter/AbstractAdapter.php (L104)@lostfocus commented on GitHub (Nov 16, 2023):
Not sure if it is the same issue but updating from 2.16.* to 2.17.* also breaks PHPstan tests in interesting ways.
@greg0ire commented on GitHub (Nov 16, 2023):
Possible next step: reproduce the issue inside a test. Copy/pasting https://github.com/doctrine/orm/blob/2.17.x/tests/Doctrine/Tests/ORM/Functional/ParserResultSerializationTest.php#L45-L67 , then replacing the calls to
serialize/unserializewith equivalentVarExportercalls (VarExporterand…eval🙈 ?) should do the trick.@greg0ire commented on GitHub (Nov 16, 2023):
@lostfocus definitely not the same.
@alcohol commented on GitHub (Nov 16, 2023):
I can't figure out why I don't have these cached files on my local development environment though. I use the same cache adapter, but nothing seems to be generated.
@alcohol commented on GitHub (Nov 16, 2023):
From what I can tell though, it breaks on production when storing and retrieving the
ParserResultinstance. So production works on first deploy (no cache). Then as soon as cache items get generated, things break.@dsands commented on GitHub (Nov 16, 2023):
From what I've determined so far, dev env works fine, prod is the problem. I've completely deleted the cache (rm -fR var/cache/*) to be sure it's really gone for each test. The only way to get prod to function is commenting out the query_cache_driver config (which I guess falls back to array type):
@greg0ire commented on GitHub (Nov 16, 2023):
Try using
APP_ENV=prod?@alcohol commented on GitHub (Nov 16, 2023):
Yeah unfortunately I cannot do that, that would connect to production database etc. Our configuration setup is still a bit of a env/files spaghetti mix. I can try turning
APP_DEBUGoff though.Edit
Jup, instantly crashed. Guess in debug mode these parser results are never cached?
@alcohol commented on GitHub (Nov 16, 2023):
Definitely related to storing and retrieving the cache item.
yields
@alcohol commented on GitHub (Nov 16, 2023):
This is the generated cache entry on disk using
2.17.0@alcohol commented on GitHub (Nov 16, 2023):
On
2.16.3it yields this file:@alcohol commented on GitHub (Nov 16, 2023):
We can see that no properties for the
SingleSelectExecutor(nor its parentAbstractSqlExecutor) are exported by the exporter when running2.17.0.@alcohol commented on GitHub (Nov 16, 2023):
Perhaps @nicolas-grekas has some insights into how/why this change breaks things?
@nicolas-grekas commented on GitHub (Nov 16, 2023):
I'd appreciate a reproducer so that I could quickly reproduce the issue.
@kerbert101 commented on GitHub (Nov 16, 2023):
@nicolas-grekas in short, this is the problem:
The (array) $this returns the properties with the internal visibility names, which does not work with VarExporter.
Returning those properties with the internal visibility names and executing serialize on it will work however.
@mbabker commented on GitHub (Nov 16, 2023):
FWIW I just added the query cache to my Symfony app's dev environment after updating the ORM to 2.17 and the profiler now has log messages like this:
No other changes in the app.
Dumping the exception the VarExporter throws here just gives a "User notice: serialize()" message with this trace:
@alcohol commented on GitHub (Nov 16, 2023):
@alcohol commented on GitHub (Nov 16, 2023):
@greg0ire It seems @kerbert101 hit the nail on the head. Any particular reason you are using the
(array) $thisapproach? The documentation @ https://www.php.net/manual/en/language.oop5.magic.php also merely documents using the exact property names (no matter if they are public, protected or private).@kerbert101 commented on GitHub (Nov 16, 2023):
One could argue that this method should not return the properties with their internal visibility names, but as @greg0ire mentioned before, get_object_vars is not an option in this case as this will skip the private properties of subclasses. Using Reflection is another option, but I'm not sure if that's the best option in this case neither 😕
@greg0ire commented on GitHub (Nov 16, 2023):
Yes, that and also I thought I would be consistent with
e8afa9f80c/lib/Doctrine/ORM/Query/ParserResult.php#@nicolas-grekas commented on GitHub (Nov 16, 2023):
Is it legal in PHP to have sleep return property identifiers? I'm not aware of it. The PHP notice also means something is odd for the engine. __serialize can but __sleep???
@alcohol commented on GitHub (Nov 16, 2023):
What about making the
__sleep()method abstract too (inAbstractSqlExecutor) and have each child be explicitly about what to serialize by implementing their own__sleep()?@alcohol commented on GitHub (Nov 16, 2023):
That's precisely how it is documented though? https://www.php.net/manual/en/language.oop5.magic.php#object.sleep
"It can clean up the object and is supposed to return an array with the names of all variables of that object that should be serialized."
@nicolas-grekas commented on GitHub (Nov 16, 2023):
it's not to me
@greg0ire commented on GitHub (Nov 16, 2023):
Weird that there is a notice, I don't see one when running the test that contains this:
e8afa9f80c/tests/Doctrine/Tests/ORM/Functional/ParserResultSerializationTest.php (L36)@alcohol commented on GitHub (Nov 16, 2023):
But do we have parent classes here? The properties are protected, not private, and they are implemented by children of the abstract class.
@alcohol commented on GitHub (Nov 16, 2023):
It's triggered by the exporter using
trigger_error(sprintf('serialize(): "%s" returned as member variable from __sleep() but does not exist', $n), \E_USER_NOTICE);.@nicolas-grekas commented on GitHub (Nov 16, 2023):
Let me be more explicit:
__sleepmust return property names, not property identifiers. The difference is the\0-based prefix.__sleepcannot return any of them.@greg0ire commented on GitHub (Nov 16, 2023):
OK. So this is a Symfony bug, right?
@nicolas-grekas commented on GitHub (Nov 16, 2023):
Nope, it's a Doctrine bug,
__sleepmust return property names without the\0-based prefix.@alcohol commented on GitHub (Nov 16, 2023):
@nicolas-grekas I do think you are correct in that
__sleepshould return just their plain names, not magic names. But I believe @greg0ire used this approach for reasons and it somehow works with just plain php serialize/unserialize.@greg0ire commented on GitHub (Nov 16, 2023):
How is it a Doctrine bug if everything works (there is not even a notice if I understood correctly) when using
serialize, but it breaks when usingVarExporter? Can somebody reproduce the bug in a test using Doctrine APIs and only them?@nicolas-grekas commented on GitHub (Nov 16, 2023):
It works because it's broken : __sleep fails and then php likely falls back to standard behavior, ignoring __sleep, thus generating a serialized payload that works, but doesn't achieve the wanted compatibility I'd say. It's my guess at least :)
@nicolas-grekas commented on GitHub (Nov 16, 2023):
And the difference of behavior might be related to VarExporter not falling back to the exact same behavior when __sleep returns a broken payload. That'd explain everything. But the root issue is in Doctrine.
@greg0ire commented on GitHub (Nov 16, 2023):
@nicolas-grekas I think it works just fine: https://3v4l.org/3ckU7 If PHP was falling back to standard behavior then we would see 2 properties, and here there is just
sqlStatements.@nicolas-grekas commented on GitHub (Nov 16, 2023):
Oh, OK! Then you're relying on undocumented behavior of __sleep! (and unimplemented in VarExporter)
@greg0ire commented on GitHub (Nov 16, 2023):
Yeah that sounds more plausible 😅
My process wasn't to go through the docs, but I stumbled upon other occurrences of
\0in the code, then on this: https://www.phpinternalsbook.com/php5/classes_objects/serialization.htmlIt's not part of the official docs, but I thought it would be good enough.
The other occurrences, for reference:
e8afa9f80c/lib/Doctrine/ORM/Query/ParserResult.php (L143-L144)I think to address this, a possibility might be to iterate of the result of
(array) $thisand strip the\0*\0part, before doing the diff with['_sqlStatements']. That prefix, is probably here for a reason though 🤔 … avoiding collisions?@nicolas-grekas commented on GitHub (Nov 16, 2023):
\0*\0is needless, __sleep works with protected properties without the prefix. The prefix might be required for private properties of parent classes. If that doesn't apply to this use case, removing the prefix is the right workaround for Doctrine.@greg0ire commented on GitHub (Nov 16, 2023):
I think it would work just fine, unless somebody external to Doctrine have their own extending class, with their own
_sqlStatementsprivate property defined in it… pretty far-fetched.@kerbert101 commented on GitHub (Nov 16, 2023):
@greg0ire https://github.com/doctrine/orm/pull/11065
@greg0ire commented on GitHub (Nov 16, 2023):
@kerbert101 thanks for working on this!
@nicolas-grekas commented on GitHub (Nov 16, 2023):
Fix for Symfony: https://github.com/symfony/symfony/pull/52618
Thank you all for the help on this one!
@symfonyaml commented on GitHub (Nov 16, 2023):
Upgrading from
2.16.3to2.17.0on my project break only in prod environment (not dev or test) with this error :which means
$this->_sqlStatementsis NULL ... if it helps.@greg0ire commented on GitHub (Nov 16, 2023):
Thanks, but we have 2 fixes on the way already, and pinpointed the issue a few messages ago. You just need to wait for either a new version of
doctrine/ormorsymfony/var-exporter.@szczyglis-dev commented on GitHub (Nov 17, 2023):
Hi, I've got the same issue after upgrade to
2.17.0. It only happens in prod env. I had to roll back to2.16.3.Stacktrace:
@greg0ire commented on GitHub (Nov 17, 2023):
@szczyglis-dev I said: YOU JUST NEED TO WAIT
So just wait. No need for more reports.
@greg0ire commented on GitHub (Nov 17, 2023):
https://github.com/doctrine/orm/releases/tag/2.17.1 released
@alcohol commented on GitHub (Nov 17, 2023):
Awesome, many thanks to everyone who got involved so quickly to resolve this 🥳 !