This undoes the change from a4b209fdcf
and addresses the original problem by dropping the unrefs during
shutdown. All objects should get unref'ed without that, and this
code path should only get hit for dangling references due to
bailout.
Alternatively we'd have to relax some assertions that check that the
object is part of the weakrefs table, which seems worse.
Fixes oss-fuzz #40090.
Nikita Popov