Mass JSON encoding => mix iterate() and resultAsArray() #5288

New Issue

admin · 2026-01-22T15:03:34+01:00

admin commented

2026-01-22 15:03:34 +01:00

Originally created by @cyrilchapon on GitHub (Oct 7, 2016).

Originally assigned to: @Ocramius on GitHub.

I've got a huge table and I just want to get this as json.

I'm able to "process the whole data table as an array"

      $q = $qb->getQuery();
      $q->setHint(Query::HINT_INCLUDE_META_COLUMNS, true);
      $data = $q->getResult(Query::HYDRATE_ARRAY);

This takes 4 seconds for 15k records

And to "process each object 1 by 1 cleaning doctrine everytime"

      $q = $qb->getQuery();
      $data = $q->iterate();

(though, this last solution leads me to manually encoding to json every row)
This takes 15 seconds for 15k records (!) even if I detach($current) and even clear() entity manager.

How would I "process each record 1 by 1 as an array" ?
I tried $data = $q->iterate(Query::HYDRATE_ARRAY); which gave me an error

Originally created by @cyrilchapon on GitHub (Oct 7, 2016). Originally assigned to: @Ocramius on GitHub. I've got a huge table and I just want to get this as json. I'm able to "process the whole data table as an array" ``` $q = $qb->getQuery(); $q->setHint(Query::HINT_INCLUDE_META_COLUMNS, true); $data = $q->getResult(Query::HYDRATE_ARRAY); ``` This takes 4 seconds for 15k records And to "process each object 1 by 1 cleaning doctrine everytime" ``` $q = $qb->getQuery(); $data = $q->iterate(); ``` (though, this last solution leads me to manually encoding to json every row) This takes 15 seconds for 15k records (!) even if I `detach($current)` and even `clear()` entity manager. How would I **"process each record 1 by 1 as an array"** ? I tried `$data = $q->iterate(Query::HYDRATE_ARRAY);` which gave me an error

admin added the Can't Fix label 2026-01-22 15:03:34 +01:00

admin closed this issue

2026-01-22 15:03:35 +01:00

admin commented

2026-01-22 15:03:35 +01:00

@Ocramius commented on GitHub (Oct 7, 2016):

You'd probably use SQL for that. If speed is necessary, the ORM is NOT designed to get you there fast, when it comes to batch processing.

Closing as can't fix.

@Ocramius commented on GitHub (Oct 7, 2016): You'd probably use SQL for that. If speed is necessary, the ORM is **NOT** designed to get you there fast, when it comes to batch processing. Closing as `can't fix`.

admin commented

2026-01-22 15:03:35 +01:00

@cyrilchapon commented on GitHub (Oct 7, 2016):

@ocramius thanks for your answer.

Even if I can see that indeed the ORM was not designed with speed mind, I don't think in 2016 web can call simple SQL select to json serialisation of 15000 rows without any joint a "batch processing"...

@cyrilchapon commented on GitHub (Oct 7, 2016): @ocramius thanks for your answer. Even if I can see that indeed the ORM was not designed with speed mind, I don't think in 2016 web can call simple SQL select to json serialisation of 15000 rows without any joint a "batch processing"...

admin commented

2026-01-22 15:03:36 +01:00

@Ocramius commented on GitHub (Oct 7, 2016):

@cyrilchapon it really depends on the amount of data, queries, etc.

We have performance tests that hydrate/persist 10k records within .2 seconds, so if you think there is a performance problem somewhere, there needs to be some sort of benchmark exposing it, or some profiling exposing a particularly slow piece of the logic.

If you have large JSON structures, then most overhead will end up being in json_decode(), which we don't have control on.

If your entities implement JsonSerializable, then you may have an issue in that logic.

It's not as simple as " I don't think in 2016 web can call simple SQL select to json serialisation of 15000 rows", there needs to be some testing and some reproducible issue.

That said, 15000 rows loaded into memory is way out of scope for this ORM: wrong tool for the job already.

@Ocramius commented on GitHub (Oct 7, 2016): @cyrilchapon it really depends on the amount of data, queries, etc. We have performance tests that hydrate/persist 10k records within .2 seconds, so if you think there is a performance problem somewhere, there needs to be some sort of benchmark exposing it, or some profiling exposing a particularly slow piece of the logic. If you have large JSON structures, then most overhead will end up being in `json_decode()`, which we don't have control on. If your entities implement `JsonSerializable`, then you may have an issue in that logic. It's not as simple as `" I don't think in 2016 web can call simple SQL select to json serialisation of 15000 rows"`, there needs to be some testing and some reproducible issue. That said, 15000 rows loaded into memory is way out of scope for this ORM: wrong tool for the job already.

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: doctrine/archived-orm#5288