findAll() don't return all elements #6568

New Issue

Open

opened 2026-01-22 15:35:04 +01:00 by admin · 7 comments

admin commented 2026-01-22 15:35:04 +01:00

Owner

Copy Link

Originally created by @secit-pl on GitHub (Nov 12, 2020).

composer.json

"doctrine/common": "^2", -> 2.13.3
"doctrine/orm": "^2", -> 2.7.4
"doctrine/doctrine-bundle": "^2", -> 2.2.0

While querying for all elements from the database (SQL Server) not all elements are returned.

The problem was noticed when I was using findAll() to get all elements from the database table:

$doctrine->getRepository('entity name...')->findAll()

After some debugging I noticed that the problem is not in findAll implementation but in processing the DQL query. See the following example (executed from the Symfony command):

1. lets build the DQL to select specified element (the one which is missed in findAll)

#~> php bin/console doctrine:query:dql "SELECT jd FROM App:Duration jd WHERE jd.id = 27616" --env=prod

 array(1) {
  [0]=>
  object(stdClass)#266 (8) {
    ["__CLASS__"]=>
    string(31) "App\Entity\Duration"
    ["id:App\Entity\Duration:private"]=>
    int(27616)
...
    ["standardDuration:App\Entity\Duration:private"]=>
    string(7) "4800000"
  }
}

all is ok

2. now query for all and grep for the specified id (27616) in output

#~> php bin/console doctrine:query:dql "SELECT jd FROM App:Duration jd" --env=prod | grep -c 27616
0

Zero matching rows found, so the element with ID 27616 was not returned

3. now repeat the same procedure from point 2 but using the SQL instead of DQL

#~> php bin/console doctrine:query:sql "SELECT * FROM Journey_Duration" --env=prod | grep -c 27616
1

One record found - the one missing while using the DQL.

ps. Some time ago I noticed the problem that while using findAll() some elements are missing, and in place of them was returned the previous element (the element which is in previous index in array) but after moving to DQL query the problem disappeared. I thing this might be related to the above problem with missing data.

Originally created by @secit-pl on GitHub (Nov 12, 2020). composer.json ``` "doctrine/common": "^2", -> 2.13.3 "doctrine/orm": "^2", -> 2.7.4 "doctrine/doctrine-bundle": "^2", -> 2.2.0 ``` While querying for all elements from the database (SQL Server) not all elements are returned. The problem was noticed when I was using findAll() to get all elements from the database table: ```php $doctrine->getRepository('entity name...')->findAll() ``` After some debugging I noticed that the problem is not in findAll implementation but in processing the DQL query. See the following example (executed from the Symfony command): 1. lets build the DQL to select specified element (the one which is missed in findAll) ``` #~> php bin/console doctrine:query:dql "SELECT jd FROM App:Duration jd WHERE jd.id = 27616" --env=prod array(1) { [0]=> object(stdClass)#266 (8) { ["__CLASS__"]=> string(31) "App\Entity\Duration" ["id:App\Entity\Duration:private"]=> int(27616) ... ["standardDuration:App\Entity\Duration:private"]=> string(7) "4800000" } } ``` all is ok 2. now query for all and grep for the specified id (27616) in output ``` #~> php bin/console doctrine:query:dql "SELECT jd FROM App:Duration jd" --env=prod | grep -c 27616 0 ``` Zero matching rows found, so the element with ID 27616 was not returned 3. now repeat the same procedure from point 2 but using the SQL instead of DQL ``` #~> php bin/console doctrine:query:sql "SELECT * FROM Journey_Duration" --env=prod | grep -c 27616 1 ``` One record found - the one missing while using the DQL. ps. Some time ago I noticed the problem that while using findAll() some elements are missing, and in place of them was returned the previous element (the element which is in previous index in array) but after moving to DQL query the problem disappeared. I thing this might be related to the above problem with missing data.

admin commented 2026-01-22 15:35:05 +01:00

Author

Owner

Copy Link

@Seb33300 commented on GitHub (Nov 13, 2020):

Any cache enabled in your doctrine config?

@Seb33300 commented on GitHub (Nov 13, 2020): Any cache enabled in your doctrine config?

admin commented 2026-01-22 15:35:06 +01:00

Author

Owner

Copy Link

@secit-pl commented on GitHub (Nov 13, 2020):

Symfony Doctrine configuration:

parameters:
    # Adds a fallback DATABASE_URL if the env var is not set.
    # This allows you to run cache:warmup even if your
    # environment variables are not available yet.
    # You should not need to change this value.
    env(DATABASE_URL): ''

doctrine:
    dbal:
        # configure these for your database server
#        driver: 'pdo_sqlsrv'
#        server_version: '5.7'
        charset: 'UTF-8'
#        default_table_options:
#            charset: utf8mb4
#            collate: utf8mb4_unicode_ci

        url: '%env(resolve:DATABASE_URL)%'

        types:
            datetime: App\Doctrine\Type\DateTime

    orm:
        auto_generate_proxy_classes: '%kernel.debug%'
        naming_strategy: doctrine.orm.naming_strategy.underscore
        auto_mapping: true
        mappings:
            App:
                is_bundle: false
                type: annotation
                dir: '%kernel.project_dir%/src/Entity'
                prefix: 'App\Entity'
                alias: App

parameters:
    memcached_host: localhost
    memcached_port: 11211

doctrine:
    orm:
        metadata_cache_driver:
            type: pool
            pool: doctrine.metadata_cache_driver

        query_cache_driver:
            type: service
            id: doctrine.system_cache_provider

        result_cache_driver:
            type: service
            id: doctrine.result_cache_provider

services:
    doctrine.result_cache_provider:
        class: Symfony\Component\Cache\DoctrineProvider
        public: false
        arguments:
            - '@doctrine.result_cache_pool'
    doctrine.system_cache_provider:
        class: Symfony\Component\Cache\DoctrineProvider
        public: false
        arguments:
            - '@doctrine.system_cache_pool'

framework:
    cache:
        pools:
            doctrine.result_cache_pool:
                adapter: cache.app
            doctrine.system_cache_pool:
                adapter: cache.system
            doctrine.metadata_cache_driver:
                adapter: cache.adapter.memcached
                provider: "memcached://%memcached_host%:%memcached_port%"

Entity definition

<?php

namespace App\Entity;

use Doctrine\ORM\Mapping as ORM;
use JMS\Serializer\Annotation as Serializer;

/**
 * Class Duration.
 *
 * @ORM\Entity(repositoryClass="App\Repository\DurationRepository")
 * @ORM\Table(name="Journey_Duration")
 */
class Duration
{
    /**
     * @var int|null
     *
     * @ORM\Column(name="Id", type="integer")
     *
     * @Serializer\Groups({"journey_duration"})
     */
    private $id;

...

}

@secit-pl commented on GitHub (Nov 13, 2020): Symfony Doctrine configuration: ```yaml parameters: # Adds a fallback DATABASE_URL if the env var is not set. # This allows you to run cache:warmup even if your # environment variables are not available yet. # You should not need to change this value. env(DATABASE_URL): '' doctrine: dbal: # configure these for your database server # driver: 'pdo_sqlsrv' # server_version: '5.7' charset: 'UTF-8' # default_table_options: # charset: utf8mb4 # collate: utf8mb4_unicode_ci url: '%env(resolve:DATABASE_URL)%' types: datetime: App\Doctrine\Type\DateTime orm: auto_generate_proxy_classes: '%kernel.debug%' naming_strategy: doctrine.orm.naming_strategy.underscore auto_mapping: true mappings: App: is_bundle: false type: annotation dir: '%kernel.project_dir%/src/Entity' prefix: 'App\Entity' alias: App ``` ```yaml parameters: memcached_host: localhost memcached_port: 11211 doctrine: orm: metadata_cache_driver: type: pool pool: doctrine.metadata_cache_driver query_cache_driver: type: service id: doctrine.system_cache_provider result_cache_driver: type: service id: doctrine.result_cache_provider services: doctrine.result_cache_provider: class: Symfony\Component\Cache\DoctrineProvider public: false arguments: - '@doctrine.result_cache_pool' doctrine.system_cache_provider: class: Symfony\Component\Cache\DoctrineProvider public: false arguments: - '@doctrine.system_cache_pool' framework: cache: pools: doctrine.result_cache_pool: adapter: cache.app doctrine.system_cache_pool: adapter: cache.system doctrine.metadata_cache_driver: adapter: cache.adapter.memcached provider: "memcached://%memcached_host%:%memcached_port%" ``` Entity definition ```php <?php namespace App\Entity; use Doctrine\ORM\Mapping as ORM; use JMS\Serializer\Annotation as Serializer; /** * Class Duration. * * @ORM\Entity(repositoryClass="App\Repository\DurationRepository") * @ORM\Table(name="Journey_Duration") */ class Duration { /** * @var int|null * * @ORM\Column(name="Id", type="integer") * * @Serializer\Groups({"journey_duration"}) */ private $id; ... } ```

admin commented 2026-01-22 15:35:06 +01:00

Author

Owner

Copy Link

@Seb33300 commented on GitHub (Nov 13, 2020):

Yes, it looks like you have configured doctrine to use cache.

You should try to clear your cache and test again:
https://symfony.com/doc/current/cache.html#clearing-the-cache

@Seb33300 commented on GitHub (Nov 13, 2020): Yes, it looks like you have configured doctrine to use cache. You should try to clear your cache and test again: https://symfony.com/doc/current/cache.html#clearing-the-cache

admin commented 2026-01-22 15:35:06 +01:00

Author

Owner

Copy Link

@secit-pl commented on GitHub (Nov 13, 2020):

After each deploy I'm clearing the cache with a following commands:

php bin/console cache:clear --env=prod --no-warmup
php bin/console doctrine:cache:clear-metadata --env=prod
php bin/console doctrine:cache:clear-query --env=prod
php bin/console doctrine:cache:clear-result --env=prod

The output:

 // Clearing the cache for the prod environment with debug                      
 // false                                                                       

 [OK] Cache for the "prod" environment (debug=false) was successfully cleared.  


 // Clearing all Metadata cache entries                                         

 [OK] Successfully deleted cache entries.                                       


 // Clearing all Query cache entries                                            

 [OK] Successfully deleted cache entries.                                       


 // Clearing all Result cache entries                                           

 [OK] Successfully deleted cache entries.  

@secit-pl commented on GitHub (Nov 13, 2020): After each deploy I'm clearing the cache with a following commands: ```bash php bin/console cache:clear --env=prod --no-warmup php bin/console doctrine:cache:clear-metadata --env=prod php bin/console doctrine:cache:clear-query --env=prod php bin/console doctrine:cache:clear-result --env=prod ``` The output: ``` // Clearing the cache for the prod environment with debug // false [OK] Cache for the "prod" environment (debug=false) was successfully cleared. // Clearing all Metadata cache entries [OK] Successfully deleted cache entries. // Clearing all Query cache entries [OK] Successfully deleted cache entries. // Clearing all Result cache entries [OK] Successfully deleted cache entries. ```

admin commented 2026-01-22 15:35:06 +01:00

Author

Owner

Copy Link

@Seb33300 commented on GitHub (Nov 13, 2020):

Something weird may have happened after the deployment.
In my case, every time I am encountering an issue like this one, clearing the doctrine cache solve it.

@Seb33300 commented on GitHub (Nov 13, 2020): Something weird may have happened after the deployment. In my case, every time I am encountering an issue like this one, clearing the doctrine cache solve it.

admin commented 2026-01-22 15:35:07 +01:00

Author

Owner

Copy Link

@beberlei commented on GitHub (Nov 13, 2020):

Why are you using the result cache? Is that needed to scale your project?

@beberlei commented on GitHub (Nov 13, 2020): Why are you using the result cache? Is that needed to scale your project?

admin commented 2026-01-22 15:35:07 +01:00

Author

Owner

Copy Link

@secit-pl commented on GitHub (Nov 13, 2020):

@Seb33300 php bin/console cache:pool:clear cache.global_clearer solves the problem.
I don't understand why executing the

php bin/console cache:clear --env=prod --no-warmup
php bin/console doctrine:cache:clear-metadata --env=prod
php bin/console doctrine:cache:clear-query --env=prod
php bin/console doctrine:cache:clear-result --env=prod

was not enough?

@beberlei 1000+ very active users synchronizing application after each action (offline mode support, so the data shot can be very big). Each synchronization is about 40+ serwer requests and makes a lot of db queries. Server is not a high end machine so I need to cache some things to handle requests in reasonable time. Moreover the database is not an ACID compatible (legacy software integration) so I had to make some hacks to handle database inconsistencies.

@secit-pl commented on GitHub (Nov 13, 2020): @Seb33300 `php bin/console cache:pool:clear cache.global_clearer` solves the problem. I don't understand why executing the ``` php bin/console cache:clear --env=prod --no-warmup php bin/console doctrine:cache:clear-metadata --env=prod php bin/console doctrine:cache:clear-query --env=prod php bin/console doctrine:cache:clear-result --env=prod ``` was not enough? @beberlei 1000+ very active users synchronizing application after each action (offline mode support, so the data shot can be very big). Each synchronization is about 40+ serwer requests and makes a lot of db queries. Server is not a high end machine so I need to cache some things to handle requests in reasonable time. Moreover the database is not an ACID compatible (legacy software integration) so I had to make some hacks to handle database inconsistencies.

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

3.6.x

2.20.x

dependabot/github_actions/2.20.x/doctrine/dot-github/dot-github/workflows/composer-lint.yml-14.0.0

3.7.x

4.0.x

2.21.x

old-prototype-3.x

3.6.2

3.6.1

3.6.0

2.20.9

3.5.8

3.5.7

3.5.6

3.5.5

2.20.8

3.5.4

3.5.3

2.20.7

3.5.2

2.20.6

3.5.1

3.5.0

3.4.4

3.4.3

3.4.2

2.20.5

3.4.1

3.4.0

3.3.4

2.20.4

3.3.3

2.20.3

3.3.2

2.20.2

3.3.1

2.20.1

3.3.0

3.2.3

2.20.0

2.19.8

3.2.2

2.19.7

3.2.1

2.19.6

3.2.0

3.1.4

3.1.3

2.19.5

3.1.2

2.19.4

3.1.1

2.19.3

2.19.2

2.19.1

3.1.0

2.19.0

3.0.3

2.18.3

3.0.2

2.18.2

3.0.1

2.18.1

3.0.0

2.18.0

2.17.5

3.0.0-RC1

2.17.4

2.17.3

2.17.2

2.17.1

3.0.0-beta2

2.17.0

2.16.3

3.0.0-beta1

2.16.2

2.16.1

2.16.0

2.15.5

2.15.4

2.15.3

2.15.2

2.15.1

2.15.0

2.14.3

2.14.2

2.14.1

2.14.0

2.13.5

2.13.4

2.13.3

2.13.2

2.13.1

2.13.0

2.12.4

2.12.3

2.12.2

2.12.1

2.12.0

2.11.3

2.11.2

2.11.1

2.11.0

2.10.5

2.10.4

2.10.3

2.10.2

2.10.1

2.10.0

2.9.6

2.9.5

2.9.4

2.9.3

2.9.2

2.9.1

2.9.0

2.8.5

2.8.4

2.8.3

2.8.2

2.8.1

2.8.0

2.7.5

2.7.4

v2.7.3

v2.7.2

v2.7.1

v2.7.0

v2.6.6

v2.6.5

v2.6.4

v2.6.3

v2.6.2

v2.6.1

v2.6.0

v2.5.14

v2.5.13

v2.5.12

v2.5.11

v2.5.10

v2.5.9

v2.5.8

v2.5.7

v2.5.6

v2.5.5

v2.5.4

v2.5.3

v2.5.2

v2.4.8

v2.5.1

v2.5.0

v2.5.0-RC2

v2.5.0-RC1

v2.5.0-beta1

v2.5.0-alpha2

v2.5.0-alpha1

v2.4.7

v2.4.6

v2.4.5

v2.4.4

v2.4.3

v2.3.6

v2.4.2

2.3.5

v2.4.1

v2.4.0

2.4.0-RC2

2.4.0-RC1

2.4.0-BETA2

2.3.4

2.3.3

2.4.0-BETA1

2.3.2

2.3.1

2.3.0

2.3.0-RC4

2.3.0-RC3

2.3.0-RC2

2.2.3

2.3.0-RC1

2.3.0-BETA1

2.1.7

2.2.2

2.2.1

2.1.6

2.2.0

2.2.0-RC1

2.2.0-BETA2

2.2.0-BETA1

2.1.5

2.1.4

2.1.3

2.1.2

2.1.1

2.0.7

2.1.0

2.1.0RC3

2.1.0RC2

2.1.0RC1

2.0.6

2.1.0BETA1

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

2.0.0RC2

2.0.0RC1

2.0.0-BETA4

2.0.0-BETA3

2.0.0-BETA2

2.0.0-BETA1

Labels

Clear labels

BC Break

Bug

Can't Fix

Deprecation

Documentation

DQL

Duplicate

Failing Test

Hacktoberfest

Improvement

Improvement

Incomplete

Invalid

Missing Tests

New Feature

PR Created

pull-request

Mirrored from GitHub Pull Request

Question

Regression

Status: Waiting feedback

Waiting feedback

WIP

Won't Fix

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

admin

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: doctrine/archived-orm#6568