FR: Support databases with limitations/stack overflow in lists - I'll provide the PRs #5990

New Issue

admin · 2026-01-22T15:24:13+01:00

admin commented

2026-01-22 15:24:13 +01:00

Originally created by @kafoso on GitHub (Jun 18, 2018).

Feature Request

Q	A
New Feature	yes
RFC	yes
BC Break	no

Summary

Some database systems have a limit to how many elements are allowed in a list, e.g. an IN statement. Proposition: Split IN statements into several IN statements separated by OR in chunks of a certain size (from %s IN (%s) to (%s IN (%s) OR %s IN(%s) ...)).

If this proposition is accepted, I will provide pull requests for all necessary versions of the Doctrine DBAL, applying the described changes.

There is no backward compatibility breakage.

Request details

Some database systems have a limit to how many elements are allowed in a list, e.g. an IN statement. Firebird (https://firebirdsql.org/), for instance. When exceeding 1500 (or 1499 due to a bug (source)) elements in a list the error looks like the following in Firebird:

Dynamic SQL Error 
SQL error code =-901 
Implementataion limit exceeded 
too many values (more than 1500) in member list to match against

This error occurs when data is extracted through the Doctrine ORM and when relations (OneToMany, etc.) contain the IDs of more than 1500 elements. Granted, it's a lot of relations, but nonetheless it's a case which causes a hard error/exception. And it's fixable.

Firebird Driver for Doctrine DBAL

I have implemented a stand-alone Firebird Driver for the Doctrine DBAL. You can review it here: https://github.com/kafoso/doctrine-firebird-driver

Proposition

Split IN statements into several IN statements separated by OR in chunks of a certain size (from %s IN (%s) to (%s IN (%s) OR %s IN(%s) ...)).

At a quick glance, it appears the list logic occurs in Doctrine\ORM\Persisters\Entity\SingleTablePersister->getSelectConditionDiscriminatorValueSQL() in version 2.4. I will, of course, thoroughly go through this, but I won't waste my time should the proposition be rejected.

If the above is the case, I propose the below change.

The platform interface (\Doctrine\DBAL\Platforms\AbstractPlatform) will be expanded with this:

/**
 * @return null|int                             When an integer it must be > 0.
 */
public function getMaximumElementCountInList()
{
    return null;
}

The extending class (e.g. \Kafoso\DoctrineFirebirdDriver\Platforms\FirebirdInterbasePlatform) may then provide an integer to enable chunking.

Doctrine\ORM\Persisters\Entity\SingleTablePersister->getSelectConditionDiscriminatorValueSQL() will be changed to:

if (is_int($this->platform->getMaximumElementCountInList())) {
    $chunks = array_chunk($values, $this->platform->getMaximumElementCountInList());
    return sprintf(
        "(%s)",
        implode(" OR ", array_map(function($chunk) use ($discrColumnType, $tableAlias, $quotedColumnName){
            return sprintf(
                '%s IN (%s)',
                $discrColumnType->convertToDatabaseValueSQL($tableAlias . '.' . $quotedColumnName, $this->platform),
                implode(', ', $chunk)
            );
        }, $chunks))
    );
}
return sprintf(
    '%s IN (%s)',
    $discrColumnType->convertToDatabaseValueSQL($tableAlias . '.' . $quotedColumnName, $this->platform),
    implode(', ', $values)
);

Logic may need to be adapted to other versions of the Doctrine DBAL, particularly those requiring PHP 7+.

I will, if the proposition is accepted, provide a pull request for version 2.4. Once reviewed and accepted, I will provide pull requests for other versions as well.

Yay or Nay?

Originally created by @kafoso on GitHub (Jun 18, 2018). ### Feature Request | Q | A |------------ | ------ | New Feature | yes | RFC | yes | BC Break | no #### Summary Some database systems have a limit to how many elements are allowed in a list, e.g. an `IN` statement. Proposition: Split `IN` statements into several `IN` statements separated by `OR` in chunks of a certain size (from `%s IN (%s)` to `(%s IN (%s) OR %s IN(%s) ...)`). If this proposition is accepted, I will provide pull requests for all necessary versions of the Doctrine DBAL, applying the described changes. There is no backward compatibility breakage. #### Request details Some database systems have a limit to how many elements are allowed in a list, e.g. an `IN` statement. Firebird (https://firebirdsql.org/), for instance. When exceeding 1500 (or 1499 due to a bug ([source](http://tracker.firebirdsql.org/browse/core-1438))) elements in a list the error looks like the following in Firebird: Dynamic SQL Error SQL error code =-901 Implementataion limit exceeded too many values (more than 1500) in member list to match against This error occurs when data is extracted through the Doctrine ORM and when relations (OneToMany, etc.) contain the IDs of more than 1500 elements. Granted, it's a lot of relations, but nonetheless it's a case which causes a hard error/exception. And it's fixable. ##### Firebird Driver for Doctrine DBAL I have implemented a stand-alone Firebird Driver for the Doctrine DBAL. You can review it here: https://github.com/kafoso/doctrine-firebird-driver #### Proposition Split `IN` statements into several `IN` statements separated by `OR` in chunks of a certain size (from `%s IN (%s)` to `(%s IN (%s) OR %s IN(%s) ...)`). At a quick glance, it appears the list logic occurs in `Doctrine\ORM\Persisters\Entity\SingleTablePersister->getSelectConditionDiscriminatorValueSQL()` in version 2.4. I will, of course, thoroughly go through this, but I won't waste my time should the proposition be rejected. If the above is the case, I propose the below change. The platform interface (`\Doctrine\DBAL\Platforms\AbstractPlatform`) will be expanded with this: ```php /** * @return null|int When an integer it must be > 0. */ public function getMaximumElementCountInList() { return null; } ``` The extending class (e.g. [`\Kafoso\DoctrineFirebirdDriver\Platforms\FirebirdInterbasePlatform`](https://github.com/kafoso/doctrine-firebird-driver/blob/a843bd709af27adc98da8cffd78a8737b7603e1f/src/Platforms/FirebirdInterbasePlatform.php)) may then provide an integer to enable chunking. `Doctrine\ORM\Persisters\Entity\SingleTablePersister->getSelectConditionDiscriminatorValueSQL()` will be changed to: ```php if (is_int($this->platform->getMaximumElementCountInList())) { $chunks = array_chunk($values, $this->platform->getMaximumElementCountInList()); return sprintf( "(%s)", implode(" OR ", array_map(function($chunk) use ($discrColumnType, $tableAlias, $quotedColumnName){ return sprintf( '%s IN (%s)', $discrColumnType->convertToDatabaseValueSQL($tableAlias . '.' . $quotedColumnName, $this->platform), implode(', ', $chunk) ); }, $chunks)) ); } return sprintf( '%s IN (%s)', $discrColumnType->convertToDatabaseValueSQL($tableAlias . '.' . $quotedColumnName, $this->platform), implode(', ', $values) ); ``` Logic may need to be adapted to other versions of the Doctrine DBAL, particularly those requiring PHP 7+. I will, if the proposition is accepted, provide a pull request for version 2.4. Once reviewed and accepted, I will provide pull requests for other versions as well. ----- Yay or Nay?

admin commented

2026-01-22 15:24:13 +01:00

@Majkl578 commented on GitHub (Jul 4, 2018):

@morozov WDYT?

@Majkl578 commented on GitHub (Jul 4, 2018): @morozov WDYT?

admin commented

2026-01-22 15:24:14 +01:00

@morozov commented on GitHub (Jul 5, 2018):

Some databases also have limitations on the number of bound parameters, it should be taken into account. I think it’s worth a try.

@morozov commented on GitHub (Jul 5, 2018): Some databases also have limitations on the number of bound parameters, it should be taken into account. I think it’s worth a try.

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: doctrine/archived-orm#5990