Johannes Wachter
6a45db5ef5
[Store] Split query string into words for TextQuery and HybridQuery in Retriever
2026-03-17 21:42:47 +01:00
Christopher Hertel
174012d169
Add features of #1778 and #1763 to changelog
2026-03-16 21:38:51 +01:00
Christopher Hertel
b8aa1ca5b8
feature #1778 [Store] Introduce batch processing in DistanceCalculator (Guikingone)
...
This PR was merged into the main branch.
Discussion
----------
[Store] Introduce batch processing in `DistanceCalculator`
| Q | A
| ------------- | ---
| Bug fix? | no
| New feature? | yes
| Docs? | yes
| Issues | --
| License | MIT
Following #1770 , local stores might be used to store / retrieve vectors, problem is, the current implementation perform the sorting strategy through the whole dataset, this PR aims to introduce a "batch and reduce" approach, `VectorDocument`s are handled per batch (default size to `100`) and returned in smaller size if `maxItems` is set.
The implementation is inspired by https://x.com/JeffDean/status/1997854093239595503 .
Commits
-------
293df7d9 feat(store): allow to sort documents as batches
2026-03-16 21:28:14 +01:00
Johannes Wachter
477439c00c
[Store] Skip missing toctree entries gracefully
2026-03-16 21:13:23 +01:00
Guillaume Loulier
280cc84477
feat(store): allow to sort documents as batches
2026-03-16 18:12:35 +01:00
Christopher Hertel
af632c6249
Consistently use symfony/event-dispatcher-contracts
2026-03-15 23:05:44 +01:00
Christopher Hertel
bc8181431f
bug #1772 [Store] Fix RstToctreeLoader trailing-slash toctree entry resolution (wachterjohannes)
...
This PR was merged into the main branch.
Discussion
----------
[Store] Fix RstToctreeLoader trailing-slash toctree entry resolution
| Q | A
| ------------- | ---
| Bug fix? | yes
| New feature? | no
| Docs? | no
| Issues | Fix #1771
| License | MIT
## Summary
Trailing-slash toctree entries like `components/` are valid Sphinx syntax and resolve to `components/index.rst`. The
loader now handles this convention instead of producing the invalid path `components/.rst`.
Commits
-------
1bd2a72e [Store] Fix RstToctreeLoader trailing-slash toctree entry resolution
2026-03-15 22:00:09 +01:00
Johannes Wachter
43ebaab9fc
[Store] Fix RstToctreeLoader trailing-slash toctree entry resolution
...
Trailing-slash toctree entries like `components/` are valid Sphinx
syntax and resolve to `components/index.rst`. The loader now handles
this convention instead of producing the invalid path `components/.rst`.
2026-03-15 21:43:33 +01:00
Johannes Wachter
5776f37099
[Store] Add PreQueryEvent for query enhancement
2026-03-15 20:10:17 +01:00
Johannes Wachter
2ab5e84e67
[Store][Platform] Add platform-level reranking and hybrid retriever
2026-03-14 16:07:15 +01:00
Johannes Wachter
d92c21f5b1
[Store] Add RstToctreeLoader for RST documentation parsing
2026-03-14 00:38:34 +01:00
Christopher Hertel
9b6cc8a5bd
Bump versions for v0.6
2026-03-05 00:27:55 +01:00
Johannes Wachter
8fd580222b
[Store] Add SummaryGeneratorTransformer for LLM-based document summaries
...
Add a document transformer that uses the Platform to generate concise
summaries and stores them in document metadata. Supports dual-indexing
mode where an additional TextDocument with the summary as content is
yielded alongside the original.
2026-02-28 21:29:26 +01:00
Christopher Hertel
38df638cd0
Prepare for v0.5.0
2026-02-19 19:34:43 +01:00
Christopher Hertel
435392b951
Prepare for release v0.4.0
2026-02-18 23:56:14 +01:00
Santiago San Martin
5f935922d9
[AI Bundle][MCP Bundle][Chat][Store] add ResetInterface to prevent memory leaks
2026-02-18 09:13:19 -03:00
Oskar Stark
a86e223342
Bump phpunit/phpunit to ^11.5.53 across all components
2026-02-18 11:36:38 +01:00
Christopher Hertel
1aa0c583b8
Keep lazy iterator alive
2026-02-16 21:27:32 +01:00
Johannes Wachter
6f0a3a902a
[Store] Add query abstraction with filter support
2026-02-16 20:33:28 +01:00
Christopher Hertel
55a8fbff11
minor #1563 [Store] Introduce integration tests for stores (chr-hertel)
...
This PR was merged into the main branch.
Discussion
----------
[Store] Introduce integration tests for stores
| Q | A
| ------------- | ---
| Bug fix? | no
| New feature? | no
| Docs? | no
| Issues |
| License | MIT
Want to tackle a few things here
* `examples/commands/store.php` is not an example, but integration tests
back then we went for the synergy about the docker setup, but with that `remove` PR it gets a bit more important for tests
* store tests are by far the slowest in our pipeline since we bring up all the container together and run it sequentially
* bring in a store conformance test with a base integration test
Initially built by Claude
Commits
-------
0e20e22f Introduce integration tests for stores
2026-02-13 23:23:09 +01:00
Christopher Hertel
badf72760f
Introduce integration tests for stores
2026-02-13 23:12:15 +01:00
bigdevlarry
7f944eef14
[Store] Add configurable json file loader
2026-02-13 21:48:38 +00:00
Christopher Hertel
d2170e29c4
feature #1537 [Store] Add CSV loader to load data from CSV file as text documents (RamyHakam)
...
This PR was squashed before being merged into the main branch.
Discussion
----------
[Store] Add CSV loader to load data from CSV file as text documents
| Q | A
| ------------- | ---
| Bug fix? | no
| New feature? | yes <!-- please update src/**/CHANGELOG.md files -->
| Docs? | no <!-- required for new features -->
| Issues | ---- <!-- prefix each issue number with "Fix #", no need to create an issue if none exist, explain below instead -->
| License | MIT
This PR adds a new `CsvLoader` class to the Store component for loading documents from CSV files. Each row in the CSV becomes a TextDocument, with configurable column mappings for content, ID, and metadata.
Features:
- Configurable content column (by name or index)
- Optional ID column mapping (generates UUID if not specified)
- Metadata columns extraction
- Custom delimiter, enclosure, and escape characters
- Header row support (enabled by default)
- Runtime options override via load() method
- Automatic skipping of empty content rows
- Proper handling of quoted content with commas and embedded quotes
```php
Usage example:
$loader = new CsvLoader(
contentColumn: 'content',
idColumn: 'id',
metadataColumns: ['author', 'category']
);
foreach ($loader->load('/path/to/data.csv') as $document) {
// $document is a TextDocument with content, ID, and metadata
}
```
Commits
-------
982e8274 [Store] Add CSV loader to load data from CSV file as text documents
2026-02-13 17:32:37 +01:00
Ramy Hakam
319f2cb881
[Store] Add CSV loader to load data from CSV file as text documents
2026-02-13 17:32:33 +01:00
Guillaume Loulier
9d4c88a1a2
feat(store): Add MarkdownLoader
2026-02-13 16:40:55 +01:00
Christopher Hertel
88964c9c11
Replace public properties by private+getters for BC promise
2026-02-13 16:00:48 +01:00
Christopher Hertel
d90745d750
minor #1575 [Store] Remove UID as type from Documents ID (chr-hertel)
...
This PR was merged into the main branch.
Discussion
----------
[Store] Remove UID as type from Documents ID
| Q | A
| ------------- | ---
| Bug fix? | no
| New feature? | no
| Docs? | no
| Issues |
| License | MIT
This might be a bit controversial, but while working on #1563 I stumbled on a few store implementations strongly assuming that `id` is a `Uuid` even tho we opened it to `string|int|Uuid` with #1397 - and i still thinkg `int|string` is def needed here.
Commits
-------
fdbecc55 Remove UID as type from Documents ID
2026-02-13 15:53:36 +01:00
Christopher Hertel
14ffae53ae
Upgrade Symfony dev dependencies in Store
2026-02-13 14:33:05 +01:00
Christopher Hertel
d1a03a9d32
Remove UID as type from Documents ID
2026-02-09 22:49:57 +01:00
Christopher Hertel
c05aad7011
[AI Bundle] Add DocumentIndexer support when no loader is configured
2026-02-05 14:39:40 +01:00
Christopher Hertel
debc193fd2
feature #1490 [Store][InMemory] Implement remove() method (chr-hertel)
...
This PR was merged into the main branch.
Discussion
----------
[Store][InMemory] Implement remove() method
| Q | A
| ------------- | ---
| Bug fix? | no
| New feature? | yes
| Docs? | no
| Issues | Fix #1469
| License | MIT
Implements document removal by ID for the InMemory store.
## Implementation
- Accepts `string|array` of document IDs
- Filters internal documents array and re-indexes with `array_values()` to prevent gaps
- Validates `$options` parameter (throws `InvalidArgumentException` if non-empty, consistent with other methods)
- Casts document IDs to strings for comparison
## Usage
```php
$store = new Store();
$store->add([
new VectorDocument($id1, new Vector([0.1, 0.1, 0.5])),
new VectorDocument($id2, new Vector([0.7, -0.3, 0.0])),
]);
// Remove single document
$store->remove((string) $id1);
// Remove multiple documents
$store->remove([(string) $id1, (string) $id2]);
```
## Tests
- Single/multiple document removal
- Non-existent ID handling
- Array re-indexing verification
- Options validation
> [!WARNING]
>
> <details>
> <summary>Firewall rules blocked me from connecting to one or more addresses (expand for details)</summary>
>
> #### I tried to connect to the following addresses, but was blocked by firewall rules:
>
> - `0e82bdfe85fcf91eb643c37c58bdd4a5666a80c1b2bed4734f6c19255616c534666659459c2f5dd3ffa29dc95e07d290f0c45ee4e978b78c360e2732dca41cd15c54750ef60cc581d4941e4f7fd7836ce41a24703d707984727bc71ecc56dfbbd9eacc08dbefd12671f16e1d5863115999dc7f1d01c49d4ea0eeab580c5cee1d3dfc92a98ada2bfb05a34cd1a004701b11e4c5a22bf41ed9e626a328cd424c5eb8afa9a97015c5dd40dc249c25bbb3183a9b26325cf245ed8623caa5777654391c61e42c95e1e86cee0326914d66dc7be5b8ee41d384aba42039c742b4ccd8571205faa86b4ca5c750386970a298763bc9396768853be331570a88b1eb038f8426f4eebcd36ad0d7821748f199f5f5b498e6316e1a43b41189dc41f1152c1ed04922118cfaaa8b18a3d4f3afc1ca381473cdf46882fb3e0404dc6b51c29005b7297d0ed22e3b415def830df37f7891fdfc727f0f5d8d13f2fcf6458abbf36abfe424c6f77d2d4e780e6075acecc687e3387bd8334211a14effec562e9169f247764275b53b8a41e38717ac76a02cddca63afe740e9bbd66f9d5571fd6a82fc59eb412e93ccba706060dc2c0eba1ad551b38811ad4daa7c3875d7043853a108a53ce084bb634857a3cc8b044a380872130d3833d7255c6d857f4d7617f6f9a6b14acd8b3205d8ced4d87521533be36cfa49bf1ca81c9d326bd6480cd1237c09dd24253745112560a31c4b10461bf83c725e727719ce8aba03a60f169c24dd4de28db7489ce515ce6a2f100chttps://gh.io/copilot/actions-setup-steps ) to set up my environment, which run before the firewall is enabled
> - Add the appropriate URLs or hosts to the custom allowlist in this repository's [Copilot coding agent settings](https://github.com/symfony/ai/settings/copilot/coding_agent ) (admins only)
>
> </details>
<!-- START COPILOT ORIGINAL PROMPT -->
<details>
<summary>Original prompt</summary>
>
> ----
>
> *This section details on the original issue you should resolve*
>
> <issue_title>[Store][InMemory] Implement remove() method</issue_title>
> <issue_description>Part of symfony/ai#1468
>
> Implement the `remove()` method for the InMemory store.
>
> **File:** `src/store/src/InMemory/Store.php`
>
> **Method signature:**
> ```php
> public function remove(string|array $ids, array $options = []): void;
> ```
>
> See Pinecone implementation in `src/store/src/Bridge/Pinecone/Store.php` for reference.</issue_description>
>
> <agent_instructions>Please make sure to extend existing tests, if there is one.</agent_instructions>
>
> ## Comments on the Issue (you are `@copilot` in this section)
>
> <comments>
> </comments>
>
</details>
<!-- START COPILOT CODING AGENT SUFFIX -->
- Fixes symfony/ai#1469
<!-- START COPILOT CODING AGENT TIPS -->
---
💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more [Copilot coding agent tips](https://gh.io/copilot-coding-agent-tips ) in the docs.
Commits
-------
9074cdfb Implement remove() method for InMemory Store
2026-01-31 17:36:10 +01:00
Christopher Hertel
32626ebb6c
Split Indexer into DocumentIndexer and SourceIndexer
2026-01-30 15:55:37 +01:00
Christopher Hertel
9dbdef915c
Fix to proper delay handling, more than once but also not after last element
2026-01-29 15:50:26 +01:00
Grégoire Pineau
96e15ef407
[Store] Allow iterable of source in Indexer::index() + does't break lazyness
2026-01-29 14:26:45 +01:00
Christopher Hertel
4caff3ce8f
Shift state out of Indexer service
2026-01-29 12:39:38 +01:00
Christopher Hertel
3522d1bdd3
Implement remove() method for InMemory Store
2026-01-28 23:06:36 +01:00
Christopher Hertel
61fbab7e54
feature #1389 [Store] Add support for "remove" operation (MolloKhan)
...
This PR was squashed before being merged into the main branch.
Discussion
----------
[Store] Add support for "remove" operation
| Q | A
| ------------- | ---
| Bug fix? | no
| New feature? | yes
| Docs? | yes
| Issues | Fix #1373
| License | MIT
Added `remove()` method to the `StoreInterface` and added real implementation to the Pinecone bridge
Commits
-------
3f380e50 [Store] Add support for "remove" operation
2026-01-28 21:47:40 +01:00
Diego Aguiar
41ed8bf792
[Store] Add support for "remove" operation
2026-01-28 21:47:29 +01:00
Christopher Hertel
94ebb7240d
Revert of bumping the branch-alias right away
2026-01-27 23:21:43 +01:00
Christopher Hertel
01507c60f2
Expose chunk size and delay via construct to actually override via container
2026-01-27 14:55:43 +01:00
Christopher Hertel
de92ee75fa
Bump versions and aliases for v0.3.0 release
2026-01-23 22:29:55 +01:00
Christopher Hertel
5c622870b4
CS follows Symfony rules update
2026-01-23 21:56:05 +01:00
Diego Aguiar
0a00012732
[Store] Allow id to be int|string|Uuid for VectorDocument and TextDocument
2026-01-18 00:29:17 +01:00
Christopher Hertel
6aa9f3febc
minor #1360 Prepare for release v0.2 (chr-hertel)
...
This PR was merged into the main branch.
Discussion
----------
Prepare for release v0.2
| Q | A
| ------------- | ---
| Bug fix? | no
| New feature? | no <!-- please update src/**/CHANGELOG.md files -->
| Docs? | no <!-- required for new features -->
| Issues | Fix #... <!-- prefix each issue number with "Fix #", no need to create an issue if none exist, explain below instead -->
| License | MIT
Commits
-------
e1ca6baf Prepare for release v0.2
2026-01-09 17:20:45 +01:00
Guillaume Loulier
c2fc46ecfc
conf(tests): executionOrder moved to random
2026-01-08 11:38:03 +01:00
Guillaume Loulier
ba1b505b29
fix(store): reword force option description
2026-01-08 09:42:03 +01:00
Christopher Hertel
51cf6f20bf
Prepare for release v0.2
2026-01-08 09:09:35 +01:00
Oskar Stark
d933217c30
[AI Bundle][Agent][Store] Require symfony/ai-platform ^0.2
2026-01-06 22:40:14 +01:00
Guillaume Loulier
70e237caf7
ref
2026-01-06 21:17:44 +01:00
Christopher Hertel
48748f341c
Bump branch alias to 0.2.x-dev
2026-01-06 15:41:54 +01:00
