|
| Summary: |
Collection search returns the text of a previous version of a page without indicating it |
| Issue Type: |
 Improvement |
| Affects Versions: |
0.11, 1.0.0 |
| Assignee: |
Unassigned |
| Created: |
29/Oct/26 11:43 |
| Priority: |
 Major |
| Reporter: |
Paul Pantiru |
| Description: |
A chunk stores the id, wiki, document id, document URL, language, collection, chunk index, character positions, content, embeddings, error message and store hint of the piece of text it was cut from (Chunk). It does not store the version or the date of the page it was indexed from. Indexing is asynchronous: a save queues a task, and the chunks are replaced when the task runs, which takes seconds on an idle instance, longer when the queue is loaded (after an import or a collection reindex), and indefinitely when the embedding model is unavailable and the task ends up abandoned. During that time the search returns the text of the previous version as if it were the current one. Neither the chat nor an MCP agent can tell that the text is out of date, and nothing points them at the live page. Rights are checked at query time against the live page, so the text is only served to users who may view the page now; the problem is correctness, not disclosure. Proposal
- Store the version and the save date of the page in every chunk at indexing time.
- At query time, where the authorization step already resolves the page of every hit, read the current version of the page and compare it with the version stored in the chunk.
- Make the handling of a stale hit a collection setting. Default label: the hit is kept and the result carries a stale marker; the search_collections tool prints a line such as "Indexed from version 3.1, current version is 4.1; re-read the page with get_document", and the chat receives the same note with the source. Setting drop: stale hits are removed, for collections where the text of a previous version must never be served.
- Chunks indexed before the change have no version: treat them as of unknown freshness, not as stale, until the page is next indexed.
- The new chunk fields require a Solr schema update. Check whether the core version bump used by the reindexer can back-fill the version from the page without re-embedding, so that the upgrade does not cost a full re-embedding of every collection.
|
|
