This issue has been created
 
 
LLM AI Integration / cid:jira-generated-image-avatar-dad6e05b-35bb-4ba3-b9a9-e60a4c03b29d LLMAI-191 Open

Search snippets can be cut inside a multi-byte character

 
View issue   ยท   Add comment
 

Issue created

 
cid:jira-generated-image-avatar-6c58f060-79f5-4975-9bc7-4b93dcd16b9e Paul Pantiru created this issue on 30/Sep/26 17:19
 
Summary: Search snippets can be cut inside a multi-byte character
Issue Type: cid:jira-generated-image-avatar-dad6e05b-35bb-4ba3-b9a9-e60a4c03b29d Bug
Affects Versions: 0.10
Assignee: Unassigned
Created: 30/Sep/26 17:19
Priority: cid:jira-generated-image-static-minor-645051d4-3ebd-43c5-930a-40d3a33f9d5e Minor
Reporter: Paul Pantiru
Description:

With hl.fragsize=200 the highlight fragment of emoji-rich content ends in U+FFFD replacement characters. (ex:  ๐Ÿš€๐ŸŒŸ๐Ÿ”ฅ๐ŸŽ‰โœจ๐ŸŒˆ๐Ÿ€๐ŸŽ๏ฟฝ๏ฟฝ๏ฟฝ....)

 

Expected: The fragment ends on a character (ideally word) boundary.

Actual: Three replacement characters before the ellipsis.