This issue has been created
There are 2 updates.
 
 
LLM AI Integration / cid:jira-generated-image-avatar-80612087-1b0a-4aa0-b489-3b7f2c82105c LLMAI-80 Open

Evaluate the output language of the LLM in the benchmark

 
View issue   ยท   Add comment
 

Issue created

 
cid:jira-generated-image-avatar-355b1bfa-9e71-46de-938e-82728aa9014b Michael Hamann created this issue on 27/May/24 15:22
 
Summary: Evaluate the output language of the LLM in the benchmark
Issue Type: cid:jira-generated-image-avatar-80612087-1b0a-4aa0-b489-3b7f2c82105c Improvement
Affects Versions: 0.4
Assignee: Unassigned
Created: 27/May/24 15:22
Priority: cid:jira-generated-image-static-major-dfdddb8b-12d7-4bcf-b92e-ceb811a3d71e Major
Reporter: Michael Hamann
Description:

The benchmark should evaluate the output language of the LLM and ensure it corresponds to the language of the question (as opposed to the language of the provided context). We should also try to improve the performance of the models by explicitly prompting them to use the question's language.

 
 

2 updates

 
cid:jira-generated-image-avatar-355b1bfa-9e71-46de-938e-82728aa9014b Changes by Michael Hamann on 27/May/24 15:22
 
Fix Version: 0.4
Assignee: Paul Pantiru