Scores & Evaluation - Langfuse
Hi, I have a query regarding scores related to "similarity to prompt injections" and "refusals" as mentioned in https://langfuse.com/docs/scores under "Kinds of scores". However none of the attributes from langchain eval or ragas attributes address these two. Have any of you implemented scores to address these aspects?
Scores & Evaluation - Langfuse
Scores serve as a metric for evaluating individual LLM executions or traces. Commonly used to measure quality, tonality, factual accuracy, completeness, and relevance, among others.
2 Replies
Hi Raji,
Welcome to our Discord community! 🙌
Regarding your query, these aren't directly covered by the current attributes in LangChain Eval or Ragas. However, you can explore the concepts behind them at LLM-Guard's input scanners for prompt injections (https://llm-guard.com/input_scanners/prompt_injection/). This resource offers insights into handling and evaluating such scenarios, which might be helpful in your implementation.
Hey, thanks Richard.🙌 I was able to use Aspect Critique of ragas metrics for refusal by simply posing a question whether the submission sounds like refusal. But that didnt work for prompt injections, probably because these questions only work on the response and not the queries. Let me try the input scanners link for a possible solution.