Aggregating user feedback
I’m leading the AI Assistant project for my employer and they’re already asking for a way to have basically summaries of the feedback being received from langfuse. Sounds like that’s something I can build on top of the API but I’d love to see that baked into langfuse itself.
6 Replies
Thanks for sharing! Should be easy to build on top of the api but definitely interesting as a feature for langfuse mid-term.
Can you expand more on what kind of feedback you collect and how it should be summarized?
Ah I didn’t think about how feedback in langfuse can be all sorts of things when I wrote this. For me I just have a thumbs up (1.00), thumbs down (0.00) and a comment (string). I haven’t gathered their exact requirements yet so I’m not sure what kinds of summarization they really want yet. We’re hoping to launch next month so I’m sure we’ll know more then
Ok. Let’s keep the thread open and feel free to share more details/ideas on this over the next weeks
Hey @Marc
I would like to add to this but in a different way.
The user feedback features from LLM outputs are really good but our org would like to use them in a different manner. Specifically for internal reviews of output.
We want specific people to simply go through a bunch of different outputs and basically just score and give feed back foreach.
Like select these specific traces/generations are sent to a person for review. Each round of review has its own results and feedback from internal reviews can be referenced later.
Before we would paste each output into a google sheets and simply have our reviewers give feedback as comments.
But now we would like to link versioned prompts and outputs to specific feedback and vice versa.
As a custom solution we are planning to just have a webpage internally that loads specific outputs and prompts by calling langfuse API.
Then our reviewers can just give feedback for each output as a custom score/comment.
If this doesnt make much sense then please let me know as I can flesh it out further.
Would love to contribute in anyway to get this idea off the ground as that would literally make langfuse an all in one for my employer.
As a custom solution we are planning to just have a webpage internally that loads specific outputs and prompts by calling langfuse API. Then our reviewers can just give feedback for each output as a custom score/comment.Super helpful! @Marlies Mayerhofer is currently working on workflow tooling to assign labelers traces to review and annotate in Langfuse as we know many teams who have built their own frontends on top of the langfuse api. If you are well-versed in Typescript, we'd appreciate working together on this or at least aligning spec. Can we chat some time soon about this together?
@stupidretard6969 you can schedule some time with Marlies and me here in case you are interested: https://cal.com/marlies-mayerhofer+marc-kl