jfarmer
Is it possible to score a full dataset run?
I'm wondering if there is a best practice for evaluations that require multiple traces. Primary use case would be running a prompt against a full dataset and wanting to evaluate the total precision/recall/f1/etc. Right now I can score each dataset item but I haven't figured out a great way to surface metrics to the UI that would encompass the full run.
The alternative I've tested is encompassing the full run in a trace and scoring that but it seems a bit hacky.
6 replies