j6m6•10mo ago

Thanks, that would be great. For now, I

Thanks, that would be great. For now, I might be able to fork langserv and add the langfuse CallbackHandler.

5 Replies

j6m6•10mo ago

I've opened https://github.com/langchain-ai/langserve/issues/311 so that perhaps the upstream project can implement changes to support request callback handlers.

GitHub

Custom CallbackHandlers when chain is invoked · Issue #311 · langch...

Currently, I haven't found a way to configure request-based CallbackHandlers described at https://python.langchain.com/docs/modules/callbacks/#where-to-pass-in-callbacks. It looks like It looks...

Max•10mo ago

Nice! This would be a great addition to the lanserve project.

j6m6•10mo ago

It looks like a fix has been proposed https://github.com/langchain-ai/langserve/pull/317. It just needs to be merged to main and then released.

GitHub

Build software better, together

GitHub is where people build software. More than 100 million people use GitHub to discover, fork, and contribute to over 420 million projects.

j6m6•10mo ago

That's now in the latest langserve 0.0.36 release. I had to implement something like this using per_req_config_modifier to set callbacks on config:

langfuse = Langfuse(
    os.environ["LANGFUSE_PUBLIC_KEY"],
    os.environ["LANGFUSE_SECRET_KEY"],
    host=os.environ["LANGFUSE_HOST"]
)


def per_req_config_modifier(
        config: Dict[str, Any],
        request: Request
) -> Dict[str, Any]:
    if "callbacks" not in config:
        config["callbacks"] = []

    trace = langfuse.trace(CreateTrace()) # TODO set id based on conversation_id from request
    trace_handler = trace.getNewHandler()

    config["callbacks"].extend([trace_handler])
    return config


add_routes(app, my_chain, path="/my-chain", per_req_config_modifier=per_req_config_modifier)

langfuse = Langfuse(
    os.environ["LANGFUSE_PUBLIC_KEY"],
    os.environ["LANGFUSE_SECRET_KEY"],
    host=os.environ["LANGFUSE_HOST"]
)


def per_req_config_modifier(
        config: Dict[str, Any],
        request: Request
) -> Dict[str, Any]:
    if "callbacks" not in config:
        config["callbacks"] = []

    trace = langfuse.trace(CreateTrace()) # TODO set id based on conversation_id from request
    trace_handler = trace.getNewHandler()

    config["callbacks"].extend([trace_handler])
    return config


add_routes(app, my_chain, path="/my-chain", per_req_config_modifier=per_req_config_modifier)

Max•10mo ago

really cool! Thanks for sharing this!