________•10mo ago

Hiya, I'm using the langchain

Hiya, I'm using the langchain OpenAiEmbeddings I want to track the calls to this using langfuse but I'm not sure whether this is supported/how to inject the CallbackHandler. Cheers

7 Replies

denis56•10mo ago

I would be interested in this as well

Marc•10mo ago

Unfortunately, Langchain's callback system does not support embeddings :/ https://github.com/langchain-ai/langchain/issues/945

GitHub

Total token count of openai callback does not count embedding usage...

When using embeddings, the total_tokens count of a callback is wrong, e.g. the following example currently returns 0 even though it shouldn't: from langchain.callbacks import get_openai_callbac...

Marc•10mo ago

We are currently working on a more generalized solution to embedding tracking and will then figure out how to make this easy to use for Langchain users Definetly on the radar

________•10mo ago

Ahh great. Thanks for letting me know. Is there a way to track cost by embeddings you currently recommend?

Marc•10mo ago

yes, if you have access to the used tokens, you can use the generation object to track embeddings. Costs are then calculated based on the token amounts Let me give you an example, 1 sec

from datetime import datetime
from langfuse.model import InitialGeneration, Usage

generationStartTime = datetime.now()


# embedding function


generation = langfuse.generation(InitialGeneration(
    name="doc-embedding",
    startTime=generationStartTime,
    endTime=datetime.now(),
    model="text-ada-001",
    prompt="whatever you want to store about the input, do not log all of the context here for embeddings",
    usage=Usage(promptTokens=50, completionTokens = 49), # you need to get them from your embedding function
    metadata={"interface": "whatsapp"} # whatever you want to add
))

from datetime import datetime
from langfuse.model import InitialGeneration, Usage

generationStartTime = datetime.now()


# embedding function


generation = langfuse.generation(InitialGeneration(
    name="doc-embedding",
    startTime=generationStartTime,
    endTime=datetime.now(),
    model="text-ada-001",
    prompt="whatever you want to store about the input, do not log all of the context here for embeddings",
    usage=Usage(promptTokens=50, completionTokens = 49), # you need to get them from your embedding function
    metadata={"interface": "whatsapp"} # whatever you want to add
))

instead of promptTokens, completionTokens, you can here also just use usage=Usage(totalTokens=50) let me know if this works for you

________•10mo ago

That worked. That's a lot

Marc•10mo ago

Perfect 👍 Let me know if you have other questions