Nils Reichardt Comments

Nils Reichardt

•Created by Nils Reichardt on 12/21/2023 in #get-support

Zero latency when using .end()

Found a reproducable code sample: https://github.com/langfuse/langfuse/issues/779 Not sure if I'm using trace.generation() after generation.end() is allowed.

8 replies

LLangfuse

•Created by Nils Reichardt on 12/27/2023 in #get-support

Is it possible to search for number

A (Langfuse) generation can create up to 40 flashards. It depends on a few factors.

10 replies

LLangfuse

•Created by Nils Reichardt on 12/27/2023 in #get-support

Is it possible to search for number

My goal is to identify expensive traces (measured by "Tokens per flashcard" - using total cost of the trace is not really helpful because a trace can generate a deck between 10 and 40 cards)

10 replies

LLangfuse

•Created by Nils Reichardt on 12/27/2023 in #get-support

Is it possible to search for number

Thanks for the response 🙏 Is sorting (desc or asc) a metadata key complex?

10 replies

LLangfuse

•Created by Nils Reichardt on 12/21/2023 in #get-support

Zero latency when using .end()

8 replies

LLangfuse

•Created by Nils Reichardt on 12/21/2023 in #get-support

Zero latency when using .end()

const openAiKey = process.env.OPENAI_KEY!;
    const openai = new OpenAI({
      apiKey: process.env.OPENAI_KEY,
    });

    let generation: LangfuseGenerationClient | undefined;

    return await backOff<GptResponse>(
      async () => {
        const model = params.model ?? "gpt-3.5-turbo-1106";

        generation = params.trace.generation({
          name: `chat-completion-${params.tryCount}`,
          model: model,
          completionStartTime: new Date(),
          modelParameters: {
            temperature: 1.0,
          },
          input: params.messages,
        });

        const chatCompletion = await openai.chat.completions.create({
          messages: params.messages,
          model: model,
          response_format: {
            type: "json_object",
          },
        });

        generation.end({
          output: chatCompletion,
          usage: {
            input: chatCompletion.usage?.prompt_tokens,
            output: chatCompletion.usage?.completion_tokens,
            total: chatCompletion.usage?.total_tokens,
          },
        });

        return {
          message: chatCompletion.choices[0].message.content ?? "",
          finishReason: chatCompletion.choices[0].finish_reason ?? "null",
          usage: {
            totalTokens: chatCompletion.usage?.total_tokens ?? 0,
            completionTokens: chatCompletion.usage?.completion_tokens ?? 0,
            promptTokens: chatCompletion.usage?.prompt_tokens ?? 0,
          },
          langfuseGenerationId: generation?.id,
        };
      },
      {
        retry(e, attemptNumber) {
          generation?.update({
            level: "ERROR",
            statusMessage: `${e}`,
          });
          generation?.end();

          return true;
        },
        numOfAttempts: 5,
      }
    );

const openAiKey = process.env.OPENAI_KEY!;
    const openai = new OpenAI({
      apiKey: process.env.OPENAI_KEY,
    });

    let generation: LangfuseGenerationClient | undefined;

    return await backOff<GptResponse>(
      async () => {
        const model = params.model ?? "gpt-3.5-turbo-1106";

        generation = params.trace.generation({
          name: `chat-completion-${params.tryCount}`,
          model: model,
          completionStartTime: new Date(),
          modelParameters: {
            temperature: 1.0,
          },
          input: params.messages,
        });

        const chatCompletion = await openai.chat.completions.create({
          messages: params.messages,
          model: model,
          response_format: {
            type: "json_object",
          },
        });

        generation.end({
          output: chatCompletion,
          usage: {
            input: chatCompletion.usage?.prompt_tokens,
            output: chatCompletion.usage?.completion_tokens,
            total: chatCompletion.usage?.total_tokens,
          },
        });

        return {
          message: chatCompletion.choices[0].message.content ?? "",
          finishReason: chatCompletion.choices[0].finish_reason ?? "null",
          usage: {
            totalTokens: chatCompletion.usage?.total_tokens ?? 0,
            completionTokens: chatCompletion.usage?.completion_tokens ?? 0,
            promptTokens: chatCompletion.usage?.prompt_tokens ?? 0,
          },
          langfuseGenerationId: generation?.id,
        };
      },
      {
        retry(e, attemptNumber) {
          generation?.update({
            level: "ERROR",
            statusMessage: `${e}`,
          });
          generation?.end();

          return true;
        },
        numOfAttempts: 5,
      }
    );

8 replies

LLangfuse

•Created by Nils Reichardt on 12/21/2023 in #get-support

Zero latency when using .end()

I'm using the latest JS SDK version (2.0.1). When looking in Chrome DevTools when loading the Langfuse web app, I'm seeing that the startTime and endTime are equal:

[
    {
        "result": {
            "data": {
                "json": {
                    "id": "eb0b1a4d-9793-4144-82d1-4854e7d70258",
                    "traceId": "f2d90b26-6bd8-43bc-82ea-7285fd50026f",
                    "projectId": "clqa3z7ox00003m2f63ecmz8k",
                    "type": "GENERATION",
                    "startTime": "2023-12-27T15:20:56.525Z",
                    "endTime": "2023-12-27T15:20:56.521Z",
                    "name": "chat-completion-1",

[
    {
        "result": {
            "data": {
                "json": {
                    "id": "eb0b1a4d-9793-4144-82d1-4854e7d70258",
                    "traceId": "f2d90b26-6bd8-43bc-82ea-7285fd50026f",
                    "projectId": "clqa3z7ox00003m2f63ecmz8k",
                    "type": "GENERATION",
                    "startTime": "2023-12-27T15:20:56.525Z",
                    "endTime": "2023-12-27T15:20:56.521Z",
                    "name": "chat-completion-1",

Very weird. But at the moment I think it's somehow a bug in my code because it only happens with the "chat-completion" generations. The previous generation works fine 🤔

8 replies

LLangfuse

•Created by Nils Reichardt on 12/27/2023 in #get-support

Is it possible to search for number

Current workaround: If I'm searching for traces with 'tokensPerCard' > 300 I can do multiple search queries with "'tokensPerCard' starts with '30'", "'tokensPerCard' starts with '40'", "'tokensPerCard' starts with '50'", etc.

10 replies

LLangfuse

•Created by Nils Reichardt on 12/27/2023 in #get-support

Is it possible to search for number

Ah, technically, it's possible but in practice not really useful because there are no >, <, >=, <= operators

10 replies