Nils Reichardt•10mo ago

Zero latency when using .end()

Anything missing in my code which causes setting the completion time for "chat-copmletion" generations to -0.00s?

8 Replies

Max•9mo ago

This looks off. I will look into this today.

Max•9mo ago

which JS SDK version are you using? I just tested it on my machine and it seems to work fine

const span = trace.span({
  name: "test-span-1",
});

// wait for 200 milliseconds
await new Promise((resolve) => setTimeout(resolve, 200));

span.end();

const span = trace.span({
  name: "test-span-1",
});

// wait for 200 milliseconds
await new Promise((resolve) => setTimeout(resolve, 200));

span.end();

Nils Reichardt•9mo ago

I'm using the latest JS SDK version (2.0.1). When looking in Chrome DevTools when loading the Langfuse web app, I'm seeing that the startTime and endTime are equal:

[
    {
        "result": {
            "data": {
                "json": {
                    "id": "eb0b1a4d-9793-4144-82d1-4854e7d70258",
                    "traceId": "f2d90b26-6bd8-43bc-82ea-7285fd50026f",
                    "projectId": "clqa3z7ox00003m2f63ecmz8k",
                    "type": "GENERATION",
                    "startTime": "2023-12-27T15:20:56.525Z",
                    "endTime": "2023-12-27T15:20:56.521Z",
                    "name": "chat-completion-1",

[
    {
        "result": {
            "data": {
                "json": {
                    "id": "eb0b1a4d-9793-4144-82d1-4854e7d70258",
                    "traceId": "f2d90b26-6bd8-43bc-82ea-7285fd50026f",
                    "projectId": "clqa3z7ox00003m2f63ecmz8k",
                    "type": "GENERATION",
                    "startTime": "2023-12-27T15:20:56.525Z",
                    "endTime": "2023-12-27T15:20:56.521Z",
                    "name": "chat-completion-1",

Very weird. But at the moment I think it's somehow a bug in my code because it only happens with the "chat-completion" generations. The previous generation works fine 🤔

Max•9mo ago

The first ts ends with 525, the second with 521. So they are different, but starttime is older than endtime.

Nils Reichardt•9mo ago

const openAiKey = process.env.OPENAI_KEY!;
    const openai = new OpenAI({
      apiKey: process.env.OPENAI_KEY,
    });

    let generation: LangfuseGenerationClient | undefined;

    return await backOff<GptResponse>(
      async () => {
        const model = params.model ?? "gpt-3.5-turbo-1106";

        generation = params.trace.generation({
          name: `chat-completion-${params.tryCount}`,
          model: model,
          completionStartTime: new Date(),
          modelParameters: {
            temperature: 1.0,
          },
          input: params.messages,
        });

        const chatCompletion = await openai.chat.completions.create({
          messages: params.messages,
          model: model,
          response_format: {
            type: "json_object",
          },
        });

        generation.end({
          output: chatCompletion,
          usage: {
            input: chatCompletion.usage?.prompt_tokens,
            output: chatCompletion.usage?.completion_tokens,
            total: chatCompletion.usage?.total_tokens,
          },
        });

        return {
          message: chatCompletion.choices[0].message.content ?? "",
          finishReason: chatCompletion.choices[0].finish_reason ?? "null",
          usage: {
            totalTokens: chatCompletion.usage?.total_tokens ?? 0,
            completionTokens: chatCompletion.usage?.completion_tokens ?? 0,
            promptTokens: chatCompletion.usage?.prompt_tokens ?? 0,
          },
          langfuseGenerationId: generation?.id,
        };
      },
      {
        retry(e, attemptNumber) {
          generation?.update({
            level: "ERROR",
            statusMessage: `${e}`,
          });
          generation?.end();

          return true;
        },
        numOfAttempts: 5,
      }
    );

const openAiKey = process.env.OPENAI_KEY!;
    const openai = new OpenAI({
      apiKey: process.env.OPENAI_KEY,
    });

    let generation: LangfuseGenerationClient | undefined;

    return await backOff<GptResponse>(
      async () => {
        const model = params.model ?? "gpt-3.5-turbo-1106";

        generation = params.trace.generation({
          name: `chat-completion-${params.tryCount}`,
          model: model,
          completionStartTime: new Date(),
          modelParameters: {
            temperature: 1.0,
          },
          input: params.messages,
        });

        const chatCompletion = await openai.chat.completions.create({
          messages: params.messages,
          model: model,
          response_format: {
            type: "json_object",
          },
        });

        generation.end({
          output: chatCompletion,
          usage: {
            input: chatCompletion.usage?.prompt_tokens,
            output: chatCompletion.usage?.completion_tokens,
            total: chatCompletion.usage?.total_tokens,
          },
        });

        return {
          message: chatCompletion.choices[0].message.content ?? "",
          finishReason: chatCompletion.choices[0].finish_reason ?? "null",
          usage: {
            totalTokens: chatCompletion.usage?.total_tokens ?? 0,
            completionTokens: chatCompletion.usage?.completion_tokens ?? 0,
            promptTokens: chatCompletion.usage?.prompt_tokens ?? 0,
          },
          langfuseGenerationId: generation?.id,
        };
      },
      {
        retry(e, attemptNumber) {
          generation?.update({
            level: "ERROR",
            statusMessage: `${e}`,
          });
          generation?.end();

          return true;
        },
        numOfAttempts: 5,
      }
    );

Max•9mo ago

From the example above, did you run into a retry state?

Nils Reichardt•9mo ago

Yes, I run into a return statement. The total time of the trace (35s) looks good

Nils Reichardt•9mo ago

Found a reproducable code sample: https://github.com/langfuse/langfuse/issues/779 Not sure if I'm using trace.generation() after generation.end() is allowed.

GitHub

bug: generation time is 0s when using trace.generation after call...

Describe the bug You can in this screenshot that the time for the "usage-playground" generation is -0.00s. This happens, when you call trace.generation() after calling generation.end(). T...