justanothergraphguy Comments

justanothergraphguy

Posts Comments

LLangfuse

•Created by justanothergraphguy on 4/29/2024 in #get-support

I have had langfuse deployed and self-

I will let you know if the issue arises again

9 replies

LLangfuse

•Created by justanothergraphguy on 4/29/2024 in #get-support

I have had langfuse deployed and self-

We are on 2.16.2 for the python sdk... I think I figured out the issue actually -- seems like we were submitting too many requests to the langfuse deployment, so I created replica service deployments which seems to have handled it

9 replies

LLangfuse

•Created by justanothergraphguy on 4/29/2024 in #get-support

I have had langfuse deployed and self-

hmm... actually I am getting some odd traces now as well. Some extraneous and erroneous traces that are nonsensical

9 replies

LLangfuse

•Created by justanothergraphguy on 4/29/2024 in #get-support

I have had langfuse deployed and self-

So I only saw this in the langserve output logs and not in langfuse logs

9 replies

LLangfuse

•Created by justanothergraphguy on 4/29/2024 in #get-support

I have had langfuse deployed and self-

Any ideas as to why this request timeout would occur? This error occurs from the requesting application -- The traces actually do make it through to the database and show appropriately in the frontend

9 replies

LLangfuse

•Created by justanothergraphguy on 4/29/2024 in #get-support

I have had langfuse deployed and self-

Giving up execute_task_with_backoff(...) after 3 tries (langfuse.request.APIError: upstream request timeout (504): None)
ERROR:backoff:Giving up execute_task_with_backoff(...) after 3 tries (langfuse.request.APIError: upstream request timeout (504): None)
ERROR:langfuse:error uploading: upstream request timeout (504): None
Traceback (most recent call last):
  File ".../lib/python3.9/site-packages/langfuse/request.py", line 99, in _process_response
    payload = res.json()
  File ".../lib/python3.9/site-packages/httpx/_models.py", line 761, in json
    return jsonlib.loads(self.content, **kwargs)
  File ".../lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File ".../lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File ".../lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../lib/python3.9/site-packages/langfuse/task_manager.py", line 138, in upload
    self._upload_batch(batch)
  File ".../lib/python3.9/site-packages/langfuse/task_manager.py", line 165, in _upload_batch
    execute_task_with_backoff(batch)
  File ".../lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File ".../lib/python3.9/site-packages/langfuse/task_manager.py", line 163, in execute_task_with_backoff
    return self._client.batch_post(batch=batch, metadata=metadata)
  File ".../lib/python3.9/site-packages/langfuse/request.py", line 52, in batch_post
    return self._process_response(
  File ".../lib/python3.9/site-packages/langfuse/request.py", line 103, in _process_response
    raise APIError(res.status_code, res.text)
langfuse.request.APIError: upstream request timeout (504): None

Giving up execute_task_with_backoff(...) after 3 tries (langfuse.request.APIError: upstream request timeout (504): None)
ERROR:backoff:Giving up execute_task_with_backoff(...) after 3 tries (langfuse.request.APIError: upstream request timeout (504): None)
ERROR:langfuse:error uploading: upstream request timeout (504): None
Traceback (most recent call last):
  File ".../lib/python3.9/site-packages/langfuse/request.py", line 99, in _process_response
    payload = res.json()
  File ".../lib/python3.9/site-packages/httpx/_models.py", line 761, in json
    return jsonlib.loads(self.content, **kwargs)
  File ".../lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File ".../lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File ".../lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../lib/python3.9/site-packages/langfuse/task_manager.py", line 138, in upload
    self._upload_batch(batch)
  File ".../lib/python3.9/site-packages/langfuse/task_manager.py", line 165, in _upload_batch
    execute_task_with_backoff(batch)
  File ".../lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File ".../lib/python3.9/site-packages/langfuse/task_manager.py", line 163, in execute_task_with_backoff
    return self._client.batch_post(batch=batch, metadata=metadata)
  File ".../lib/python3.9/site-packages/langfuse/request.py", line 52, in batch_post
    return self._process_response(
  File ".../lib/python3.9/site-packages/langfuse/request.py", line 103, in _process_response
    raise APIError(res.status_code, res.text)
langfuse.request.APIError: upstream request timeout (504): None

9 replies

LLangfuse

•Created by justanothergraphguy on 3/4/2024 in #self-host-discussion

AWS RDS IAM Connection

Yeah, I'm not super familiar with the backend framework, but if the connection password is read in each time rather than once at the beginning of the server start, then that would allow flexibility to have the creds rotate

10 replies

LLangfuse

•Created by justanothergraphguy on 3/4/2024 in #self-host-discussion

AWS RDS IAM Connection

Well, yeah I can rotate them but the variables are set at the container level, right? I can create like a sidecar (or some background CLI process) to refresh the token, but that won't update the process env in the node app (I don't think?)... sorry I am not super familiar with ins-and-outs of node.js

10 replies

LLangfuse

•Created by justanothergraphguy on 3/4/2024 in #self-host-discussion

AWS RDS IAM Connection

RDS IAM Auth rotates the auth token every 15 minutes, so the string you pass as an env variable will expire unless explicitly refreshed. It's easy to refresh the token... I could probably fork the repo and implement it, but I would rather not have to do that

10 replies