shenghan
shenghan•5mo ago

Google Cloud Vertex AI | 🦜🔗 LangChain

Yes they provide token counts but looks like only if using the Langchain generate() method to call. See: https://python.langchain.com/docs/integrations/llms/google_vertex_ai_palm/ GenerationChunk can return the following 
'usage_metadata': {'prompt_token_count': 15, 'candidates_token_count': 647, 'total_token_count': 662}})
'usage_metadata': {'prompt_token_count': 15, 'candidates_token_count': 647, 'total_token_count': 662}})
Can langfuse capture token counts from the langchain GenerationChunk on vertex AI models? Any pointers where to start?
Google Cloud Vertex AI | 🦜️🔗 LangChain
Note: This is separate from the Google Generative AI integration,
6 Replies
shenghan
shenghan•5mo ago
Any guidance on the above? @Marc
Marc
Marc•5mo ago
sorry for slow response here this is good to know, I'll create an issue to add this to our langchain js integration
Marc
Marc•5mo ago
added this here: https://github.com/langfuse/langfuse/issues/2032
GitHub
feat: capture token counts from google vertex in langchain-python ·...
When using generate the GenerationChunk object contains usage_metadata. It'd be useful to capture the token counts from there. Example: [[GenerationChunk([...] generation_info={'usage_metad...
Marc
Marc•5mo ago
If you want to contribute to this, we're happy to support
shenghan
shenghan•5mo ago
no worries @Marc! thanks for creating the issue - I would be happy to contribute this to langchain-python. just need some pointers to where does token capturing typically occur and happy to figure out the rest!
shenghan
shenghan•5mo ago