Flexible cost calculation
i understand you have in hosue cost calculations. But how about models you don't support? Where should I store the cost of generations once I calculate it? having cost in 2 different tables/places in the backend will make cost tracking over time very hard
8 Replies
cc'ing @Max here who is cooking on something regarding this.
Hi Gokusan, yes we want to streamline this process and enable our users to change costs per models themselves. This is very high on our roadmap. This means, that right now it is only possible to ingest tokens manually. If the model is not known to us, we wont calculate the costs. Which model are you looking at? I will take it onto our list then.
I think my pain is more to have 1 single place in your data model to store the costs, whether YOU calculate it, or the user does
In 1 year, I want to be able to easily do something like "select sum(cost) from generations" or something like that, whether YOU calculated the cost or I did. I don't want to have to pull cost data wherever YOU stored it for your known models,. and have a different pull like "select sum(metadata['custom_cost_calculation) from generations"
hope that clarifies it
your auto cost calculation should only happen if the user doesn't already provide a cost.
Hi Gokusan, sorry for the late reply. I understand, that you want to have a dedicated column to store the cost of generations so that you can pull the data and are independent from the models we know and support. Thanks for the clarification. Would a solution work for you where you can specify the cost (e.g. 1 ct per 1k characters for a certain model), you provide us the usage in case we dont support the tokenizer via the SDK, and we calculate the USD cost for you wherever needed in the UI?
This way, you could define how costs should be calculated and we apply that throughout the UI. As mentioned above, this is high on the prio list. Expect news on that within the next week.
I think it wld make more sense to allow users to add a "prompt_cost" and "completion_cost" to the Usage object (or Cost object or something)
If user provides it, then just use that. If user doesnt provide it, then try to auto calculate cost using YOUR formulas
With that said, the point is more about having cost always being in the same place in the DB. whether user provides actual cost, or the $/token and you calculate it im indifferent
Hi gokusan, we had a session around that topic today and the plan is to allow such inputs via the SDK. We would only calculate cost if it was not provided byt the user 🙂
This would also mean there is a place for cost in the db that you can query 🙂
Noyyceeeee
great stufff
congrats on the new hire btw!
I saw the post on linkedin
Thanks a lot!! Hopefully we can fully focus on shipping more features soon 🙂