Help Needed with Calculating Pricing for Processing Documents with Langchain #26640
Unanswered
DavidNavarroSaiz
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi Langchain Team,
I’m working on a project where I load documents (PDF, DOCX, TXT), split them into smaller chunks using the RecursiveCharacterTextSplitter, and then convert them into graph nodes and relationships with LLMGraphTransformer to store in a graph database.
Here’s a simplified version of my process:
Load the document (different formats like PDF, DOCX, TXT).
Split the document into chunks using RecursiveCharacterTextSplitter (chunk size: 1500, overlap: 30).
Extract nodes and relationships using LLMGraphTransformer.
Store the nodes and relationships in a graph database (e.g., Neo4j).
I would like to calculate the cost for processing each document, considering the following:
Each chunk of text processed by the model contributes to the cost.
I’m using OpenAI’s API for the LLM transformation.
I need to understand how to calculate or estimate the pricing for each document based on its size, the number of tokens, and the number of API calls.
Questions:
Is there an existing Langchain function or utility that helps calculate costs based on the number of tokens or API calls made during the document processing?
What’s the best way to estimate or calculate costs for each document processed, especially when the document is split into multiple chunks?
I appreciate any guidance or examples on how to approach pricing for document conversion with Langchain.
Thank you in advance!
code :
`class DocumentProcessor:
def init(self, llm, allowed_nodes, allowed_relationships):
self.llm = llm
self.allowed_nodes = allowed_nodes
self.allowed_relationships = allowed_relationships
`
Beta Was this translation helpful? Give feedback.
All reactions