Replies: 3 comments 1 reply
-
Yes, it is possible to configure LangChain to utilize the increased token limit of 8192 tokens for Claude 3.5 Sonnet with Amazon Bedrock. Here is the relevant code snippet for setting the from langchain_community.llms.bedrock import Bedrock
# Initialize the Bedrock LLM with the appropriate model ID and client
llm = Bedrock(
model_id="anthropic.claude-3.5-sonnet",
client=<bedrock_client>,
model_kwargs={
"max_tokens_to_sample": 8192 # Set the token limit to 8192
}
)
# Example usage
response = llm._call(prompt="Your prompt here")
print(response) In the @classmethod
def prepare_input(
cls,
provider: str,
model_kwargs: Dict[str, Any],
prompt: Optional[str] = None,
system: Optional[str] = None,
messages: Optional[List[Dict]] = None,
) -> Dict[str, Any]:
input_body = {**model_kwargs}
if provider == "anthropic":
if messages:
input_body["anthropic_version"] = "bedrock-2023-05-31"
input_body["messages"] = messages
if system:
input_body["system"] = system
if "max_tokens" not in input_body:
input_body["max_tokens"] = 1024
if prompt:
input_body["prompt"] = _human_assistant_format(prompt)
if "max_tokens_to_sample" not in input_body:
input_body["max_tokens_to_sample"] = 1024
elif provider in ("ai21", "cohere", "meta", "mistral"):
input_body["prompt"] = prompt
elif provider == "amazon":
input_body = dict()
input_body["inputText"] = prompt
input_body["textGenerationConfig"] = {**model_kwargs}
else:
input_body["inputText"] = prompt
return input_body To utilize the increased token limit, ensure that the |
Beta Was this translation helpful? Give feedback.
-
The implementation mentioned above did not work for me. As of August 19, 2024, while this is not for Amazon Bedrock, the original Anthropic API has officially supported this feature. Additionally, with the addition of "stop_reason" to the response_metadata of Bedrock (Anthropic), it's now possible to implement features such as generating continuations when the token count is insufficient. However, when actually executed, you can see that stop_reason has been added:
|
Beta Was this translation helpful? Give feedback.
-
I am still facing this issue, when when I set max_tokens in Chat Bedrock to 8192, I still get the following error:
|
Beta Was this translation helpful? Give feedback.
-
Checked other resources
Description
Anthropic's Claude 3.5 Sonnet now supports a maximum output of 8192 tokens. I'd like to know if there's currently a way to set the max_tokens parameter to 8192 when using Claude 3.5 Sonnet through Amazon Bedrock via LangChain.
For reference, Anthropic's documentation about Claude models can be found here:
https://docs.anthropic.com/en/docs/about-claude/models
Has anyone successfully configured LangChain to utilize this increased token limit with Amazon Bedrock's implementation of Claude 3.5 Sonnet? If so, could you please share the method or any relevant code snippets?
Thank you in advance for any insights or assistance!
Beta Was this translation helpful? Give feedback.
All reactions