Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating readme + refactoring #478

Merged
merged 2 commits into from
Feb 18, 2025
Merged

Updating readme + refactoring #478

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0a1a66d
Updating readme + refactoring
ssinghgai Feb 7, 2025
f54dc74
Minor update in Readme
ssinghgai Feb 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions ...bases/features-examples/00-zero-setup-chat-with-your-document/chat_with_document_kb.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
"collapsed": false
},
"source": [
"## Chat with your document using Knowledge Bases for Amazon Bedrock - RetrieveAndGenerate API\n",
"## Chat with your document using Amazon Bedrock Knowledge Bases - RetrieveAndGenerate API\n",
" With `chat with your document` capability, you can securely ask questions on single documents, without the overhead of setting up a vector database or ingesting data, making it effortless for businesses to use their enterprise data. You only need to provide a relevant data file as input and choose your FM to get started.\n",
"\n",
"For details around use cases and benefits, please refer to this [blogpost](#https://aws.amazon.com/blogs/machine-learning/knowledge-bases-in-amazon-bedrock-now-simplifies-asking-questions-on-a-single-document/).\n",
Expand Down Expand Up @@ -278,11 +278,11 @@
"metadata": {},
"source": [
"## Next Steps\n",
"In this notebook, we covered how Knowledge Bases for Amazon Bedrock now simplifies asking questions on a single document. We also demonstrated how to configure and use this capability through the Amazon Bedrock - AWS SDK, showcasing the simplicity and flexibility of this feature, which provides a zero-setup solution to gather information from a single document, without setting up a vector database.\n",
"In this notebook, we covered how Amazon Bedrock Knowledge Bases now simplifies asking questions on a single document. We also demonstrated how to configure and use this capability through the Amazon Bedrock - AWS SDK, showcasing the simplicity and flexibility of this feature, which provides a zero-setup solution to gather information from a single document, without setting up a vector database.\n",
"\n",
"To further explore the capabilities of Knowledge Bases for Amazon Bedrock, refer to the following resources:\n",
"To further explore the capabilities of Amazon Bedrock Knowledge Bases, refer to the following resources:\n",
"\n",
"[Knowledge bases for Amazon Bedrock](#https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html)\n"
"[Amazon Bedrock Knowledge Bases](#https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html)\n"
]
},
{
Expand Down
12 changes: 6 additions & 6 deletions ...bases/features-examples/01-rag-concepts/01_create_ingest_documents_test_kb_multi_ds.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Knowledge Bases for Amazon Bedrock - End to end example using multiple data sources as data source(s)\n",
"# Amazon Bedrock Knowledge Bases - End to end example using multiple data sources as data source(s)\n",
"\n",
"This notebook provides sample code for building an empty OpenSearch Serverless (OSS) index,Knowledge bases for Amazon Bedrock and ingest documents into the index from various data sources (S3, Confluence, Sharepoint, Salesforce, and Web). Please note that you can add upto 5 data sources.\n",
"This notebook provides sample code for building an empty OpenSearch Serverless (OSS) index,Amazon Bedrock Knowledge Bases and ingest documents into the index from various data sources (S3, Confluence, Sharepoint, Salesforce, and Web). Please note that you can add upto 5 data sources.\n",
"\n",
"\n",
"#### Notebook Walkthrough\n",
Expand Down Expand Up @@ -62,7 +62,7 @@
},
"outputs": [],
"source": [
"# %pip install --force-reinstall -q -r ../requirements.txt"
"%pip install --force-reinstall -q -r ../requirements.txt"
]
},
{
Expand Down Expand Up @@ -93,7 +93,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
Expand Down Expand Up @@ -328,7 +328,7 @@
},
{
"cell_type": "code",
"execution_count": 19,
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
Expand Down Expand Up @@ -421,7 +421,7 @@
"outputs": [],
"source": [
"# delete role and policies\n",
"print(\"===============================Knowledge base with fixed chunking==============================\\n\")\n",
"print(\"===============================Deleting Knowledge Base and associated resources==============================\\n\")\n",
"knowledge_base.delete_kb(delete_s3_bucket=True, delete_iam_roles_and_policies=True)"
]
}
Expand Down
4 changes: 2 additions & 2 deletions ...ases/features-examples/01-rag-concepts/02_managed_retreiveandgenerate_and_streamapi.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,10 +47,10 @@
},
"source": [
"## Pre-requisites\n",
"Before being able to answer the questions, the documents must be processed and stored in a knowledge base. For this notebook, we use a `synthetic dataset for 10K financial reports` to create the Knowledge Bases for Amazon Bedrock. \n",
"Before being able to answer the questions, the documents must be processed and stored in a knowledge base. For this notebook, we use a `synthetic dataset for 10K financial reports` to create the Amazon Bedrock Knowledge Bases. \n",
"\n",
"1. Upload your documents (data source) to Amazon S3 bucket.\n",
"2. Knowledge Bases for Amazon Bedrock using [01_create_ingest_documents_test_kb_multi_ds.ipynb](/knowledge-bases/01-rag-concepts/01_create_ingest_documents_test_kb_multi_ds.ipynb)\n",
"2. Amazon Bedrock Knowledge Bases using [01_create_ingest_documents_test_kb_multi_ds.ipynb](/knowledge-bases/01-rag-concepts/01_create_ingest_documents_test_kb_multi_ds.ipynb)\n",
"3. Note the Knowledge Base ID\n"
]
},
Expand Down
18 changes: 9 additions & 9 deletions ...res-examples/01-rag-concepts/03_customized_rag_retreive_api_hybrid_search_langchain.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Building Q&A application using Knowledge Bases for Amazon Bedrock - Retrieve API"
"## Building Q&A application using Amazon Bedrock Knowledge Bases - Retrieve API"
]
},
{
Expand All @@ -13,7 +13,7 @@
"source": [
"### Context\n",
"\n",
"In this notebook, we will dive deep into building Q&A application using Knowledge Bases for Amazon Bedrock - Retrieve API. Here, we will query the knowledge base to get the desired number of document chunks based on similarity search. We will then augment the prompt with relevant documents and query which will go as input to Anthropic Claude V2 for generating response.\n",
"In this notebook, we will dive deep into building Q&A application using Amazon Bedrock Knowledge Bases - Retrieve API. Here, we will query the knowledge base to get the desired number of document chunks based on similarity search. We will then augment the prompt with relevant documents and query which will go as input to Anthropic Claude V2 for generating response.\n",
"\n",
"With a knowledge base, you can securely connect foundation models (FMs) in Amazon Bedrock to your company\n",
"data for Retrieval Augmented Generation (RAG). Access to additional data helps the model generate more relevant,\n",
Expand All @@ -29,10 +29,10 @@
"\n",
"### Pre-requisite\n",
"\n",
"Before being able to answer the questions, the documents must be processed and stored in a knowledge base. For this notebook, we use a `synthetic dataset for 10K financial reports` to create the Knowledge Bases for Amazon Bedrock. \n",
"Before being able to answer the questions, the documents must be processed and stored in a knowledge base. For this notebook, we use a `synthetic dataset for 10K financial reports` to create the Amazon Bedrock Knowledge Bases. \n",
"\n",
"1. Upload your documents (data source) to Amazon S3 bucket.\n",
"2. Knowledge Bases for Amazon Bedrock using [01_create_ingest_documents_test_kb_multi_ds.ipynb](/knowledge-bases/01-rag-concepts/01_create_ingest_documents_test_kb_multi_ds.ipynb)\n",
"2. Amazon Bedrock Knowledge Bases using [01_create_ingest_documents_test_kb_multi_ds.ipynb](/knowledge-bases/01-rag-concepts/01_create_ingest_documents_test_kb_multi_ds.ipynb)\n",
"3. Note the Knowledge Base ID\n",
"\n",
"\n",
Expand All @@ -44,7 +44,7 @@
"\n",
"\n",
"\n",
"For our notebook we will use the `Retreive API` provided by Knowledge Bases for Amazon Bedrock which converts user queries into\n",
"For our notebook we will use the `Retreive API` provided by Amazon Bedrock Knowledge Bases which converts user queries into\n",
"embeddings, searches the knowledge base, and returns the relevant results, giving you more control to build custom\n",
"workflows on top of the semantic search results. The output of the `Retrieve API` includes the the `retrieved text chunks`, the `location type` and `URI` of the source data, as well as the relevance `scores` of the retrievals. \n",
"\n",
Expand All @@ -56,7 +56,7 @@
"\n",
"#### Dataset\n",
"\n",
"In this example, you will use Octank's financial 10k reports (sythetically generated dataset) as a text corpus to perform Q&A on. This data is already ingested into the Knowledge Bases for Amazon Bedrock. You will need the `knowledge base id` to run this example.\n",
"In this example, you will use Octank's financial 10k reports (sythetically generated dataset) as a text corpus to perform Q&A on. This data is already ingested into the Amazon Bedrock Knowledge Bases. You will need the `knowledge base id` to run this example.\n",
"In your specific use case, you can sync different files for different domain topics and query this notebook in the same manner to evaluate model responses using the retrieve API from knowledge bases.\n",
"\n",
"\n",
Expand Down Expand Up @@ -151,7 +151,7 @@
"source": [
"### Part 1 - Retrieve API with foundation models from Amazon Bedrock\n",
"\n",
"Define a retrieve function that calls the `Retreive API` provided by Knowledge Bases for Amazon Bedrock which converts user queries into\n",
"Define a retrieve function that calls the `Retreive API` provided by Amazon Bedrock Knowledge Bases which converts user queries into\n",
"embeddings, searches the knowledge base, and returns the relevant results, giving you more control to build custom\n",
"workflows on top of the semantic search results. The output of the `Retrieve API` includes the the `retrieved text chunks`, the `location type` and `URI` of the source data, as well as the relevance `scores` of the retrievals. You can also use the `overrideSearchType` option in `retrievalConfiguration` which offers the choice to use either `HYBRID` or `SEMANTIC`. By default, it will select the right strategy for you to give you most relevant results, and if you want to override the default option to use either hybrid or semantic search, you can set the value to `HYBRID/SEMANTIC`.\n",
"\n",
Expand Down Expand Up @@ -328,7 +328,7 @@
"metadata": {},
"source": [
"## Part 2 - LangChain integration\n",
"In this notebook, we will dive deep into building Q&A application using Retrieve API provided by Knowledge Bases for Amazon Bedrock and LangChain. We will query the knowledge base to get the desired number of document chunks based on similarity search, integrate it with LangChain retriever and use `Anthropic Claude 3 Sonnet` model for answering questions."
"In this notebook, we will dive deep into building Q&A application using Retrieve API provided by Amazon Bedrock Knowledge Bases and LangChain. We will query the knowledge base to get the desired number of document chunks based on similarity search, integrate it with LangChain retriever and use `Anthropic Claude 3 Sonnet` model for answering questions."
]
},
{
Expand All @@ -350,7 +350,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Create a `AmazonKnowledgeBasesRetriever` object from LangChain which will call the `Retreive API` provided by Knowledge Bases for Amazon Bedrock which converts user queries into embeddings, searches the knowledge base, and returns the relevant results, giving you more control to build custom workflows on top of the semantic search results. The output of the `Retrieve API` includes the the `retrieved text chunks`, the `location type` and `URI` of the source data, as well as the relevance `scores` of the retrievals."
"Create a `AmazonKnowledgeBasesRetriever` object from LangChain which will call the `Retreive API` provided by Amazon Bedrock Knowledge Bases which converts user queries into embeddings, searches the knowledge base, and returns the relevant results, giving you more control to build custom workflows on top of the semantic search results. The output of the `Retrieve API` includes the the `retrieved text chunks`, the `location type` and `URI` of the source data, as well as the relevance `scores` of the retrievals."
]
},
{
Expand Down
16 changes: 8 additions & 8 deletions ...-examples/01-rag-concepts/04_customized_rag_retreive_api_langchain_evaluation_ragas.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Building and evaluating Q&A Application using Knowledge Bases for Amazon Bedrock using RAG Assessment (RAGAS) framework"
"# Building and evaluating Q&A Application using Amazon Bedrock Knowledge Bases using RAG Assessment (RAGAS) framework"
]
},
{
Expand All @@ -13,9 +13,9 @@
"source": [
"### Context\n",
"\n",
"In this notebook, we will dive deep into building Q&A application using Retrieve API provide by Knowledge Bases for Amazon Bedrock, along with LangChain and RAGAS for evaluating the responses. Here, we will query the knowledge base to get the desired number of document chunks based on similarity search, prompt the query using Anthropic Claude, and then evaluate the responses effectively using evaluation metrics, such as faithfulness, answer_relevancy, context_recall, context_precision, context_entity_recall, answer_similarity, answer_correctness, harmfulness, maliciousness, coherence, correctness and conciseness.\n",
"In this notebook, we will dive deep into building Q&A application using Retrieve API provide by Amazon Bedrock Knowledge Bases, along with LangChain and RAGAS for evaluating the responses. Here, we will query the knowledge base to get the desired number of document chunks based on similarity search, prompt the query using Anthropic Claude, and then evaluate the responses effectively using evaluation metrics, such as faithfulness, answer_relevancy, context_recall, context_precision, context_entity_recall, answer_similarity, answer_correctness, harmfulness, maliciousness, coherence, correctness and conciseness.\n",
"\n",
"### Knowledge Bases for Amazon Bedrock Introduction\n",
"### Amazon Bedrock Knowledge Bases Introduction\n",
"\n",
"With knowledge bases, you can securely connect foundation models (FMs) in Amazon Bedrock to your company\n",
"data for Retrieval Augmented Generation (RAG). Access to additional data helps the model generate more relevant,\n",
Expand All @@ -28,10 +28,10 @@
"\n",
"### Pre-requisite\n",
"\n",
"Before being able to answer the questions, the documents must be processed and stored in a knowledge base. For this notebook, we use a `synthetic dataset for 10K financial reports` to create the Knowledge Bases for Amazon Bedrock. \n",
"Before being able to answer the questions, the documents must be processed and stored in a knowledge base. For this notebook, we use a `synthetic dataset for 10K financial reports` to create the Amazon Bedrock Knowledge Bases. \n",
"\n",
"1. Upload your documents (data source) to Amazon S3 bucket.\n",
"2. Knowledge Bases for Amazon Bedrock using [01_create_ingest_documents_test_kb_multi_ds.ipynb](/knowledge-bases/01-rag-concepts/01_create_ingest_documents_test_kb_multi_ds.ipynb)\n",
"2. Amazon Bedrock Knowledge Bases using [01_create_ingest_documents_test_kb_multi_ds.ipynb](/knowledge-bases/01-rag-concepts/01_create_ingest_documents_test_kb_multi_ds.ipynb)\n",
"3. Note the Knowledge Base ID\n",
"\n",
"<!-- ![data_ingestion.png](./images/data_ingestion.png) -->\n",
Expand All @@ -42,7 +42,7 @@
"\n",
"\n",
"\n",
"For our notebook we will use the `Retreive API` provided by Knowledge Bases for Amazon Bedrock which converts user queries into\n",
"For our notebook we will use the `Retreive API` provided by Amazon Bedrock Knowledge Bases which converts user queries into\n",
"embeddings, searches the knowledge base, and returns the relevant results, giving you more control to build custom\n",
"workflows on top of the semantic search results. The output of the `Retrieve API` includes the the `retrieved text chunks`, the `location type` and `URI` of the source data, as well as the relevance `scores` of the retrievals. \n",
"\n",
Expand Down Expand Up @@ -119,7 +119,7 @@
"source": [
"### Follow the steps below to set up necessary packages\n",
"\n",
"1. Import the necessary libraries for creating `bedrock-runtime` for invoking foundation models and `bedrock-agent-runtime` client for using Retrieve API provided by Knowledge Bases for Amazon Bedrock. \n",
"1. Import the necessary libraries for creating `bedrock-runtime` for invoking foundation models and `bedrock-agent-runtime` client for using Retrieve API provided by Amazon Bedrock Knowledge Bases. \n",
"2. Import Langchain for: \n",
" 1. Initializing bedrock model `anthropic.claude-3-haiku-20240307-v1:0` as our large language model to perform query completions using the RAG pattern. \n",
" 2. Initializing bedrock model `anthropic.claude-3-sonnet-20240229-v1:0` as our large language model to perform RAG evaluation. \n",
Expand Down Expand Up @@ -174,7 +174,7 @@
"source": [
"### Retrieve API: Process flow \n",
"\n",
"Create a `AmazonKnowledgeBasesRetriever` object from LangChain which will call the `Retreive API` provided by Knowledge Bases for Amazon Bedrock which converts user queries into\n",
"Create a `AmazonKnowledgeBasesRetriever` object from LangChain which will call the `Retreive API` provided by Amazon Bedrock Knowledge Bases which converts user queries into\n",
"embeddings, searches the knowledge base, and returns the relevant results, giving you more control to build custom\n",
"workflows on top of the semantic search results. The output of the `Retrieve API` includes the the `retrieved text chunks`, the `location type` and `URI` of the source data, as well as the relevance `scores` of the retrievals. "
]
Expand Down
4 changes: 2 additions & 2 deletions rag/knowledge-bases/features-examples/01-rag-concepts/05_document_level_kb_ingestion.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
"metadata": {},
"outputs": [],
"source": [
"%pip install --force-reinstall -q -r ../../requirements.txt --quiet\n",
"%pip install --force-reinstall -q -r ../requirements.txt --quiet\n",
"%pip install --upgrade boto3"
]
},
Expand Down Expand Up @@ -1099,7 +1099,7 @@
],
"instance_type": "ml.t3.medium",
"kernelspec": {
"display_name": "Python 3",
"display_name": "bedrock_env",
"language": "python",
"name": "python3"
},
Expand Down
4 changes: 2 additions & 2 deletions rag/knowledge-bases/features-examples/01a-kb-rag-evaluation/01_qna_generation_from_pdf.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
"metadata": {},
"outputs": [],
"source": [
"%pip install --force-reinstall -q -r ../../requirements.txt --quiet"
"%pip install --force-reinstall -q -r ../requirements.txt --quiet"
]
},
{
Expand Down Expand Up @@ -1150,7 +1150,7 @@
],
"instance_type": "ml.t3.medium",
"kernelspec": {
"display_name": "Python 3",
"display_name": "bedrock_env",
"language": "python",
"name": "python3"
},
Expand Down
Loading