feature(platform) Smart Decision Maker Block #9490

Swiftyos · 2025-02-17T10:29:22Z

Task

The SmartDecisionMakerBlock is a specialized block in a graph-based system that leverages a language model (LLM) to make intelligent decisions about which tools or functions to invoke based on a user-provided prompt. It is designed to process input data, interact with a language model, and dynamically determine the appropriate tools to call from a set of available options, making it a powerful component for AI-driven workflows.

How It Works in Practice

Scenario: Imagine a workflow where a user inputs, "Send an email to John about the meeting." The SmartDecisionMakerBlock is connected to tools like send_email, schedule_meeting, and search_contacts.
Execution:
1. The block receives the prompt and system instructions (e.g., "Choose a function to call").
  2.It identifies the available tools from the graph and constructs their signatures (e.g., send_email(recipient, subject, body)).
2. The LLM analyzes the prompt and decides to call send_email with arguments like recipient: "John", subject: "Meeting", body: "Let’s discuss...".
3. The block yields these tool-specific outputs, which can be picked up by downstream nodes to execute the email-sending action.

Changes 🏗️

Add the Smart Decision Maker (SDM) block.
Break circular imports in integration code.

Work in Progress

⚠️ Important note this is a temporary UX for the system - UX will be addressed in a future PR ⚠️

Current Status

I’m currently focused on the smart decision logic. The main additions in the ongoing PR include:

Defining function signatures for OpenAI function-calling schemas based on node links and the linked blocks.
Adding tests for function signature generation.
Force all tool calls to be made via an agent. (Need to uncomment)
Restrict each tool call entry to a single node.
simplify the output emission process, to emit each parameter one at a time.
Change test to use agents and hardcode output how I think it should work to test it does actually work
Hook up openai, in a simplified way, to test the function calling (mock for testing)
Once all the above is working, use credentials system and build of llm.py

What’s Next

Review Process

Reviewers Phase 1

This PR is now ready for review, during the first phase of reviews I'm looking for comments on approach and logic.

Out of scope: code style and organization at this stage

Reviewers Phase 2

Once we are all happy with the approach and logic. We can open the review process to general code quality and nits, to be considered.

supabase · 2025-02-17T10:29:30Z

This pull request has been ignored for the connected project bgwpwdsxblryihinutbx because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

github-actions · 2025-02-17T10:29:35Z

This PR targets the master branch but does not come from dev or a hotfix/* branch.

Automatically setting the base branch to dev.

netlify · 2025-02-17T10:30:07Z

✅ Deploy Preview for auto-gpt-docs canceled.

Name	Link
🔨 Latest commit	`267b96d`
🔍 Latest deploy log	https://app.netlify.com/sites/auto-gpt-docs/deploys/67b8a1a43c576d000875bd31

deepsource-io · 2025-02-17T10:30:07Z

Here's the code health analysis summary for commits a692eed..267b96d. View details on DeepSource ↗.

Analysis Summary

Analyzer	Status	Summary	Link
JavaScript	✅ Success	❗ 2 occurences introduced 🎯 2 occurences resolved	View Check ↗
Python	✅ Success	❗ 12 occurences introduced 🎯 2 occurences resolved	View Check ↗

💡 If you’re a repository administrator, you can configure the quality gates from the settings.

autogpt_platform/backend/backend/blocks/smart_decision_maker.py

netlify · 2025-02-17T13:29:13Z

✅ Deploy Preview for auto-gpt-docs-dev canceled.

Name	Link
🔨 Latest commit	`267b96d`
🔍 Latest deploy log	https://app.netlify.com/sites/auto-gpt-docs-dev/deploys/67b8a1a45bd5df000883b0f9

ntindle · 2025-02-17T15:45:14Z

autogpt_platform/backend/backend/blocks/smart_decision_maker.py

+        if not graph:
+            raise ValueError(f"Graph not found {graph_id}")
+
+        tool_functions = self._create_function_signature(node_id, graph)


Test this works with mem0

AutoGPT/autogpt_platform/backend/backend/blocks/mem0.py

Line 121 in 016ec0f

) -> BlockOutput:

since it uses the kwargs based on its place in the graph

@ntindle It seems that high level decision is that tools must be an AgentExecutionBlock.

If you feel strongly that this is a bad idea, let me know and we can discuss it with Toran

Why should it require that? I can see plenty of use for this not needing a whole agent and the overhead that would come with it.

The main reason why is to make a smart decision, a certain amount of information is required about a tool. We can backprogate this information from an agent - name, description, required params, etc.

But with a normal block, we can't really do this unless we assume that block can't be connected to another block...

There are a fair bit of subtleties with this system

…l-discovery-input-mapping-for-smart-decision-maker-block

github-actions · 2025-02-20T12:40:53Z

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

…l-discovery-input-mapping-for-smart-decision-maker-block

github-actions · 2025-02-20T13:57:46Z

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

Swiftyos · 2025-02-21T09:33:01Z

@Pwuts tests are failing due to an issue with the credentials system that does not occur locally. Can you advise on how to fix this please

qodo-merge-pro · 2025-02-21T09:33:40Z

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 Security concerns API Key Exposure: The error handling in llm_call() function includes API keys in error messages (e.g., "Anthropic API error: {str(e)}"). This could lead to sensitive API keys being exposed in logs or error responses. The error messages should be sanitized to remove any sensitive information before logging or returning to users.
⚡ Recommended focus areas for review Error Handling The LLM response handling in the SmartDecisionMakerBlock needs validation to ensure proper error handling when the LLM returns invalid or unexpected responses. credentials=credentials, llm_model=input_data.model, prompt=prompt, json_format=False, max_tokens=input_data.max_tokens, tools=tool_functions, ollama_host=input_data.ollama_host, ) if not response.tool_calls: yield "finished", f"No Decision Made finishing task: {response.response}" Security Risk The llm_call function exposes API keys in error messages which could lead to sensitive information disclosure in logs. except anthropic.APIError as e: error_message = f"Anthropic API error: {str(e)}" logger.error(error_message) raise ValueError(error_message) Input Validation The validate_graph method needs to verify that tool names are unique across the entire graph to prevent naming conflicts. tool_name = link.source_name.split("_#_")[0] if tool_name in tool_name_to_node: if tool_name_to_node[tool_name] != link.sink_id: raise ValueError( f"Tool name {tool_name} links to multiple nodes: {tool_name_to_node[tool_name]} and {link.sink_id}" ) else: tool_name_to_node[tool_name] = link.sink_id

majdyz

Are these true:

SMD can't be connected to non-AgentExecutorBlock, if so, why not? Would it calling non-agent block would be simpler, e.g: We have a block definition, type, description, etc, plus it's a more common pattern.
the tools output pin will connected to N other input pins, and then on each yield, N input will be produced, but N-1 of it will raise an exception due to the validation failure since it doesn't match the tool that the SMD is using. Is this the flow we are expecting ?

majdyz · 2025-02-21T14:16:36Z

autogpt_platform/backend/backend/blocks/llm.py

+    prompt: list[dict],
+    json_format: bool,
+    max_tokens: int | None,
+    tools: list[dict] | None = None,


What is tools here how do I use it? can we add it to the docstring too?

majdyz · 2025-02-21T14:18:29Z

autogpt_platform/backend/backend/blocks/llm.py

+    completion_tokens: int
+
+
+def llm_call(


This is copied function, can we remove the old one and make the function caller use this?

I really want too, but I know how much people hate me making bigger PRs. So I was planning on refactoring llm.py in a separate PR if thats okay.

Okay sounds good, though if the change is just removing self. in self.llm_call and removing the llm_call function I definitely don't mind doing it in a single PR. Both are fine for me 👍

majdyz · 2025-02-21T14:20:55Z

autogpt_platform/backend/backend/blocks/smart_decision_maker.py

+        tool_links = {
+            link.sink_id
+            for link in graph.links
+            if link.source_name.startswith("tools_") and link.source_id == node_id


Can we make tools_ constant, and this is so prone to conflict, can we use a special character?

Sure good idea

autogpt_platform/backend/backend/data/graph.py

majdyz · 2025-02-21T14:26:26Z

autogpt_platform/backend/backend/data/graph.py

        for link in self.links:
            input_links[link.sink_id].append(link)

+            if (


Add comments here?

Swiftyos added 2 commits February 17, 2025 11:01

Add generation of function signitures

643cb11

break circular import

90cf771

github-actions bot changed the base branch from master to dev February 17, 2025 10:29

github-actions bot added platform/backend AutoGPT Platform - Back end platform/blocks size/l labels Feb 17, 2025

Swiftyos changed the title ~~Swiftyos/open 2373 tool discovery input mapping for smart decision maker block~~ feature(platform) Smart Decision Maker Block Feb 17, 2025

majdyz reviewed Feb 17, 2025

View reviewed changes

autogpt_platform/backend/backend/blocks/smart_decision_maker.py Outdated Show resolved Hide resolved

add docs and tests for parse_execution_data and merge_execution_inputs

f0b98eb

Added a mechanism for linking specific tool params to an input node

59c6825

ntindle reviewed Feb 17, 2025

View reviewed changes

Swiftyos added 3 commits February 18, 2025 10:08

simplify output parsing

f7f3534

Updated system to work using AgentExecutionBlocks

946e3b1

Merge remote-tracking branch 'origin/dev' into swiftyos/open-2373-too…

d9778be

…l-discovery-input-mapping-for-smart-decision-maker-block

github-actions bot added size/xl and removed size/l labels Feb 19, 2025

Swiftyos added 5 commits February 19, 2025 12:25

fixed tests

cba8cfe

working conceptual execution of smart decision making block

671faf1

moving towards full llm call setup

f08b66c

added credentails to sdm

6de9e21

remove test that wont work without real credentials

f5e8e0a

github-actions bot added the conflicts Automatically applied to PRs with merge conflicts label Feb 20, 2025

Merge remote-tracking branch 'origin/dev' into swiftyos/open-2373-too…

f029049

…l-discovery-input-mapping-for-smart-decision-maker-block

github-actions bot removed the conflicts Automatically applied to PRs with merge conflicts label Feb 20, 2025

handle dynamic link naming

6a134a7

github-actions bot added the platform/frontend AutoGPT Platform - Front end label Feb 21, 2025

Swiftyos added 2 commits February 21, 2025 10:20

remove debug messages

b9e0b95

fixed case in test

a23cc38

Swiftyos marked this pull request as ready for review February 21, 2025 09:33

Swiftyos requested a review from a team as a code owner February 21, 2025 09:33

Swiftyos requested review from Pwuts and majdyz and removed request for a team February 21, 2025 09:33

qodo-merge-pro bot added the Review effort 4/5 label Feb 21, 2025

Swiftyos requested a review from ntindle February 21, 2025 09:35

majdyz requested changes Feb 21, 2025

View reviewed changes

Avoid hardcoded ID

267b96d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature(platform) Smart Decision Maker Block #9490

feature(platform) Smart Decision Maker Block #9490

Swiftyos commented Feb 17, 2025 •

edited

Loading

supabase bot commented Feb 17, 2025

github-actions bot commented Feb 17, 2025

netlify bot commented Feb 17, 2025 •

edited

Loading

deepsource-io bot commented Feb 17, 2025 •

edited

Loading

Analysis Summary

netlify bot commented Feb 17, 2025 •

edited

Loading

ntindle Feb 17, 2025

Swiftyos Feb 19, 2025

ntindle Feb 19, 2025

Swiftyos Feb 20, 2025

github-actions bot commented Feb 20, 2025

github-actions bot commented Feb 20, 2025

Swiftyos commented Feb 21, 2025

qodo-merge-pro bot commented Feb 21, 2025

majdyz left a comment

majdyz Feb 21, 2025

majdyz Feb 21, 2025

Swiftyos Feb 21, 2025

majdyz Feb 21, 2025

majdyz Feb 21, 2025

Swiftyos Feb 21, 2025

majdyz Feb 21, 2025

feature(platform) Smart Decision Maker Block #9490

Are you sure you want to change the base?

feature(platform) Smart Decision Maker Block #9490

Conversation

Swiftyos commented Feb 17, 2025 • edited Loading

Task

How It Works in Practice

Changes 🏗️

Work in Progress

Current Status

What’s Next

Reviewers Phase 1

Reviewers Phase 2

supabase bot commented Feb 17, 2025

github-actions bot commented Feb 17, 2025

netlify bot commented Feb 17, 2025 • edited Loading

✅ Deploy Preview for auto-gpt-docs canceled.

deepsource-io bot commented Feb 17, 2025 • edited Loading

Analysis Summary

netlify bot commented Feb 17, 2025 • edited Loading

✅ Deploy Preview for auto-gpt-docs-dev canceled.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 20, 2025

github-actions bot commented Feb 20, 2025

Swiftyos commented Feb 21, 2025

qodo-merge-pro bot commented Feb 21, 2025

PR Reviewer Guide 🔍

majdyz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Swiftyos commented Feb 17, 2025 •

edited

Loading

netlify bot commented Feb 17, 2025 •

edited

Loading

deepsource-io bot commented Feb 17, 2025 •

edited

Loading

netlify bot commented Feb 17, 2025 •

edited

Loading