Which is low/medium/high/max about?
- Autonomy over what the agent is allowed to do
- Which LLM it uses and therefore how intelligent it is
I believe it is 1. Reading some comment here I think people are (wrongly) assuming it is 2.
I posted on another thread and it looks like i’m getting some progress, I’m still having issues but I can worth through issues as I go, at least its doing something today.
1 Like
I have graduated to high mode and joined the dark side @Gipity-Steve 
Of course I am not implementing features here, I am using it for highly targeted bug fixes… the run times are surprisingly quick, and I don’t mind paying the few extra dollars for a high quality bug fix.
1 Like
Medium is running me ~$3-$7/prompt. So far, significantly less time spent than Agent 2. High has been $7-$20/prompt, and seems to act like me going down a Reddit rabbit hole. It does seem to fix things wrong with the app, but highly unrelated to my request.
I’ll probably be using High in the same way @Marshal_Thompson is saying, targeting bugs and cleaning up code.
1 Like
I have been testing different modes with Agent, so far the low/medium ones have been most effective on existing codebase.
The autonomy level switches seem to be a bit glued on solution. They appear to be mostly different system instruction reminders for modes.
The low setting is constraining agent for focusing on the issue/task and to avoid creating any bigger changes but it will also the drive agent to take shortcuts.
For example it created a task list for itself about adding backend functionality but later decided to just add a notification toast that feature is disabled because one route didnt exist.
What disturbs me the most is the “variable costs” for the task, one simple task could take 0.40$ and another same level task could be 0.80$ and user has no idea what caused the difference. (Unless agent clearly struggled and debugged a lot)
I would want to see the autonomy levels to actually affect the reasoning/effort levels too. So low/medium modes should be kind of workhorse versions that you can task for work without thinking what will the cost be this time.
I think the agent uses too much unnecessary tools and searches especially in the low/medium modes, I’ve been testing it by prohibiting some tools in prompts and it appears to lower the costs.
It would be nice if we had more transparency what the “General work” is for agent to justify the variable costs. I took some numbers from different tasks:
Total Cost: $3.76
Total Actions: 79
Total Lines Read: 2116
Lines Added: 901
Lines Removed: 1
Calculated Prices:
Per Action Price: $0.0476
Per Line Price: $0.0018
Per Code Line Changed Price: $0.0042
---
Total Cost: $0.75
Total Actions: 10
Total Lines Read: 278
Lines Added: 89
Lines Removed: 56
Calculated Prices:
Per Action Price: $0.0750
Per Line Price: $0.0027
Per Code Line Changed Price: $0.0052
---
Total Cost: $6.31
Total Actions: 185
Total Lines Read: 6458
Lines Added: 1241
Lines Removed: 733
Calculated Prices:
Per Action Price: $0.0341
Per Line Price: $0.0010
Per Code Line Changed Price: $0.0032
---
Total Cost: $1.50
Total Actions: 38
Total Lines Read: 519
Lines Added: 1861
Lines Removed: 50
Calculated Prices:
Per Action Price: $0.0395
Per Line Price: $0.0029
Per Code Line Changed Price: $0.0008
I also tested to create a new app (Start with design option) and noticed that the agent is much more cost aware during the initial build, there are constant reminders in its thoughts about cost-effectiveness and parallel tool usage. Few examples:
The system reminder also emphasizes that I MUST execute all independent operations in parallel within a single function_calls block. Let me create the remaining components in parallel and then fix any issues.
Let me create multiple components at once to optimize efficiency as instructed.
But you don’t see such behaviour in the agent after that.
1 Like