seems OTT, maybe inspect the changes it’s going to make more often?
Agent? It makes them w/out notice, once it’s done, it’s done - or you must rollback.
You must be talking about Assistant.
Also, I always check all the code commits IF POSSIBLE. This is why I prefer assistant.
To this point - similar to Eric, I just say things like “Please make the code more efficient and refactor without losing any core functionality. Ensure that we’re operating coding best practices but make sure to test so that we have identical style and full functionality before making major changes.”
That combined with (A) rollbacks (B) a staging server that Agent developed so nothing ever breaks on Production and (C) a habit of incremental prompting such that no prompt becomes too complex and prone to error means that I rarely have major issues.
agreed, this has been a hair-on-fire problem for me. Would love if there was a way to mitigate this.
I asked my ai genetic code expert GEM the question. Here is the output. You’re facing a common challenge with AI agents in development environments like Replit: unpredictable code modifications. Here’s a breakdown of strategies to gain more control and mitigate unwanted changes:
- Version Control is Your Best Friend:
- Git (Essential): Absolutely crucial. Commit frequently. Even small changes should be tracked. This allows you to revert to previous, working states easily. Replit integrates well with Git. Use it religiously.
- Branches (Recommended): Don’t work directly on your main branch. Create branches for new features, bug fixes, or agent experiments. This isolates changes and prevents them from impacting your stable code.
- Sandboxing and Isolation:
- Separate Replit Projects: For major experiments or if you’re very concerned about the agent going rogue, create a separate Replit project specifically for agent interaction. This prevents it from accidentally modifying your primary codebase.
- Containers (Advanced, but Powerful): If Replit supports it (some plans do), or if you move to a more robust hosting environment, Docker containers are excellent for isolation. You can define the exact environment the agent operates in, limiting its access to specific files and directories.
- Code Structure and Organization:
- Modular Design: Break your code into smaller, independent modules or functions. This makes it easier to understand what each part does and reduces the risk of unintended side effects from agent modifications.
- Clear Interfaces: Define clear interfaces (APIs) between modules. This provides a controlled way for the agent to interact with your code.
- Configuration Files: Store important parameters and settings in configuration files. This makes it easier to manage and modify settings without changing the core code.
- Agent Interaction Control:
- Limited Permissions: If possible, configure your agent to have only the necessary permissions. Don’t give it full access to your entire codebase. Restrict its ability to write files or execute certain commands.
- Input Validation: Carefully validate any input the agent provides before using it. This prevents the agent from injecting malicious code or making unexpected changes.
- Monitoring and Logging: Implement thorough logging to track the agent’s actions. This helps you understand what the agent is doing and identify any problematic behavior.
- Human-in-the-Loop: Consider a human-in-the-loop approach. The agent proposes changes, but a human reviews and approves them before they are applied. This gives you ultimate control.
- Prompt Engineering: Experiment with different prompts to guide the agent’s behavior. Be very specific about what you want the agent to do and what you don’t want it to do. Reinforce good behavior with positive feedback.
- Replit-Specific Tips:
- .replit file: Use the .replit configuration file to customize your Replit environment, including setting up run commands and specifying dependencies.
- Packager: If you’re using a language with a package manager (e.g., npm for Node.js, pip for Python), use it to manage your project’s dependencies. This helps ensure consistency and makes it easier to reproduce your environment.
Example (Conceptual - Python):
main.py (Your core application)
def process_data(data):
… your core logic …
return result
def safe_apply_agent_modification(data, agent_code):
try:
# Validate agent_code (crucial!) - e.g., check for dangerous functions
# … validation logic …
# Execute agent_code in a sandboxed environment (if possible)
# ... sandboxing approach ...
modified_data = agent_code(data) # Agent's code acts on the data
return process_data(modified_data) # Process with your core logic
except Exception as e:
print(f"Agent error: {e}")
return data # Return original data on error
agent_interaction.py (How you interact with the agent)
data = {“value”: 10}
agent_code = lambda x: {“value”: x[“value”] * 2} # Example agent code
result = safe_apply_agent_modification(data, agent_code)
print(result)
By combining these strategies, you’ll gain much more control over how your AI agent interacts with your Replit projects, reducing the risk of unwanted changes and making your development process more predictable. Remember to start with version control (Git) as your foundation.
I create an “instructions for agent.md” file in every project I create and keep a running list of Do’s & Don’ts for the agent, including what are “protected files & processes”. I then tell the agent every time to reference the .md file of instructions before beginning. In that file, I also tell it explicitly that it is in read-only mode until I give it explicit instructions to proceed.
What is your honest take on how well this is respected?
Just to lob a grenade into the mix you should try to get to a point where you stop using the agent except to automate simple stuff that you yourself can edit later if need be.
Also, you’ll be hearing this a lot out of me and I’m not super familiar with react JavaScript and html yet. The LLM, agent or AI is not there yet. At the very least we should be taking the approach where we manually are learning the ins and outs and ultimately stop relying on the agent or are able to use it for speed and you run through after each document it writes, have it stop after each one and manually change, inspect and push through each block of code to make sure the structure and design is correct. Speaking as someone who openly used LLMs to make step by step interactive guides for their computer science class Java labs and projects and got the grades back for it… it’s about 80% for a beginner coder when it comes to the more advanced topics it’s just not there yet… it hallucinates and you’ll spend more time and money.
You might want to spend the money on using the agent to do as I mentioned above as a sort of “pair programming” and use this as an opportunity to learn the languages. Also, look around for a tutors, bootcamps, or sites, in the languages you’re actively using. Share small blocks or snippets of your code.
If truly worried ask if they’d sign an NDA, get a boiler plate off the internet for your area and let that be at least something (I suggest you get a professional to write you one that you can plug details in and use for multiple projects) then use all of the sources you have to spend the next few years making it MVP ready.
We can deploy and play around all day long for the sake of education. Tbh this is a better, more interactive hands on approach to codecadamy or alongside it because you get to see a wider view of the whole chabang.
The agent will only be able to take you so far until they manage to improve its coding capabilities. Until then it’s proven and known that the LLM’s just can’t quite handle more complex and coding errors. It will lie to you all day long and say it does and spin you in circles.
That’s what they are talking about when they say it’s “hallucinating” it’s not out in the woods tripping on acid while some other old hippie catches him tripping on acid. SOHCAHTOA
it’s just blowing steam up your pipes to see what’s under your skirt per sei lol
If you really want to make these ready to deploy, baked and ready to ship out of the box when the customers open them. Get ready to spend years, and years and when you thought you were done everything’s changed and now you have to start again… it’s like this seriously. This is why Mark Zuckerburger and Elon Tusk and Bill the Gates Keeper all had deep pockets and generational wealth helping them. Its becoming a lot more sustainable and easier for us to be able to break through these systemic barriers that plague software engineering, however we are far away from that reality… sadly, even the professors in the industry are superficial gate keepers who are terrified of LLMs replacing them rather than leaning into them and using them as the dank tool they are.
Edit: I was told right off the bat. Software and most companies have a lifespan of 20-30 years. Most successful apps are weird niche apps (like this even though it’s honestly a trivial concept and idea Replit went boom). Plan for the day you’re no longer relevant anymore otherwise you might find yourself being old washed up, and unable to communicate like a decent human being. No matter how successful you were, or how much money the apps make you. Please remember these days. The beginning days. The days that you felt so much vigor and passion. It might help you from becoming a monster lol
Oh, that’s handy! This setup would help create, for example, an admin dashboard and an app side by side, separating concerns and then use GitHub Actions and Bash to spin up, say Azure resources?
Depending on how complex rich your app
Is! I’ve been experimenting which might help: I’ve started creating a little .json “plan” for each feature before the agent starts work. It includes the expected files, components, or fields that should change. Then I run a small script that checks if only those things were touched, kind of like giving the agent a checklist and guardrails. Maybe this could help with your issues with naming commits correctly!? You can also run all this from Replits bash shell.