I’ve been using Replit Agent to build a mobile 3D Bible learning game using Expo and Three.js with procedural geometry. Overall the experience has been incredible — Agent is capable of building surprisingly complex 3D environments through conversation alone.
As the scene becomes more complex, however, one major limitation becomes clear: Agent edits the scene as raw code instead of understanding it as a structured collection of 3D objects.
Because of this, several issues appear when modifying the world:
-
Adding a new object sometimes changes or breaks nearby objects in the code
-
Objects are sometimes added without collision detection, so the player can walk directly through them
-
The AI occasionally loses track of existing objects, causing overlapping placement
-
Maintaining scene hierarchy becomes harder as the world grows
-
New objects may accidentally modify existing meshes or materials
The core issue is that the AI treats the scene like code text rather than a structured 3D world.
A major improvement would be introducing scene-aware editing, where Agent understands objects as separate entities with properties instead of just editing code lines.
For example, Agent could maintain a lightweight scene manifest or scene graph describing the objects in the environment. Something like:
player
type: character
collider: capsule
position: (0,0,0)
fountain
type: environment
collider: mesh
blocking: true
tree_01
type: environment
collider: cylinder
blocking: true
path_tiles
type: terrain
collider: walkable
If Agent could reference a structured scene description like this before making changes, it could:
-
Add objects without modifying unrelated meshes
-
Maintain scene hierarchy and object identities
-
Automatically classify whether objects block player movement
-
Prevent overlapping placements
-
Safely modify only the intended object
Currently my workaround has been creating documentation files that describe every object in the scene, including positions, collision rules, and construction guidelines. Agent does reference these files, and they help significantly, but they require manual upkeep — I have to update them by hand after every change, and Agent can still miss steps even with the documentation in place. If this kind of world memory were a built-in, automatically maintained feature rather than something I build and update manually, it would prevent many of these issues before they happen.
Even a lightweight version — such as an automatically generated JSON scene manifest or a simple visual scene editor that Agent references before editing — could prevent the majority of these problems.
Another improvement that could help large 3D projects would be persistent project memory at a deeper level. As scenes and game systems grow, it becomes harder for the AI to keep track of all mechanics, assets, and rules across many files. Agent could maintain a project knowledge layer that automatically tracks:
-
Existing game systems and mechanics
-
Scene structure and object inventory
-
Asset types and their properties
-
Collision rules and spatial relationships
-
Gameplay rules and progression logic
This would help Agent make safer edits and avoid introducing regressions as projects grow, without requiring the developer to manually maintain that knowledge base.
For context, my project is intended to grow into a fairly large interactive world, so tools that help Agent maintain structural understanding of the scene and systems would make a huge difference as the project scales.
Replit Agent is already one of the most powerful tools I’ve used for building apps. Adding scene awareness and persistent project memory could make it dramatically better for building games and interactive 3D worlds directly inside Replit.
