Over the past few weeks, I’ve been deep in the weeds building a proof-of-concept that uses an LLM agent to autonomously write and execute Python code inside isolated containers. The goal: take raw data in, get a polished, self-contained report out \u2014 fully automated, fully secure.<\/p>\n\n\n\n
It was one of those projects that looks<\/em> straightforward on a whiteboard and then humbles you the moment you open a terminal. Here are the most important things I learned along the way.<\/p>\n\n\n\n The first architectural decision I had to make was: where does the agent live relative to the sandbox?<\/em><\/p>\n\n\n\n The intuitive answer is to put everything inside the container \u2014 agent, code, data, all of it. But this creates a serious problem: your LLM API key ends up inside an isolated container. That expands your attack surface in ways you don’t want.<\/p>\n\n\n\n The better pattern is “Sandbox as Tool”<\/strong>: the agent runs on the host, and the sandbox is just one of several tools it can call. The container only ever receives code to execute \u2014 no secrets, no direct LLM API access, no external network calls. The agent stays on the host where it can be monitored, rate-limited, and controlled.<\/p>\n\n\n\n This feels like a subtle distinction, but it has massive downstream implications for security architecture, observability, and cost management.<\/p>\n\n\n\n I ended up evaluating multiple different sandbox providers. What saved me enormous amounts of time was defining a minimal, common interface upfront:<\/p>\n\n\n\n Every provider implemented those four methods. Switching between them became a one-line config change. The agent code, the prompts, the tools \u2014 all of them remained completely provider-agnostic.<\/p>\n\n\n\n If you’re building anything like this, define your abstraction boundary before<\/em> you start integrating providers. The refactoring cost if you don’t is brutal.<\/p>\n\n\n\n This one surprised me more than anything else. My initial load tests came back with costs roughly 10x higher than they needed to be<\/strong>. The model itself wasn’t the problem. The prompts<\/em> were.<\/p>\n\n\n\n Breaking it down, the waste was almost entirely from:<\/p>\n\n\n\n After rewriting the system prompt to explicitly address the stateless execution model, enforce batching rules, and enabling prompt caching (which gives a significant discount on cached tokens), cost dropped by 89%<\/strong> with no change in output quality.<\/p>\n\n\n\n The lesson: treat your system prompt as production code. It needs the same rigor as the infrastructure around it.<\/p>\n\n\n\n LLMs are trained to be helpful, which sometimes means they fill in gaps with plausible-but-wrong assumptions. In an agentic coding context, this is dangerous.<\/p>\n\n\n\n In my case, the model assumed code state persisted between execution calls \u2014 a completely reasonable assumption in interactive Python (Jupyter-style), but dead wrong in an architecture where each execution call spawns a fresh process. The fix was being brutally explicit in the system prompt:<\/p>\n\n\n\n “CRITICAL: Each execute_code call is a fresh process. No variables, no imports, no state persists between calls.”<\/em><\/p>\n<\/blockquote>\n\n\n\n More broadly: don’t rely on the model inferring constraints from context. If there’s a hard rule about how your tools work, state it explicitly. Ambiguity at the prompt level becomes expensive retries at runtime.<\/p>\n\n\n\n One of the best architectural decisions I made was splitting the output into two distinct layers:<\/p>\n\n\n\n This two-layer approach solved several problems at once:<\/p>\n\n\n\n If you’re using LLMs to generate any kind of structured document, this pattern is worth stealing.<\/p>\n\n\n\n I also compared a classic ReAct agent against a more sophisticated “deep agent” framework designed for open-ended coding tasks \u2014 the kind of thing you’d use for an autonomous coding assistant.<\/p>\n\n\n\n The deep agent had more built-in capabilities: planning middleware, sub-agent orchestration, automatic context management. But for a well-defined pipeline with a clear start and end state, all of that overhead became a liability:<\/p>\n\n\n\n The simpler ReAct agent \u2014 cheaper, faster, more focused \u2014 won on every metric that mattered.<\/p>\n\n\n\n The lesson isn’t that sophisticated agent frameworks are bad. It’s that they’re designed for open-ended tasks where planning and recovery are genuinely needed. If your task is bounded and well-structured, a tightly scoped ReAct loop with a well-crafted prompt will almost always beat a general-purpose agent.<\/p>\n\n\n\n Cold-starting an isolated container takes time. Depending on your infrastructure, that can range from under a second (Firecracker microVMs) to several seconds (Kubernetes pod scheduling + runtime boot).<\/p>\n\n\n\n I explored two strategies to address this:<\/p>\n\n\n\n Both approaches work well. But here’s the important nuance: agent execution dominates total runtime by an order of magnitude<\/strong>. In my benchmarks, sandbox init was measured in single-digit seconds. Agent execution was measured in minutes. Optimizing cold start feels good but moves the needle much less than optimizing the agent itself.<\/p>\n\n\n\n Fix the agent first. Then fix the infrastructure.<\/strong><\/p>\n\n\n\n This one is a pure gotcha for anyone building on container orchestration platforms. Some execution APIs stream data as UTF-8 strings. If you’re downloading binary files \u2014 like PNG charts embedded in HTML \u2014 non-UTF-8 bytes get silently corrupted in transit.<\/p>\n\n\n\n The fix is to base64-encode the output inside the container<\/em> before streaming it, then decode on the client side:<\/p>\n\n\n\n Simple fix, but it’s the kind of thing that only surfaces during load testing with real binary outputs \u2014 not during happy-path local dev runs. Always test file transfers with binary content. Don’t assume that because text files work, everything works.<\/p>\n\n\n\n This seems obvious in retrospect, but it’s worth stating clearly: do not let the agent read raw data into its context window.<\/strong><\/p>\n\n\n\n A large CSV can be tens of megabytes of text. Even if it fit in context, the cost would be absurd and the useful signal-to-noise ratio would be terrible. The correct pattern:<\/p>\n\n\n\n The agent never needs to see the raw data. It needs to understand the structure, write code that processes the data, and reason about the outputs. Those are very different things \u2014 and treating them as different things is what makes the system scalable to any dataset size.<\/p>\n\n\n\n The thing that struck me most about this project is how much the non-ML<\/em> parts of the system matter. The sandbox architecture, the prompt design, the abstraction layers, the output structure \u2014 these are fundamentally software engineering problems, and they have more impact on the system’s cost, reliability, and quality than the choice of model.<\/p>\n\n\n\n LLMs are extraordinarily capable, but they’re also very literal. They do what you tell them. If your prompts are ambiguous, your costs will be high. If your architecture leaks assumptions, your agent will hallucinate workarounds. If your abstractions are clean, everything else becomes easier.<\/p>\n\n\n\n Build the boring parts well. The model will handle the rest.<\/strong><\/p>\n\n\n\n Have questions or thoughts on any of these patterns? Drop them in the comments below.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":" Introduction Over the past few weeks, I’ve been deep in the weeds building a proof-of-concept that uses an LLM agent to autonomously write and execute Python code inside isolated containers….<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[8],"tags":[],"class_list":["post-169","post","type-post","status-publish","format-standard","hentry","category-learn-with-me"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/posts\/169","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/comments?post=169"}],"version-history":[{"count":1,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/posts\/169\/revisions"}],"predecessor-version":[{"id":171,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/posts\/169\/revisions\/171"}],"wp:attachment":[{"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/media?parent=169"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/categories?post=169"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/balamurali.in\/blog\/wp-json\/wp\/v2\/tags?post=169"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\n\n\n\n1. “Sandbox as Tool” Is a Better Pattern Than “Agent in Sandbox”<\/h2>\n\n\n\n
\n\n\n\n2. A Clean Sandbox Abstraction Pays for Itself Immediately<\/h2>\n\n\n\n
execute(command)\nupload_file(local_path, container_path)\ndownload_file(container_path, local_path)\ncleanup()<\/code><\/pre>\n\n\n\n
\n\n\n\n3. Prompt Engineering Is Your Biggest Cost Lever \u2014 By Far<\/h2>\n\n\n\n
\n
\n\n\n\n4. Tell the LLM Exactly What It Cannot Assume<\/h2>\n\n\n\n
\n
\n\n\n\n5. Separate Deterministic Presentation from LLM-Generated Content<\/h2>\n\n\n\n
\n
\n
\n\n\n\n6. Simpler Agents Often Win for Bounded Tasks<\/h2>\n\n\n\n
\n
\n\n\n\n7. Cold Start Latency Is Solvable \u2014 But Understand What Actually Dominates<\/h2>\n\n\n\n
\n
\n\n\n\n8. Watch Out for Binary Data Over Streaming Interfaces<\/h2>\n\n\n\n
# Inside container\ntar cf - \/output | base64\n\n# On the client\nimport base64\ndata = base64.b64decode(raw_stream)<\/code><\/pre>\n\n\n\n
\n\n\n\n9. Never Let the LLM Read the Full Dataset<\/h2>\n\n\n\n
\n
\n\n\n\nClosing Thoughts<\/h2>\n\n\n\n
\n\n\n\n