Self-Hosting an Open-Weight Coding Agent on Your Own AWS

Should you self host?

A hosted API bills per token: every prompt, every re-sent file of context, every tool call, every generated line. A self-hosted GPU bills per hour, whether it is answering one request or twenty. That difference, variable cost versus fixed cost, is the whole economic tradeoff. Below a certain amount of usage the API is cheaper because you only pay for what you use; above it, the flat cost of a GPU you keep busy wins.

The chart below gives an illustrative trade off as of mid-2026: one always-on L40S instance on a savings plan at roughly $800 a month, against a heavy agentic-coding developer costing roughly $200 a month on a Sonnet-class API. Your real numbers will differ, but the crossover behavior is roughly what would be expected.

The reason the self-hosted line stays flat across a whole team is that interactive coding is bursty: a developer reads, thinks, and types far more than they wait on generation, so a single GPU is idle between requests and can interleave many people's calls. That is also the answer to the most common question about this setup, which we return to below: yes, everyone can use it at once.

Architecture of self-hosted coding agent

A simple architecture has four parts, and only the first two are strictly required. A GPU instance in your VPC runs the model. On it, vLLM serves an OpenAI-compatible HTTP endpoint, the same request and response shape as the OpenAI or Anthropic APIs, which is what makes everything downstream seamless. In each developer's editor, the Continue extension points at that endpoint instead of a hosted provider. Optionally, and this is where the real value tends to live, an internal gateway sits in front for authentication and audit logging, and one or more MCP servers running in the VPC give the agent scoped access to your own systems.

Model & GPU Choice

For a team on a single GPU, the practical choice in the current Qwen line is Qwen3-Coder-30B-A3B, a mixture-of-experts model with about 30 billion total parameters but only around 3 billion active per token. The sparsity buys inference speed, but it does not shrink the memory you must load: all the expert weights sit in VRAM regardless of how few fire per token. That is the number that sizes your GPU. At full precision the weights are far too big for one card; at FP8 they land near 30 GB, and at 4-bit quantization near 18–22 GB, on top of which you need room for the KV cache that grows with context length and concurrency.

In AWS terms that points at an L40S instance (the g6e family, 48 GB) running FP8, which leaves comfortable headroom for context and several concurrent sessions. If budget is tighter, a 24 GB card (the g5 A10G or g6 L4 families) runs the same model at 4-bit with less room to spare. If you need frontier-adjacent quality and can afford it, the much larger Qwen3-Coder-480B-A35B is the top open option, but it needs multiple H100-class GPUs and changes the economics entirely. The practical advice is to start with the 30B and only move up if the quality actually becomes a blocker.

Frontier-size Model Cost

Just for perspective, it is interesting to know what the largest open models cost to host, even if most teams should not do that. A frontier-size model like Kimi K2 is a one trillion parameter mixture of experts with about 32 billion active per token. Even quantized to INT4 it is around 600 GB, so serving it needs a full eight-GPU node such as 8 by H200, roughly 1.1 TB of VRAM. On current cloud GPU pricing that runs on the order of 16,000 to 24,000 dollars a month for one always-on node, before storage and redundancy. For most teams, this is not viable, and in practice this cost should always be compared to the estimated gain there is in using a frontier model versus a 30B Qwen.

Provisioning the GPU instance

One thing worth checking before beginning: new or low-usage AWS accounts often have a GPU vCPU quota of zero. In the console, open Service Quotas, choose Amazon EC2, and confirm "Running On-Demand G and VG Instances" is at least 4, since a g6e.xlarge needs 4 vCPUs. If it reads zero, request an increase and wait for approval before continuing.

The instance is a single GPU box with a security group that lets only your team reach it. You can define the whole thing as code with the AWS CDK, or click through the console. Both do the same work: launch a g6e.xlarge with a GPU-ready image, open port 8000 to your office or VPN range only, give it enough disk for the weights, and have it install and start vLLM on boot.

# infra/coding_agent_stack.py   (AWS CDK, Python)
from aws_cdk import Stack, CfnOutput, aws_ec2 as ec2
from constructs import Construct

OFFICE_CIDR = "203.0.113.0/24"      # your office or VPN range, never 0.0.0.0/0

class CodingAgentStack(Stack):
    def __init__(self, scope: Construct, id: str, **kwargs):
        super().__init__(scope, id, **kwargs)

        vpc = ec2.Vpc.from_lookup(self, "Vpc", is_default=True)

        sg = ec2.SecurityGroup(self, "VllmSg", vpc=vpc, description="vLLM endpoint")
        sg.add_ingress_rule(ec2.Peer.ipv4(OFFICE_CIDR), ec2.Port.tcp(8000), "vLLM API")
        sg.add_ingress_rule(ec2.Peer.ipv4(OFFICE_CIDR), ec2.Port.tcp(22), "SSH")

        # install vLLM and run it as a service, so it restarts after every stop/start.
        # a kernel compiles itself the first time this model is served, which needs Python's
        # development headers and the CUDA compiler present. this was verified on Amazon Linux
        # 2023 (see "Provisioning the GPU instance" below for the exact dnf packages); on this
        # Ubuntu image, confirm both are already present (python3 --version; which nvcc) before
        # assuming this will start cleanly unattended, since that was not verified here.
        user_data = ec2.UserData.for_linux()
        user_data.add_commands(
            "pip install vllm",
            "cat >/etc/systemd/system/vllm.service <<'UNIT'",
            "[Unit]",
            "After=network-online.target",
            "[Service]",
            "ExecStart=/usr/local/bin/vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8 --max-model-len 65536 --gpu-memory-utilization 0.90 --api-key team-shared-secret --enable-auto-tool-choice --tool-call-parser qwen3_coder",
            "Restart=always",
            "[Install]",
            "WantedBy=multi-user.target",
            "UNIT",
            "systemctl enable --now vllm.service",
        )

        instance = ec2.Instance(self, "CodingAgent", vpc=vpc,
            instance_type=ec2.InstanceType("g6e.xlarge"),
            machine_image=ec2.MachineImage.from_ssm_parameter(
                # confirm the current Deep Learning base GPU AMI parameter for your region
                "/aws/service/deeplearning/ami/x86_64/base-oss-nvidia-driver-gpu-ubuntu-24.04/latest/ami-id"),
            security_group=sg, user_data=user_data,
            block_devices=[ec2.BlockDevice(device_name="/dev/sda1",
                volume=ec2.BlockDeviceVolume.ebs(200))])   # room for the weights

        CfnOutput(self, "InstanceId", value=instance.instance_id)

Before the first run, two one-time build prerequisites are worth checking, since vLLM compiles a couple of kernels itself the first time it serves this model. It needs Python's development headers for one kernel and the CUDA compiler (nvcc) for another, and it fails with a plain compiler error if either is missing. On Amazon Linux 2023 these are a few extra dnf packages, using AWS's own AL2023-maintained NVIDIA repository rather than NVIDIA's generic one:

# on the GPU instance (inside your VPC), one-time setup on Amazon Linux 2023
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y python3.11-devel        # needed by a Triton kernel compiled on first run
sudo dnf install -y nvidia-release          # enables the AL2023-maintained NVIDIA repo
sudo dnf install -y cuda-toolkit            # installs nvcc, needed by a second kernel

export CUDA_HOME=/usr/local/cuda
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
echo 'export CUDA_HOME=/usr/local/cuda' >> ~/.bashrc
echo 'export PATH=$CUDA_HOME/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc

pip install vllm
pip install -U flashinfer-python            

vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.90 \
  --api-key "team-shared-secret" \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder
# serves an OpenAI-compatible API on http://0.0.0.0:8000/v1

If the last of these steps, the flashinfer upgrade, still leaves the server failing on a TypeError inside flashinfer's comm module, that is a known Python 3.11 packaging bug in older flashinfer builds; the fallback is a one-line patch, adding a single from __future__ import annotations line to the top of the file the traceback names, which defers that one annotation instead of evaluating it.

Before wiring anything into an editor, confirm the server is reachable and note the exact model id it reports.

curl http://<your-server>:8000/v1/models \
  -H "Authorization: Bearer team-shared-secret"
# note the "id" field in the response; you will need it verbatim

Keep the instance private. Expose the endpoint only inside the VPC (or over a VPN, or behind an internal load balancer with the gateway in front), never to the open internet. The bearer token is a coarse gate, not real access control; the network boundary is what actually protects it.

Scheduling it off outside work hours

A single always-on box is the simplest setup, but if your team shares working hours you can cut the bill by roughly two thirds by stopping the instance at night and on weekends. A box that runs ten hours on weekdays is about 220 hours a month instead of 730. The clean way to do this is EventBridge Scheduler calling the EC2 start and stop APIs directly, with no Lambda in between.

# add to the same stack, after the instance is defined
from aws_cdk import aws_iam as iam, aws_scheduler as scheduler

sched_role = iam.Role(self, "SchedulerRole",
    assumed_by=iam.ServicePrincipal("scheduler.amazonaws.com"))
sched_role.add_to_policy(iam.PolicyStatement(
    actions=["ec2:StartInstances", "ec2:StopInstances"], resources=["*"]))

def schedule(name, cron, api):
    scheduler.CfnSchedule(self, name,
        flexible_time_window=scheduler.CfnSchedule.FlexibleTimeWindowProperty(mode="OFF"),
        schedule_expression=cron,
        schedule_expression_timezone="America/New_York",
        target=scheduler.CfnSchedule.TargetProperty(
            arn="arn:aws:scheduler:::aws-sdk:ec2:" + api,
            role_arn=sched_role.role_arn,
            input=self.to_json_string({"InstanceIds": [instance.instance_id]})))

schedule("StopNightly",  "cron(0 20 ? * MON-FRI *)", "stopInstances")   # 8 pm weekdays
schedule("StartMorning", "cron(0 7 ? * MON-FRI *)",  "startInstances")  # 7 am weekdays

Cron expressions default to UTC, so either set the schedule time zone as shown or convert your hours. Because vLLM runs as a service, the box serves requests again a minute or two after each morning start, with no manual step.

Connecting VS Code

The Continue extension for VS Code reads a small YAML config and can point at any OpenAI-compatible endpoint. Each developer sets the apiBase to your server, the apiKey to the shared token, and the model to the exact id from the step above. The roles list controls where the model is used: chat, inline edits, and apply are the sensible ones for an agentic coding model.

# ~/.continue/config.yaml
name: Team Assistant
version: 1.0.0
schema: v1
models:
  - name: Qwen3 Coder (self-hosted)
    provider: openai
    apiBase: http://<your-server>:8000/v1
    apiKey: team-shared-secret
    model: Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8
    roles:
      - chat
      - edit
      - apply

That is the whole integration. From the developer's side it now behaves like any other coding agent in the editor (chat, edits, agent runs), except the model answering is yours.

Can several engineers use it at the same time?

Yes. vLLM uses continuous batching: rather than waiting for one request to finish before starting the next, it slots new requests into the running batch at every generation step, so the GPU stays busy and many conversations advance together. In practice a single GPU reliably serves dozens of concurrent developers with streaming responses, and throughput holds steady as concurrency climbs rather than collapsing into a queue. Because coding is bursty, the effective ceiling for interactive use is higher than raw benchmarks suggest, and the constraint you actually hit first is VRAM for the KV cache at long contexts, which is why context length and concurrency are the two dials you size together.

The practical implication for cost is the one the chart showed: adding a developer to a self-hosted box that is not yet saturated is free at the margin, whereas adding a developer to an API bill is another full per-seat cost. That is the entire reason the economics tip toward self-hosting as a team grows.

Simplified Cost Model

We provide simplified estimate of cost to give a sense of the economics, but it is recommended that teams compute their own exact costs.

Two levers move the self-hosted side a lot. First, scheduling the box to run only during working hours, instead of always-on, can roughly halve its cost if your team shares a time zone. And a one-year compute savings plan or reserved capacity cuts the on-demand rate substantially in exchange for commitment. On the API side, prompt caching is the big lever, since coding agents re-send large, stable context, and caching that context can cut the input bill by most of its value, which is why the per-developer figure has such a wide band.

Privacy Benefits

If cost were the only axis, many teams would land on the hosted API and never look back, because it is zero-operations and frontier-quality. The case for self-hosting gets much stronger on the axes a token price cannot capture: the model runs inside your account, so your code and your data never leave it. For a team under real data-residency or contractual constraints, that alone can be decisive, since the prompts, the proprietary source, the internal context all stay within your boundary.

Because the model is in your VPC, the agent can be given scoped access to your own systems through MCP tools that run alongside it, the same local-tool pattern from our MCP server piece, but with the server holding IAM credentials and sitting next to your data. A tool that runs a read-only query against your warehouse, a tool that fetches an object from a specific S3 bucket, a tool that calls an internal deployment script and reports back what it saw: the coding agent can call these mid-conversation, and none of it is ever exposed to a third party or the public internet. The permissions can be scoped through IAM, so the agent gets exactly the access you grant and nothing more.

# an MCP tool running in your VPC, next to your data (sketch)
# (see the FastMCP piece for the full server setup)
import sys
from fastmcp import FastMCP
import boto3

mcp = FastMCP("internal-tools")

@mcp.tool()
def read_s3_object(bucket: str, key: str) -> str:
    """Return the text of an object from an approved internal bucket."""
    print(f"read_s3_object {bucket}/{key}", file=sys.stderr)   # log to stderr
    body = boto3.client("s3").get_object(Bucket=bucket, Key=key)["Body"]
    return body.read().decode("utf-8", errors="replace")[:20000]

if __name__ == "__main__":
    mcp.run()

Lastly, because the weights are yours, you can fine-tune the model on your own codebase and serve the adapted version, so the agent knows your conventions and internal libraries out of the box.

Trade-Offs

Four main costs come with all of this. The first is quality: the strongest open coding models are competitive with a mid-tier hosted model on many agentic coding tasks, but a single-GPU model like the 30B sits below the frontier on the hardest, longest-horizon work, and you will feel that difference on genuinely difficult problems. The second is operations: an API key is something you buy, while a GPU box is something you run: patching, monitoring, upgrading vLLM, and being on call when it falls over. A single instance is also a single point of failure, so a team that depends on it needs redundancy, which changes the cost math. The third is capacity: if the team grows past what one GPU serves, you add and balance more.

The fourth is a context budget that shows up faster than you would expect, often on the very first agent request. An agent turn in VS Code sends far more than the literal question: the system prompt, the full schema for every available tool, and whatever the editor auto-attaches for codebase awareness can add up to tens of thousands of tokens before your actual prompt even starts. Something as plain as asking the agent to list the files in a directory can be enough to overflow a 65,536-token cap on its own. There is no way around sizing for this: either raise --max-model-len (which trades away some of the KV cache headroom that gives you concurrency, so re-check the numbers vLLM prints at startup after doing so) or reduce what the editor sends per turn.

This is why the most common landing spot is not all-or-nothing but hybrid: run the self-hosted model for the constant, context-heavy, privacy-sensitive work (autocomplete, routine edits, anything touching internal data), and keep a hosted frontier key for the occasional hard agent run where the quality gap actually matters. That captures most of the cost saving and the data-control benefit while keeping frontier quality available for the cases that need it.

Common questions

When is self-hosting actually cheaper than a team Claude key?

Once you have more than a handful of heavy users sharing one GPU. Because the box is a fixed cost and the API scales per developer, the crossover in the illustrative numbers above is around four heavy agentic users; beyond that, one box carries the whole team more cheaply, and the gap widens as the team grows. For a small team of light users, the API is both cheaper and simpler.

Do I need a huge GPU?

No. The team-scale sweet spot is a single 48 GB L40S instance running Qwen3-Coder-30B-A3B at FP8, or a 24 GB card at 4-bit if you are cost-constrained. You only need multi-GPU, H100-class infrastructure if you insist on the largest open models, which most teams do not need to start.

Will it feel the same as Copilot or Claude in the editor?

Mechanically, yes. Through Continue it is chat, edits, and agent runs like any other provider. The difference you will notice is quality on hard tasks, where a single-GPU open model trails the frontier, and speed, which is usually fine or better for interactive use because the model is small and nearby. The parts that are strictly better are the ones that come from it being yours: your data stays in, and the agent can reach your internal systems.

References

• vLLM documentation: docs.vllm.ai (OpenAI-compatible server, continuous batching, quantization, engine arguments).
• Continue self-hosting guide: docs.continue.dev (pointing the extension at a vLLM endpoint).
• Qwen3-Coder model card: huggingface.co/Qwen (sizes, context, quantized VRAM).
• AWS EC2 GPU pricing: aws.amazon.com (g5, g6, g6e, p5 on-demand and savings-plan rates).
• Anthropic API pricing: anthropic.com/pricing (per-token rates, batch and caching discounts).

Prices and model options in this space move monthly (GPU rates, token rates, and the current best open coding model all drift), so treat every number here as an illustrative input to your own worksheet, confirm the instance and token rates against the live pricing pages.