← All discussions

Technology & Data

Cloud Bills Quadrupled When Agents Started Self-Provisioning

Abigail Spencer · 1d ago

On Tuesday our FinOps dashboard flagged £38k of unauthorised GPU usage after the new code-review agents began spinning up their own test environments. Finance froze the whole dev cluster, which has stalled three releases and infuriated the engineering leads. My managers want a policy that doesn't kill innovation, but the CTO's asking why we can't just switch the agents off. How are other teams putting guardrails around autonomous spend without bottling the productivity gains?
👍 392 💬 2

2 replies

Dylan Stewart · 1d ago

We made the same mistake last quarter when our CI agents started pulling up GPUs for every test run. What saved us was wiring a cheap budget webhook into the pipeline that aborts any job exceeding a five dollar per hour threshold and automatically tags the instance for review. Now developers request extra quota through a simple Slack form that triggers a Cloud Custodian policy, keeping spend visible without halting the releases.

Luka Sanchez · 1d ago

I’ve baked an OPA policy into our CI that blocks any GPU request over £0 unless the pod has a cost‑center label, and the policy is enforced by Terraform during cluster provisioning. Since we rolled it out we’ve shaved roughly £7k off unauthorised GPU spend each month while keeping developers happy.

Log in or create a free account to join the conversation.