Measure first, buy last: how we keep our infrastructure lean

Every server, network and permission behind AdZoic is written down as code. We have had to grow that platform without a large budget, and when you read back through the history of changes, one pattern keeps turning up: most of our best fixes were a measurement, not a purchase. This is what that looks like in practice — the decisions, the prices behind them, and a place where the code and reality had quietly stopped agreeing.

Why the whole platform is code

“Infrastructure as code” means we don't build servers by clicking through a console. We describe what should exist — machines, networks, databases, the rules for who can reach what — in plain text files, using a tool called Terraform. The tool then makes the real world match the files.

Three things follow from that, and all three save money:

• Nothing runs forgotten. If something exists, it is written down, so nobody discovers a machine on the bill that no one remembers starting.
• Every change is reviewed before it happens, like any other code change, instead of being discovered afterwards.
• An environment can be rebuilt from scratch. A test environment that isn't needed can be taken down and brought back later exactly as it was.

The architecture post covers what the platform does. This one is about how we keep it lean.

One definition, every environment

We run separate test and production environments, and the same foundation serves more than one product. Early on, each environment had its own full copy of the infrastructure definition: one long file each, hundreds of lines long.

Copies are expensive in a way that never shows up on an invoice. Every fix had to be made several times, and it was easy to make it in two places and miss the third. Over time the copies quietly stopped matching, and “the same setup” meant something slightly different in each place.

So we turned the whole platform into a single module — one definition of the complete stack. Each environment is now a short file that states only what is different about it: its name, how large it should be, and which parts are switched on. A fix is made once and reaches every environment. A new environment is a new short file, not a new copy of everything.

Before, every environment carried its own copy. After, there is one definition, and each environment's file says only what makes it different.

It also changes how a new environment comes up. Each major piece — the network, the cluster, the front door, monitoring, the databases, the data pipeline — sits behind its own switch. We turn them on one at a time and check each before the next. When something fails, you know which piece it was.

The price list is part of the design

Cloud pricing is detailed and public, which means a lot of the cost is decided when you choose a design, long before any traffic arrives. These are the choices where the price list made the call. Every figure is AWS's published on-demand price for the Singapore region, where we run.

ARM processors, because the software doesn't mind

AWS sells two broad families of general-purpose machine: ones built on x86 chips, like most laptops, and ones built on its own ARM chips, called Graviton. For the same size of machine, the ARM one costs less:

Relative on-demand list prices for comparable x86 and ARM machines of the same size, Linux, AWS Asia Pacific (Singapore) region, as checked on 19 September 2026. Shown as a proportion, not a price.

Our services are built for ARM, so every machine in our production cluster is an ARM one. It is the least exciting decision in this post, and it applies to every hour of every machine we run.

Test environments don't get production's plumbing

Machines on a private network need a way out to the internet — to download updates, for instance — without the internet being able to reach in. On AWS that is usually a NAT gateway, and it bills from the moment it exists: tens of dollars a month before a single byte passes through it, plus a charge on every gigabyte it handles.

Production has one. Our test environment is built to a simpler network layout that doesn't need it, and one setting switches between the two. Paying production prices for an environment that sits idle overnight is money spent on nothing.

Storage traffic takes the free road

In production, all outbound traffic used to go through that NAT gateway — including traffic to S3, AWS's file storage. That is where our logs and backups are kept, and where the layers of the software images each machine downloads come from. Every one of those gigabytes paid the handling charge.

AWS offers a second route for exactly this case: a gateway endpoint, a private shortcut from our own network to S3. It has no hourly charge and no per-gigabyte charge. S3 traffic now takes the shortcut, and everything else still uses the NAT gateway as before. We explain how, why and when we made the switch in a separate post, with the checks worth running first.

Falling behind has a price tag

AWS runs the “brain” of our Kubernetes cluster — the part that decides where each piece of software runs. On a supported version of Kubernetes, AWS charges a modest hourly fee for it. Let the version fall out of standard support and AWS adds an extended-support charge on top — about six times the price in total, for exactly the same thing.

Relative Amazon EKS cluster pricing, standard support versus extended support, AWS Asia Pacific (Singapore) region, as checked on 19 September 2026. Shown as a multiple, not a price.

That turns keeping up to date from a chore into a cost decision. Because upgrades are written as code, they are a routine change rather than a project — and that is what stops them being put off.

Every resource carries a label

Everything the code creates is automatically labelled with the product, the environment and the fact that it is managed by code. It sounds like bookkeeping, and it is. It is also the difference between “the AWS bill went up” and “the test environment's storage went up”. You can't make something cheaper if you can't see what it costs.

Most of our fixes were a measurement, not a purchase

This is the pattern that stands out when you read back through the history. When something was short of capacity, the obvious move was to buy more of it. More often than not, when we measured first, the capacity was already there — something was simply using it badly.

Three copies of every message, without new machines

Our event stream — Redpanda, the pipeline that carries auction events from the bidder towards our analytics — started life on a single server, holding a single copy of each message. That is a reasonable way to start small. It is also a single point of failure, so the goal was three servers, each holding a copy.

The first attempt failed for lack of room: no machine had enough spare capacity for another server. The obvious conclusion was that we needed to buy more machines.

Then we measured. Each server was reserving a large, fixed share of processor time for itself — the default setting — while even at its busiest it used only a small fraction of that. Machines were “full” of reservations that nothing was using. Reserving a sensible amount instead, still with generous headroom above its busiest moment, freed enough room for three servers to fit on machines we already had.

Today every message is held on three servers, on three separate machines. The number of machines we bought to get there was zero.

Make the unsafe option impossible

Once every message had three copies, one more step remained. The stream would still accept a request to create a new channel — a topic — with a single copy, if someone asked for one. Nobody intends to. But “nobody intends to” is exactly how single points of failure come back.

So the minimum is now enforced by the system itself: it refuses to create anything with fewer than three copies. A rule people have to remember is a rule that eventually gets forgotten. A rule the system enforces doesn't need remembering.

Batching, not a bigger processor

Our analytics database, ClickHouse, looked short of processing power: it was regularly being held back at its limit. The obvious fix was to raise the limit.

When we looked at what it was actually doing, the work was housekeeping, not answering questions. This kind of database writes each incoming batch of rows as a small file, then keeps merging small files into larger ones in the background. Our services were sending it only a few dozen rows at a time — so each row ended up being rewritten hundreds of times as the files were merged and merged again. Tiny writes don't cost a little extra. They cost a storm of housekeeping.

So instead of buying more processor time, we changed how the data arrives: the database now gathers incoming rows for a brief moment and writes them together. Each write still waits until the data is safely stored before it is confirmed — this is batching, not “send and hope”. The only thing our services see is a very short delay, on writes they already make in the background.

Sometimes the answer really is more machines

Measuring first doesn't mean never buying. As traffic has grown, there have been times when the cluster genuinely needed more machines, and we raised the ceiling on how many it is allowed to add. The difference is that extra machines arrive automatically when there is work for them and leave when there isn't. We pay for the traffic we get, not the traffic we might get one day.

Keep the code honest

Infrastructure as code only works if the code describes what is actually running. The gap between the two has a name — drift — and it is quieter and more dangerous than it sounds.

During one fix, our analytics database's memory was raised on the running system and never written back into the code. So the code said one amount, and the database was actually running with more. Nobody noticed, because nothing looked wrong.

Here is why it mattered. The next routine update — made for some completely unrelated reason — would have “corrected” the running system to match the code. That would have cut the database's memory below what it actually uses, and crashed it. A change meant to be harmless would have caused an outage, and nobody would have thought to look at the database first.

We caught it before that happened and wrote the real value back into the code. The rule we took from it: when the code and the running system disagree, the difference is a bug. Fix one or the other, on purpose. Never leave them disagreeing.

A change should do only what it says

Two smaller fixes share a lesson:

• Pin what you don't control. Our servers start from a base image that AWS publishes and updates. Left set to “always use the newest”, any change we made — to monitoring, say — would also swap every server onto AWS's latest image, whether we wanted that on that day or not. That happened to us once, which is how we found out. The image version is now fixed in the code, and upgrading it is a deliberate change of its own.
• Don't depend on someone else's good day to make your own change. One component was downloaded from GitHub every time we applied an update, so when GitHub had a slow moment, completely unrelated changes failed. We now keep a copy of that component in our own repository.

The principle is the same in both: a change to monitoring should change monitoring, and nothing else.

Back up what has no other copy

Backups are insurance, and insurance has a premium. Disk snapshots on AWS are charged roughly by how much data has changed since the last one, so the cost depends heavily on what you snapshot — and some data changes far more than it appears to. A database that constantly reorganises its files, like the one above, makes daily snapshots expensive out of all proportion to the new data it actually holds.

So before backing something up, we ask one question: does it already have another copy?

• A cache that can be rebuilt from the main database, and is saved separately every night anyway, doesn't need a disk snapshot as well.
• The event stream already keeps three copies on three machines, so losing one is survivable without a restore.
• Monitoring data is regenerated by the running system, and the dashboards are themselves written as code, so they can be recreated rather than restored.

That roughly halved what we back up. Each thing we left out is left out for a specific reason, written next to the setting — not because it seemed unimportant. The same goes for trade-offs we choose to accept: the reasoning, including the case against it, is recorded where the next person will see it, so a decision can be revisited rather than rediscovered.

Levers we haven't pulled yet

Two well-known ways to cut an AWS bill are still on the table, on purpose:

• Spare-capacity machines. AWS sells its unused capacity at a steep discount, on the condition that it can take a machine back at short notice. That suits work that can be interrupted. It needs real care for anything that holds data, so it is a decision for a specific workload, not a switch to flip everywhere.
• Committing in advance. AWS discounts machines you commit to using for a year or more. That only saves money once you know your steady baseline, and ours is still moving as traffic grows. Committing too early is a way of paying for machines you might not need.

What we took away

• Measure before you buy. The capacity you need is often already there, busy doing something it doesn't need to.
• Read the price list as a design input. Many costs are set when you choose a design, not when traffic arrives.
• Make the unsafe option impossible, not merely unlikely.
• Treat drift as a bug. Code that doesn't describe what is running is worse than no code, because it looks trustworthy.
• A change should do only what it says. Pin what you don't control, and keep your own copy of what you depend on.
• Size a saving honestly. Some of these saved a lot and one saved a few dollars. Both are worth writing down, at their real size.

For how the pieces fit together, there is the architecture post. For more of the same habit — checking what a number actually measures before trusting it — see how we load test the bidder.