How I Cut Our Cloud Bill by $800 a Month — Without Reducing Reliability

Nobody tells you that observability is one of the most expensive line items in a cloud infrastructure budget until you get your first Datadog invoice and have to have an awkward conversation with your manager.

We were spending about $5,400/month on Datadog — APM, log management, infrastructure monitoring. Not outrageous for a production system, but there was clearly room to optimise. After two months of careful analysis and incremental changes, we brought it down to around $4,600. Here's exactly what we did and why.

Step 1: Audit What You Are Actually Paying For

Before touching anything, spend a week understanding your usage. In Datadog, go to Plan and Usage → Usage Summary. Look for:

• Which services are generating the most log volume?
• Which log pipelines have the highest indexed log count?
• Which traces are being retained at full fidelity?
• Are there any hosts still reporting metrics that no longer exist?

In our case, we found that 60% of our log volume came from three services — and most of those logs were DEBUG level that we never actually looked at in production.

Step 2: Stop Shipping Debug Logs to Datadog

This was the biggest single win. Debug logs are useful in development. In production, they're expensive noise.

We updated our log configuration to only ship INFO level and above to Datadog:

# Datadog Agent configuration
logs:
  - type: container
    service: my-app
    source: node
    log_processing_rules:
      - type: exclude_at_match
        name: exclude_debug
        pattern: '"level":"debug"'

For the services that genuinely needed debug logs in production for troubleshooting, we sent them to S3 instead — orders of magnitude cheaper, and we could query them with Athena when needed:

# Monthly cost comparison for 50GB of logs
# Datadog Log Management: ~$120
# S3 + Athena: ~$1.15

Step 3: Tune Log Retention Tiers

Datadog has three retention tiers: hot (indexed, immediately searchable), rehydration (archived to S3, searchable on demand), and deleted.

We had everything on 15-day hot retention. After looking at our actual usage patterns — when did we last search logs older than 7 days? — we made these changes:

• Application logs: 7 days hot, then rehydration archive for 90 days
• Access logs: 3 days hot, then rehydration archive for 30 days
• Debug logs from S3: 90-day archive, no Datadog involvement at all

Rehydration costs money when you use it, but if you're rarely searching old logs, paying per-use is much cheaper than always-on hot storage.

Step 4: APM Sampling

By default, Datadog APM samples 100% of traces. For high-traffic services, this is expensive and not actually that useful — you don't need a trace for every successful request. You need traces for errors and slow requests.

We switched to head-based sampling at 10% for normal traffic, but kept 100% sampling for errors and anything slower than 500ms:

# datadog-values.yaml
apm:
  portEnabled: true
  traceBuffer: 8192
  errorTrackingStandalone:
    enabled: true
  sampling:
    rules:
    - sample_rate: 1.0
      name: "error-traces"
      resource: "*"
    - sample_rate: 0.1
      name: "default"

We lost zero visibility into actual problems. We kept full fidelity on everything that mattered. APM costs dropped by about 40%.

Step 5: Clean Up Unused Infrastructure Hosts

We had 12 hosts still reporting metrics to Datadog that had been decommissioned. Datadog bills per host — that was roughly $180/month going nowhere. A quick audit and cleanup fixed this.

# Find hosts not seen in 24 hours via API
curl -G "https://api.datadoghq.com/api/v1/hosts" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
  --data-urlencode "filter=status:OFFLINE"

Step 6: Commit to a Savings Plan

After 60 days of real usage data with the new configuration, we knew roughly what we'd spend each month. We committed to a Datadog annual plan, which gave us about 18% off the on-demand rate.

Don't do this before you've optimised. Commit to what you'll actually use, not what you're currently spending.

The Final Numbers

We ended up slightly above the $800 target. All of these changes took about three weeks to implement carefully, testing each one before moving to the next.

The most important rule: don't reduce visibility on things that matter. Every change we made was based on actual usage data — we could see that nobody was querying debug logs, that most traces were identical successful requests. If you don't have that data first, you're just guessing.