Designing Ingestion Pipelines That Keep Usage Counts Honest

How we handle retries, idempotency, and observation so ingestion stays reliable under load.

10 min read

usage ingestionevent streamingbilling reliability

Usage ingestion looks like a logging problem until the numbers become billable. Then ordinary distributed-systems behavior — retries, duplicate delivery, late events, partial failures, and clock skew — becomes financial correctness.

The contract a billing-grade ingest endpoint needs

A useful usage API should make five guarantees explicit:

  1. Idempotent retries. The producer can retry a request after a timeout without creating extra usage.
  2. Tenant-scoped identity. An idempotency key used by one account cannot collide with the same key used by another.
  3. Event-time semantics. A late event updates the period in which the work actually happened, not simply the month in which it arrived.
  4. Raw-event retention. The aggregate is not the only record. You can still inspect the source events behind it.
  5. Actionable partial-failure reporting. If a batch contains invalid records, the response tells you which records were skipped and why.

Idempotency belongs at both request and record level

A request-level marker prevents a complete retry from replaying the whole batch. A per-record marker handles the harder case: the server commits the first records, crashes, and receives the same batch again. The retry should cheaply recognize the records that already landed while attempting only the missing ones.

The important detail is scope. The marker identity must include the account as well as the request or event identifier. Otherwise two customers can accidentally block one another by choosing the same idempotency key.

Event time is different from ingest time

Queues pause. Mobile clients reconnect. Batch processors report yesterday's work today. If a billing system groups solely by arrival time, delayed delivery changes the customer's bill. Store both timestamps and aggregate by event time.

The same rule applies to subscription lifecycle. Whether an event is valid should be evaluated against the subscription interval at the time of the event, not merely the subscription's status today. A cancellation should not erase legitimate usage that occurred before cancellation.

Do not expire state that defines the bill

Transport dedupe markers can have a finite retry window. Unique-member state is different. If a monthly unique-count meter accepts late events but forgets which members have already been counted, the same member can be counted twice when an old event arrives after the membership record expires.

Retention policy therefore has to follow semantics: retry bookkeeping may expire; state required to reproduce an open billing period must not.

What UsageBox implements today

UsageBox accepts usage through POST /api/v1/usage, supports idempotency keys, resolves events to subscription/product-item/meter/charge context, stores raw events, and produces monthly rollups. Supported meter aggregations are sum, count, max, and unique count.

UsageBox deliberately stops at metered quantity. It does not currently perform monetary rating or generate customer invoices. Read the API documentation for the exact request contract and the usage metering API architecture guide for the wider design.

Key Topics

  • •usage ingestion
  • •event streaming
  • •billing reliability

Related Articles

Explore more articles on similar topics to deepen your understanding of usage-based billing.

GitLab + Docker Hub Pipeline for Usage Ingestion Services

Ship containerized usage ingestion with GitLab CI, Docker Hub images, and environment promotions that keep billing data ...

8 min readRead more

Implementation Checklist We Use for Every UsageBox Rollout

Week-by-week tasks that keep engineering, finance, and support aligned during a billing migration.

9 min readRead more

Gemini 3.8 Flash Pricing & Free Tier (October 2026)

Gemini 3.8 Flash is free on the Gemini API Free Tier and costs $0.75 input and $3.75 output per 1M tokens through Decemb...

9 min readRead more

Explore More Articles

Discover our complete collection of usage-based billing guides and implementation patterns.

View all articles