Heterogeneous Ad Data Ingestion: Configuration Still Needs a Release Process

One source sends Kafka messages, another writes object-storage files, and a third calls an HTTP endpoint. Their payloads may be JSON, CSV, or Protobuf. Copying a job for each source is quick at first, until a timestamp bug needs to be fixed in a dozen branches. A more maintainable approach shares the processing core and contains source differences in adapters and mapping rules.

Moving behavior into configuration does not remove the need for software releases. One incorrect configuration line can affect an entire batch of events. The following is a reference design.

Share processing semantics, not source cursors

Split the pipeline into read → decode → map → validate → enrich → write. Each stage should return a result or a structured error, with minimal direct manipulation of source offsets. Offline replay and streaming consumption can then use the same mapping and validation logic.

The reader must retain its own progress semantics. Kafka uses partitions and offsets; a file reader needs object identity, version, and record position. Collapsing these into an ambiguous lastPosition loses useful guarantees. For example, S3 event notifications use at-least-once delivery. A repeated notification is not a new file, so file processing needs its own idempotency record. AWS S3 Event Notifications

A practical source envelope can contain source_id, a source locator, read time, a digest of the raw payload, and a parser version. The digest helps with lookup and verification; it does not replace controlled retention of the original input. Sensitive raw payloads should stay out of ordinary error logs.

Treat mapping rules with the rigor of code

source: demo_partner
mapping_version: 12
fields:
  event_id:
    from: $.tracking.id
    required: true
  event_time:
    from: $.timestamp
    transform: epoch_millis_to_utc
  amount_minor:
    from: $.price
    transform: decimal_major_to_minor
    currency_from: $.currency

This illustrative configuration makes units and conversion functions explicit. Avoid arbitrary script execution, and do not turn every failure into a null. At load time, check field paths, target types, enum values, and transformation parameters. At execution time, record the specific rule that failed.

Java’s DynamicMessage can represent message types using a Descriptor, reducing the need to regenerate and release code whenever a target structure changes. It addresses structural representation, however; it does not automatically provide correct mappings, thread safety, or lossless rollback. Protobuf DynamicMessage

Package the Descriptor, mapping, and validation rules into an immutable version. An event should use one snapshot throughout its processing. Otherwise, a hot update can cause the first half of the event’s processing to use old rules and the second half to use new ones.

Test compatibility and business meaning separately

Deleted Protobuf field numbers must not be casually reused. Even wire-compatible changes can violate business meaning: changing an amount from cents to whole currency units while keeping the same integer type introduces a hundredfold error downstream. Protobuf proto3 guide

I would maintain a small, stable contract fixture set for each source: normal input, missing fields, invalid types, unknown fields, boundary amounts, time-zone differences, duplicate records, and historical versions. After structural validation, check business assertions such as amount ranges, event times, required IDs, and output enums.

An LLM can suggest field mappings or additional test cases, but its suggestions should not directly replace production rules. The useful output is a candidate mapping, supporting evidence, and unresolved questions, followed by deterministic validation and a diff. If the source unit is unclear, an unresolved result is more reliable than a plausible guess.

Releases need traceability and a way back

Before release, process a controlled sample with both versions and compare parsing rates, defaulted values, event distributions, and field-level differences. Use source-based or stable-hash cohorts for staged rollout so retries of one event do not randomly receive different rules. Record the rule version on every output to identify the results affected by a problem.

Rolling back a configuration changes future processing; it does not repair incorrect data already written. A release record also needs the impact scope, replayable source locations, downstream idempotency keys, and a repair plan. Before promising one-click replay, demonstrate that repeating an external write cannot double-charge or double-count.

After onboarding, slice quality dashboards by source and version. A small source’s missing-required-field rate can climb from zero to ten percent with almost no visible effect on global success rates. Alongside percentages, show absolute error counts, first-seen times, and sanitized sample locations to help identify the upstream change.

Acceptance testing should deliberately simulate three failures: a partially loaded configuration, an object overwritten during processing, and a process exit after successful output. Verify atomic version switching, object identity checks, and replay idempotency respectively. Configuration reduces maintenance costs when the system can explain and handle these boundaries.

References

• AWS S3 Event Notifications , for at-least-once notification semantics.
• Protobuf DynamicMessage Java API , for Descriptor-based message handling.
• Protobuf proto3 Language Guide , for field numbers, reserved fields, and evolution constraints.