Feed Ingestion Failure Troubleshooting

A feed-ingestion failure is broader than “the file did not arrive”. The artifact may exist while parsing, validation, persistence or downstream publishing fails later in the pipeline.

Observable symptom

New inventory does not appear, import counts drop to zero, rejected items spike, the ingestion job fails, or source changes never become visible in the consumer API.

Most likely causes by layer

• Transport: SFTP/HTTP auth, DNS, TLS, timeout or missing artifact.
• File: encoding, delimiter, compression or truncated upload.
• Schema: renamed fields, type changes or enum drift.
• Parsing: malformed JSON/XML/CSV.
• Validation: required fields, duplicates, invalid currency/date.
• Persistence: constraints, transactions, deadlocks or storage.
• Publish: index/cache/event propagation failed.

Fast triage order

1. Confirm expected feed arrival.
2. Capture raw artifact checksum and size.
3. Compare parser input/output counts.
4. Group rejected records by reason.
5. Compare valid count with persisted insert/update count.
6. Verify downstream publish/index counts.

Evidence to collect

Every run should have a control record containing feed ID, source version, received timestamp, checksum, raw/parsed/valid/rejected/persisted/published counts and final status.

Diagnostic decision tree

No artifact means transport.

Artifact present but low parsed count means format/parser.

Parsed count correct but validation low means schema/data quality.

Valid count correct but persistence low means database/persistence.

Persistence correct but users see old data means publish/cache/index.

False positives

An intentionally empty feed is not necessarily a technical failure, though it can be a business anomaly. Distinguish EMPTY_SUCCESS from FAILED_RUN.

Fix and validation

Depending on the contract, quarantine bad records rather than failing an entire batch. Gate incompatible schema versions explicitly.

Replay the same raw artifact deterministically and verify stage counts.

Metrics proving resolution

Track feed lateness, parse reject rate, validation reject rate, persistence errors, end-to-end ingestion lag and source-to-published count delta.

Prevention

Use schema contract tests, fixtures, checksums, versioning and run-level reconciliation as standard pipeline controls.