The upload succeeded, the processing didn't: async ingestion design

Someone uploads a spreadsheet. Your API returns 200. What have you promised them?

If the answer is “the file arrived”, say so in the response. If the answer is “the data is now in the system and searchable”, you have made a promise you cannot keep inside an HTTP request, because parsing and validating a large spreadsheet takes longer than anyone should hold a connection open, and the work can fail long after the connection closes.

Conflating those two events is the most common design fault I see in ingestion systems, and it produces a specific, miserable support conversation: the user says “I uploaded it yesterday”, the system says the upload succeeded, and the data is not there. Both parties are right.

This is a pattern writeup from multi-tenant platform work where organizations upload spreadsheet data that gets validated, stored, indexed, and audited. The architecture is production work; I do not have public metrics to attach to it and will not invent any.

Two events, not one

  POST /uploads
      │
      ├─► validate the request, not the contents
      ├─► retain the original file (S3-compatible)
      ├─► create a record: job id, status = accepted
      └─► 202, here is your job id          <── event 1: ACCEPTED
              │
              v
        queue message
              │
              v
        worker parses and validates rows
              │
              ├─► valid rows        -> PostgreSQL
              ├─► searchable form   -> Elasticsearch
              ├─► row-level errors  -> validation report
              └─► audit event, status = processed / failed / partial
                                                  <── event 2: PROCESSED

202 Accepted with a job id, rather than 200 OK, is not pedantry about status codes. It changes what the client builds. A 200 invites a UI that says “done”. A 202 with an id invites a UI that shows status, which is the truthful one.

Decisions that matter more than they look

Keep the original file forever. Whatever your parser produced, the source of truth is what the user actually sent. When someone disputes the result in six months, or when you fix a parser bug and need to reprocess, the original is the only artifact that can settle it. Storage is cheaper than an argument you cannot win.

Every upload gets a job id at the moment of acceptance. The id exists before any processing happens, so there is always something to reference: status endpoint, support ticket, audit trail, log correlation. An upload without an identifier is unobservable.

Workers must be idempotent. Queues redeliver. A worker will be killed mid-job, a message will be processed twice, and a user will double-click. If processing the same job twice produces two copies of the data, you have built a system that corrupts itself under normal operating conditions rather than exceptional ones.

Row-level errors, not a single verdict. “Validation failed” on a 4,000-row spreadsheet is useless. Which rows, which columns, what was wrong, and preferably enough context to fix it in the source file. This is most of the perceived quality of the whole feature, and it is the part that gets built last.

Partial success is a real state. 3,800 rows valid and 200 rejected is the common case, not an edge case. If your model only has succeeded and failed, you will be forced to choose between discarding good data and importing bad data. Model it explicitly.

Record the parser and schema version. When validation rules change, you need to know which version processed a given dataset. Otherwise reprocessing old uploads produces different results and nobody can explain why.

Keeping two datastores agreed

Records go to PostgreSQL and a searchable representation goes to Elasticsearch. That is two writes and no distributed transaction, so they will disagree.

The naive version writes to both in the worker and hopes. When the second write fails, you have a record that exists but cannot be found, which users experience as data loss even though the data is right there.

Postgres is the source of truth and the search index is derived. The index can be rebuilt from the database; the database can never be rebuilt from the index. Every design decision follows from that asymmetry: an outbox or reconciliation path so a failed index write is retried rather than lost, and the ability to reindex a tenant from scratch as a routine operation rather than an emergency.

If the index is derived and rebuildable, an inconsistency is a delay. If it is authoritative for anything, an inconsistency is data loss.

Fanout, when one action means five things

An upload finishing means: update the search index, write an audit event, notify the user, refresh analytics. Doing that inline couples them into one fragile sequence where a slow notification service delays indexing and a failing analytics call fails the whole job.

  processing complete
         │
         v
   fanout exchange
         │
         ├─► indexing queue
         ├─► audit queue
         ├─► notification queue
         └─► analytics queue

Each consumer has its own queue, own retry policy, and own dead-letter destination. Analytics being down stops analytics. It does not stop indexing.

The settings that make this survive contact with production: durable exchanges and queues, persistent messages, explicit acknowledgement after the work is done rather than on receipt, dead-letter exchanges so poison messages leave the main queue instead of blocking it forever, idempotent consumers because all of the above implies redelivery, correlation ids so one user action can be traced across four consumers, and versioned event schemas because consumers deploy on their own timetable.

Explicit acknowledgement placement is the one that bites hardest. Acknowledge on receipt and a worker crash silently drops the work. Acknowledge after completion and a crash redelivers it, which is why idempotency is not optional.

Two queue systems, on purpose

The platforms I work on tend to end up running both RabbitMQ and a Redis-backed job queue, and that looks like indecision until you look at what each is doing.

Broadcasting “dataset processed” to four independent consumers is a routing problem. Running “parse this spreadsheet” with progress reporting and retries inside one application is a job problem. Using one tool for both means either building routing on top of a job queue or building job ergonomics on top of a broker, and both of those are worse than running two things that each do their own job well.

The general shape

Strip out the spreadsheets and this is the pattern for any work that outlives its request:

Accepting work and completing work are separate events with separate identities, separate failure modes, and separate places in the UI. A system that reports only the first one is lying by omission.

File uploads, video encoding, report generation, sending a bulk email, an export, a payment that settles asynchronously. In every case the request-time response can honestly say “I have this”, and only a later, addressable status can say “I did it”.

If your API returns 200 for work that has not happened yet, the bug is in the contract, not in the worker.