Submit, poll, read: what a long analysis looks like over HTTP
Polling, done properly
Page 2 of 4
Polling has a bad reputation it mostly earned by being done badly. Three rules make it unremarkable.
Start slow. A job that takes four minutes does not benefit from being asked about eight times a second. Two to five seconds is right for anything measured in minutes; back off as the job ages.
Treat every terminal status as terminal. A job ends succeeded or failed, and a failed job is a result too — it has an error on it that says what went wrong. Code that only handles success will hang on the one case you wrote it for.
Keep the job id. It is the only handle you have on that piece of work. Losing it does not lose the work — the worker finishes regardless and the output lands in your store — but it does mean you have to go and look for something you could have asked about directly.
There is no WebSocket, no callback URL and no push notification. That is a deliberate omission rather than a gap: a callback means we hold a queue of your endpoints, retry against them, and become an outbound HTTP client on your behalf, which is a great deal of machinery to avoid a five-second poll.