Kentro Tech

Async Background Tasks

Hey this is Isaac,

This week I was fixing some basic and classic async problems: a "processing" pill that never updated to "complete." Blech.

Images of video processing ui showing a processing state and log and then compleded status

The goal is to prevent long-running tasks from freezing the user interface. If something takes a second, it’s fine to expect the user to stay on the page until it completes. But if it takes 5 minutes? Users will want to be able to do other things while it’s processing. So we have background tasks that run independently of the UI the user sees. But the user will still expect to see updates and watch the job though!

For a project I am working on, Devrelifier, the updates were not working for this processing pipeline:

  1. User uploads a video
  2. Video goes to an API for transcription and timestamping
  3. Screenshots are taken every 3 seconds and uploaded to S3
  4. Chapter markers are created
  5. All that + other context is sent to LLM to create a draft (blog, newsletter, etc)

For a 90-minute video, that can take quite a while! I needed a clean and simple way to update the UI after a background job finished..

The Build: Choosing Simplicity Over Infrastructure

The user clicks “Generate” to start background jobs that generate content. It can take a while, especially if it’s processing a long video, and the user may want to navigate to other pages or wait and watch the job in the UI until it completes. Once the background job is running, there are two main options to deliver updates to the UI:

  1. Server-Sent Events (SSE): The server pushes events to the client in real-time via a persistent connection. This is the best choice if there are lots of updates (like streaming a response) or if near-real-time updates are valuable.
  2. Polling: The front end asks for updates periodically. It’s super simple with HTMX, but it’s a tiny bit less performant and not perfectly real-time (though close enough for many things).

I chose simplicity of logic. A 1-second delay in the UI updating for a long-running background task is inconsequential for what I was doing. For many things, like a chat application, the real-time updates SSE gives are really important.

SSE requires managing active connections, which is typically implemented using something like the HTMX SSE extension. The extension allows you to keep code very light, but I think regular HTTP requests with fewer dependencies make for easier testing, debugging, and maintenance.

The Learn: The "Self-Polling Pill"

Here’s the pattern I used. It's a self-contained component that polls every second and then stops itself.

Here’s how it works:

  1. The Trigger: A user clicks Generate.
  2. The Initial Response: The server immediately starts the background job and returns only the processing pill: Processing...
  3. The Poll: On load, this new waits 1 seconds, then hits the /status/123 endpoint to check the job.
  4. The Loop or Finish: The status endpoint returns one of two things:
    • Still Processing? It returns the exact same . The pill swaps itself, waits 1 second, and polls again.
    • Complete? It returns the "✓ Complete" pill plus the "Edit" button using an OOB (Out-of-Band) swap. This means the UI is updated and the polling loop is broken.

The Refactor: From 4x Duplication to 1 Config

The self-polling pill worked quite well, but after a while I realized that I had almost the same logic in 4 places! blog_actions, newsletter_actions, twitter_actions, and linkedin_actions.

The old way: Each content type had its own duplicated logic for the "Generate" button and the completion button ("Edit" vs. "View"). This logic was mirrored in the main view and in the status endpoint. It was a headache-in-waiting.

The new way: I refactored all of it into a single, config-driven helper: build_content_actions(project, task_type).

This function uses a small config dictionary that maps a task_type (like "blog") to:

  1. A check for existing content (e.g., project.content is not None).
  2. A function to build the correct completion button (e.g., an "Edit" button for "blog," a "View" button for "twitter").

This simple refactor replaced about 100 lines of duplicated code. More importantly, adding a new content type is much easier (and I plan on adding many more!)

Until next week,

Isaa