Async Background Tasks

Hey this is Isaac,

This week I fixed classic async problems: a "processing" pill that never updated to "complete."

The goal: prevent long-running tasks from freezing the UI. A one-second task can block the user. A five-minute task cannot. So we have background tasks that run independently of the UI. But users still expect to see updates.

For a project I am working on, Devrelifier, the updates were not working for this processing pipeline:

1. User uploads a video
2. Video goes to an API for transcription and timestamping
3. Screenshots are taken every 3 seconds and uploaded to S3
4. Chapter markers are created
5. All that + other context is sent to LLM to create a draft (blog, newsletter, etc)

For a 90-minute video, that can take a while. I needed a clean way to update the UI after a background job finished.

The Build: Choosing Simplicity Over Infrastructure

The user clicks "Generate" to start background jobs. Long videos take time, and users may navigate away or wait and watch. Two options to deliver updates:

1. Server-Sent Events (SSE): The server pushes events to the client in real-time via a persistent connection. This is the best choice if there are lots of updates (like streaming a response) or if near-real-time updates are valuable.
2. Polling: The front end asks for updates periodically. It's super simple with HTMX, but it's a tiny bit less performant and not perfectly real-time (though close enough for many things).

I chose simplicity. A 1-second delay in UI updates for a long-running background task is fine for my use case. For chat applications, SSE's real-time updates matter more.

SSE requires managing active connections, typically with the HTMX SSE extension. The extension keeps code light, but regular HTTP requests with fewer dependencies make testing, debugging, and maintenance easier.

The Learn: The "Self-Polling Pill"

Here's the pattern I used. It's a self-contained component that polls every second and then stops itself.

Here's how it works:

1. The Trigger: A user clicks Generate.
2. The Initial Response: The server immediately starts the background job and returns only the processing pill: Processing...
3. The Poll: On load, this new waits 1 seconds, then hits the /status/123 endpoint to check the job.
4. The Loop or Finish: The status endpoint returns one of two things:
   • Still Processing? It returns the exact same . The pill swaps itself, waits 1 second, and polls again.
   • Complete? It returns the "✓ Complete" pill plus the "Edit" button using an OOB (Out-of-Band) swap. This means the UI is updated and the polling loop is broken.

The Refactor: From 4x Duplication to 1 Config

The self-polling pill worked well, but I realized I had almost the same logic in 4 places: blog_actions, newsletter_actions, twitter_actions, and linkedin_actions.

The old way: Each content type had its own duplicated logic for the "Generate" button and the completion button ("Edit" vs. "View"). This logic was mirrored in the main view and in the status endpoint. It was a headache-in-waiting.

The new way: I refactored all of it into a single, config-driven helper: build_content_actions(project, task_type).

This function uses a small config dictionary that maps a task_type (like "blog") to:

1. A check for existing content (e.g., project.content is not None).
2. A function to build the correct completion button (e.g., an "Edit" button for "blog," a "View" button for "twitter").

This refactor replaced about 100 lines of duplicated code. Adding a new content type is now much easier.

---

Until next week,

Isaac