# Keep the request path fast: AI generation in BullMQ workers

> Generating a pitch deck takes far longer than a web request should. How StartupDeckAI moves analysis and generation into BullMQ workers, with retries, progress and one job per idea.

By Yash Suthar · 2026-10-18 · https://www.yashsuthar.dev/writing/background-jobs-bullmq

Turning an idea into market analysis and a pitch deck in StartupDeckAI takes a chain of slow steps: enrichment, scoring, generation. Any one of them can take longer than a web request should, and any one can fail. Run them inside the request and the user stares at a spinner, and a single timeout throws away all the work done so far. So the request path does almost nothing. It records what was asked for, queues the work, and returns.

## The shape

- The API validates the request, saves it, and adds a job to a BullMQ queue. It answers right away with the job's ID.
- Workers, in separate processes, take jobs from the queue in Redis and run the slow steps.
- The client follows the job's progress and loads the result when it completes.

BullMQ keeps queues in Redis and handles the parts that are tedious to get right: giving each job to exactly one worker, retries, delays, and putting back jobs whose worker crashed halfway.

## Adding a job

```ts
// queue.ts
import { Queue } from "bullmq";

export const connection = { host: process.env.REDIS_HOST, port: 6379 };
export const decks = new Queue("decks", { connection });

// Called from the API handler. Returns as soon as the job is stored.
export async function requestDeck(ideaId: string) {
  await decks.add(
    "generate",
    { ideaId },
    {
      jobId: `deck-${ideaId}`,          // one job per idea, however often the button is pressed
      attempts: 3,                        // AI providers fail; retry before giving up
      backoff: { type: "exponential", delay: 5_000 },
      removeOnComplete: 1_000,            // keep recent finished jobs for debugging
      removeOnFail: 5_000,
    },
  );
}
```

The custom `jobId` is the cheap kind of idempotency: while a job with that ID exists, adding it again does nothing. A double click or a retried HTTP request can't start a second, expensive generation.

## The worker

```ts
import { Worker } from "bullmq";
import { connection } from "./queue";

new Worker(
  "decks",
  async (job) => {
    const { ideaId } = job.data;

    const research = await enrich(ideaId);          // market and competitors
    await job.updateProgress(40);

    const scores = await score(ideaId, research);   // one isolated module per signal
    await job.updateProgress(70);

    const deck = await generateDeck(ideaId, research, scores);
    await job.updateProgress(100);

    return { deckId: deck.id };
  },
  { connection, concurrency: 4 },
);
```

Write the steps so a retry is cheap: each one saves its output, and checks for a saved result before doing the work again. Then a failure in generation doesn't redo the research, and three attempts don't cost three times as much.

## Progress the user can see

`job.updateProgress` publishes an event. On the API side, `QueueEvents` hears progress, completion and failure from every worker, and the server passes them on to whichever browser is watching that job. Server-sent events fit well here: one direction, plain HTTP, automatic reconnects. The client never polls, and a page refresh just picks the same job back up by its ID.

```ts
import { QueueEvents } from "bullmq";
import { connection } from "./queue";

const events = new QueueEvents("decks", { connection });

// notify() pushes to whichever browser is watching that job.
events.on("progress", ({ jobId, data }) => notify(jobId, { progress: data }));
events.on("completed", ({ jobId, returnvalue }) => notify(jobId, { done: true, result: returnvalue }));
events.on("failed", ({ jobId, failedReason }) => notify(jobId, { error: failedReason }));
```

## What to watch

- Concurrency is a budget, not a speed dial. Set it from the AI provider's rate limits and the worker's memory, not from how many CPUs you have; past that point, more concurrency only turns queue time into 429s.
- Failed jobs are data. Keeping recent failures with their errors makes most production bugs a matter of reading, not reproducing.
- Keep job payloads small: an ID, not the document. The worker loads the current state when it runs, so a retry an hour later doesn't act on stale data.

The request path stays fast because it refuses to do slow work.

---

*Originally published at [yashsuthar.dev](https://www.yashsuthar.dev/writing/background-jobs-bullmq). I'm Yash Suthar, a software engineer in Mumbai building [LimitYourAPI](https://limityourapi.tech), [ScopeCI](https://scopeci.onrender.com) and [StartupDeckAI](https://startupdeck.in).*