Keep the request path fast: AI generation in BullMQ workers

Turning an idea into market analysis and a pitch deck in StartupDeckAI takes a chain of slow steps: enrichment, scoring, generation. Any one of them can take longer than a web request should, and any one can fail. Run them inside the request and the user stares at a spinner, and a single timeout throws away all the work done so far. So the request path does almost nothing. It records what was asked for, queues the work, and returns.

The shape

• The API validates the request, saves it, and adds a job to a BullMQ queue. It answers right away with the job's ID.
• Workers, in separate processes, take jobs from the queue in Redis and run the slow steps.
• The client follows the job's progress and loads the result when it completes.

BullMQ keeps queues in Redis and handles the parts that are tedious to get right: giving each job to exactly one worker, retries, delays, and putting back jobs whose worker crashed halfway.

Adding a job

// queue.ts
import { Queue } from "bullmq";

export const connection = { host: process.env.REDIS_HOST, port: 6379 };
export const decks = new Queue("decks", { connection });

// Called from the API handler. Returns as soon as the job is stored.
export async function requestDeck(ideaId: string) {
  await decks.add(
    "generate",
    { ideaId },
    {
      jobId: `deck-${ideaId}`,          // one job per idea, however often the button is pressed
      attempts: 3,                        // AI providers fail; retry before giving up
      backoff: { type: "exponential", delay: 5_000 },
      removeOnComplete: 1_000,            // keep recent finished jobs for debugging
      removeOnFail: 5_000,
    },
  );
}

The custom jobId is the cheap kind of idempotency: while a job with that ID exists, adding it again does nothing. A double click or a retried HTTP request can't start a second, expensive generation.

The worker

import { Worker } from "bullmq";
import { connection } from "./queue";

new Worker(
  "decks",
  async (job) => {
    const { ideaId } = job.data;

    const research = await enrich(ideaId);          // market and competitors
    await job.updateProgress(40);

    const scores = await score(ideaId, research);   // one isolated module per signal
    await job.updateProgress(70);

    const deck = await generateDeck(ideaId, research, scores);
    await job.updateProgress(100);

    return { deckId: deck.id };
  },
  { connection, concurrency: 4 },
);

Write the steps so a retry is cheap: each one saves its output, and checks for a saved result before doing the work again. Then a failure in generation doesn't redo the research, and three attempts don't cost three times as much.

Progress the user can see

job.updateProgress publishes an event. On the API side, QueueEvents hears progress, completion and failure from every worker, and the server passes them on to whichever browser is watching that job. Server-sent events fit well here: one direction, plain HTTP, automatic reconnects. The client never polls, and a page refresh just picks the same job back up by its ID.

import { QueueEvents } from "bullmq";
import { connection } from "./queue";

const events = new QueueEvents("decks", { connection });

// notify() pushes to whichever browser is watching that job.
events.on("progress", ({ jobId, data }) => notify(jobId, { progress: data }));
events.on("completed", ({ jobId, returnvalue }) => notify(jobId, { done: true, result: returnvalue }));
events.on("failed", ({ jobId, failedReason }) => notify(jobId, { error: failedReason }));

What to watch

• Concurrency is a budget, not a speed dial. Set it from the AI provider's rate limits and the worker's memory, not from how many CPUs you have; past that point, more concurrency only turns queue time into 429s.
• Failed jobs are data. Keeping recent failures with their errors makes most production bugs a matter of reading, not reproducing.
• Keep job payloads small: an ID, not the document. The worker loads the current state when it runs, so a retry an hour later doesn't act on stale data.

The request path stays fast because it refuses to do slow work.