Queue-Based Job Processing

Problem this solves

Long-running tasks inside API requests cause timeouts and poor user experience.

When to use it

• You run document processing, report generation, or batch sync tasks.
• Task duration can exceed normal request timeouts.
• You need retry semantics and failure observability.

Code snippet

// lib/jobs/process-documents.ts
import { queue } from '@/lib/queue';

export async function enqueueDocumentProcessing(documentId: string) {
  await queue.add(
    'process-document',
    { documentId },
    { attempts: 5, backoff: { type: 'exponential', delay: 3000 } }
  );
}

queue.process('process-document', async (job) => {
  await updateStatus(job.data.documentId, 'processing');
  await runExtractionPipeline(job.data.documentId);
  await updateStatus(job.data.documentId, 'completed');
});

Integration guide

1. Enqueue work from the request path and return quickly.
2. Store job IDs and status for client polling or subscriptions.
3. Configure retries with exponential backoff and dead-letter handling.
4. Instrument queue depth, failure rate, and processing latency.