Bull Queues in NestJs

I have been working with NestJs and Bull queues individually for quite a time. Recently, I thought of using Bull in NestJs. The problem involved using multiple queues which put up following challenges:

* Abstracting each queue using modules.
* Importing queues into other modules.
* Using Bull UI for realtime tracking of queues.

Starting with async way of creating queues, for those not familiar with bull queue creation can follow https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md#queue...

BullModule.registerQueueAsync({
  name: 'queue-one',
    imports: [ConfigModule],
    useFactory: async (configService: ConfigService) => ({
    	name: 'queue-one',
        redis: configService.get('REDIS_URL'),
        prefix: 'prefix', 
        defaultJobOptions: {
        	removeOnComplete: true,
            removeOnFail: true,
        },
        settings: {
        	lockDuration: 300000,
        }
    }),
    inject: [ConfigService]
})

Earlier, I was not using ‘name’ field inside useFactory while creating queue. It led to creation of default queue in Nest’ BullModule with no name. And if you are creating more than one queues, only one queue is created: ‘default’. This caused problems when writing respective consumers for queues, as only one consumer was called every time. Moreover, Nest was not able to resolve dependencies when importing these queue modules into other modules.

Coming to the part where you can abstract these queues inside modules. It can be done as follows:

@Module({
  imports: [ConfigModule, ...queueOne],
    exports: [...queueOne, QueueOneConsumer],
    providers: [QueueOneConsumer],
})

export class QueueOneModule { }

where queueOne is array of BullModule.registerQueueAsync. As you can see, queueOne has been exported and now can be used in other modules as well. (Please make sure to add QueueOneModule in ApplicationModule).

Let’s import QueueOne into QueueTwo, QueueTwoModule would look like

@Module({
  imports: [ConfigModule, ...queueTwo],
    exports: [...queueTwo, QueueOneConsumer],
    providers: [QueueTwoConsumer],
})

export class QueueTwoConsumer { }

and QueueTwoConsumer can use @InjectQueue decorator to import QueueOne.

@Processor('queue-two')
export class QueueTwoConsumer {
  constructor (
    	@InjectQueue('queue-one') private readonly queueOne: Queue,
    ) {}   
    ...
         await this.queueOne.add('name', data, opts);

Bull UI can come handy for realtime tracking of jobs inside queues and can be easily integrated with Nest framework. Let’s start with creating a Nest module for UI

@Module({    
  imports: [QueueOneModule, QueueTwoModule],  
    exports: [],     
    providers: [QueueUIProvider],
})

export class QueueUIModule { }

And QueueUIProvider would look like

import { setQueues } from 'bull-board';
...
@Injectable()
export class QueueUIProvider {    
  constructor (     
    	@InjectQueue('queue-one') private readonly queueOne: Queue,     				@InjectQueue('queue-two') private readonly queueTwo: Queue
    ) {
    	setQueues([queueOne, queueTwo]);
    }
}

Finally, creating rest route for UI inside main.ts or index.ts (wherever is your app is bootstrapping)

import { UI as bullUI } from 'bull-board';
...
app.use('/admin/queues', basicAuth({
  users: {
    		[config.BULL_UI_USERNAME]: config.BULL_UI_PASSWORD,
    },
    challenge: true,
}),  bullUI);

Voila !! When you go to /admin/queues you can see all your queues working smoothly.