Sidekiq is an open-source job scheduler written in Ruby. It uses Redis as a data store for queue management and job processing.
Have you ever wondered how sidekiq uses Redis to store and process these jobs?? Well, let me go through it step by step.
How are queues stored in Redis?
Each queue is stored as a separate key inside the Redis data store. The key is the queue name with the text "queue:" as the prefix. For example, the key for the default queue is "queue:default"
You can list the available queues in Redis using this command>> $redis.keys("queue*")[ [0] "queue:default",[1] "queue:test" ]
Adding a job to the queue:
Sidekiq uses the perform_async method to push the job inside a queue for it to run asynchronously. The job is stored as a JSON object inside the Redis queue.
You can add a job to the Redis queue using this command>> TestJob.perform_async("my_attributes")
or>> TestJob.perform_async
By default, the job goes into the default queue. We can override the queue name using sidekiq_options for a custom queue. In this case, the job will go inside the test queue while calling the perform_async method on the TestJob.
>> class TestJob>> include Sidekiq::Worker>> sidekiq_options queue: :test
>> end
We can also override the queue name at run time by setting the queue name during job creation.
>> TestJob.set(queue: 'new_queue').perform_async("my_attributes")
Note*: perform method runs the job synchronously without pushing it to the queue, whereas perform_async pushes the job inside the queue for it to run asynchronously.
perform method can be used as mentioned below:*
>> TestJob.new.perform("my_attributes")
What happens in Redis while adding a job?
On calling a perform_async method, the sidekiq converts the job data into a JSON string and adds it to the job list for a queue in Redis. The new job gets added to the end of this list on creation.
You can check the list of waiting jobs for a queue in Redis using the below command.
>> $redis.lrange(key, start, stop)
To get a list of all waiting jobs in the test queue, run the below command>> $redis.lrange("queue:test", 0, -1)[ [0] "{"retry":false,"queue":"test","class":"TestJob","args":["my_attributes"],"jid":"9df2c184146ab351609c0974","created_at":1675228990.2853692,"enqueued_at":1675228990.2855182}",[1] "{"retry":false,"queue":"test","class":"TestJob","args":[],"jid":"9aa982bfe1df0cdcac771eda","created_at":1675228901.3451672,"enqueued_at":1675228901.345802}" ]
Deleting a job from the queue
Jobs from a sidekiq queue can be deleted by iterating linearly through the queue.>> queue = Sidekiq::Queue.new("test")>> queue.each do |job|>> job.klass # => 'TestJob'>> job.args # => ["my_attributes"]>> job.delete if job.jid == '9df2c184146ab351609c0974' >> end
But is it worth it to delete a job from the queue?
Deleting the job from sidekiq is a very cumbersome operation as it uses a linear search to find the job in Redis and then deletes it. Thus to delete a job in the queue, ruby first searches for that job in this list linearly and then deletes its first occurrence. This happens for every single job, making the complexity o(n2). This can lead to a lot of Redis connection issues while trying to delete the jobs from large queues.
Hope, this was helpful. Happy Coding!!!