Rails activejob with Sidekiq is spawning multiple jobs on perform_later
|Added at||2016-10-11 12:10|
I have an fairly simple Rails 5 app for monitoring sites speeds. It sends jobs off to an external page speed testing service and periodically checks to see if the jobs are complete, and if so it calls and stores the data about the page.
Each project can have many pages, and each page can have many jobs, which is where the response from the testing service is stored.
There is an activejob, with a sidekiq backend, that is supposed to run every minute, check for pages that are due to run, and if any are found launch a job to enqueue it. It also checks if there are any enqueued jobs, and if they are found, it spools up a job to check the status and save data if it's complete.
#Check any jobs in the queue, then schedule the next task GetrunningtasksJob.perform_later FindjobstorunJob.set(wait: 1.minute).perform_later() end
This seems to work as expected for a while, but after 5 minutes or so two jobs seem to end up spawning at the same time. Eventually each of those spawn more of their own, and after a few days I end up with tens of thousands trying to run per hour. There's no errors or failing jobs as best I can tell, I can't find any reason why it'd be happening. Any help would be mundo appreciated :)