FAQ Database Discussion Community


Sidekiq Rails 4.2 Use Active Job or Worker? What's the difference

ruby-on-rails,asynchronous,sidekiq,whenever,rails-activejob
This is my first processing jobs asynchronously I am implementing Sidekiq for background processing in my app. I will use it for reminder emails and in-app notifications. I am confused as to whether I should use Active Job to create a job that sends an email or a Sidekiq Worker...

Using view_context inside the Mailer

ruby-on-rails,sidekiq,rails-activejob
I'm using ActiveJob to send mails: Using deliver_now method: invoices_controller.rb def send_invoice #other stuff Members::InvoicesMailer.send_invoice(@invoice.id, view_context).deliver_now end invoices_mailer.rb require 'open-uri' class Members::InvoicesMailer < ApplicationMailer def send_invoice(invoice_id, view_context) @invoice = Invoice.find(invoice_id) attachments["#{@invoice.identifier}.pdf"] = InvoicePdf.new(@invoice, view_context).render mail :to => @invoice.client.email, :subject => "Invoice" end end Notice here that I'm...

How to clear ActiveJob queue

sidekiq,rails-activejob
I have jobs that are trying and failing repeatedly..how to I clear them? I now have the retries count set to 0, but that doesn't help the jobs that are already queued.

Sidekiq: how to pass a queue as a variable into a worker

sidekiq
I have a worker like so: class MassDispatchServiceWorker include Sidekiq::Worker sidekiq_options queue: :bulk def perform(message_params) DispatchService.new(message_params).dispatch! end end However I have a need for the same exact method but to pass in a queue as a variable. Sometimes I want the queue to be bulk, other times I'll need something...

Kill multiple Sidekiq jobs from the same worker

ruby-on-rails,sidekiq
I would like to know how to kill many Sidekiq jobs from the same worker at once. I deployed a bug on a production environment and there are queued jobs that are bugging out. I can simply fix the bug and deploy again, but the jobs are time-sensitive (they send...

Using Sidekiq with Nokogiri for scraping

ruby-on-rails,nokogiri,delayed-job,sidekiq
I'm using Rails with Nokogiri. I have some heavy scraping tasks that I would like to execute in the background with Sidekiq. The problem is, I followed the three steps mentioned on sidekiq.org but nothing happened. What am I missing? What follows is one of my scrapes without using Sidekiq,...

Sidekiq worker's responsibility

ruby-on-rails,sidekiq
I would like to know if it is a best design practice to put many database interactions in Sidekiq. I have a service object that delivers SMS messages to User. It just retrieves id of users and calls .perform_async on a Sidekiq worker class. The SMS contains many information about...

Do sidekiq jobs run inside a large transaction?

ruby-on-rails,ruby,sidekiq
I have a large record import job running on our web application. The import is run by a PORO which calculates how many objects it needs to import and divides the number by an index of the current object it is presently on, leaving me with a clean way to...

Paperclip with Sidekiq not working

ruby-on-rails,paperclip,sidekiq
I have created a Sidekiq worker that will duplicate an existing record with Paperclip attachment, but it doesn't seem to be working. #controller product = Product.find(2) SuperWorker.perform_in(5.minutes, product.id) #worker class SuperWorker include Sidekiq::Worker def perform(product_id) product = Product.find(product_id) product.generate_clone end end #product model ... has_attached_file :front_image, :styles => { :medium...

How to avoid duplicates from saving in database parsed from external JSON file with sidekiq in Rails

ruby-on-rails,json,ruby-on-rails-4,sidekiq
I have a small To Do list in a .json file that I´m reading, parsing, and saving to a rails app with Sidekiq. Everytime I refresh the browser, the worker executes and duplicates the entries on the database. How do I maintain a unique database that is synchronized with the...

Display image uploaded to S3 with Carrierwave/Sidekiq after form submit

ruby-on-rails,amazon-s3,carrierwave,sidekiq
I'm using Sidekiq with CarrierWave and CarrierWaveBackgrounder to upload my images to S3 in the background. This works fine except for one thing: after I submit my form, the image gets enqueued in Sidekiq and processed, but the page reload happens faster than the background job, resulting in a 404...

Get value of atomic counter (increment) with Rails and Postgres

ruby-on-rails,postgresql,rails-activerecord,atomic,sidekiq
I need to atomically increment a model counter and use its new value (processed by a Sidekiq job). At the moment, I use Group.increment_counter :tasks_count, @task.id in my model which atomically increments the counter. But I also need its new value to send a notification, if the counter has e.g....

Rails 4.2 - Sidekiq not sending emails in development

ruby,ruby-on-rails-4,actionmailer,sidekiq,rails-activejob
I have a rails app in which I have a method where I send a lot of emails. I would like to perform this action asynchronously. To do it I've tried to use Sidekiq, but I can't get it to work properly - it doesn't send any emails. Sending email...

rails block a record for change for another places in the code

ruby-on-rails,ruby,postgresql,activerecord,sidekiq
I have a ruby on rails application with a lot of sidekiq workers. Some of workers can work for a while (at least few minutes). How can I block some record for changes from another places (ie controllers), to avoid data conflict when I save this record in the worker?...

perform_async not working in sidekiq

ruby,sidekiq
I'm trying to use sidekiq to process some requests in background (it's integrated with padrino framework). My expectation was that once the worker got called, the request handler would return immediately the answer to user. In order to be sure it was working properly I implemented a worker that would...

Rails minify/optimize files on the fly using Grunt or Gulp

ruby-on-rails,gruntjs,gulp,sidekiq
My Rails 4 application allow it's users to import some Adobe Edge animation files (advertisement). While it works, Adobe Edge is loading a bunch of files that could be minified and optimized (images) using a tool like Grunt or Gulp. I'm using CarrierWave to upload the files and then I...

Clear memory after some number of records passed

ruby-on-rails,ruby,garbage-collection,sidekiq
I have few m of records in db, and need to process it from time to time. However this operation takes all memory on my server. I'm running this operation using sidekiq. So while this task using all memory, my rails app becomes very slow. In general(no logic included) my...

Sidekiq with Clockwork triggers but doesn't process job on heroku

ruby-on-rails,heroku,sidekiq,clockwork
Running: sidekiq 2.1.17, rails 3.2.21 I'm trying to use clockwork to schedule some recurring tasks to be done by sidekiq workers. Running foreman in development, everything runs perfectly as scheduled. When I deploy to heroku, however, I get the "Triggering 'NameWorker.perform_async'" message in the logs at the appropriate times but...

Block/Re-queue other sidekiq jobs from processing when existing sidekiq job is processing a particular resource

ruby-on-rails,postgresql,ruby-on-rails-4,synchronized,sidekiq
I have sidekiq jobs doing processing on a many types of resources. However, for a particular type of resource, eg: Resource X, I need to ensure that only 1 sidekiq job can process that particular resource at any given time. For example, if i have 3 sidekiq jobs that gets...

Disable automatic retry with ActiveJob, used with Sidekiq

ruby-on-rails,sidekiq,rails-activejob
Is there a way to disable automatic retry with ActiveJob and Sidekiq ? I know that with Sidekiq only, we just have to put sidekiq_options :retry => false as mentioned here : https://github.com/mperham/sidekiq/wiki/Error-Handling#configuration but it doesn't seem to work with ActiveJob and Sidekiq. I also know the solution to entierly...

How to prevent parallel Sidekiq jobs from executing code in Rails

ruby-on-rails,multithreading,sidekiq,rails-activejob
I have around 10 workers that performs a job that includes the following: user = User.find_or_initialize_by(email: '[email protected]') if user.new_record? # ... some code here that does something taking around 5 seconds or so elsif user.persisted? # ... some code here that does something taking around 5 seconds or so end...

Prevent sidekiq from executing queued up jobs when starting from command line?

ruby-on-rails,sidekiq
When I start sidekiq in my development environment (Rails 3.2), I use the following command: bundle exec sidekiq When I do this, sidekiq will execute all jobs that have been queued up when it was not running. e.g. If I have created a bunch of new user accounts during testing,...

Sidekiq / Rails : PG::ConnectionBad: PQconsumeInput() SSL error: system lib

postgresql,ruby-on-rails-4,sidekiq
I'm getting this error from sidekiq / rails / postgresql combination after it processes about 2000 jobs. PG::ConnectionBad: PQconsumeInput() SSL error: system lib It's on simple / random SQL queries, sometimes the ActiveRecord table schema queries. Things that work OK for 2000 or so queries suddenly start to fail for...

how to import millions of users without compromise sidekiq

ruby-on-rails,ruby,redis,sidekiq
I have a rake task like this task import_all: :environment do Person.find_each do |p| UserWorker.perform_async(p.to_global_id) end end each user creates around 20 new jobs on Sidekiq and these 20 jobs also create more 4 or 5 new jobs. Turns out I have 980k users and my redis is using almost...

Sidekiq process failing

ruby-on-rails,ruby,sidekiq
I have a rails 4.1 application with Sidekiq 3.3.2 in production mode with default config. Everything works fine, but sometimes, I see in Sidekiq admin panel that Sidekiq process, not listed in the "busy" in the panel and does not exist in the list of processes. That seems like a...

Is stats.queues method removed in Sidekiq 3.3.1? How to fetch queues now?

ruby-on-rails,heroku,sidekiq
I just upgraded to Rails 4.1.9 and Sidekiq 3.3.1 (running on Heroku). I've been using the Sidekiq API and based on the documentation https://github.com/mperham/sidekiq/wiki/API the stats.queues method should still be there, but in practice it doesn't seem to be: irb(main):001:0> stats = Sidekiq::Stats.new => #<Sidekiq::Stats:0x007f73db350290 @stats={:processed=>464, :failed=>3, :scheduled_size=>2, :retry_size=>0, :dead_size=>2,...

Rails / Rspec: How would I test #any? for Sidekiq?

rspec,sidekiq
I have a block: def already_syncing? Sidekiq::Queue.new(queue_name).any? { |q| q.args[0] == car_id } end How would I test this? If I stub it out like this: allow(Sidekiq::Queue).to receive_message_chain(:new, :any?) { true } Then the condition in my actual block is not tested....

Sidekiq + Rails: How to see queue of “Busy” jobs?

ruby-on-rails,sidekiq
I have a script that looks for jobs: Sidekiq::Queue.new(queue_name).any? { |q| q.args[0] == blah_id } || Sidekiq::RetrySet.new.any? { |r| r.queue == queue_name && r.args[0] == blah_id } || Sidekiq::ScheduledSet.new.any? { |s| s.queue == queue_name && s.args[0] == blah_id } However, this doesn't look inside of Busy: How would I look...

What kinds of issues can I expect to see when using Sidekiq with 100+ queues?

ruby,sidekiq
The Sidekiq documentation recommends against having more than a few queues. I'm considering attempting to migrate a Resque setup that currently uses 107 queues. Can anyone provide a more detailed explanation of why I shouldn't use Sidekiq with 107 queues? The above linked documentation just says that Sidekiq is "is...

Postgres connection not closing after sidekiq Ruby script

ruby-on-rails,ruby,postgresql,sidekiq
It's a small Ruby script running under Sidekiq. It opens a connection with db_connect = Sequel.connect(@db_credential, search_path: @namespace) It never explicitly closes the connection; I think this is not supposed to be necessary? After the script has has been run many times, and they have all completed, and the Sidekiq...

Using multiple sidekiq databases

ruby-on-rails,ruby-on-rails-3,sidekiq
Hey I am attempting to spawn a sidekiq worker that connects to a completely separate Redis database. I know with 3.0's connection pooling this is possible, and I have been able to successfully push a job onto the correct Redis DB, but the problem is the Sidekiq web UI is...

How to set multiple pidfile in sidekiq.yml file?

ruby-on-rails,queue,task,jobs,sidekiq
From the official wiki page of sidekiq I found: https://github.com/mperham/sidekiq/wiki/Advanced-Options --- :concurrency: 5 :pidfile: tmp/pids/sidekiq.pid staging: :concurrency: 10 production: :concurrency: 20 :queues: - default - [myqueue, 2] Now I want to create multiple tasks using the same sidekiq.yml file. But use different pid name, such as: /tmp/pids/task1.pid /tmp/pids/task2.pid How to...

Efficiently reschedule ActiveJob (resque/sidekiq)

ruby-on-rails,resque,sidekiq,rails-activejob
I'm playing with Rails 4.2 app which uses ActiveJob backed by resque/sidekiq for email scheduling. When a user creates newsletter campaign a new job is created and scheduled on certain date. That's all great but what happens when the user changes the delivery date. In this case every job could...

Golang go-workers custom logging middleware?

logging,go,sidekiq,middleware
I'm building a Golang app that implements a Sidekiq-compatible jrallison/go-workers work queue and a custom logging wrapper around Sirupsen/logrus for marshaled JSON logs. Now, all of my app (except for go-workers so far) uses my logger-wrapper in a central place to ensure that 100% of its output is JSON compatible....

delayed_paperclip is not hitting sidekiq

paperclip,sidekiq,rails-activejob
I have set up delayed paperclip and sidekiq, however when I try the photo upload again, the log's out put is below: [ActiveJob] Enqueued DelayedPaperclip::Jobs::ActiveJob (Job ID: f9bf0100-2c5b-4c30-af04-9fb9b4ee8c5e) to Inline(paperclip) with arguments: "CarPhoto", 32, "image" [ActiveJob] [DelayedPaperclip::Jobs::ActiveJob] [f9bf0100-2c5b-4c30-af04-9fb9b4ee8c5e] Performing DelayedPaperclip::Jobs::ActiveJob from Inline(paperclip) with arguments: "CarPhoto", 32, "image" [ActiveJob] [DelayedPaperclip::Jobs::ActiveJob]...

Sidekiq keeps rebooting Cloud66

ruby-on-rails,redis,sidekiq,bluepill,cloud66
I have been struggling with this issue for a while and just can't figure it out. I am trying to get Redis and Sidekiq to handle background jobs for my Rails Project hosted on Cloud66 w/ Digital Ocean. All needed gems seem to be present and the setup works perfectly...

How can I detect whether my code is running “inside” Sidekiq server or Puma?

ruby,ruby-on-rails-4,connection-pooling,sidekiq,puma
I'm using Puma as a web server, and Sidekiq as my queue runner. For multiple things (Database connections, Redis connections, other external services) I'm using the ConnectionPool gem to manage safe access to connections. Now, depending on whether I'm running in the context of Sidekiq or of Puma, I need...

How to set up 'workers' server responsible for running jobs from another servers

ruby-on-rails,ruby,redis,sidekiq
Let's say that we have 2 servers. One app server where is http server, app server and application, it would be opened for world. Another server worker is supposed to run the Sidekiq jobs only (has application, Redis and Sidekiq). How to configure the Sidekiq to enqueue jobs from app...

rails, Cannot run sidekiq

ruby-on-rails,sidekiq
I have created sidekiq.yml file inside config folder: --- :concurrency: 25 :pidfile: ./tmp/pids/sidekiq.pid :logfile: ./log/sidekiq.log :queues: -default I have added foreman gem in my Rails project and created Procfile in my root folder: web: bundle exec rails s worker: sidekiq -C config/sidekiq.yml When I run foreman in my command line...

Sidekiq ActiveJob track spawned jobs in

ruby-on-rails,sidekiq
A lot of time I will have background worker code that looks like this. class MyJob < ActiveJob::Base def perform User.find_in_batches.each do |batch| MyOtherJobToProcessBatchOfUsers.perform_later(batch) end end end Basically, one jobs that spawns many child jobs. Is there an easy way to know when all of the jobs are finished? I...

Sidekiq and Rails model validations

ruby-on-rails-4,activerecord,sidekiq
I have a model validation on :create that validates a certain attribute is unique and I create many elements using some background jobs on sidekiq using 50 concurrency on a pool of 50 mysql connections. For some reasons I see many duplicates where I have the validation, is there a...

Sidekiq not processing more than one queue

ruby-on-rails,redis,sidekiq
I have successfully configured Sidekiq for my use but now it is not processing more than one queue. It is only processing the queue that is written first in configuration file. For instance: --- :verbose: true :concurrency: 100 :pidfile: tmp/pids/sidekiq.pid staging: :concurrency: 10 production: :concurrency: 20 :queues : - [kritikal...

Not start the queue by wisper-sidekiq

ruby-on-rails,sidekiq,wisper
I'm trying to make a small example with wisper-sidekiq gem. The queue is created, but it does not start. Why? I will describe the steps that you are doing, I think it will be easier. I hope for your help. A. controller: class BooksController < ApplicationController def create service =...

Rails 4 Pass Arguments To Sidekiq Worker

ruby-on-rails,sidekiq
I am new to Sidekiq and currently using a worker to send push notifications asynchronously. Push notifications are sent when a message is sent by a user to another. Since Sidekiq best practices are to make worker params small and simple, is it fine to pass the message text (<140...

Odd behaviour with sidekiq returned jid on a mailer when pushing into a serialised field

ruby-on-rails,ruby,ruby-on-rails-4,sidekiq
I am getting some odd behaviour when trying to push a returned jid from a worker into a serialised field on a model. Best explained by this code: class Foo < ActiveRecord::Base serialize :jids, Array def do_job test_jids = [] jids << "s" test_jids << "s" Rails.logger.info("s has been pushed")...

Error starting sidekiq on Heroku, missing sidekiq.log file crash

ruby-on-rails,ruby-on-rails-4,heroku,redis,sidekiq
I'm new to using sidekiq & redis and am trying to get sidekiq to run on Heroku, but it seems to crash each time I deploy my Rails 4 app. I get this error in Papertrail: May 03 11:16:37 myapp app/worker.1: No such file or directory @ rb_sysopen - ./log/sidekiq.log...

Testing Sidekiq/ActiveJob - Job in ActiveJob but not Sidekiq

ruby-on-rails,sidekiq,rails-activejob
I monkey patched my user class (backed by devise) to use ActiveJob like so: class User < ActiveRecord::Base # Omitted def send_devise_notification(notification, *args) devise_mailer.send(notification, self, *args).deliver_later end end I tried testing this with the following test: class UserTest < ActiveSupport::TestCase include ActiveJob::TestHelper def setup @user = User.new(email: '[email protected]', password: 'password',...

Can't connect to redis on heroku

ruby-on-rails,ruby,heroku,redis,sidekiq
I'm using sidekiq with active_job in my rails application. The job currently works but I am unable to get connect to the Redis when deploying to heroku. In heroku I have the RedisToGo addon. When pushing to heroku I get the following error. error /app/vendor/bundle/ruby/2.2.0/gems/redis-3.2.1/lib/redis/client.rb:331:in `rescue in establish_connection': Error connecting...

How do I have a ruby program call into a rails app?

ruby-on-rails,ruby,irc,sidekiq,cinch
I want to write a cinch IRC bot that sends information to a rails application. The rails application will essentially function as a monitor for users, and theoretically the data displayed will continually refresh. I have the basic IRC bot working on it's own and have written standalone rails applications...