Beanpicker is a project mainly written in Ruby, it's free.
A stalker inspired gem to use with beanstalk
Beanpicker is a job queueing DSL for Beanstalk
beanstalk(d) is a fast, lightweight queueing backend inspired by memcached. The Ruby Beanstalk client is a bit raw, however, so Beanpicker provides a thin wrapper to make job queueing from your Ruby app easy and fun.
Yes, it is inspired in stalker and in Minion
Beanpicker work with subprocess. It create a fork for every request and destroy it in the end.
The vantages of Beanpicker are:
My machine:
The speed with 10000 requests given 'with fork' is :fork => :every and 'without fork' is :fork => :master/false
# time / requests per second / cpu load
Fork is activated by default, it should slow down your application but keep safe from memory leaks.
You can easy active or desactive the fork for a job with:
job "job.without.fork", :fork => false do |args|
debug "Running on a thread in main process"
warn "This process will grow because of any job running on main process"
end
job "job.with.fork.every.time", :fork => :every do |args|
debug "Running on a fork of main process"
debug "This process will be killed on end of this job"
debug "This decrease the peformance but save from memory leaks"
debug "All extra memory used by this process will vanish in end"
end
job "job.with.fork.once", :fork => :master do |args|
debug "Running on a fork of main process"
debug "This process will not be killed on end of this job"
debug "This increase the performance but don't save from memory leaks"
debug "This process will only grow in memory because of code executed in 'job.with.fork.once'"
end
You can pass :fork_every => true(default)/false and :fork_master => true/false(default)
The :fork argument overwrite :fork_every and :fork_master
The default :fork_every and :fork_master are setted on Beanpicker::default_fork_[master|every]
Beanpicker::fork_every and Beanpicker::fork_master overwrite the job options, so, if you set they false the jobs will run in the main thread even if they specify the :fork, :fork_every and/or :fork_master
From anywhere in your app:
require 'beanpicker'
Beanpicker.enqueue('email.send', :to => '[email protected]')
Beanpicker.enqueue('post.cleanup.all')
Beanpicker.enqueue('post.cleanup', :id => post.id)
If you have a task that requires more than one step just pass an array of queues when you enqueue.
require 'beanpicker/job_server'
Beanpicker::Worker.new do
# this is a slow job, so we'll spawn 10 forks of it :)
job "email.fetch_attachments", :childs => 10 do |args|
attachment_ids = Email.fetch_attachments_for args[:email_id]
{ :attachment_ids => attachment_ids }
end
# by default :childs is 1
job "email.send" do |args|
Email.send({
:id => args[:email_id],
:attachments => args[:attachment_ids].map { |a| Attachment.find(a) }
})
end
end
Beanpicker.enqueue(["email.fetch_attachments", "email.send"], :email_id => 10)
Inside of a job you can use debug, info, warn, error and fatal. It will be redirected to logger(STDOUT by default)
All options are inside of module Beanpicker
Beanpicker ships with "combine", "A Beanpicker server"
Try combine --help to see all options
e.g. command:
combine -r config.rb -l log/jobs.log sandwich_jobs.rb email_jobs.rb
Beanpicker look in ENV variables BEANSTALK_URL and BEANSTALK_URLS.
In BEANSTALK_URL it expect a url like "server[:port]" or "beanstalk://server[:port]".
In BEANSTALK_URLS it expect a list of urls separed by comma. e.g. "localhost,localhost:11301,10.1.1.9,10.1.1.10:3000"
Created by Renan Fernandes
Released under the MIT License