This is an attempt at writing an Array method that runs a block on each of its elements on a farm.

Syntax might look like this:

  list_of_files.parallelize(:queue => 'urgent',
                            :max_concurrent => 5,
                            :select => '[mem >= 1000]') do |file_name|
    File.open(file_name).each do |line|
      # do something
    end
  end

There should also be a way to know what the exit code of the job is.

This would be equivalent to the approach followed at the moment:

• Write a script that takes a filename as argument (e.g. called script.rb).

  File.open(ARGV[0]).each do |line|
  # do something
  end
  

• Write a bash script that calls script.rb for each file:

  script.rb file_1.txt
  script.rb file_2.txt
  script.rb file_3.txt
  script.rb file_4.txt
  ...
  

• Submit this bash script as a job array:

  bsub -J"foobar[1-50]%10" ...