Running Galaxy jobs remotely via ssh / rsh

Question: Running Galaxy jobs remotely via ssh / rsh

2.9 years ago by

United States

danielfortin86 • 110 wrote:

Hi,

I would like to run the Galaxy framework on server A, while performing all of the jobs on server B using ssh.

Looking at the documentation here: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#CLI, this seems like it should be possible. However, the documentation states that the cli runner requires, at a minimum, two parameters, one for the shell (which I'm selecting SecureShell) and a Job plugin. I'm not sure what this should be since the ones available are Torque, Slurm, and SlurmTorque and I'm not running any of these. Can anyone give me any hints? My current job_conf.xml looks like this:

<?xml version="1.0"?>
<job_conf>
<plugins>

<plugin id="cli" type="runner" load="galaxy.jobs.runners.cli:ShellJobRunner"/>
</plugins>
<handlers>
<handler id="main"/>
</handlers>
<destinations default="cli_default">
<destination id="cli_default" runner="cli">
<param id="shell_plugin">SecureShell</param>
<param id="job_plugin">cli</param>
<param id="shell_hostname">computehost</param>
<param id="shell_username">galaxy</param>
</destination>
</destinations>
</job_conf>

galaxy.ini secureshell cli job_conf.xml cluster • 996 views

ADD COMMENT • link •

modified 2.9 years ago • written 2.9 years ago by danielfortin86 • 110

2.9 years ago by

matthias.desmet • 150

Belgium

matthias.desmet • 150 wrote:

I don't think what you want is really possible.

You're going to need te setup a job submission system (e.g. Torque or Slurm) if both servers have shared storage resources.
If both servers don't share any resources you'll need to setup a Pulsar jobqueue.

For more info, you can go over the page you linked in your question.

Greetings
M

ADD COMMENT • link written 2.9 years ago by matthias.desmet • 150

2.9 years ago by

danielfortin86 • 110

United States

danielfortin86 • 110 wrote:

As an alternative, in a tool xml I tried directly sshing between the <command> tags e.g.:

<command>ssh galaxy@serverB 'sh $__tool_directory__/runTool.sh --verbose > $output 2>&1' </command>

This appears to work. Is it possible to "inject" this ssh command into tool execution in Galaxy?

ADD COMMENT • link written 2.9 years ago by danielfortin86 • 110

Do you have any shared resources between both servers? A networked drive that is mapped to both?

I don't really think the method you propose is going to work.
Perhaps it would be best is you submit this question to galaxy-dev@lists.galaxyproject.org (just send an email to it). There are more accomplished Galaxy developers and admins there who will be able to help you out.

ADD REPLY • link written 2.9 years ago by matthias.desmet • 150

Yes, there is a networked drive that is mapped in the same location for both servers.

Thanks for the suggestion to submit the question to the galaxy-dev list. I will do that!

ADD REPLY • link written 2.9 years ago by danielfortin86 • 110

If you install a jobqueue on the second server (eg torque) you can submit your jobs over the DRMAA jobrunner.

You can easily use one server as both submission and execution node (We do the same for development purposes)
you can reference pages like this on (https://help.ubuntu.com/community/TorquePbsHowto) on how to install a jobqueue.

M

ADD REPLY • link written 2.9 years ago by matthias.desmet • 150

Please log in to add an answer.

Similar posts • Search »