REST API - can't find workflow step_id in a workflow invocation

Question: REST API - can't find workflow step_id in a workflow invocation

3.0 years ago by

sfischer • 50

United States

sfischer • 50 wrote:

Hi,

i am using the REST API for workflows, as documented here:

https://galaxy.readthedocs.org/en/master/lib/galaxy.webapps.galaxy.api.html#module-galaxy.webapps.galaxy.api.workflows

I need to:

GET a workflow invocation
scrape its parameter values
fire off a new run of that workflow based on those parameter values. (Please don't ask why....)
(I don't know in advance which workflow or invocation to use, so i can't hard-code anything)

This entails matching the steps in the invocation with the steps in the workflow, which is what I am having trouble with.

In the JSON returned by the api/workflow/{workflow_id} endpoint, i get a bunch of steps, each with what looks like a "step index", such as 1909417:

"steps": {
 "1909417": {
    "tool_id": "Cut1",
    "tool_version": "1.0.2",
    "id": 1909417,
    "input_steps": { },
    "tool_inputs":  {
        "columnList": "{\"__class__\": \"RuntimeValue\"}",
        "input": "null",
        "delimiter": "\"T\""
    },
    "type": "tool",
    "annotation": null

},
...

The JSON to run a workflow looks like this:

{
  "workflow_id" :"ba06423ff18d37cc",
  "history" : "groovy",
  "parameters" : {"1909417" : {"columnList" : "C1,C2"}},
  "ds_map":{"1909417" : {"src" : "hda", "id": "bbd44e69cb8906b587c210f637326cca"}}
}

The parameters and ds_map are keyed with the step index from the workflow:

My problem is that when I look at a workflow invocation, or its steps in detail, or its jobs in detail, i never see that step index.

This is what the JSON for an invocation looks like (for the step above):

"steps": [
    {
        "workflow_step_uuid": "f2498dae-6e41-4dba-b1c2-74e984661980",
        "update_time": "2015-11-12T15:07:08.602957",
        "job_id": "bbd44e69cb8906b57a8032d7e2c63780",
        "state": "ok",
        "workflow_step_label": null,
        "order_index": 0,
        "action": null,
        "model_class": "WorkflowInvocationStep",
        "workflow_step_id": "925f0024d97df8bb",
        "id": "7965aefdfdc0734d"
    }

]

And here is the associated job:

{
    "tool_id": "Cut1",
    "update_time": "2015-11-12T15:08:28.150810",
    "inputs": 
{
    "input": 
    {
        "src": "hda",
        "id": "bbd44e69cb8906b587c210f637326cca",
        "uuid": "acf3626f-c89d-4c55-918c-b1a2c746ef54"
    }
},
"outputs": 
{
    "out_file1": 
    {
        "src": "hda",
        "id": "bbd44e69cb8906b5f9839f2acf8a48d6",
        "uuid": "464c5960-3ec9-44db-9141-85f22c9b9f35"
    }
},
"exit_code": 0,
"state": "ok",
"create_time": "2015-11-12T15:07:07.576559",
"params": 
    {
        "__workflow_invocation_uuid__": "\"09b2c6e0894f11e5a641005056a52a46\"",
        "delimiter": "\"T\"",
        "dbkey": "\"?\"",
        "columnList": "\"c1,c2\"",
        "chromInfo": "\"/galaxy-repl/localdata/chrom/?.len\""
    },
    "model_class": "Job",
    "id": "bbd44e69cb8906b57a8032d7e2c63780"

MY QUESTION IS: how can i correlate a step in a workflow invocation with a step in its workflow?

Thanks,

Steve

api • 1.2k views

ADD COMMENT • link •

modified 3.0 years ago • written 3.0 years ago by sfischer • 50

3.0 years ago by

Dannon Baker ♦ 3.7k

United States

Dannon Baker ♦ 3.7k wrote:

So the problem here is that the workflow endpoint does not encode step
ids, while the invocation one does. I'll look into the right way to
resolve this going forward (probably encode at both endpoints), though
we need to do it in a backwards compatible way.

Anyway, in short, if you want to do this *right now*, the easiest
thing to do would be to use the encoding/decoding routines from Galaxy
to manually decode the workflowinvocation step ids to match back up to
workflow steps.

ADD COMMENT • link written 3.0 years ago by Dannon Baker ♦ 3.7k

3.0 years ago by

sfischer • 50

United States

sfischer • 50 wrote:

Dannon,

Thanks very much. I had a guess that was it, but couldn't find it in the code base.

I do think it would be good, as you suggest, for galaxy to encode this value in the workflow info returned by the API call, to make it consistent with the invocation.

In the meantime, I can try the workaround you suggest.

Can you point me to the method that does the encoding? Which file is it in, and which method is it?

Clarification: I am using the REST API remotely, so my program does not have access to the galaxy codebase. Instead, to do the recommended workaround, I'll need to duplicate the encoding algorithm in my program. To do that, I need to see exactly how galaxy is doing its encoding.

Thanks very much,

Steve

ADD COMMENT • link modified 3.0 years ago • written 3.0 years ago by sfischer • 50

Yep, so what you'll want to do is use the SecurityHelper from galaxy.web.security (lib/galaxy/web/security/__init__.py)

In [1]: from galaxy.web.security import SecurityHelper

In [2]: sh = SecurityHelper(id_secret="USING THE DEFAULT IS NOT SECURE!")

In [3]: sh.decode_id("6fc9fbb81c497f69")

Out[3]: 82

(this, of course, only works if you know the id_secret for the server in question -- just saw your followup direct email)

ADD REPLY • link modified 3.0 years ago • written 3.0 years ago by Dannon Baker ♦ 3.7k

Dannon,

Thanks.

It looks like this workaround isn't going to work for me:

1) i am operating remotely, so cannot import that code. (i am running my code, via REST, against usegalaxy.org)

2) i tried this in perl, locally:

% perl -e 'use Crypt::Blowfish; my $c = new Crypt::Blowfish "USING THE DEFAULT IS NOT SECURE!"; $e = $c->encrypt("!2091734"); print unpack("H16", $e), "\n";'

55fdda0bb0ad06b1

that value is not the right answer.

maybe the problem is that usegalaxy.org is not using the above value for its secret? it seems like without the id_secret, I can't do the workaround, is that right?

i sure hope you guys can fix this bug soon :)

Steve

ADD REPLY • link modified 3.0 years ago • written 3.0 years ago by sfischer • 50

Yep, I assumed you were talking about a server you administered. The code was just an example -- usegalaxy.org definitely does not use the default secret_id and nobody else should either :)

I'm trying to think of another workaround for you, since even if I pushed a changeset flipping the encoding today it would be some time before we'd get it into the stable branch and update usegalaxy.org.

ADD REPLY • link written 3.0 years ago by Dannon Baker ♦ 3.7k

Hi Dannon,

I have been prototyping on usegalaxy.org. You're right we will soon have our own instance running.

While we wait for this to be fixed in the stable branch, can you provide a patch we can apply to our (upcoming) instance? Just let me know what file to edit, and what the change would be. I imagine it might be a single line of code.

Similar posts • Search »