Question: Workflow import error - from release 16.07, to release 17.01
1
matt.chambers42 • 50 wrote:
I'm trying to import workflows from a 16.07 instance to a 17.01 one. The tools are slightly different, so I'm getting a lovely Internal Server Error which tells me nothing. So I go to the log and find more detail that is almost, but not entirely, completely unhelpful:
129.114.104.5 - - [24/May/2017:12:13:47 -0500] "POST /workflow/import_workflow HTTP/1.0" 500 - "http://129.114.18.61/workflow/import_workflow" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
Error - <type 'exceptions.KeyError'="">: 'input_opts'
URL: http://129.114.18.61/workflow/import_workflow
File '/opt/galaxy/galaxy-app/lib/galaxy/web/framework/middleware/error.py', line 154 in __call__
app_iter = self.application(environ, sr_checker)
File '/opt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/paste/recursive.py', line 85 in __call__
return self.application(environ, start_response)
File '/opt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/paste/httpexceptions.py', line 640 in __call__
return self.application(environ, start_response)
File '/opt/galaxy/galaxy-app/lib/galaxy/web/framework/base.py', line 134 in __call__
return self.handle_request( environ, start_response )
File '/opt/galaxy/galaxy-app/lib/galaxy/web/framework/base.py', line 193 in handle_request
body = method( trans, **kwargs )
File '/opt/galaxy/galaxy-app/lib/galaxy/webapps/galaxy/controllers/workflow.py', line 956 in import_workflow
workflow, missing_tool_tups = self._workflow_from_dict( trans, data, source=src, add_to_menu=add_to_menu )
File '/opt/galaxy/galaxy-app/lib/galaxy/web/base/controller.py', line 1180 in _workflow_from_dict
exact_tools=exact_tools,
File '/opt/galaxy/galaxy-app/lib/galaxy/managers/workflows.py', line 206 in build_workflow_from_dict
exact_tools=exact_tools,
File '/opt/galaxy/galaxy-app/lib/galaxy/managers/workflows.py', line 298 in _workflow_from_dict
module, step = self.__track_module_from_dict( trans, steps, steps_by_external_id, step_dict, exact_tools=exact_tools )
File '/opt/galaxy/galaxy-app/lib/galaxy/managers/workflows.py', line 850 in __track_module_from_dict
module, step = self.__module_from_dict( trans, step_dict, exact_tools=exact_tools )
File '/opt/galaxy/galaxy-app/lib/galaxy/managers/workflows.py', line 897 in __module_from_dict
module.save_to_step( step )
File '/opt/galaxy/galaxy-app/lib/galaxy/workflow/modules.py', line 880 in save_to_step
step.tool_inputs = self.tool.params_to_strings( self.state.inputs, self.trans.app )
File '/opt/galaxy/galaxy-app/lib/galaxy/tools/__init__.py', line 1378 in params_to_strings
return params_to_strings( self.inputs, params, app )
File '/opt/galaxy/galaxy-app/lib/galaxy/tools/parameters/__init__.py', line 149 in params_to_strings
value = params[ key ].value_to_basic( value, app )
File '/opt/galaxy/galaxy-app/lib/galaxy/tools/parameters/grouping.py', line 91 in value_to_basic
rval_dict[ input.name ] = input.value_to_basic( d[input.name], app )
KeyError: 'input_opts'
CGI Variables
-------------
CONTENT_LENGTH: '531'
CONTENT_TYPE: 'multipart/form-data; boundary=----WebKitFormBoundary3FMOXG9DsshH6EHO'
HTTP_ACCEPT: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
HTTP_ACCEPT_ENCODING: 'gzip, deflate'
HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4,so;q=0.2,hu;q=0.2'
HTTP_CACHE_CONTROL: 'max-age=0'
HTTP_CONNECTION: 'close'
HTTP_COOKIE: 'galaxysession=c6ca0ddb55be603a14cbe07075a87ab2ebb9de99b46f305bd035b4cf143e2201a77f6cdf42ac961a'
HTTP_HOST: '129.114.18.61'
HTTP_ORIGIN: 'http://129.114.18.61'
HTTP_REFERER: 'http://129.114.18.61/workflow/import_workflow'
HTTP_UPGRADE_INSECURE_REQUESTS: '1'
HTTP_USER_AGENT: 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'
PATH_INFO: '/workflow/import_workflow'
REMOTE_ADDR: '129.114.104.5'
REQUEST_METHOD: 'POST'
SERVER_NAME: '127.0.0.1'
SERVER_PORT: '8080'
SERVER_PROTOCOL: 'HTTP/1.0'
WSGI Variables
--------------
application: <paste.recursive.RecursiveMiddleware object="" at="" 0x7f757f01bf90="">
controller_action_key: u'web.workflow.import_workflow'
is_api_request: False
paste.cookies: (<SimpleCookie: galaxysession="c6ca0ddb55be603a14cbe07075a87ab2ebb9de99b46f305bd035b4cf143e2201a77f6cdf42ac961a">, 'galaxysession=c6ca0ddb55be603a14cbe07075a87ab2ebb9de99b46f305bd035b4cf143e2201a77f6cdf42ac961a')
paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'="">]
paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object="" at="" 0x7f757f01bf10="">
paste.httpserver.proxy.host: 'dummy'
paste.httpserver.proxy.scheme: 'http'
paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object="" at="" 0x7f757fb1d210="">
paste.recursive.forward: <paste.recursive.Forwarder from=""/>
paste.recursive.include: <paste.recursive.Includer from=""/>
paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from=""/>
paste.recursive.script_name: ''
paste.throw_errors: True
request_id: '590675c640a411e78eacfa163e8ef98e'
webob._body_file: (<_io.BufferedReader>, <socket._fileobject object="" at="" 0x7f758c077250="" length="531">)
webob._parsed_post_vars: (MultiDict([('url', u'http://js-156-29.jetstream-cloud.org/u/chambm/w/human-sequence-database-search-variant-confirmation-jetstream-imported-from-uploaded-file/json'), ('file_data', u''), ('import_button', u'Import')]), <_io.BytesIO object at 0x7f7580124a10>)
webob._parsed_query_vars: (GET([]), '')
webob.is_body_seekable: True
wsgi process: 'Multithreaded'
------------------------------------------------------------
Now, if I could see the argument values in the traceback instead of just the argument names, it should give me a strong idea about which part of the workflow is causing the problem. Apparently the inspect or cgitb modules in python will accomplish this, but I have no idea where to insert it so that it will take effect for the whole of Galaxy.
ADD COMMENT
• link
•
modified 18 months ago
by
Jennifer Hillman Jackson ♦ 25k
•
written
18 months ago by
matt.chambers42 • 50
After a few false starts, I found that editing some code in lib/galaxy/web/framework/middleware/error.py achieved my desired effect. It is quite verbose because the workflow dictionary is huge, but it actually allowed me to pinpoint the problematic tool: Query to Tabular.
Here is the original error, cgitb style:
Will the workflow load if the latest version of the tool is installed first? https://toolshed.g2.bx.psu.edu/view/jjohnson/query_tabular/49ca06d66762
You may be doing that already, but just checking.
There have been some changes in the way dependencies are handled during tool installs in Galaxy version 17.01 and further improved in version 17.05 (the most current release). Upgrading the Galaxy instance would also be a good idea and might be enough to handle this case (load the workflow successfully, then edit it to resolve tool mismatches between the two Galaxy servers), but that would need to be tested.
I posted as an answer intentionally. :) My main concern was with being able to more quickly clean information about exceptions from the Galaxy log (now and in the future). Although I should have posted the code change I made!
To fix this problem, I tried to manually update the JSON syntax of the old workflow (which used Query Tabular 0.1.3 while new instance has 3.0.0). I made a new workflow with 3.0.0 and copied parameters manually. Then I downloaded the new workflows' JSON and copied the tool_state part over to the old workflow. No dice, still got the input_opts KeyError when importing. So I tried importing the new workflow and I got the same error, i.e. I couldn't import the workflow I had just exported a minute earlier. So this has to be a Galaxy bug. I can't try this on main because it doesn't have Query Tabular installed. I'm running a Jetstream instance and 17.05 isn't an option yet. This is on 17.01.
Np :) I wanted to make sure we got this settled out as a usage versus code problem (that needs to be fixed) before this dropped off as answered.
I'll recruit some developer help for more troubleshooting. Thanks for the follow up! Jen