Commit b8ff28c8 authored by Audrey Gibert's avatar Audrey Gibert
Browse files
parents 13164c27 ff867bc4
......@@ -15,6 +15,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
# TEST de software update
1. WHAT IS NG6
......
......@@ -67,7 +67,9 @@ dbname = typo3
[storage]
# where should be written the log file
log_file = <path>/ng6.log
# Where should ngspipelines data be stored
#level of logs to display in log_file
log_level = logging.DEBUG|logging.WARNING|logging.ERROR|...
# Where should data be stored
save_directory = <path>/ng6
# Where should the pipelines write results, should be accessible
# by all cluster nodes
......@@ -103,7 +105,7 @@ user_base_directory = /work/
#fastx_reverse_complement = /usr/bin/fastx_reverse_complement
#STAR = /usr/bin/STAR
#fastuniq = /usr/local/bin/fastuniq
#
#javaPICARD = /tools/java/jre1.8.0_45/bin/java
# workflow specific
#
......@@ -150,6 +152,8 @@ user_base_directory = /work/
#BWA.batch_options = -q unlimitq
#AlignmentStats.batch_options = -q unlimitq
#ContaminationSearch.batch_options = -q unlimitq
#Trim_porechop.batch_options = --mem=4G -p wflowq -t 30
#Trim_porechop.modules = compiler/gcc-7.2.0,bioinfo/Porechop-0.2.1
# Set workflows group
[workflows]
......
......@@ -53,8 +53,8 @@
<li class="active"><a href="./advanced-configuration.html">Advanced configuration</a></li>
</ul>
</li>
<li><a target="_blank" href="https://mulcyber.toulouse.inra.fr/plugins/mediawiki/wiki/ng6/index.php/Accueil">Sources</a></li>
<li><a href="mailto:support.genopole@toulouse.inra.fr">Contact</a></li>
<li><a target="_blank" href="https://forgemia.inra.fr/genotoul-bioinfo/ng6/">Sources</a></li>
<li><a href="mailto:support.bioinfo.genotoul@inra.fr">Contact</a></li>
</ul>
</div><!--/.nav-collapse -->
</div>
......
......@@ -53,8 +53,8 @@
<li><a href="./advanced-configuration.html">Advanced configuration</a></li>
</ul>
</li>
<li><a target="_blank" href="https://mulcyber.toulouse.inra.fr/plugins/mediawiki/wiki/ng6/index.php/Accueil">Sources</a></li>
<li><a href="mailto:support.genopole@toulouse.inra.fr">Contact</a></li>
<li><a target="_blank" href="https://forgemia.inra.fr/genotoul-bioinfo/ng6/">Sources</a></li>
<li><a href="mailto:support.bioinfo.genotoul@inra.fr">Contact</a></li>
</ul>
</div><!--/.nav-collapse -->
</div>
......@@ -100,7 +100,7 @@
<section id="what" class="group">
<h1 class="page-header">What is an Analysis</h1>
<p>
An analysis presents the results of the execution of one or multiple softwares, which can be external scripts or python code, using NGS data.
An analysis presents the results of the execution of one or multiple software, which can be external scripts or python code, using NGS data.
It is represented by two main objects :
</p>
<ul>
......@@ -113,7 +113,7 @@
<span class="glyphicon glyphicon-exclamation-sign" aria-hidden="true"></span>
In a NG6Workflow, the developper can also add instances of <code>jflow.component.Component</code> object. It is recommended to use <code>Component</code>
when the result of the execution has not to be presented to the final user (for example indexing a reference file) and <code>Analysis</code> when
results must be presented to the user, like a fastqc report. The differences are the necessity of developping <span style="text-decoration: underline;">define_analysis()</span>, and <span style="text-decoration: underline;">post_process()</span> functions and a <span style="text-decoration: underline;">template HTML</span> file for <code>Analysis</code> component.
results must be presented to the user, like a fastqc report. The differences are the necessity to develop <span style="text-decoration: underline;">define_analysis()</span>, <span style="text-decoration: underline;">post_process()</span> functions and a <span style="text-decoration: underline;">template HTML</span> file for <code>Analysis</code> component.
</div>
</section>
......@@ -121,16 +121,16 @@
<section id="analysis-architecture" class="group">
<h1 class="page-header">Where to add a new Analysis</h1>
<p>
The new analysis must be added in a Python package. Two different locations are possible in order to be imported by ng6:
The new analysis must be added in a Python package. Two different locations are possible in order to be imported by ng6 :
</p>
<ul>
<li><code>workflows.components</code>: the analysis will be visible by all workflows,</li>
<li><code>workflows.myWorkflow.components</code>: the analysis will only be available for<code>myWorkflow</code>.</li>
</ul>
<p>
The template file presenting the results of the analysis must be added in ui/nG6/pi1/analyzes/. The tempalte file must
The template file presenting the results of the analysis must be added in ui/nG6/pi1/analyzes/. The template file must
have the same name as the Python class of the Analysis.
The following code represent the structure of the source, and the location where to add specific files.
The following code represents the structure of the source, and the location where to add specific files.
</p>
<pre class="pre-hl "><code class="ini">nG6/
......@@ -971,7 +971,7 @@ class MyComponent (Analysis):
Overloading <code>process()</code> can be omitted. NG6 offers, for analyzes and components with easy command lines,
an automatic built of the <code>process()</code> method.
In this case, options <code>argpos</code> and <code>cmd_format</code> must be provided by the developper for each parameter.
Also, two other methods of must be overloaded. <code>get_command()</code>, which must returns
Also, two other methods must be overloaded. <code>get_command()</code>, which must returns
the execution path and <code>get_abstraction()</code> which returns the abstraction name to use.
</div>
......@@ -1256,7 +1256,7 @@ fastaTrim.pl --length 50 splC.fasta > splC_trim.fasta</code></pre>
<h3>MultiMap</h3>
<p>The <code>MultiMap</code> abstraction allows to map n input file lists to n output file lists, all of the same lenght.</p>
<p>The <code>MultiMap</code> abstraction allows to map n input file lists to n output file lists, all of the same length.</p>
<h4>Example</h4>
<pre class="pre-hl "><code class="python">fastq_list = ["splA.fastq", "splB.fastq", "splC.fastq"]
......
......@@ -53,8 +53,8 @@
<li><a href="./advanced-configuration.html">Advanced configuration</a></li>
</ul>
</li>
<li><a target="_blank" href="https://mulcyber.toulouse.inra.fr/plugins/mediawiki/wiki/ng6/index.php/Accueil">Sources</a></li>
<li><a href="mailto:support.genopole@toulouse.inra.fr">Contact</a></li>
<li><a target="_blank" href="https://forgemia.inra.fr/genotoul-bioinfo/ng6/">Sources</a></li>
<li><a href="mailto:support.bioinfo.genotoul@inra.fr">Contact</a></li>
</ul>
</div><!--/.nav-collapse -->
</div>
......
......@@ -53,8 +53,8 @@
<li><a href="./advanced-configuration.html">Advanced configuration</a></li>
</ul>
</li>
<li><a target="_blank" href="https://mulcyber.toulouse.inra.fr/plugins/mediawiki/wiki/ng6/index.php/Accueil">Sources</a></li>
<li><a href="mailto:support.genopole@toulouse.inra.fr">Contact</a></li>
<li><a target="_blank" href="https://forgemia.inra.fr/genotoul-bioinfo/ng6/">Sources</a></li>
<li><a href="mailto:support.bioinfo.genotoul@inra.fr">Contact</a></li>
</ul>
</div><!--/.nav-collapse -->
</div>
......
......@@ -50,8 +50,8 @@
<li><a href="./advanced-configuration.html">Advanced configuration</a></li>
</ul>
</li>
<li><a target="_blank" href="https://mulcyber.toulouse.inra.fr/plugins/mediawiki/wiki/ng6/index.php/Accueil">Sources</a></li>
<li><a href="mailto:support.genopole@toulouse.inra.fr">Contact</a></li>
<li><a target="_blank" href="https://forgemia.inra.fr/genotoul-bioinfo/ng6/">Sources</a></li>
<li><a href="mailto:support.bioinfo.genotoul@inra.fr">Contact</a></li>
</ul>
</div><!--/.nav-collapse -->
</div>
......@@ -106,7 +106,7 @@
<p><a class="btn btn-primary" href="./core-workflow.html" role="button">View details &raquo;</a></p>
</div>
<div class="col-lg-6">
<p><a class="btn btn-primary" href="./core-component.html" role="button">View details &raquo;</a></p>
<p><a class="btn btn-primary" href="./core-anaysis.html" role="button">View details &raquo;</a></p>
</div>
</div>
......
......@@ -53,8 +53,8 @@
<li><a href="./advanced-configuration.html">Advanced configuration</a></li>
</ul>
</li>
<li><a target="_blank" href="https://mulcyber.toulouse.inra.fr/plugins/mediawiki/wiki/ng6/index.php/Accueil">Sources</a></li>
<li><a href="mailto:support.genopole@toulouse.inra.fr">Contact</a></li>
<li><a target="_blank" href="https://forgemia.inra.fr/genotoul-bioinfo/ng6/">Sources</a></li>
<li><a href="mailto:support.bioinfo.genotoul@inra.fr">Contact</a></li>
</ul>
</div><!--/.nav-collapse -->
</div>
......@@ -138,7 +138,7 @@ CREATE USER 'typo3'@' %' IDENTIFIED BY 'typo3';
GRANT ALL ON typo3.* TO 'typo3'@' %';
#Get last ng6 source code
svn checkout svn://scm.mulcyber.toulouse.inra.fr/svnroot/ng6/nG6
Download the latest version from https://forgemia.inra.fr/genotoul-bioinfo/ng6/tags/
</code></pre>
Now go to step #4.
</div>
......@@ -146,7 +146,7 @@ Now go to step #4.
<section id="download">
<h1 class="page-header">Step #2 <small>download the latest copy</small></h1>
<p>The latest version can be downloaded from the <a href="https://mulcyber.toulouse.inra.fr/frs/?group_id=102" target="_blank">download page</a>.</p>
<p>The latest version can be downloaded from the <a href="https://forgemia.inra.fr/genotoul-bioinfo/ng6/tags/" target="_blank">page</a>.</p>
</section>
<section id="extract">
......@@ -184,6 +184,7 @@ dbname = typo3
[storage]
# where should be written the log file
log_file = /path/to/jflow.log
log_level = logging.INFO
# Where should the pipelines write results, should be accessible
# by all cluster nodes
work_directory = /path/to/work
......
......@@ -53,8 +53,8 @@
<li><a href="./advanced-configuration.html">Advanced configuration</a></li>
</ul>
</li>
<li><a target="_blank" href="https://mulcyber.toulouse.inra.fr/plugins/mediawiki/wiki/ng6/index.php/Accueil">Sources</a></li>
<li><a href="mailto:support.genopole@toulouse.inra.fr">Contact</a></li>
<li><a target="_blank" href="https://forgemia.inra.fr/genotoul-bioinfo/ng6/">Sources</a></li>
<li><a href="mailto:support.bioinfo.genotoul@inra.fr">Contact</a></li>
</ul>
</div><!--/.nav-collapse -->
</div>
......
......@@ -53,8 +53,8 @@
<li><a href="./advanced-configuration.html">Advanced configuration</a></li>
</ul>
</li>
<li><a target="_blank" href="https://mulcyber.toulouse.inra.fr/plugins/mediawiki/wiki/ng6/index.php/Accueil">Sources</a></li>
<li><a href="mailto:support.genopole@toulouse.inra.fr">Contact</a></li>
<li><a target="_blank" href="https://forgemia.inra.fr/genotoul-bioinfo/ng6/">Sources</a></li>
<li><a href="mailto:support.bioinfo.genotoul@inra.fr">Contact</a></li>
</ul>
</div><!--/.nav-collapse -->
</div>
......
......@@ -22,8 +22,6 @@ import os
from jflow.config_reader import JFlowConfigReader
# Define some Error classes
class InvalidFormatError(Exception): pass
jflowconf = JFlowConfigReader()
......@@ -32,7 +30,9 @@ log_directory = os.path.dirname(jflowconf.get_log_file_path())
if not os.path.isdir(log_directory):
os.makedirs(log_directory, 0o751)
logging.basicConfig(level=logging.DEBUG,
#log_level = eval(jflowconf.get_log_level())
#if (log_level = logging.DEBUG
logging.basicConfig(level=eval(jflowconf.get_log_level()),
format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
datefmt='%m-%d %H:%M',
filename=jflowconf.get_log_file_path(),
......
......@@ -27,7 +27,66 @@ from weaver.options import Options
from weaver.abstraction import Abstraction
class MultiMap(Abstraction):
class AbstractionWargs(Abstraction):
""" The base Abstraction class.
**Positional Arguments**:
- `function` -- Function to apply (Function, string, string format)
**Keyword Arguments**:
- `inputs` -- Inputs to function
- `outputs` -- Output of function
- `includes` -- Files to include for each task.
- `native` -- Whether or not to use native abstraction if available.
- `group` -- Number of tasks to inline.
- `collect` -- Whether or not to mark files for garbage collection.
- `local` -- Whether or not to force local execution.
`inputs` and `includes` are parsed using
:func:`~weaver.data.parse_input_list` and must be in a form acceptable tot
hat function. Likewise, `outputs` is parsed by
:func:`~weaver.data.parse_output_list` and `function` is parsed by
:func:`~weaver.function.parse_function`.
"""
Counter = None
def __init__(self, function, inputs=None, outputs=None, includes=None,
native=False, group=None, collect=False, local=False, arguments=None):
Abstraction.__init__(self, function, inputs, outputs, includes, native, group, collect, local)
self.arguments = arguments
class Map(AbstractionWargs):
""" Weaver Map Abstraction.
This Abstraction enables the following pattern of execution:
Map(f, inputs, outputs)
In this case, the :class:`Function` *f* is applied to each item in
*inputs* to generate the corresponding *outputs*.
"""
Counter = itertools.count()
@cache_generation
def _generate(self):
with self:
debug(D_ABSTRACTION, 'Generating Abstraction {0}'.format(self))
function = parse_function(self.function)
inputs = parse_input_list(self.inputs)
outputs = parse_output_list(self.outputs, inputs)
includes = parse_input_list(self.includes)
for i, o in zip(inputs, outputs):
with Options(local=self.options.local, collect=[i] if self.collect else None):
yield function(i, o, self.arguments, includes)
class MultiMap(AbstractionWargs):
""" Weaver MultiMap Abstraction.
This Abstraction enables the following pattern of execution:
......@@ -110,5 +169,5 @@ class MultiMap(Abstraction):
iteration_outputs = parse_output_list(self.outputs, input_pattern)
with Options(local=self.options.local):
yield function(iteration_inputs, iteration_outputs, None, includes)
yield function(iteration_inputs, iteration_outputs, self.arguments, includes)
#
# Copyright (C) 2015 INRA
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
import argparse
class JflowArgumentParser (argparse.ArgumentParser):
def _read_args_from_files(self, arg_strings):
# expand arguments referencing files
new_arg_strings = []
for arg_string in arg_strings:
# if it's not a comment or an empty line
if not arg_string.startswith("#") and arg_string:
# for regular arguments, just add them back into the list
if not arg_string or arg_string[0] not in self.fromfile_prefix_chars:
new_arg_strings.append(arg_string)
# replace arguments referencing files with the file content
else:
try:
with open(arg_string[1:]) as args_file:
arg_strings = []
# give to the convert_arg_line_to_args a table of lines instead of line per line
for arg in self.convert_arg_line_to_args(args_file.read().splitlines()):
arg_strings.append(arg)
arg_strings = self._read_args_from_files(arg_strings)
new_arg_strings.extend(arg_strings)
except OSError:
err = _sys.exc_info()[1]
self.error(str(err))
# return the modified argument list
return new_arg_strings
\ No newline at end of file
......@@ -21,6 +21,7 @@ import sys
import inspect
import tempfile
import types
import shutil
from operator import attrgetter
......@@ -33,26 +34,36 @@ from jflow.abstraction import MultiMap
from weaver.util import parse_string_list
from weaver.function import ShellFunction
from weaver.abstraction import Map
from jflow.abstraction import Map
from weaver.function import PythonFunction
from pathlib import Path
import inspect
class Component(object):
TRACE_FILE_NAME = "trace.txt"
def __init__(self):
self.prefix = "default"
self.__prefix = "default"
self.params_order = []
self.output_directory = None
self.description = None
self.config_reader = JFlowConfigReader()
try:
self.prg_name = self.get_command()
except NotImplementedError:
self.prg_name = self.__class__.__name__
self.version = self.get_version()
if isinstance(self.version, bytes):
self.version = self.version.decode()
self.batch_options = self.config_reader.get_component_batch_options(self.__class__.__name__)
self.modules = self.config_reader.get_component_modules(self.__class__.__name__)
# in case of SGE, parse the cpu and memory parameter
self.__cpu=None
self.__memory=None
self.__memory_per_cpu=None
type, options, limit_submission = self.config_reader.get_batch()
if type.lower() == "sge" :
try:
......@@ -68,13 +79,34 @@ class Component(object):
try:
self.__memory = re.match( r'.*\s?mem=(\d+\w)\s?.*', self.batch_options).group(1)
except: pass
elif type.lower() == "slurm" :
try:
self.__cpu = int(re.match( r'.*-c\s+(\d+).*', self.batch_options).group(1))
except: pass
try:
self.__memory = re.match( r'.*--mem=(\d+\S+)\s?.*', self.batch_options).group(1)
except: pass
try:
self.__memory_per_cpu = re.match( r'.*--mem-per-cpu=(\d+\S+)\s?.*', self.batch_options).group(1)
except: pass
def get_prefix(self):
return self.__prefix
def set_prefix(self, prefix):
self.__prefix = prefix
def get_description(self):
return self.description
def get_cpu(self):
return self.__cpu
def get_memory(self):
return self.__memory
def get_memory_per_cpu(self):
return self.__memory_per_cpu
def is_dynamic(self):
return len(self.get_dynamic_outputs()) != 0
......@@ -92,11 +124,13 @@ class Component(object):
outputs = {}
for attribute_value in list(self.__dict__.values()):
if ( issubclass( attribute_value.__class__, DynamicOutput ) or
issubclass( attribute_value.__class__, OutputFileList) ):
issubclass( attribute_value.__class__, OutputFileList)):
for f in attribute_value:
outputs[os.path.basename(f)] = f
elif issubclass( attribute_value.__class__, OutputFile):
outputs[os.path.basename(attribute_value)] = attribute_value
elif issubclass( attribute_value.__class__, OutputDirectory):
outputs[os.path.basename(attribute_value)] = attribute_value
return outputs
def add_input_directory(self, name, help, default=None, required=False, flag=None,
......@@ -128,7 +162,11 @@ class Component(object):
def reset(self):
for file in os.listdir(self.output_directory):
os.remove(os.path.join(self.output_directory, file))
final_file = os.path.join(self.output_directory, file)
if os.path.isdir(final_file):
shutil.rmtree(final_file)
else:
os.remove(final_file)
def add_input_file_list(self, name, help, file_format="any", default=None, type="inputfile",
required=False, flag=None, group="default", display_name=None,
......@@ -236,6 +274,17 @@ class Component(object):
# add it to the class itself
self.params_order.append(name)
self.__setattr__(name, new_param)
def add_output_directory(self, name, help, dirname=None, group="default", display_name=None,
cmd_format="", argpos=-1):
dirname = os.path.basename(dirname)
new_param = OutputDirectory(name, help, default=os.path.join(self.output_directory, dirname),
group=group, display_name=display_name, cmd_format=cmd_format, argpos=argpos)
# store where the parameter is coming from
new_param.linkTrace_nameid = self.get_nameid()
# add it to the class itself
self.params_order.append(name)
self.__setattr__(name, new_param)
def add_output_file(self, name, help, file_format="any", filename=None, group="default", display_name=None,
cmd_format="", argpos=-1):
......@@ -393,7 +442,7 @@ class Component(object):
for e in file :
commandline += ' %s $%s ' % (file.cmd_format, cpt)
cpt+=1
function = ShellFunction( commandline, cmd_format='{EXE} {IN} {OUT}')
function = ShellFunction( commandline, cmd_format='{EXE} {IN} {OUT}', modules=self.modules)
function(inputs=inputs, outputs=outputs)
# weaver map abstraction
elif abstraction == 'map' :
......@@ -410,7 +459,7 @@ class Component(object):
if isinstance(file, ParameterList) :
outputs = file
function = ShellFunction( commandline, cmd_format='{EXE} {IN} {OUT}')
function = ShellFunction( commandline, cmd_format='{EXE} {IN} {OUT}', modules=self.modules)
exe = Map(function, inputs=inputs, outputs=outputs)
# jflow multimap
......@@ -422,7 +471,7 @@ class Component(object):
commandline += ' %s $%s ' % (file.cmd_format, cpt)
cpt+=1
function = ShellFunction( commandline, cmd_format='{EXE} {IN} {OUT}')
function = ShellFunction( commandline, cmd_format='{EXE} {IN} {OUT}', modules=self.modules)
exe = MultiMap(function, inputs=inputs, outputs=outputs)
# anything other than that will be considered errored
else :
......@@ -477,16 +526,16 @@ class Component(object):
raise Exception("'" + software + "' path connot be retrieved either in the PATH and in the application.properties file!")
elif exec_path is None and which(software) != None:
exec_path = software
elif exec_path != None and not os.path.isfile(exec_path):
elif exec_path != None and not os.path.exists(exec_path):
logging.getLogger("jflow").exception("'" + exec_path + "' set for '" + software + "' does not exists, please provide a valid path!")
raise Exception("'" + exec_path + "' set for '" + software + "' does not exists, please provide a valid path!")
return exec_path
def get_nameid(self):
return self.__class__.__name__ + "." + self.prefix
return self.__class__.__name__ + "." + self.__prefix
def __eq__(self, other):
return self.__class__ == other.__class__ and self.prefix == other.prefix
return self.__class__ == other.__class__ and self.__prefix == other.get_prefix()
def __getattribute__(self, attr):
# an IOobject is a specific object defined by the presence of the dump_path attribute
......@@ -540,12 +589,13 @@ class Component(object):
return new_ios,includes
def add_python_execution(self, function, inputs=[], outputs=[], arguments=[], includes=[],
add_path=None, collect=False, local=False, map=False, cmd_format=""):
add_path=set(), collect=False, local=False, map=False, cmd_format=""):
workflow_dir = Path(os.path.dirname(inspect.getfile(self.__class__))).parent
lib_dir = str(workflow_dir) + os.path.sep + "lib"
add_path.add(lib_dir)
if map:
if arguments != [] :
logging.getLogger("jflow").exception("add_python_execution: '" + function.__name__ + "' arguments parameter not allowed with map!")
raise Exception("add_python_execution: '" + function.__name__ + "' arguments parameter not allowed with map!" )
if not issubclass(inputs.__class__, list) or not issubclass(outputs.__class__, list):
logging.getLogger("jflow").exception("add_python_execution: '" + function.__name__ + "' map requires a list as inputs and output!")
raise Exception("add_python_execution: '" + function.__name__ + "' map requires a list as inputs and output!")
......@@ -558,7 +608,7 @@ class Component(object):
cmd_format += " {IN}"
if (isinstance(outputs, list) and len(outputs)>0) or (outputs is not None and outputs != []):
cmd_format += " {OUT}"
py_function = PythonFunction(function, add_path=add_path, cmd_format=cmd_format)
py_function = PythonFunction(function, add_path=add_path, cmd_format=cmd_format, modules=self.modules)
new_inputs,includes_in = self.__generate_iolist(inputs, map)
......@@ -566,25 +616,22 @@ class Component(object):
if not isinstance(includes, list):
includes=[includes]
if map:
MultiMap(py_function, inputs=new_inputs, outputs=new_outputs, includes=includes+includes_in, collect=collect, local=local)
MultiMap(py_function, inputs=new_inputs, outputs=new_outputs, includes=includes+includes_in,
collect=collect, local=local, arguments=arguments)
else:
py_function(inputs=new_inputs, outputs=new_outputs, arguments=arguments, includes=includes+includes_in)
self.__write_trace(function.__name__, inputs, outputs, arguments, cmd_format, map, "PythonFunction")
def add_shell_execution(self, source, inputs=[], outputs=[], arguments=[], includes=[],
cmd_format=None, map=False, shell=None, collect=False, local=False):
shell_function = ShellFunction( source, shell=shell, cmd_format=cmd_format )
shell_function = ShellFunction(source, shell=shell, cmd_format=cmd_format, modules=self.modules)
# if abstraction is map or multimap
if map :
# if input and output are list or filelist
if issubclass(inputs.__class__, list) and issubclass(outputs.__class__, list) :
# arguments cannot be set with
if arguments != [] :
logging.getLogger("jflow").exception("add_shell_execution: '" + source + "' arguments parameter not allowed with map")
raise Exception("add_shell_execution: '" + source + "' arguments parameter not allowed with map" )
MultiMap(shell_function,inputs=inputs, outputs=outputs, includes=includes, collect=collect, local=local)
MultiMap(shell_function,inputs=inputs, outputs=outputs, includes=includes, collect=collect, local=local, arguments=arguments)
else :
logging.getLogger("jflow").exception("add_shell_execution: '" + source + "' map requires a list as inputs and output")
raise Exception("add_shell_execution: '" + source + "' map requires a list as inputs and output")
......@@ -613,12 +660,11 @@ class Component(object):
if isinstance(element[0], list):
for i in range(len(element)) :
to_write+="List"+str(i+1)+": \n"
to_write+="\n".join(element[i])+"\n"
to_write+="\n".join([str(x) for x in element[i]])+"\n"
else :
to_write+="\n".join(element)+"\n"
to_write+="\n".join([str(x) for x in element])+"\n"
else :
to_write+= element+"\n"
to_write+= str(element)+"\n"
if to_write != "" :
fh.write(title+" :\n")
fh.write(to_write)
\ No newline at end of file