quickstart.html 47 KB
Newer Older
Penom Nom's avatar
Penom Nom committed
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <title>NG6</title>

    <!-- Bootstrap core CSS -->
    <link href="css/bootstrap.min.css" rel="stylesheet">
    <!-- Bootstrap theme -->
    <link href="css/bootstrap-theme.min.css" rel="stylesheet">

    <!-- Custom styles for this template -->
    <link href="css/theme.css" rel="stylesheet">

	<!--  Highlight code -->
	<link href="css/tomorrow-night-eighties.css" rel="stylesheet">

    <!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries -->
    <!--[if lt IE 9]>
      <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
      <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
    <![endif]-->
  </head>

  <body role="document">
  
    <!-- Fixed navbar -->
    <div class="navbar navbar-inverse navbar-fixed-top" role="navigation">
      <div class="container">
        <div class="navbar-header">
          <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target=".navbar-collapse">
            <span class="sr-only">Toggle navigation</span>
            <span class="icon-bar"></span>
            <span class="icon-bar"></span>
            <span class="icon-bar"></span>
          </button>
          <a class="navbar-brand" href="./index.html">NG6</a>
        </div>
        <div class="navbar-collapse collapse">
          <ul class="nav navbar-nav">
            <li><a href="./index.html">About</a></li>
            <li><a href="./install.html">Install</a></li>
            <li class="active"><a href="./quickstart.html">Quick start</a></li>
            <li class="dropdown">
              <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation<span class="caret"></span></a>
              <ul class="dropdown-menu" role="menu">
                <li><a href="./core-workflow.html">Add a workflow</a></li>
                <li><a href="./core-analysis.html">Add an analysis</a></li>
                <li><a href="./type.html">Add a data type</a></li>
                <li><a href="./format.html">Add a file format</a></li>
                <li><a href="./advanced-configuration.html">Advanced configuration</a></li>
              </ul>
            </li>
56 57
			<li><a target="_blank" href="https://forgemia.inra.fr/genotoul-bioinfo/ng6/">Sources</a></li>
			<li><a href="mailto:support.bioinfo.genotoul@inra.fr">Contact</a></li>
Penom Nom's avatar
Penom Nom committed
58 59 60 61 62
          </ul>
        </div><!--/.nav-collapse -->
      </div>
    </div>

Penom Nom's avatar
Penom Nom committed
63 64 65 66 67 68
	<div class="container">
      <div class="row">
 		<!--Nav Bar -->
	    <nav class="col-xs-3 bs-docs-sidebar">
	        <ul id="sidebar" class="nav nav-stacked fixed">
		    	<li><a href="#introduction" class="active">Introduction</a></li>
69
		    	<li><a href="#first_step">Step 1: folder tree</a></li>
70
		    	
71
		    	<li><a href="#second_step">Step 2: first component</a>
Penom Nom's avatar
Penom Nom committed
72
		    		<ul class="nav nav-stacked">
73 74 75 76
		    			<li><a href="#QilluminaFilter_analysis">QilluminaFilter analysis</a></li>
			       		<li><a href="#QilluminaFilter_template">QilluminaFilter template</a></li>
	                </ul>
		    	</li>
77
		    	<li><a href="#third_step">Step 3: second component</a>
78 79 80
		    		<ul class="nav nav-stacked">
		    			<li><a href="#QFastQC_analysis">QFastQC analysis</a></li>
			       		<li><a href="#QFastQC_template">QFastQC template</a></li>
Penom Nom's avatar
Penom Nom committed
81 82
	                </ul>
		    	</li>
83 84
		    	<li><a href="#fourth_step">Step 4: workflow </a></li>
		    	<li><a href="#fifth_step">Step 5: workflow test</a></li>
Penom Nom's avatar
Penom Nom committed
85 86 87 88 89 90 91 92
	        </ul>
		</nav>
        
      	
      	<div class="col-xs-12 col-sm-9">
			<section id="introduction" class="group">
				<h1 class="page-header">Introduction <small>to the quickstart</small></h1>
				<p>
93
					The following <code>quickstart</code> intends to implement a ng6workflow which will filter
Jerome Mariette's avatar
Jerome Mariette committed
94 95
					reads using <a target="_blank" href="http://cancan.cshl.edu/labmembers/gordon/fastq_illumina_filter/">fastq_illumina_filter</a>. 
					The workflow will also run <code>fastqc</code> after filtering. Two analyses with two templates will be created: 
Penom Nom's avatar
Penom Nom committed
96
				</p>
97 98 99 100 101 102 103 104

				<ul>
					<li><code>QIlluminaFilter</code> will be the analysis used to run the <code>fastq_illumina_filter</code> binary. Its template
						will present the number of sequence before and after filtering. </li>
					<li><code>QFastQC</code> which will execute the <code>fastqc</code> binary on filtered FASTQ files coming from <code>QIlluminaFilter</code>. 
					    The template file <code>QFastQC.tpl</code> will present some of the generated charts of fastqc.</li>
				</ul>
				
Penom Nom's avatar
Penom Nom committed
105
				<p>
Jerome Mariette's avatar
Jerome Mariette committed
106
					A NG6Workflow will be created to link those analyses together.
Penom Nom's avatar
Penom Nom committed
107
				</p>
108
				
Penom Nom's avatar
Penom Nom committed
109 110 111 112
			</section>
			
			
			<section id="first_step" class="group">
Jerome Mariette's avatar
Jerome Mariette committed
113
				<h1 class="page-header">Step #1 <small>create the folder tree</small></h1>
Penom Nom's avatar
Penom Nom committed
114 115
				
				<p>
Jerome Mariette's avatar
Jerome Mariette committed
116
					The first thing to do before implementing the analyses and the NG6Workflow, is to create the folder tree. 
117
					A <a href="./core-workflow.html">ng6workflow</a> is a <code>Python package</code> defined by a folder with a <code>__init__.py</code> file.
Penom Nom's avatar
Penom Nom committed
118 119
				</p>
				<p>
120
					Within NG6 sources, add a package named <code>quickstart</code> and within this package create an other package named <code>components</code> (with <code>__init__.py</code> inside)
Jerome Mariette's avatar
Jerome Mariette committed
121 122 123
					where all the analyses and components specific to <code>quickstart</code> will be stored. Note that if the analysis is shared by multiple workflows, you should 
					add it in the <code>workflows/components/</code> folder. The template files for the visualization will be created in <code>ui/nG6/pi1/analyses/</code>.
					These templates can have associated javascript files that must be created in the same directory.
Penom Nom's avatar
Penom Nom committed
124 125
				</p> 
				<p>
126
					The list of files that have to be created are : 
Penom Nom's avatar
Penom Nom committed
127 128 129
				</p>
				
				<ul>
130
					<li><code>qfastqc.py</code> in <code>workflows/quickstart/components/</code> for the <code>fastqc</code> analysis.</li>
Jerome Mariette's avatar
Jerome Mariette committed
131
					<li><code>QFastQC.tpl</code> in <code>ui/nG6/pi1/analyses/</code> for the <code>fastqc</code> template.</li>
132
					<li><code>qilluminafilter.py</code> in <code>workflows/quickstart/components/</code> for the <code>fastq_illumina_filter</code> analysis.</li>
Jerome Mariette's avatar
Jerome Mariette committed
133
					<li><code>QIlluminaFilter.tpl</code> in <code>ui/nG6/pi1/analyses/</code> for the <code>fastq_illumina_filter</code> template.</li>
Penom Nom's avatar
Penom Nom committed
134 135 136
				</ul>
				
				<p>
Jerome Mariette's avatar
Jerome Mariette committed
137
					You should obtained the following folder tree:
Penom Nom's avatar
Penom Nom committed
138
				</p> 
139 140
				<div>
					<pre class="pre-hl"><code class="ini">ng6/
Penom Nom's avatar
Penom Nom committed
141 142 143 144 145 146
├── bin/
├── docs/
├── src/
├── ui/
│   └── nG6/
│       └── pi1/
Jerome Mariette's avatar
Jerome Mariette committed
147
│           └── analyses/
148 149
│               ├── QIlluminaFilter.tpl [ QIlluminaFilter template file ]
│               └── QFastQC.tpl         [ QFastQC template file ]
Penom Nom's avatar
Penom Nom committed
150 151
├── workflows/
│   ├── components/
152
│   │   ├── __init__.py             [ empty __init__.py file  to make this directory a package ]
Penom Nom's avatar
Penom Nom committed
153
│   ├── extparsers/
154 155 156 157 158 159
│   ├── quickstart/                 [ the folder quickstart ]
│   │   ├── components/             [ the folder components where the analysis will be]
│   │   │   ├── __init__.py         [ empty __init__.py file to make this directory a package ]
│   │   │   ├── qfastqc.py          [ empty qfastqc.py file for the QFastQC anaysis ]
│   │   │   ├── qilluminafilter.py  [ empty qilluminafilter.py file for the QIlluminaFilter anaysis ]
│   │   ├── __init__.py             [ empty __init__.py file for the ng6workflow definition ]
Penom Nom's avatar
Penom Nom committed
160 161 162 163 164
│   ├── __init__.py
│   ├── formats.py
│   └── types.py
├── applications.properties
└── README</code></pre>				
165 166
				</div>
				
Penom Nom's avatar
Penom Nom committed
167 168 169 170
			</section>
			
			
			<section id="second_step" class="group">
171
				<h1 class="page-header">Step #2 <small>Create QilluminaFilter analysis</small></h1>
Penom Nom's avatar
Penom Nom committed
172 173
				
				<p>
Jerome Mariette's avatar
Jerome Mariette committed
174
					An analysis in NG6 consist of two files : 
Penom Nom's avatar
Penom Nom committed
175 176
				</p>
				<ul>
177
					<li>a <code>Python</code> class that will contain the process of the analysis and define wich information should be presented in the interface</li>
Jerome Mariette's avatar
Jerome Mariette committed
178
					<li>a template file, named excalty as the python class name, which is a smarty template that will present the results</li>
Penom Nom's avatar
Penom Nom committed
179 180 181
				</ul>
				
				<p>
182
					The <code>Python</code> class must inherit <code>ng6.analysis.Analysis</code>. While writting the <a href="./core-analysis.html">Analysis</a>, several methods must be overloaded:
Penom Nom's avatar
Penom Nom committed
183 184 185
				</p>
				
				<ul>
186
					<li><code>define_parameters()</code> which will describe the inputs, outputs and arguments of the analysis.</li>
Penom Nom's avatar
Penom Nom committed
187 188 189 190 191 192
					<li><code>define_analysis()</code> which is used to describe the analysis.</li>
					<li><code>get_version()</code> to return the version of this analysis.</li>
					<li><code>process()</code> to define the command line to be executed.</li>
					<li><code>post_process()</code> to make add results to the database.</li>
				</ul>
				
193 194 195 196 197 198
				<p>
					First a <code>Python</code> file <code>qilluminafilter.py</code> will be written for the analysis, and then the template file associated with this 
					analysis will be created.
				</p>
				
				<div id="QilluminaFilter_analysis" class="subgroup">
Jerome Mariette's avatar
Jerome Mariette committed
199
					<h2>QilluminaFilter analysis</h2>
200 201
					
					<p>
202
						The version of Illumina's CASAVA pipeline (Version 1.8) produces FASTQ files with both reads that pass filtering and reads that don't.
203 204 205 206 207 208
						The new READ-ID contains many new fields, one of them indicates whether the read is filtered or not. The <code>fastq_illumina_filter</code> program can 
						filter FASTQ files produced by CASAVA 1.8, and keep/discard reads based on this filter flag. 
					</p>
					
					<p>
						The command line that will be executed with <code>fastq_illumina_filter</code> looks like:
209 210 211 212 213 214 215 216 217 218 219 220 221 222 223
					</p>
					<div>
						<pre class="pre-hl "><code class="ini">fastq_illumina_filter --keep [Y/N] -v -o output_filtered.fastq input.fastq > input.stdout</code></pre>
					</div>
					<p>
						Where  <code>--keep</code> option tells which type of read must be keeped (N for reads which pass the filter and Y with read which do not pass the filter),
						<code>-v</code> allows to print a repport in STDOUT, <code>-o</code> specify the filtered output FASTQ file. <code>> input.stdout</code> allows to
						redirect the output of <code>-v</code> option to a file, this output contains information abound the number of initial and filtered sequences.
					</p>
					
					<p>
						Splitting the command line will give the following parameters, input and outputs:
					</p>
					
					<ul>
224 225 226 227
						<li><code>keep</code> which will tells which type of reads whould be keeped.</li>
						<li><code>output_filtered</code> which is the filtered output FASTQ files.</li>
						<li><code>input</code> which is the FASTQ files used as input file.</li>
						<li><code>stdout</code> which will be the file containing information generated using <code>-v</code> option.</li>
228 229 230
					</ul>
					
					<p>
Jerome Mariette's avatar
Jerome Mariette committed
231
						All those parameters must be added in <code>define_parameters()</code> method as following: 
232 233 234
					</p>
					
					<div>
Jerome Mariette's avatar
Jerome Mariette committed
235
						<pre class="pre-hl "><code class="python">from ng6.analysis import Analysis
236 237 238 239 240 241 242 243 244 245 246 247 248 249
	
class QIlluminaFilter (Analysis):
    
    def define_parameters(self, input_fastqs, keep="N"):
        self.add_parameter( "keep", "keep option for fastq_illumina_filter", 
            default=keep, choices=["N", "Y"])
        self.add_input_file_list( "input_fastqs", "Input fastq files", 
            default=input_fastqs, required=True, file_format = 'fastq')
        self.add_output_file_list( "fastq_files_filtered", "fastq_files_filtered", 
            pattern='{basename}', items=self.input_fastqs, file_format = 'fastq')
        self.add_output_file_list( "stdout", "stdout", 
            pattern='{basename_woext}.stdout', items=self.input_fastqs)</code></pre>
					</div>
					
250
					<p>Using those <a href="./core-analysis.html#define_parameters">parameters definition</a> will lead to this command structure:</p>
251 252 253 254 255 256 257 258 259 260 261 262
					
					<div>
						<pre class="pre-hl "><code class="ini">fastq_illumina_filter --keep [keep] -v -o [output_filtered] [input] > [stdout]</code></pre>
					</div>

					<p>Since <code>fastq_illumina_filter</code> can only process unziped FASTQ files, the command structure is a little different for gziped FASTQ files:</p>
					
					<div>
						<pre class="pre-hl "><code class="ini">gunzip -c [input] | fastq_illumina_filter -v --keep [keep] 2> [stdout] | gzip > [output_filtered]</code></pre>
					</div>
					
					<p>
Jerome Mariette's avatar
Jerome Mariette committed
263
			    		In the following, this structure will be used to help us to build the command line. To build a command line, NG6 provides a
264 265 266 267 268
				    	function named <code>ShellFunction</code> in which the command line structure can be given (nb: there is other functions available as 
				    	the <code>PythonFunction</code> to run an internal function). The <code>ShellFunction</code> takes 2 arguments: the command line 
				    	structure, wich is required, and the <code>cmd_format</code> defining the parameter ordering.
			    	</p>
			    	<p>
Jerome Mariette's avatar
Jerome Mariette committed
269
				    	Considering <code>cmd_format="{EXE} {IN} {OUT}"</code>, which is a classic value for this option, NG6 will consider the following 
270
				    	inputs and outputs order: <code>input_fastqs</code>, <code>fastq_files_filtered</code>, and then <code>stdout</code> resulting
271
				    	to the following command structure:
272 273 274 275 276 277
			    	</p>
					
					<div>
						<pre class="pre-hl "><code class="ini">fastq_illumina_filter --keep [keep] -v -o $2 $1 > $3</code></pre>
					</div>

278
					<p>and for gzipped FASTQ files:</p>
279 280 281 282 283 284 285 286 287 288 289
					
					<div>
						<pre class="pre-hl "><code class="ini">gunzip -c $1 | fastq_illumina_filter -v --keep [keep] 2> $3 | gzip > $2</code></pre>
					</div>
					
					<p>
						All execution path are accessible using the method <code>get_exec_path</code>. This leads to the implementation of the following 
						<code>process()</code> function:
					</p>
					
					<div>
Jerome Mariette's avatar
Jerome Mariette committed
290
						<pre class="pre-hl "><code class="python">from weaver.function import ShellFunction	
291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311
from ng6.analysis import Analysis
from jflow.abstraction import MultiMap

class QIlluminaFilter (Analysis):
    
    def process(self):
        if self.input_fastqs[0].endswith(".gz") :
            fastq_illumina_filter = ShellFunction("gunzip -c $1 | " + 
                self.get_exec_path("fastq_illumina_filter") + 
                " -v --keep  " + self.keep + "  2> $3 | gzip > $2", 
                cmd_format='{EXE} {IN} {OUT}')
        else :
            fastq_illumina_filter = ShellFunction( self.get_exec_path("fastq_illumina_filter") + 
                " --keep  " + self.keep + " -v -o $2 > $3 $1", 
                cmd_format='{EXE} {IN} {OUT}')
        MultiMap(fastq_illumina_filter, 
            inputs = self.input_fastqs, 
            outputs = [self.fastq_files_filtered, self.stdout])</code></pre>
					</div>
					
					<p>
312 313
        				We want <code>fastq_illumina_filter</code> to itterate through the input files. For this we will use an abstraction: the MultiMap. 
        				The abstraction must be called on the defined ShellFunction with inputs and outputs. 
314 315 316 317 318
        			</p>
					
					<p>
						The <code>stdout</code> file will contain informations that can be presented to the user via the web interface. Those informations are the
						number of processed sequences (Input) and the number of sequences after filtering (Output). A method <code>__parse_stat_file()</code> will
319
						be used to retrieve those informations. The method <code>_add_result_element(name, key, val, group)</code> allows to add grouped informations to the database.
Jerome Mariette's avatar
Jerome Mariette committed
320
						Here for each input FASTQ file, the number of sequences before and after filtering will be added to the NG6 database. All those operation 
321
						must be executed in <code>post_process()</code> method, since they must wait for the output files to be generated. This lead to the implementation
322
						of the following <code>post_process()</code>:
323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354
					</p>
					
					<div>
						<pre class="pre-hl "><code class="python">from weaver.function import ShellFunction
from ng6.analysis import Analysis
from jflow.abstraction import MultiMap

class QIlluminaFilter (Analysis):
    
    def post_process(self):
        for file in self.stdout :
            file_name = os.path.splitext(os.path.basename(file))[0] 
            input, output = self.__parse_stat_file(file)
            self._add_result_element(file_name, "input", str(input))
            self._add_result_element(file_name, "output", str(output))
    
    def __parse_stat_file (self, stat_file):
        input, output = 0, 0
        for line in open(stat_file, "r").readlines():
            line = line.strip()
            input_reg = re.search("Input: (.*) reads", line)
            output_reg = re.search("Output: (.*) reads \(.*\)", line)
            if input_reg:
                input = input_reg.group(1).replace(",", "")
            if output_reg:
                output = output_reg.group(1).replace(",", "")
        
        return input, output</code></pre>
					</div>
					
					<p>
						Two other methods must be overloaded, <code>get_version()</code> and <code>define_analysis()</code>, which respectivly give the version of the analysis and
355
						describe the analysis parameters, name options and software. The global class definition for <code>QilluminaFilter</code> is the following:
356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428
					</p>
					
					<div>
						<pre class="pre-hl "><code class="python">import re, os
from subprocess import Popen, PIPE
from jflow.abstraction import MultiMap
from ng6.analysis import Analysis
from weaver.function import ShellFunction

class QIlluminaFilter (Analysis):
    
    def define_parameters(self, input_fastqs, keep="N"):
        self.add_parameter( "keep", "keep option for fastq_illumina_filter", 
            default=keep, choices=["N", "Y"])
        self.add_input_file_list( "input_fastqs", "Input fastq files", 
            default=input_fastqs, required=True, file_format = 'fastq')
        self.add_output_file_list( "fastq_files_filtered", "fastq_files_filtered", 
            pattern='{basename}', items=self.input_fastqs, file_format = 'fastq')
        self.add_output_file_list( "stdout", "stdout", pattern='{basename_woext}.stdout', 
            items=self.input_fastqs)
        
    def define_analysis(self):
        self.name = "QIlluminaFilter"
        self.description = "Filters FASTQ file generated by CASAVA 1.8"
        self.software = "fastq_illumina_filter"
        self.options = "--keep " + self.keep + " -v"

    def get_version(self):
        cmd = [self.get_exec_path("fastq_illumina_filter"), "-help"]
        p = Popen(cmd, stdout=PIPE, stderr=PIPE)
        stdout, stderr = p.communicate()
        return stdout.split()[2][:-1]
    
    def process(self):
        if self.input_fastqs[0].endswith(".gz") :
            fastq_illumina_filter = ShellFunction("gunzip -c $1 | " + 
                self.get_exec_path("fastq_illumina_filter") + " -v --keep  " + 
                self.keep + "  2> $3 | gzip > $2", 
                cmd_format='{EXE} {IN} {OUT}')
        else :
            fastq_illumina_filter = ShellFunction( self.get_exec_path("fastq_illumina_filter") + 
                " --keep  " + self.keep + " -v -o $2 > $3 $1", 
                cmd_format='{EXE} {IN} {OUT}')
        MultiMap(fastq_illumina_filter, 
            inputs = self.input_fastqs, 
            outputs = [self.fastq_files_filtered, self.stdout])
        
    def post_process(self):
        for file in self.stdout :
            file_name = os.path.splitext(os.path.basename(file))[0] 
            input, output = self.__parse_stat_file(file)
            self._add_result_element(file_name, "input", str(input))
            self._add_result_element(file_name, "output", str(output))
    
    def __parse_stat_file (self, stat_file):
        input, output = 0, 0
        for line in open(stat_file, "r").readlines():
            line = line.strip()
            input_reg = re.search("Input: (.*) reads", line)
            output_reg = re.search("Output: (.*) reads \(.*\)", line)
            if input_reg:
                input = input_reg.group(1).replace(",", "")
            if output_reg:
                output = output_reg.group(1).replace(",", "")
        
        return input, output
        </code></pre>
					</div>
					
				</div>
			
				
				<div id="QilluminaFilter_template" class="subgroup">
429
					<h2>QIlluminaFilter template</h2>
430 431
					
					<p>
432
						To present the results of <code>QIlluminaFilter</code> analysis in the NG6 web interface, a <a href="./core-analysis.html#analysis_template">template file</a> must be written.
433 434
						To be associated to an analysis, a template file must have the exact same name as the <code>Python</code> class name of the
						analysis, so the template file for <code>QIlluminaFilter</code> will be named <code>QIlluminaFilter.tpl</code>.
Jerome Mariette's avatar
Jerome Mariette committed
435
						This template has to be written using <a target="_blank" href="http://www.smarty.net/">PHP Smarty</a> paradigm.
436 437 438
					</p>
					
					<p>
Jerome Mariette's avatar
Jerome Mariette committed
439
						The first thing to do while writing a template file for NG6 is to extends the default analysis template. This is done using the command
440
						<code>{extends file="AnalysisTemplate.tpl"}</code>. Doing this provide you an empty template with default tabulations:
441
					</p>
442
					<div>
443
					<pre class="pre-hl "><code class="xml">{extends file="AnalysisTemplate.tpl"}</code></pre>
444
					</div>
445 446 447
					<div class="bs-docs-example" style="text-align: center;">
				  		<img width="90%" src="img/empty_analysis.png" class="img-rounded">
				  	</div> 
448
	
449
					<p>
Jerome Mariette's avatar
Jerome Mariette committed
450
						To populate this template, the <code>results</code> block must  be overloaded using the <code>{block name="results""}</code> instruction. An HTML table will be written. 
451 452 453 454 455
						Assuming that each input file used in <code>QIlluminaFilter</code> analysis represents a sample,
						each line of the table will present the number of sequence before and after filtering and also the number of filtered sequences.
					</p>
					
					<p>
456 457
						Attributes can be passed to a Smarty template. All the previously generated analysis results are available in <code>$analyse_results</code>
						attribute. This attribute is a dictionnary which contains the results for each sample regrouped by the group, with the sample name as key.
458 459 460 461 462
						In the Python class, we added our results using <code>_add_result_element(sample, key, value, group)</code>. The value can be retrieved in 
						<code>$analyse_results</code> as <code>$analyse_results['group'].key</code>. 
					</p>
					
					<p>
463 464
						<code>PHP</code> functions can be used in a <code>Smarty</code> template. In the template the PHP function <code>number_format()</code> will 
						be used to format the number of sequences. The template looks like:				
465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498
					</p>
					
					<div>
					<pre class="pre-hl "><code class="xml">{extends file="AnalysisTemplate.tpl"}

{block name=results_title} Filtering results {/block}
{block name=results}
&lt;table class="table table-striped table-bordered dataTable analysis-result-table"&gt;
 &lt;thead&gt;
  &lt;tr&gt;
   &lt;th class="string-sort"&gt;Samples&lt;/th&gt;
   &lt;th class="numeric-sort"&gt;Before filtering&lt;/th&gt;
   &lt;th class="numeric-sort"&gt;After filtering&lt;/th&gt;
   &lt;th class="numeric-sort"&gt;Filtered&lt;/th&gt;
  &lt;/tr&gt;
 &lt;/thead&gt;
 {assign var="analyse_results_sorted" value=$analyse_results|@ksort}
 &lt;tbody&gt;
 {foreach from=$analyse_results_sorted key=sample item=sample_results}
  &lt;tr&gt;
   &lt;td&gt;{$sample|get_description:$descriptions}&lt;/td&gt;
   &lt;td&gt;{$sample_results["default"].input|number_format:0:' ':' '}&lt;/td&gt;
   &lt;td&gt;{$sample_results["default"].output|number_format:0:' ':' '}&lt;/td&gt;
   &lt;td&gt;{($sample_results["default"].input-$sample_results["default"].output)|number_format:0:' ':' '}
    &lt;/td&gt;
  &lt;/tr&gt;
 {/foreach}
 &lt;/tbody&gt;
&lt;/table&gt;
{/block}

{block name=download}{/block}</code></pre>	
					</div>
					
499 500 501 502 503 504 505 506 507
					
					<p>
						Once executed, the analysis will have the following layout in the web interface:
					</p>
					
					<div class="bs-docs-example" style="text-align: center;">
				  		<img width="90%" src="img/ng6-qilluminafilter.png" class="img-rounded">
				  	</div> 
					
508 509 510 511 512 513 514 515 516 517
				</div>
				
			</section>
				
				
			<section id="third_step" class="group">
				<h1 class="page-header">Step #3 <small>Create QFastQC analysis</small></h1>				
				
				<div id="QFastQC_analysis" class="subgroup">
					<h2>QFastQC analysis</h2>		
Penom Nom's avatar
Penom Nom committed
518 519 520
				
					<p>
						FastQC aims to provide a simple way to do some quality control checks on raw sequence data. Several charts quality charts are generated. 
521 522
						To make it as simple as possible, we will only focus on two of them: "Per base sequence quality" and "Per base sequence content".
						The executed fastqc command line looks like:
Penom Nom's avatar
Penom Nom committed
523 524 525 526 527 528 529
					</p>
					<pre class="pre-hl "><code class="ini">fastqc --extract --outdir output_directory input_file.fastq > input_file.stdout 2>> input_file.stderr</code></pre>
					<p>
						Where <code>--extract</code> tells fastqc to extract the generated pictures from its archive, <code>--outpdir</code> sets the output directory. <code>input_file.fastq</code>
						is the raw sequence file that will be used. <code>input_file.stdout</code> and <code>input_file.stderr</code> are log files.
					</p>
					<p>
530
						Splitting the command line into parameter will give the following parameters:
Penom Nom's avatar
Penom Nom committed
531 532
					</p>
					<ul>
533 534 535 536
						<li><code>input_file</code>: which will be the input file used to generate our repport.</li>
						<li><code>output_directory</code>: which will be the working directory of fastqc.</li>
						<li><code>stdout</code>: which will contain the standart output stream.</li>
						<li><code>stderr</code>: which will contain the standart error stream.</li>
Penom Nom's avatar
Penom Nom committed
537 538 539
					</ul>
					
					<p>
Jerome Mariette's avatar
Jerome Mariette committed
540
						By default, fastqc generates a file named <code>input_file_fastqc.zip</code> which contains several files (summary, images, etc...). Using 
541 542 543
						the <code>--extract</code> option will automaticaly extract the content of this file in a folder named <code>input_file_fastqc</code>, which
						can then be browsed to retrieve images or parse generated informations files.  This will lead to the following <code>define_parameters()</code> definition,
						allong with the definition of the analysis class: 
Penom Nom's avatar
Penom Nom committed
544 545 546 547 548
					</p>
					
			    	<pre class="pre-hl "><code class="python">from weaver.function import ShellFunction
from ng6.analysis import Analysis
	
549
class QFastQC (Analysis):
Penom Nom's avatar
Penom Nom committed
550 551 552 553 554 555 556 557 558
	    
    def define_parameters(self, input_files):
        self.add_input_file_list("input_files", \
            "Fastq file", default=input_files, required=True)
        self.add_output_file_list( "output_fastqc", \
            "outputs", pattern='{basename_woext}_fastqc', items=input_files)
        self.add_output_file_list( "stdouts", \
            "stdout", pattern='{basename_woext}.stdout', items=input_files)
        self.add_output_file_list( "stderrs", \
Jerome Mariette's avatar
Jerome Mariette committed
559
            "stderr", pattern='{basename_woext}.stderr', items=input_files)</code></pre>
Penom Nom's avatar
Penom Nom committed
560 561
			
					<p>
562
						The resulting command line should have the following structure:
Penom Nom's avatar
Penom Nom committed
563 564 565 566
			    	</p>
				    <pre class="pre-hl "><code class="ini">[EXE] --extract --outdir [outdir] [input_file] > [stdout] 2>> [stderr]</code></pre>
				    	
			    	<p>
567
			    		This structure will be used to help us to build the command line. 
Jerome Mariette's avatar
Jerome Mariette committed
568
			    		As said before, NG6 provide multiple function (ShellFunction, PythonFunction) to build the comand line. Here the <code>ShellFunction</code> 
569 570
			    		will be used again. The <code>ShellFunction</code> takes 2 arguments: the command line structure, wich is required, and the <code>cmd_format</code> 
			    		defining the parameter ordering.
Penom Nom's avatar
Penom Nom committed
571 572
			    	</p>
			    	<p>
Jerome Mariette's avatar
Jerome Mariette committed
573
				    	Considering <code>cmd_format="{EXE} {IN} {OUT}"</code>, which is a classic value for this option, NG6 will consider the following 
Penom Nom's avatar
Penom Nom committed
574 575 576 577 578 579 580 581 582 583 584 585 586
				    	inputs and outputs order: <code>input_file</code>, <code>stdout</code>, and then <code>stderr</code> resulting
				    	to the following command structure:
			    	</p>
			    		<pre class="pre-hl "><code class="ini">[EXE] --extract --outdir [outdir] $1 > $2  2>> $3 </code></pre>
					<p>
						All execution path are accessible using the method <code>get_exec_path</code>. This leads to the implementation of the following 
						<code>process()</code> function:
					</p>	
			    	
			    	<pre class="pre-hl "><code class="python">from weaver.function import ShellFunction
from ng6.analysis import Analysis
from jflow.abstraction import MultiMap

587
class QFastQC (Analysis):
Penom Nom's avatar
Penom Nom committed
588 589 590 591 592 593 594 595
	
    def process(self):
        fastqc = ShellFunction( self.get_exec_path("fastqc") + ' --extract --outdir ' + \
            self.output_directory + '  $1 > $2 2>> $3 ', cmd_format='{EXE} {IN} {OUT}')
        MultiMap(fastqc, inputs=[self.input_files], 
            outputs=[self.stdouts, self.stderrs, self.output_fastqc])</code></pre>
        
        			<p>
596
        				We also use again the abstraction <code>MultiMap</code> because we want fastqc to itterate through the input files.
Penom Nom's avatar
Penom Nom committed
597 598 599
        			</p>
        			
        			<p>
600 601 602
        				<code>self.output_fastqc</code> is added as an output but will not be used in the command line. This must be done because the analysis
        				expect <code>self.output_fastqc</code> to be generated, and setting it as an output tells that this file will be generated by
        				the fastqc command.
Penom Nom's avatar
Penom Nom committed
603 604 605
        			</p>
	        		
	        		<p>
606
	        			When the process is done, the database must be updated with relevant informations to present to the user. We will need the
Penom Nom's avatar
Penom Nom committed
607 608 609 610 611 612 613 614 615
	        			generated png image for "Per base sequence quality" and "Per base sequence content" as said before. The fastqc also gives us informations on
	        			wheter those metrics FAIL or PASS. Those informations are available in a file called <code>summary.txt</code>, contained in each fastqc subdirectory.
	        			To parse this file, we write a method named <code>__parse_summary_file()</code>:
	        		</p>
	        		
	        		<p>
	        			Now we can save our png files. The function <code>_save_file()</code> allows us to save a file to the storage directory. It also return the webpath of this
	        			saved file, and then make it available in the template. The function <code>_add_result_element()</code> allows us to update the database with element for each
	        			sample. Here a sample is defined by a fastq file. This method takes 4 arguments: a sample name, a key string, a value associated with the key, and an optional group
Jerome Mariette's avatar
Jerome Mariette committed
616
	        			to group key values pairs. The default value for the group is "default".
Penom Nom's avatar
Penom Nom committed
617 618 619 620 621
	        		</p>
			    	<pre class="pre-hl "><code class="python">from weaver.function import ShellFunction
from ng6.analysis import Analysis
from jflow.abstraction import MultiMap

622
class QFastQC (Analysis):
Penom Nom's avatar
Penom Nom committed
623
	
624 625 626 627 628 629 630 631 632
    def __parse_summary_file (self, summary_file):
        per_base_quality = ""
        per_base_sequence_content = ""
        for line in open(summary_file, 'r').readlines():
            parts = line.strip().split("\t")
            if parts[1] == "Per base sequence quality": per_base_quality = parts[0]
            if parts[1] == "Per base sequence content": per_base_sequence_content = parts[0]
        return per_base_quality, per_base_sequence_content
        
Penom Nom's avatar
Penom Nom committed
633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652
    def post_process(self):
        for output_fastqc_dir in self.output_fastqc :
            sample = re.sub( "_fastqc$", "", os.path.basename(output_fastqc_dir))
            per_base_quality, per_base_sequence_content = self.__parse_summary_file(
                os.path.join( output_fastqc_dir, "summary.txt" ))
            
            self._add_result_element(sample, "img", 
                self._save_file(os.path.join(output_fastqc_dir, "Images", \
                "per_base_quality.png"), \
                sample + ".per_base_quality.png"), "pbqpng")
            self._add_result_element(sample, "result", str(per_base_quality), "pbqpng")

            self._add_result_element(sample, "img", 
                self._save_file(os.path.join(output_fastqc_dir, "Images", \ 
                "per_base_sequence_content.png"), \
                sample + ".per_base_sequence_content.png"), "pbspng")     
            self._add_result_element(sample, "result", str(per_base_sequence_content), "pbspng")</code></pre>

					<p>
						Two last methods must be defined: <code>get_version()</code> which will give us the version of the analysis, and <code>define_analysis()</code> which will
653
						define the analysis. The global QFastQC class is now:
Penom Nom's avatar
Penom Nom committed
654 655 656 657 658 659 660
					</p>

			    	<pre class="pre-hl "><code class="python">import subprocess, os, re
from weaver.function import ShellFunction
from jflow.abstraction import MultiMap
from ng6.analysis import Analysis
    
661
class QFastQC (Analysis):
Penom Nom's avatar
Penom Nom committed
662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679
        
    def define_parameters(self, input_files):
        self.add_input_file_list("input_files", 
            "Fastq file", default=input_files, required=True)
        self.add_output_file_list( "output_fastqc", 
            "outputs", pattern='{basename_woext}_fastqc', items=input_files)
        self.add_output_file_list( "stdouts", 
            "stdout", pattern='{basename_woext}.stdout', items=input_files)
        self.add_output_file_list( "stderrs", 
            "stderr", pattern='{basename_woext}.stderr', items=input_files)

    def get_version(self):
        p = subprocess.Popen([self.get_exec_path("fastqc"), 
            "--version"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        stdout, stderr = p.communicate()
        return stdout.split()[1]
    
    def define_analysis(self):
680
        self.name = "QFastQC"
Penom Nom's avatar
Penom Nom committed
681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721
        self.description = "Statistics on reads and their qualities. M"
        self.software = "fastqc"
        self.options = ""
    
    def __parse_summary_file (self, summary_file):
        per_base_quality = ""
        per_base_sequence_content = ""
        for line in open(summary_file, 'r').readlines():
            parts = line.strip().split("\t")
            if parts[1] == "Per base sequence quality": per_base_quality = parts[0]
            if parts[1] == "Per base sequence content": per_base_sequence_content = parts[0]
        return per_base_quality, per_base_sequence_content
        
    def process(self):
        fastqc = ShellFunction( self.get_exec_path("fastqc") + 
            ' --extract --outdir ' + self.output_directory + 
            '  $1 > $2 2>> $3 ', cmd_format='{EXE} {IN} {OUT}')
        MultiMap(fastqc, inputs=[self.input_files], 
            outputs=[self.stdouts, self.stderrs, self.output_fastqc])

    def post_process(self):
        for output_fastqc_dir in self.output_fastqc :
            sample = re.sub( "_fastqc$", "", os.path.basename(output_fastqc_dir))
            per_base_quality, per_base_sequence_content = self.__parse_summary_file(
                os.path.join( output_fastqc_dir, "summary.txt" ))
            
            self._add_result_element(sample, "img", 
                self._save_file(os.path.join(output_fastqc_dir, 
                "Images", "per_base_quality.png"), 
                sample + ".per_base_quality.png"), "pbqpng")
            self._add_result_element(sample, "result", str(per_base_quality), "pbqpng")

            self._add_result_element(sample, "img", 
                self._save_file(os.path.join(output_fastqc_dir, 
                "Images", "per_base_sequence_content.png"), 
                sample + ".per_base_sequence_content.png"), "pbspng")     
            self._add_result_element(sample, "result", str(per_base_sequence_content), "pbspng")</code></pre>

				</div>

			
722 723
				<div id="QFastQC_template" class="subgroup">
					<h2>QFastQC template</h2>
Penom Nom's avatar
Penom Nom committed
724 725
					
					<p>
726 727
						Now we must write a template file for <code>QFastQC</code> analysis. This file must be named as the analysis class and placed in the right 
						directory. Our template will be named <code>QFastQC.tpl</code>. The goal of this template is to present the previously saved png files to the user
Jerome Mariette's avatar
Jerome Mariette committed
728
						in the web interface. This template has to be written using <a target="_blank" href="http://www.smarty.net/">PHP Smarty</a> paradigme.
Penom Nom's avatar
Penom Nom committed
729 730 731
					</p>
					
					<p>
732 733 734 735 736 737
						The analysis template must inherit a global defined template using smarty command <code>{extends file="AnalysisTemplate.tpl"}</code>.
						The block <code>{block name=results} {/block}</code> must be overloaded with HTML content to presents the fastqc results.
						We will write a table, and for each line in our table, we will present the fastqc results for one FASTQ file.
						Each png file will be available by the click on a link. To do this, we will have to write a javascript file associated with the template. 
						This javascript will open a modal  with an image when the user click on a link with the CSS class <code>image-link</code>. The image path
						must be setted in tht <code>href</code> attribute of the link.
Penom Nom's avatar
Penom Nom committed
738 739 740 741
					</p>
					
					<p>
						Attributes can be passed to the Smarty template. All the previously generated analysis results are available in <code>$analyse_results</code>
742
						attribute. This attribute is a dictionnary which contains the results for each sample regrouped by the group, with the sample name as key.
Penom Nom's avatar
Penom Nom committed
743
						In the Python class, we added our results using <code>_add_result_element(sample, key, value, group)</code>. The value can be retrieved in 
744
						<code>$analyse_results</code> as <code>$analyse_results['group'].key</code>. The global template looks like:
Penom Nom's avatar
Penom Nom committed
745 746
					</p>
					
747
					<div>
Penom Nom's avatar
Penom Nom committed
748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775
			    	<pre class="pre-hl "><code class="xml">{extends file="AnalysisTemplate.tpl"}
{block name=results_title} Reads and quality statistics {/block}

{block name=results}
	&lt;table class="table table-striped table-bordered dataTable analysis-result-table"&gt;
		&lt;thead&gt;
			&lt;tr&gt;
				&lt;th class="string-sort" &gt;Sample&lt;/th&gt;
				&lt;th class="string-sort" &gt;Per base quality&lt;/th&gt;
				&lt;th class="string-sort" &gt;Per base sequence content&lt;/th&gt;
			&lt;/tr&gt;
		&lt;/thead&gt;
		
		{assign var="analyse_results_sorted" value=$analyse_results|@ksort}
		&lt;tbody&gt;
			{foreach from=$analyse_results_sorted key=sample item=sample_results}
		        &lt;tr&gt;
			        &lt;td class="sample_name"&gt;{$sample|get_description:$descriptions}&lt;/td&gt;
			        &lt;td&gt;&lt;a class="imglink" href="{$sample_results['pbqpng'].img}"&gt; 
			            {$sample_results["pbqpng"].result} &lt;/a&gt;&lt;/td&gt;
			        &lt;td&gt;&lt;a class="imglink" href="{$sample_results['pbspng'].img}"&gt; 
			            {$sample_results["pbspng"].result} &lt;/a&gt;&lt;/td&gt;
		        &lt;/tr&gt;
			{/foreach}
		&lt;/tbody&gt;
	&lt;/table&gt;
{/block}</code></pre>					
					
776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809
					</div>
					
					<p>
						The javascript file associated with the template must also have the exact same name as the <code>Python</code> class name of the analysis.
						This file, created next to <code>QFastQC.tpl</code> will be named <code>QFastQC.js</code>. The <code>jQuery</code> library is available in
						all template javascript file. We will use this library to write our javascript file: 
					</p>
					
					<div>
					<pre class="pre-hl "><code class="javascript">$(function () {

    $(".image-link").click(function() {
        var img_href = $(this).attr('href') ;
        var imgLoad = $("&lt;img /&gt;")
            .attr("src", img_href)
            .attr('alt', 'Cannot display')
            .unbind("load")
            .bind("load", function () {
                var imgwidth = ( this.width &gt; 900 ) ? 900 : this.width ;
                resize_center_btmodal('#ng6modal',imgwidth + 50 );
                $("#modal-label-tmpl").html("NG6 &lt;small&gt; " + $("#analyse_name").val() + "&lt;/small&gt;");
                $("#modal-body-tmpl").html('&lt;div id="img_container" '+
                   'style="overflow-x : auto"&gt;&lt;img src="' +img_href+ '" alt="Cannot display"&gt;&lt;/div&gt;');
                $("#modal-foot-tmpl").html('&lt;button class="btn btn-default" ' +
                   'data-dismiss="modal" aria-hidden="true"&gt;' +
                   '&lt;i class="glyphicon glyphicon-remove"&gt;&lt;/i&gt; Close&lt;/button&gt;');
                $("#ng6modal").modal();
            });
        return false;
    });
	
});</code></pre>
					</div>
					<p>
810
						All analysis template files have access to a <code>bootstrap</code> modal (dialog box/popup window) with the id <code>ng6modal</code>. 
811 812 813 814 815 816 817 818 819 820 821
						This modal is shown using <code>$("#ng6modal").modal();</code>. It also has 3 main parts that can be selected
						and updated, each part with a uniq id:
					</p>
					
					<ul>
						<li><code>modal-label-tmpl</code>: updating this element will update the label of the modal.</li>
						<li><code>modal-body-tmpl</code>: represents the main content of the label.</li>
						<li><code>modal-foot-tmpl</code>: represents the footer content of the label.</li>
					</ul>
					
					<p>This will give us the following interface, a click to a link will display the desired png: </p>
Penom Nom's avatar
Penom Nom committed
822 823 824 825 826 827 828 829 830 831 832 833 834
					
					<div class="bs-docs-example" style="text-align: center;">
				  		<img width="90%" src="img/mfastqc_analysis.png" class="img-rounded">
				  	</div>
				  	
				  	<div class="bs-docs-example" style="text-align: center;">
				  		<img width="80%" src="img/mfastqc_quality.png" class="img-rounded">
				  	</div> 
				  	
				</div>
			
			</section>
			
835
			<section id="fourth_step" class="group">
Jerome Mariette's avatar
Jerome Mariette committed
836
				<h1 class="page-header">Step #4 <small>create the workflow</small></h1>
Penom Nom's avatar
Penom Nom committed
837 838
				
				<p>
Jerome Mariette's avatar
Jerome Mariette committed
839
					Now that the analyses have been written, the <code>quickstart</code> NG6Workflow will be created to link those analyses together.
Penom Nom's avatar
Penom Nom committed
840
					A NG6Workflow is a Python class defined in a <code>__init__.py</code> file in the workflows package. This class must inherit from
841
					<code>ng6.ng6workflow.NG6Workflow</code> class, and must overload several methods (more info <a href="./core-workflow.html#workflows">here</a>):
Penom Nom's avatar
Penom Nom committed
842 843 844 845 846 847 848 849
				</p>
				<ul>
					<li><code>get_description()</code> to give a description to the workflow in the command line and in the interface.</li>
					<li><code>define_parameters(function="process")</code> to define input parameters of the workflow</li>
					<li><code>process()</code> to define the workflow execution by linking components</li>
				</ul>
				
				<p>
850 851
					The <code>get_name()</code> method can also be overloaded to change the call name of the NG6Workflow. 
					Overloading NG6Workflow will automaticaly provides the class with default parameters for the description of a run and parameters to define samples and input files.
852 853
					The analysis <code>QIlluminaFilter</code> has a <code>keep</code> parameter. To make it available in the command line and in the web interface, 
					it will be added in <code>define_parameters()</code>. The skeleton of the class is: 
Penom Nom's avatar
Penom Nom committed
854 855
				</p>
				
856
				<div>
Penom Nom's avatar
Penom Nom committed
857 858
				<pre class="pre-hl "><code class="python">from ng6.ng6workflow import NG6Workflow

859
class QuickStart(NG6Workflow):
Penom Nom's avatar
Penom Nom committed
860 861
    
    def get_name(self):
862
        return 'quickstart'
Penom Nom's avatar
Penom Nom committed
863 864 865 866 867
    
    def get_description(self):
        return "Quick start pipeline"
    
    def define_parameters(self, function="process"):
868 869
        self.add_parameter( "keep", "keep option for fastq_illumina_filter", 
            default="N", choices=["N", "Y"])</code></pre>
Penom Nom's avatar
Penom Nom committed
870
				
871
				</div>
Penom Nom's avatar
Penom Nom committed
872
				<p>
Jerome Mariette's avatar
Jerome Mariette committed
873
					The <code>process</code> method is used to add analyses and components and link them together to make the workflow. To add a component or an analysis, use the function
874
					<code>add_component()</code>, which takes as first argument the name of the <code>Python</code> class of the component or analysis to be added, and the second argument which
875 876
					is an array of parameters that have to be used as input parameters for the analysis/component. This method also have a parent argument that can be passed to 
					specify the parent analysis. This will result in a tree view in the web interface.
877
					The <code>QIlluminaFilter</code> analysis takes as input a list of FASTQ files and produces as output a list of FASTQ files. The <code>QFastQC</code> component
Jerome Mariette's avatar
Jerome Mariette committed
878
					will take those output files as input, thus linking the two analyses together.
879
					NG6Workflow has many utilie methods, one of them <code>get_all_reads()</code> allows to retrieve all input files provided. The workflow definition is the following:
Penom Nom's avatar
Penom Nom committed
880 881
					
				</p>
882 883
				
				<div>
Penom Nom's avatar
Penom Nom committed
884 885
				<pre class="pre-hl"><code class="python">from ng6.ng6workflow import NG6Workflow

886
class QuickStart(NG6Workflow):
Penom Nom's avatar
Penom Nom committed
887 888
    
    def get_name(self):
889
        return 'quickstart'
Penom Nom's avatar
Penom Nom committed
890 891 892 893 894
    
    def get_description(self):
        return "Quick start pipeline"
    
    def define_parameters(self, function="process"):
895 896
        self.add_parameter( "keep", "keep option for fastq_illumina_filter", 
            default="N", choices=["N", "Y"])
Penom Nom's avatar
Penom Nom committed
897 898
    
    def process(self):
899 900 901 902
        illumina_filter = self.add_component("QIlluminaFilter", [self.get_all_reads()])
        fastqc = self.add_component("QFastQC", [illumina_filter.fastq_files_filtered], 
            parent = illumina_filter)</code></pre>
				</div>
Penom Nom's avatar
Penom Nom committed
903 904 905
				
			</section>
			
906 907
			<section id="fifth_step" class="group">
				<h1 class="page-header">Step #5 <small>test your workflow</small></h1>
Penom Nom's avatar
Penom Nom committed
908
				<p>
909
					A nG6 workflow can be run from the command line (<code>ng6_cli.py</code>) or via the server (<code>ng6_server.py</code>). From your install directory enter
Penom Nom's avatar
Penom Nom committed
910
				</p>
911
				<div>
Penom Nom's avatar
Penom Nom committed
912
				<pre class="pre-hl "><code class="ini">python bin/ng6_cli.py -h</code></pre>
913 914
				</div>
				<div>
Penom Nom's avatar
Penom Nom committed
915 916 917 918 919
				<pre class="pre-hl "><code class="ini">usage: ng6_cli.py [-h]
[...]
Available sub commands:
  {rerun,reset,delete,execution-graph,status,quickstart,myquickstart}
    [...]
920
    quickstart        Quick start pipeline
Penom Nom's avatar
Penom Nom committed
921
    [...]</code></pre>
922
    			</div>
Penom Nom's avatar
Penom Nom committed
923
				<p>
Jerome Mariette's avatar
Jerome Mariette committed
924
					A new line with "quickstart" should be displayed! Run it on your own data. An example could be:
Penom Nom's avatar
Penom Nom committed
925 926
				</p>
				
927
				<div>
928
				<pre class="pre-hl "><code class="ini">python bin/ng6_cli.py myquickstart --project-name "Demo project"
Penom Nom's avatar
Penom Nom committed
929 930 931
--name "my_quick_start" --description "quick start" --data-nature "DNA" --sequencer "Unknown" 
--species "test" --type "quick start" --sample read1=/path/to/sample1_file.fastq.gz 
--sample read1=/path/to/sample2_file.fastq.gz --admin-login &lt;login&gt;</code></pre>
932
				</div>
933
				
934
				<p>
935 936
					Our quickstart workflow inherit from NG6Workflow class which inherit from BasicNG6Workflow class, so it also inherit of required arguments which are : 
					<code>--project-name</code>, <code>--name</code>, <code>--description</code>, <code>--data-nature</code>, <code>--sequencer</code>, <code>--species</code>, <code>--type</code>, </code>--admin-login</code> and at least one <code>--sample</code>
Jerome Mariette's avatar
Jerome Mariette committed
937
					The <code>quickstart</code> workflow is already implemented in the NG6 sources.
938 939
				</p>

Penom Nom's avatar
Penom Nom committed
940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955
				
			</section>
			
		</div>
      	
      
      </div>
      
      
      <hr>
      <footer>
      	<div class="container" style="text-align: center;">
		  <p>Copyright © 2015, INRA | Designed by <a target="_blank" href="http://bioinfo.genotoul.fr/">GenoToul Bioinfo platform</a> and <a target="_blank" href="http://www.sigenae.org/">Sigenae platform</a>.</p>
		</div>
      </footer>
    </div>
Penom Nom's avatar
Penom Nom committed
956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977

    <!-- Bootstrap core JavaScript
    ================================================== -->
    <!-- Placed at the end of the document so the pages load faster -->
    <script src="js/jquery.min.js"></script>
    <script src="js/bootstrap.min.js"></script>
    <!-- <script src="../../assets/js/docs.min.js"></script> -->
    <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
    <!-- <script src="../../assets/js/ie10-viewport-bug-workaround.js"></script> -->
	
	<!--  Highlight code -->
	<script src="js/highlight.pack.js"></script>
	<script>hljs.initHighlightingOnLoad();</script>
	
	<script type='text/javascript'>
		$('body').scrollspy({
    		target: '.bs-docs-sidebar',
    		offset: 150
		});
	</script>
</body>
</html>