updated README for validation + added launch_example.sh

functional_tests/README.md

@@ -2,63 +2,36 @@
 
 ## I. Pre-requisites
 
-1. Install metagwgs as describe here:[installation doc](../docs/installation.md)
-2. Get datasets : two datasets are currently available for these functional tests, 
-    - ones in source code : [here](../test)) 
-    - thoses from [nf-core/mag pipeline](https://github.com/nf-core/test-datasets/tree/mag/test_data)
-    #TODO : lien direct sur les x fichier, ou sinon indiquer quels fichier telecharger/copier depuis test_data
-    ```
-    wget 
+1. Install metagwgs as described here: [installation doc](../docs/installation.md)
+2. Get datasets: two datasets are currently available for these functional tests at `https://forgemia.inra.fr/genotoul-bioinfo/metagwgs-test-datasets.git`
+
+    Replace "\<dataset\>" with either "small" or "mag":
     ```
+    git clone --branch <dataset> git@forgemia.inra.fr:genotoul-bioinfo/metagwgs-test-datasets.git
+
+    or
 
-3. Download the expected results from [link-to-test-datasets]. #TODO
+    wget https://forgemia.inra.fr/genotoul-bioinfo/metagwgs-test-datasets/-/archive/<dataset>/metagwgs-test-datasets-<dataset>.tar.gz
+    ```
 
 ## II. Run functional tests
 
 Each step of metagwgs produces a series of files. We want to be able to determine if the modifications we perform on metagwgs have an impact on any of these files (presence, contents, format, ...). You'll find more info about how the files are tested at the end of this page.
 
-#TODO Mettre l'usage du script
-
-
-There is two way to launch functionnal test :
-- by providing result of a pipeline already exectuted
-- by providing a script which launch the nextflow pipeline (see example) #TODO mettre un fichier de soumission d'exemple dans les sources
-
-
-### Launch from pipeline already exectuted
-
-If you have already launched metagwgs [see metagwgs README and usage] on test data:
-
-```
-cd project-directory
-python [work_dir]/metaG/metagwgs/functional_tests/main.py -step 07_taxo_affi -exp_dir [work_dir]/test_expected_logs -obs_dir ./results
-```
-
-
-### Launch with script
-
-Exemples below use the slurm job manager and launch all 7 steps of metagwgs to ensure all parts of main.nf work as intended.
-
-1. Create a new directory (project-directory) containing a shell script to be used by functional tests:
-
+To launch functional tests, you need to be located at the root of the folder where you want to perform the tests. There are two ways to launch functionnal tests (testing all steps to 07_taxo_affi):
+- by providing the results folder of a pipeline already exectuted
 ```
-#!/bin/bash
-
-sbatch -W -p workq -J metagwgs --mem=6G \
-    --wrap="module load bioinfo/Nextflow-v21.04.1 ; module load system/singularity-3.7.3 ; nextflow run -profile test_genotoul_workq [work_dir]/metaG/metagwgs/main.nf --step '01_clean_qc,02_assembly,03_filtering,04_structural_annot,05_alignment,06_func_annot,07_taxo_affi' --reads '../metagwgs/test/*_{R1,R2}.fastq.gz' --host_fasta '[work_dir]/human_ref/Homo_sapiens.GRCh38_chr21.fa' --host_bwa_index '[work_dir]/human_ref/Homo_sapiens.GRCh38_chr21.fa.{amb,ann,bwt,pac,sa}' --kaiju_db_dir '/bank/kaijudb/kaijudb_refseq_2020-05-25' --taxonomy_dir '[work_dir]/taxonomy' --eggnog_mapper_db_dir '/bank/eggnog-mapper/eggnog-mapper-2.0.4-rf1/data' --min_contigs_cpm 1000 --diamond_bank "[work_dir]/refseq_bacteria_2021-05-20/refseq_bacteria.dmnd" -with-report -with-timeline -with-trace -with-dag -resume"
+cd test_folder
+python <metagwgs-src>/functional_tests/main.py -step 07_taxo_affi -exp_dir metagwgs-test-datasets-small/output -obs_dir ./results
 ```
-
-*In this exemple, [work_dir] = "/home/pmartin2/work"*
-*"--min_contigs_cpm 1000" is mandatory to have the same results as exp_dir for step 03_filtering* 
-
-2. Run functionnal test by providing the script :
-
+- by providing a script which will launch the nextflow pipeline [see example](./launch_example.sh) (this example is designed for the "small" dataset with --min_contigs_cpm>1000, using slurm)
 ```
-cd project-directory
-python [work_dir]/metaG/metagwgs/functional_tests/main.py -step 07_taxo_affi -exp_dir [work_dir]/test_expected_logs -obs_dir ./results --script launch_07_taxo_affi.sh
+mkdir test_folder
+cd test_folder
+cp <metagwgs-src>/functional_tests/launch_example.sh ./
+python <metagwgs-src>/functional_tests/main.py -step 07_taxo_affi -exp_dir metagwgs-test-datasets-small/output -obs_dir ./results --script launch_example.sh
 ```
 
-
 ## III. Output
 
 A ft_\[step\].log file is created for each step of metagwgs. It contains information about each test performed on given files.
@@ -72,13 +45,13 @@ Observed directory: /work/pmartin2/metaG/refac_09_13.2/results/01_clean_qc
 
 ------------------------------------------------------------------------------
 
-File:           01_1_cleaned_reads/cleaned_a_R1.fastq.gz
+File:           01_1_cleaned_reads/cleaned_c_R1.fastq.gz
 Test method:    zdiff
 Test result:    Passed
 
 ------------------------------------------------------------------------------
 
-File:           01_1_cleaned_reads/cleaned_a_R2.fastq.gz
+File:           01_1_cleaned_reads/cleaned_c_R2.fastq.gz
 Test method:    zdiff
 Test result:    Passed
 

functional_tests/launch_example.sh

+#!/bin/bash
+
+sbatch -W -p workq -J functional_test --mem=6G \
+	--wrap="module load bioinfo/Nextflow-v21.04.1 ; module load system/singularity-3.7.3 ; nextflow run -profile test_genotoul_workq <metagwgs-src>/main.nf --step '01_clean_qc,02_assembly,03_filtering,04_structural_annot,05_alignment,06_func_annot,07_taxo_affi' --reads 'metagwgs-test-datasets-small/input/*_{R1,R2}.fastq.gz' --host_fasta 'metagwgs-test-datasets-small/input/host/Homo_sapiens.GRCh38_chr21.fa' --host_bwa_index 'metagwgs-test-datasets-small/input/host/Homo_sapiens.GRCh38_chr21.fa.{amb,ann,bwt,pac,sa}' --min_contigs_cpm 1000 --kaiju_db_dir '<bank>/kaijudb_refseq_2020-05-25' --taxonomy_dir '<bank>/taxonomy' --eggnog_mapper_db_dir '<bank>/eggnog-mapper-2.0.4-rf1/data' --diamond_bank '<bank>/refseq_bacteria_2021-05-20/refseq_bacteria.dmnd' -with-report -with-timeline -with-trace -with-dag"
\ No newline at end of file