What it does

This is the primary DNA-mapping pipeline. It can be used both alone or upstream of the ATAC-seq and ChIP-seq pipelines. This has a wide array of options, including trimming and various QC steps (e.g., marking duplicates and plotting coverage and PCAs). In addition, basic coverage tracks are created to facilitate viewing the data in IGV.


Input requirements

The only requirement is a directory of gzipped fastq files. Files could be single or paired end, and the read extensions could be modified using the keys in the defaults.yaml file below.

Configuration file

There is a configuration file in snakePipes/workflows/DNA-mapping/defaults.yaml:

## General/Snakemake parameters, only used/set by wrapper or in Snakemake cmdl, but not in Snakefile
pipeline: dna-mapping
local: False
maxJobs: 5
## directory with fastq files
## preconfigured target genomes (mm9,mm10,dm3,...) , see /path/to/snakemake_workflows/shared/organisms/
## Value can be also path to your own genome config file!
## FASTQ file extension (default: ".fastq.gz")
ext: '.fastq.gz'
## paired-end read name extension (default: ['_R1', "_R2"])
reads: [_R1, _R2]
## mapping mode
mode: mapping
aligner: Bowtie2
## Number of reads to downsample from each FASTQ file
## Options for trimming
trim: False
trimmer: cutadapt
## Bin size of output files in bigWig format
bwBinSize: 25
## Run FASTQC read quality control
fastqc: false
## Run computeGCBias quality control
GCBias: false
## Retain only de-duplicated reads/read pairs
dedup: false
## Retain only reads with at least the given mapping quality
mapq: 0
## Retain only reads mapping in proper pairs
properPairs: false
## Mate orientation in paired-end experiments for Bowtie2 mapping
## (default "--fr" is appropriate for Illumina sequencing)
mateOrientation: --fr
## other Bowtie2 stuff
insertSizeMax: 1000
plotFormat: png
UMIBarcode: False
bcPattern: NNNNCCCCCCCC #default: 4 base umi barcode, 8 base cell barcode (eg. RELACS barcode)
UMIDedup: False
UMIDedupSep: "_"
## Median/mean fragment length, only relevant for single-end data (default: 200)
fragmentLength: 200
qualimap: false
verbose: false

Many of these options can be more conveniently set on the command-line (e.g., --qualimap sets qualimap: true). However, you may need to change the reads: setting if your paired-end files are not denoted by sample_R1.fastq.gz and sample_R2.fastq.gz, but rather sample_1.fastq.gz and sample_2.fastq.gz.

Understanding the outputs

The DNA mapping pipeline will generate output of the following structure:

├── bamCoverage
├── Bowtie2
├── deepTools_qc
│   ├── bamPEFragmentSize
│   ├── estimateReadFiltering
│   ├── multiBamSummary
│   ├── plotCorrelation
│   ├── plotCoverage
│   └── plotPCA
├── FastQC
├── filtered_bam
├── multiQC
│   └── multiqc_data
└── Sambamba

In addition to the FASTQ module results (see Running snakePipes), the workflow produces the following outputs:

  • Bowtie2 : Contains the BAM files after mapping with Bowtie2 and indexed by Samtools.
  • filtered_bam : Contains the BAM files filtered by the provided criteria, such as mapping quality (--mapq) or PCR duplicates (--dedup). This file is used for most downstream analysis in the DNA-mapping and ChIP-seq/ATAC-seq pipeline.
  • bamCoverage : Contains the coverage files (bigWig format) produced from the BAM files by deepTools bamCoverage . The files are either raw, or 1x normalized (by sequencing depth). They are useful for plotting and inspecting the data in IGV.
  • deepTools_qc : Contains various QC files and plots produced by deepTools on the filtered BAM files. These are very useful for evaluation of data quality. The folders are named after the tools. Please look at the deepTools documentation on how to interpret the outputs from each tool.
  • Sambamba : Contains the alignment metrices evaluated on the BAM files by Sambamba.

A number of other directories may optionally be present if you specified read trimming, using Qualimap, or a variety of other options. These are typically self-explanatory.

A fair number of useful QC plots are or can be generated by the pipeline. These include correlation and PCA plots as well as the output from MultiQC.


Command line options

MPI-IE workflow for DNA mapping

usage example:
DNA-mapping -i input-dir -o output-dir mm10

usage: DNA-mapping -i INDIR -o OUTDIR [-h] [-v] [--ext EXT]
                   [--reads READS READS] [-c CONFIGFILE]
                   [--clusterConfigFile CLUSTERCONFIGFILE] [-j INT] [--local]
                   [--keepTemp] [--snakemakeOptions SNAKEMAKEOPTIONS] [--DAG]
                   [--version] [--emailAddress EMAILADDRESS]
                   [--smtpServer SMTPSERVER] [--smtpPort SMTPPORT] [--onlySSL]
                   [--emailSender EMAILSENDER] [--smtpUsername SMTPUSERNAME]
                   [--smtpPassword SMTPPASSWORD] [--VCFfile VCFFILE]
                   [--strains STRAINS] [--SNPfile SNPFILE]
                   [--NMaskedIndex NMASKEDINDEX] [-m MODE] [--downsample INT]
                   [--trim] [--trimmer {cutadapt,trimgalore,fastp}]
                   [--trimmerOptions TRIMMEROPTIONS] [--fastqc] [--bcExtract]
                   [--bcPattern BCPATTERN] [--UMIDedup]
                   [--UMIDedupSep UMIDEDUPSEP] [--UMIDedupOpts UMIDEDUPOPTS]
                   [--bwBinSize BWBINSIZE] [--plotFormat STR]
                   [--alignerOpts ALIGNEROPTS] [--cutntag]
                   [--mateOrientation MATEORIENTATION] [--qualimap] [--dedup]
                   [--properPairs] [--mapq INT]
                   [--insertSizeMax INSERTSIZEMAX] [--GCBias]
                   [--aligner {Bowtie2,bwa}]

Positional Arguments

GENOME Genome acronym of the target organism. Either a yaml file or one of: dm3, hg38, dm6, mm10_gencodeM13, SchizoSPombe_ASM294v2, mm9, hs37d5, mm10, GRCz10

Required Arguments

-i, --input-dir
 input directory containing the FASTQ files, either paired-end OR single-end data
-o, --output-dir
 output directory

General Arguments

-v, --verbose verbose output (default: 'False')
--ext Suffix used by input fastq files (default: '".fastq.gz"').
--reads Suffix used to denote reads 1 and 2 for paired-end data. This should typically be either '_1' '_2' or '_R1' '_R2' (default: '['_R1', '_R2']). Note that you should NOT separate the values by a comma (use a space) or enclose them in brackets.
-c, --configFile
 configuration file: config.yaml (default: 'None')
 configuration file for cluster usage. In absence, the default options specified in defaults.yaml and workflows/[workflow]/cluster.yaml would be selected (default: 'None')
-j, --jobs maximum number of concurrently submitted Slurm jobs / cores if workflow is run locally (default: '5')
--local run workflow locally; default: jobs are submitted to Slurm queue (default: 'False')
--keepTemp Prevent snakemake from removing files marked as being temporary (typically intermediate files that are rarely needed by end users). This is mostly useful for debugging problems.
 Snakemake options to be passed directly to snakemake, e.g. use --snakemakeOptions='--dryrun --rerun-incomplete --unlock --forceall'. WARNING! ONLY EXPERT USERS SHOULD CHANGE THIS! THE DEFAULT VALUE WILL BE APPENDED RATHER THAN OVERWRITTEN! (default: '['--use-conda']')
--DAG If specified, a file ending in _pipeline.pdf is produced in the output directory that shows the rules used and their relationship to each other.
--version show program's version number and exit

Email Arguments

--emailAddress If specified, send an email upon completion to the given email address
--smtpServer If specified, the email server to use.
--smtpPort The port on the SMTP server to connect to. A value of 0 specifies the default port.
--onlySSL The SMTP server requires an SSL connection from the beginning.
--emailSender The address of the email sender. If not specified, it will be the address indicated by --emailAddress
--smtpUsername If your SMTP server requires authentication, this is the username to use.
--smtpPassword If your SMTP server requires authentication, this is the password to use.

Allele-specific mapping arguments

--VCFfile VCF file to create N-masked genomes (default: 'None')
--strains Name or ID of SNP strains separated by comma (default: 'None')
--SNPfile File containing SNP locations (default: 'None')
--NMaskedIndex N-masked index of the reference genome (default: 'None')


-m, --mode workflow running modes (available: 'mapping,allelic-mapping')(default: '"mapping"')
--downsample Downsample the given number of reads randomly from of each FASTQ file (default: 'False')
--trim Activate fastq read trimming. If activated, Illumina adaptors are trimmed by default. Additional parameters can be specified under --trimmerOptions. (default: 'False')

Possible choices: cutadapt, trimgalore, fastp

Trimming program to use: Cutadapt, TrimGalore, or fastp. Note that if you change this you may need to change --trimmerOptions to match! (default: '"cutadapt"')

 Additional option string for trimming program of choice. (default: '')
--fastqc Run FastQC read quality control (default: 'False')
--bcExtract To extract umi barcode from fastq file via UMI-tools and add it to the read name (default: 'False')
--bcPattern The pattern to be considered for the barcode. 'N' = UMI position (required) 'C' = barcode position (optional) (default: '"NNNNCCCCCCCC"')
--UMIDedup Deduplicate bam file based on UMIs via umi_tools dedup that are present in the read name. (default: 'False')
--UMIDedupSep umi separation character that will be passed to umi_tools.(default: '"_"')
--UMIDedupOpts Additional options that will be passed to umi_tools.(default: '')
--bwBinSize Bin size of output files in bigWig format (default: '25')

Possible choices: png, pdf, None

Format of the output plots from deepTools. Select 'none' for no plots (default: '"png"')

--alignerOpts Options that will be passed to Bowtie2 or bwa. You can specify things such as --local or --very-sensitive here. The mate orientation and maximum insert size are specified elsewhere. Read group information is set automatically. Note that you may need to escape the first - (e.g., '--very-fast'). Default: ''.
--cutntag if set, Bowti2 is used for mapping with parameters as has been used in the method section of Kaya-okur et al. 2019. ('--local --very-sensitive-local --no-mixed --no-discordant --phred33 -I 10 -X 700')Setting this flag overwrites the '--alignerOpts' and '--insertSizeMax'. Default is 'False'.
 The --fr, --ff, or --rf option for bowtie2 (default: '"--fr"')
--qualimap activate Qualimap (default: 'False')
--dedup retain only de-duplicated reads/read pairs (given single-/paired-end data), recommended for ChIP-seq data (default: 'False')
--properPairs retain only reads mapping in proper pairs (default: 'False')
--mapq retain only reads with at least the given mapping quality. We recommend usingmapq of 3 or more for ChIP-seq to remove all true multimapping reads. (default: '0')
 Maximum insert size allowed during mapping (default: '1000')
--GCBias run computeGCBias quality control (long runtime!). Note that GCBias analysis is skipped if downsampling is specified (default: 'False')

Possible choices: Bowtie2, bwa

Program used for mapping: Bowtie2 or bwa (default: '"Bowtie2"').

code @ github.