Reads QC Workflow (v1.0.1)

../_images/1_RQC_rqc_workflow.png

Workflow Overview

This workflow utilizes the program “rqcfilter2” from BBTools to perform quality control on raw Illumina reads. The workflow performs quality trimming, artifact removal, linker trimming, adapter trimming, and spike-in removal (using BBDuk), and performs human/cat/dog/mouse/microbe removal (using BBMap).

Workflow Availability

The workflow from GitHub uses all the listed docker images to run all third-party tools. The workflow is available in GitHub: https://github.com/microbiomedata/ReadsQC; the corresponding Docker image is available in DockerHub: https://hub.docker.com/r/microbiomedata/bbtools.

Requirements for Execution

(recommendations are in bold)

  • WDL-capable Workflow Execution Tool (Cromwell)
  • Container Runtime that can load Docker images (Docker v2.1.0.3 or higher)

Hardware Requirements

  • Disk space: 106 GB for the RQCFilterData database
  • Memory: >40 GB RAM

Workflow Dependencies

Third party software (This is included in the Docker image.)

Requisite database

The RQCFilterData Database must be downloaded and installed. This is a 106 GB tar file which includes reference datasets of artifacts, adapters, contaminants, the phiX genome, and some host genomes.

The following commands will download the database:

mkdir refdata
wget http://portal.nersc.gov/dna/microbial/assembly/bushnell/RQCFilterData.tar
tar -xvf RQCFilterData.tar -C refdata
rm RQCFilterData.tar

Sample dataset(s)

Zymobiomics mock-community DNA control (SRR7877884); the original gzipped dataset is ~4 GB.

Note

If the input data is paired-end data, it must be in interleaved format. The following command will interleave the files, using the above dataset as an example:

paste <(zcat SRR7877884_1.fastq.gz | paste - - - -) <(zcat SRR7877884_2.fastq.gz | paste - - - -) | tr '\t' '\n' | gzip -c > SRR7877884-int.fastq.gz

For testing purposes and for the following examples, we used a 10% sub-sampling of the above dataset: SRR7877884-int-0.1.fastq.gz. This dataset is already interleaved.

Inputs

A JSON file containing the following information:

  1. the path to the database
  2. the path to the interleaved fastq file (input data)
  3. the path to the output directory
  4. (optional) parameters for memory
  5. (optional) number of threads requested

An example input JSON file is shown below:

{
    "jgi_rqcfilter.database": "/path/to/refdata",
    "jgi_rqcfilter.input_files": [
        "/path/to/SRR7877884-int-0.1.fastq.gz "
    ],
    "jgi_rqcfilter.outdir": "/path/to/rqcfiltered",
    "jgi_rqcfilter.memory": "35G",
    "jgi_rqcfilter.threads": "16"
}

Note

In an HPC environment, parallel processing allows for processing multiple samples. The “jgi_rqcfilter.input_files” parameter is an array data structure. It can be used for multiple samples as input separated by a comma (,). Ex: “jgi_rqcfilter.input_files”:[“first-int.fastq”,”second-int.fastq”]

Output

A directory named with the prefix of the FASTQ input file will be created and multiple output files are generated; the main QC FASTQ output is named prefix.anqdpht.fastq.gz. Using the dataset above as an example, the main output would be named SRR7877884-int-0.1.anqdpht.fastq.gz. Other files include statistics on the quality of the data; what was trimmed, detected, and filtered in the data; a status log, and a shell script documenting the steps implemented so the workflow can be reproduced.

Part of an example output JSON file is shown below:

SRR7877884-int-0.1
|-- SRR7877884-int-0.1.anqdpht.fastq.gz
|-- filterStats.txt
|-- filterStats.json
|-- filterStats2.txt
|-- adaptersDetected.fa
|-- reproduce.sh
|-- spikein.fq.gz
|-- status.log
|-- ...

Below is an example of all the output directory files with descriptions to the right.

FileName Description
SRR7877884-int-0.1.anqdpht.fastq.gz main output (clean data)
adaptersDetected.fa adapters detected and removed
bhist.txt base composition histogram by position
cardinality.txt estimation of the number of unique kmers
commonMicrobes.txt detected common microbes
file-list.txt output file list for rqcfilter2.sh
filterStats.txt summary statistics
filterStats.json summary statistics in JSON format
filterStats2.txt more detailed summary statistics
gchist.txt GC content histogram
human.fq.gz detected human sequence reads
ihist_merge.txt insert size histogram
khist.txt kmer-frequency histogram
kmerStats1.txt synthetic molecule (phix, linker, lamda, pJET) filter run log
kmerStats2.txt synthetic molecule (short contamination) filter run log
ktrim_kmerStats1.txt detected adapters filter run log
ktrim_scaffoldStats1.txt detected adapters filter statistics
microbes.fq.gz detected common microbes sequence reads
microbesUsed.txt common microbes list for detection
peaks.txt number of unique kmers in each peak on the histogram
phist.txt polymer length histogram
refStats.txt human reads filter statistics
reproduce.sh the shell script to reproduce the run
scaffoldStats1.txt detected synthetic molecule (phix, linker, lamda, pJET) statistics
scaffoldStats2.txt detected synthetic molecule (short contamination) statistics
scaffoldStatsSpikein.txt detected skipe-in kapa tag statistics
sketch.txt mash type sketch scanned result against nt, refseq, silva database sketches.
spikein.fq.gz detected skipe-in kapa tag sequence reads
status.log rqcfilter2.sh running log
synth1.fq.gz detected synthetic molecule (phix, linker, lamda, pJET) sequence reads
synth2.fq.gz detected synthetic molecule (short contamination) sequence reads

Version History

  • 1.0.1 (release date 02/16/2021; previous versions: 1.0.0)

Point of contact