Workflow Example using S3

Local Run using S3

The previous run was using the local file system for results.

The script also allows to overwrite the output destination and thus, we can use an S3 bucket as a destination.

export BUCKET_NAME_RESULTS=nextflow-spot-batch-result-${RANDOM}-$(date +%s)
aws --region ${AWS_REGION} s3 mb s3://${BUCKET_NAME_RESULTS}
aws s3api put-bucket-tagging --bucket ${BUCKET_NAME_RESULTS} --tagging="TagSet=[{Key=nextflow-workshop,Value=true}]"
aws s3 ls

Mind you that we are tagging the resource S3 to be able to find it for clean-up.

The last command will present you with the name of the bucket.

$ export BUCKET_NAME_RESULTS=nextflow-spot-batch-result-${RANDOM}-$(date +%s)
$ aws --region ${AWS_REGION} s3 mb s3://${BUCKET_NAME_RESULTS}
make_bucket: nextflow-spot-batch-result-20033-1587976463
$ aws s3api put-bucket-tagging --bucket ${BUCKET_NAME_RESULTS} --tagging="TagSet=[{Key=nextflow-workshop,Value=true}]"
$ aws s3 ls
2020-04-27 08:34:25 nextflow-spot-batch-result-20033-1587976463

With the bucket created, we can add the parameter to the Nextflow run:

nextflow run script7.nf --reads 'data/ggal/*_{1,2}.fq' --outdir=s3://${BUCKET_NAME_RESULTS}/simple

The output will look like this.

$ nextflow run script7.nf --reads 'data/ggal/*_{1,2}.fq' --outdir=s3://${BUCKET_NAME_RESULTS}/simple
N E X T F L O W  ~  version 20.01.0
Launching `script7.nf` [peaceful_joliot] - revision: ce58523d1d
R N A S E Q - N F   P I P E L I N E
===================================
transcriptome: /home/ec2-user/environment/nextflow-tutorial/nextflow-tutorial/data/ggal/transcriptome.fa
reads        : data/ggal/*_{1,2}.fq
outdir       : s3://nextflow-spot-batch-result-23641-1587713021/simple
executor >  local (8)
[ce/510041] process > index          [100%] 1 of 1 ✔
[17/d1b1d9] process > quantification [100%] 3 of 3 ✔
[08/207c77] process > fastqc         [100%] 3 of 3 ✔
[b3/169553] process > multiqc        [100%] 1 of 1 ✔
Done! Open the following report in your browser --> s3://nextflow-spot-batch-result-23641-1587713021/simple/multiqc_report.html
$

Now please go to the S3 console (deep link) and check for the report within the bucket.

report_in_s3

Within this workshop we used a local Docker executor in the small example - for the remainder of the workshop we are going to use the awsbatch executor to submit jobs to AWS Batch.

Site Info