The previous run was using the local file system for results.
The script also allows to overwrite the output
destination and thus, we can use an S3 bucket as a destination.
export BUCKET_NAME_RESULTS=nextflow-spot-batch-result-${RANDOM}-$(date +%s)
aws --region ${AWS_REGION} s3 mb s3://${BUCKET_NAME_RESULTS}
aws s3api put-bucket-tagging --bucket ${BUCKET_NAME_RESULTS} --tagging="TagSet=[{Key=nextflow-workshop,Value=true}]"
aws s3 ls
Mind you that we are tagging the resource S3 to be able to find it for clean-up.
The last command will present you with the name of the bucket.
$ export BUCKET_NAME_RESULTS=nextflow-spot-batch-result-${RANDOM}-$(date +%s)
$ aws --region ${AWS_REGION} s3 mb s3://${BUCKET_NAME_RESULTS}
make_bucket: nextflow-spot-batch-result-20033-1587976463
$ aws s3api put-bucket-tagging --bucket ${BUCKET_NAME_RESULTS} --tagging="TagSet=[{Key=nextflow-workshop,Value=true}]"
$ aws s3 ls
2020-04-27 08:34:25 nextflow-spot-batch-result-20033-1587976463
With the bucket created, we can add the parameter to the Nextflow run:
nextflow run script7.nf --reads 'data/ggal/*_{1,2}.fq' --outdir=s3://${BUCKET_NAME_RESULTS}/simple
The output will look like this.
$ nextflow run script7.nf --reads 'data/ggal/*_{1,2}.fq' --outdir=s3://${BUCKET_NAME_RESULTS}/simple
N E X T F L O W ~ version 20.01.0
Launching `script7.nf` [peaceful_joliot] - revision: ce58523d1d
R N A S E Q - N F P I P E L I N E
===================================
transcriptome: /home/ec2-user/environment/nextflow-tutorial/nextflow-tutorial/data/ggal/transcriptome.fa
reads : data/ggal/*_{1,2}.fq
outdir : s3://nextflow-spot-batch-result-23641-1587713021/simple
executor > local (8)
[ce/510041] process > index [100%] 1 of 1 ✔
[17/d1b1d9] process > quantification [100%] 3 of 3 ✔
[08/207c77] process > fastqc [100%] 3 of 3 ✔
[b3/169553] process > multiqc [100%] 1 of 1 ✔
Done! Open the following report in your browser --> s3://nextflow-spot-batch-result-23641-1587713021/simple/multiqc_report.html
$
Now please go to the S3 console (deep link) and check for the report within the bucket.
Within this workshop we used a local Docker executor in the small example - for the remainder of the workshop we are going to use the awsbatch executor to submit jobs to AWS Batch.