Learn how to upload a Nextflow workflow on Latch.
>=3.8
and <=3.11
>= 2.52.3
nextflow_schema.json
file.
If your workflow does not have a nextflow_schema.json
file, you must manually define the Nextflow metadata and input parameters.
nextflow_schema.json
and generates two files:
NextflowMetadata
object, and the second file contains the input parameter definitions.
Before continuing, we will need to make a few updates to the generated files to ensure that the input parameters are correctly defined:
hisat2_index
, salmon_index
, and rsem_index
are typing.Optional[str]
instead of typing.Optional[LatchFile]
. Update these parameters to their correct types.test
configuration profile.
To simplify the user interface, remove any input parameters not defined in conf/test.config
from your latch_metadata/parameters.py
.latch_metadata/parameters.py
file should now look like this:
Example:
NextflowMetadata
object:
display_name
: The display name of the workflow, as it will appear on the Latch UI.
author
: Name of the person or organization that publishes the workflow
parameters
: Input parameters to the workflow, defined as NextflowParameter
objects. The Latch Console will expose these parameters to scientists before they execute the workflow.
Input parameters are passed to Nextflow as command line arguments via --param-name param-value
. Therefore, the key
of the parameters
dictionary should match the name of the parameter in the Nextflow script.
runtime_resources
: The resources the Nextflow Runtime requires to execute the workflow. The storage_gib
field will configure the storage size in GiB for the shared filesystem.
log_dir
: Latch directory to dump .nextflow.log
file on workflow failure.
input
parameter in our rnaseq workflow currently accepts a samplesheet as a CSV formatted LatchFile. Generating and handling
samplesheet files can be cumbersome and error-prone.
Latch Registry is a friendly table interface that allows users to fill out a sample sheet and link sequencing file for each sample.
The section below outlines how to configure a Nextflow workflow to accept a Latch Registry samplesheet (instead of a LatchFile).
nf-core/rnaseq
, add the following snippet to your latch_metadata/parameters.py
file:
input
parameter in the generated_parameters
dictionary in latch_metadata/parameters.py
and make the following changes:
latch_metadata/parameters.py
should now look like this:
latch register .
: Searches for a Latch workflow in the current directory and registers it to Latch.
--nf-script main.nf
: Specifies the Nextflow script passed to the Nextflow command at runtime. For this workflow: nextflow run main.nf
--nf-execution-profile docker,test
: Defines the execution profile to use when running the workflow on Latch. We specify the docker
configuration profile to execute processes in a containerized environment.
latch.config
- a Nextflow configuration file passed to Nextflow via the -config
flag.wf/entrypoint.py
- the generated Latch SDK workflow code that executes the Nextflow pipeline.latch register
command. This will take you to an interface like the one below:
Dockerfile
. Normally this Dockerfile
is autogenerated and stored in .latch
, but if there is already a Dockerfile
in the workflow directory prior to registering, it will be used to build this image. This can result in errors down the line if the Dockerfile is not generated by Latch.Copy to Workspace
button in the top right corner.
samplesheet.csv
file you copied from the provided test data.
Launch Workflow
in the bottom right corner to execute the workflow.
Executions
tab of your workflow.
Graph & Logs
tab, you can view the generated two-stage DAG with the initialization step and the Nextflow runtime task.
If you click on the Nextflow runtime node, you can view the runtime logs generated by Nextflow.
Process Nodes
tab will appear in the menu bar where you can monitor the status of each process in the workflow.
Each node in the DAG represents a process in the Nextflow pipeline.
outdir
parameter
to prepend publishDir paths. For example, if we set our outdir
parameter to latch:///nf-rnaseq/outputs
, all pipeline outputs will be published to the
nf-rnaseq/outputs
directory in Latch Data.
entrypoint.py
latch register
command generates a Latch workflow that runs the Nextflow workflow. In order to provide developers with flexibility
over how their Nextflow pipelines are executed on Latch, the generated workflow code can be modified to execute custom pre- and post-processing logic.
In this tutorial, we will modify the generated wf/entrypoint.py
file to add a Run Name
parameter that will be used to namespace the outputs of the Nextflow pipeline.
To do this, add the run_name
parameter to your entrypoint.py
file as follows:
run_name
parameter to the nf_nf_core_rnaseq
workflow function. All parameters defined in the nf_nf_core_rnaseq
function are
exposed the the user in the Latch UI.run_name
parameter to the nextflow_runtime
task in the body of the workflow function.run_name
parameter to the nextflow_runtime
task signature.nextflow_runtime
task to append the run_name
parameter to the outdir
parameter before executing the Nextflow pipeline.
--nf-script
flag in the latch register
command to avoid
re-generating the Latch SDK workflow code (which will overwrite our updates).
latch generate-metadata
) converts nextflow_schema.json
into Python definitions for parameters and UI configuration.nextflow_schema.json
and adjust parameter types for correctness.latch register --nf-script main.nf
and required execution profiles.entrypoint.py
for additional pre- or post-processing logic.--nf-script
to preserve your code modifications.