The following guide will walk you through how to upload an existing Snakemake workflow to the Latch console in four simple steps. If you do not already have a Snakemake workflow, see the Tutorial to get started.
- Register for an account and log into the Latch Console
- Install a compatible version of Python. The Latch SDK is currently only supported for Python
- Install the Latch SDK with Snakemake support. We recommend installing Latch SDK in a fresh environment for best behavior:
$ python3 -m venv env
$ source env/bin/activate
$ pip install "latch[snakemake]"
- Verify the Latch SDK version >= 2.38.8
$ latch --version
latch, version 2.38.8
Step 1: Ensure your Snakefile is Cloud Compatible
Update your workflow’s
Snakefile to ensure compatibility with cloud execution on Latch.
Step 2: Define Metadata and Input Parameters
To construct a graphical interface from a Snakemake workflow, the input parameters need to be explicitly identified and defined so that they can be presented to scientists through a web application.
The Latch SDK expects these parameters to be defined as a Python
SnakemakeMetadata object. Fortunately, the Latch SDK provides a command to automatically generate this object from an existing config file. Run the following command in the root directory of your project:
latch generate-metadata <path_to_config.yaml>
This command will generate two files:
The first file holds the
SnakemakeMetadata object, and the second file contains the input parameter definitions. Be sure to inspect the resulting files to verify that the input parameters are as expected.
Below is an explanation of the most relevant fields of the
Display name of the workflow, as it will appear on the Latch UI.
Points to the folder in Latch Data where Snakemake outputs are stored after the workflow finishes executing.
Input parameters to the workflow, defined as
SnakemakeParameter objects. The Latch Console will expose these parameters to scientists before they execute the workflow. The
default fields for each parameter are inferred from the config file when the
generate-metadata command is run.
Every input parameter of type LatchFile or LatchDir must have a corresponding
SnakemakeFileMetadata in the
file_metadata field. LatchFile and LatchDir are pointers to remote files that are hosted on the Latch platform. The
SnakemakeFileMetadata object provides required metadata about each remote file, such as:
path: The local path inside the container where the workflow engine will copy Latch Data files/directories before the job executes
True, exposes the local file path in the Snakemake config
True, downloads the file in the JIT step instead of creating an empty file. Note: To limit network consumption, the
downloadfield should only be
Truefor files that the Snakefile reads at compilation time (such as configuration files).
Step 3: Define Container Environment
When registering a Snakemake workflow on Latch, we need to build a single container image containing all your runtime dependencies and the Latch packages. By default, all tasks (including the JIT step) will run inside this container.
The Latch SDK provides a command to generate a Dockerfile with all the Latch-specific dependencies required to run Snakemake workflows in the cloud. This command also attempts to infer and install the workflow’s runtime dependencies based on the configuration files in your root directory. Read here for more details.
Run the following command in the root directory of your project:
latch dockerfile . --snakemake
Inspect the resulting
Dockerfile and verify that it installs all required runtime dependencies for your workflow.
Sometimes, it is preferable to use isolated environments for each Snakemake rule using the
conda Snakemake directives instead of building one large image.
When using the
container directive, the Latch workflow will:
- Launch the workflow container.
- Execute Latch-specific commands to setup the Snakemake job.
- Pull the user-specified image and execute the Snakemake job in a sub-container of the workflow container.
To configure your environment on Latch, add the
env_config field to your workflow’s
SnakemakeMetadata object in
latch_metadata/__init__.py (this field is similar to the
--use-singularity flags in Snakemake). For example:
from latch.types.metadata import SnakemakeMetadata, SnakemakeFileParameter, EnvironmentConfig
from latch.types.directory import LatchDir
from latch.types.metadata import LatchAuthor, LatchMetadata, LatchParameter
from pathlib import Path
Step 4: Register the Workflow
To register a Snakemake workflow to Latch, type:
latch register . --snakefile Snakefile
After registering your workflow to Latch, click on the link provided in the output of the
latch register command. This will take you to an interface like the one below:
To execute the workflow, provide appropriate input parameters and click
Launch Workflow in the bottom right.