Development and Debugging

Available in latch >= 2.65.7

Workflows on Latch run inside a container built from your Dockerfile. This environment can differ significantly from your local machine (different OS, Python version, system packages, or installed dependencies). These differences can cause issues that don’t appear during local testing. To avoid surprises after registration, it’s best to debug inside the exact environment your workflow will use. The latch develop command lets you build your workflow image and drop into an interactive shell within that container. From there, you can:

Run individual tasks or workflows directly
Check installed packages and dependencies
Test imports, data access, and other environment-specific behavior

This approach ensures that the code you test is running in the same conditions it will see when executed on Latch, helping you catch both environment and logic issues before running the workflow end-to-end.

Setup

Install latch version 2.65.7 or later.
Start a development shell for your workflow by using the following commands:

$ cd test-wf # Navigate to your workflow directory
$ latch register --staging . # The `--stagign` flag builds image without releasing a new version to Latch Console
$ latch develop . # Open interactive shell inside the workflow environment

This opens an interactive shell on a remote instance running the environment built from your Dockerfile.

Example Usage

On your local machine, create test scripts first, then run them from within the latch develop interactive shell to validate your workflow. Below is a sample test script for the nf-core/atac-seq workflow. This script runs nf_nf_core_atacseq function with multiple input samples stored in public S3 buckets:

nf-core/atacseq Test Example

from wf.__init__ import *

nf_nf_core_atacseq(
    input=[
        SampleSheet(
            sample="Sample_6",
            fastq_1=LatchFile("s3://latch-public/test-data/35929/Test_Dataset_Verified/Sample_6/Sample_6.Rep_1.R1.fastq.gz"),
            fastq_2=LatchFile("s3://latch-public/test-data/35929/Test_Dataset_Verified/Sample_6/Sample_6.Rep_1.R2.fastq.gz"),
            replicate=1,
        ),
        SampleSheet(
            sample="Sample_6",
            fastq_1=LatchFile("s3://latch-public/test-data/35929/Test_Dataset_Verified/Sample_6/Sample_6.Rep_2.R1.fastq.gz"),
            fastq_2=LatchFile("s3://latch-public/test-data/35929/Test_Dataset_Verified/Sample_6/Sample_6.Rep_2.R2.fastq.gz"),
            replicate=2,
        ),
        SampleSheet(
            sample="Sample_1",
            fastq_1=LatchFile("s3://latch-public/test-data/35929/Test_Dataset_Verified/Sample_1/Sample_1.Rep_1.R1.fastq.gz"),
            fastq_2=LatchFile("s3://latch-public/test-data/35929/Test_Dataset_Verified/Sample_1/Sample_1.Rep_1.R2.fastq.gz"),
            replicate=1,
        ),
        SampleSheet(
            sample="Sample_1",
            fastq_1=LatchFile("s3://latch-public/test-data/35929/Test_Dataset_Verified/Sample_1/Sample_1.Rep_2.R1.fastq.gz"),
            fastq_2=LatchFile("s3://latch-public/test-data/35929/Test_Dataset_Verified/Sample_1/Sample_1.Rep_2.R2.fastq.gz"),
            replicate=2,
        ),
    ],
    genome_source="latch_genome_source",
    run_name="Test-1",
    read_length=50,
    aligner=Aligner.bowtie2,
    fasta=None,
    gtf=None,
    outdir=LatchOutputDir("latch:///ATAC_Seq_Test"),
    latch_genome=Reference.hg19,
    with_control=False,
    skip_trimming=False,
    skip_qc=False,
)

Inside latch-develop shell

python3 ~/atacseq/tests/main.py

See the full GitHub repository here.

If you are using local files while testing with latch develop, the workflow function signature still requires a LatchFile object. Pass the local file path to LatchFile, for example LatchFile("/root/path/to/local_test_file.fastq.gz")

`latch develop` Sync Behavior

latch develop syncs files from your local workflow directory using rsync. Files outside this directory are not synced. If you need test data, place it in a subfolder inside your workflow directory.
Updated or new local files overwrite matching files in the container. All code changes must be made locally. Edits made inside the latch develop container are not saved back to your machine and may be overwritten during sync.
Deleted local files are not removed from the container.
If you change your Dockerfile, run latch register --staging to rebuild the image, then run latch develop again to enter the updated environment.

Advanced Notes for Nextflow Users

1. The `initialize` task will always fail in `latch develop`

In a normal Latch production run, the initialize task provisions a shared POSIX filesystem volume (OFS) and returns an identifier (usually passed as pvc_name) so downstream Nextflow tasks can read and write to a common mount. This works because the task is running inside Latch’s execution environment, which has access to the internal dispatcher service that provisions storage and Kubernetes infrastructure to mount the volumeWhen you run latch develop, none of those services exist — you’re just inside an interactive shell in the workflow container. As a result, any attempt to call initialize will fail.The solution is to either run the runtime_task directly using "" as the value of pvc_name or adding a conditional if check in the initialize task as below:

@custom_task(...)
def initialize(run_name: str) -> str:
    if os.environ.get("LATCH_NF_DEBUG") is not None:
        return ""
    
    ...

Full GitHub repository example here.

2. Update `latch.config` default executor

In your latch.config, set the executor to local in debug mode so tasks can run without k8s. This ensures you can still test task logic inside latch develop without depending on infrastructure that only exists in production.

    process {
    executor = 'k8s'
}

if (System.getenv("LATCH_NF_DEBUG") == "true") {
    process {
        executor = 'local'
    }
}

Full GitHub repository example here

3. Use a compatible Nextflow base image

4. Use the Python environment with Latch installed

When running tasks, the Python interpreter must have the latch SDK. If your container uses another environment (e.g., mamba/conda), it may:

Not include latch package
Override $PATH so the wrong Python is used

Best practice:

Use /usr/local/bin/python3 (default in Latch base images)
Install extra packages into this env instead of creating a separate one
Check with:

which python3

5. Reduce resource requests

6. Optionally, lower Java CPU usage

Choosing Instance Size

This feature is available in latch versions 2.59.0 and later.

Use the --instance-size flag to run latch develop on a specific instance type. This is useful when debugging code that depends on certain hardware specs. Example:

$ latch develop . --instance-size small_gpu_task

runs on an instance with 1 NVIDIA T4 GPU. Supported instance sizes:

Instance Size	Cores	RAM	GPU
`small_task`	2	4 GiB	–
`medium_task`	30	100 GiB	–
`small_gpu_task`	7	30 GiB	1× NVIDIA T4
`large_gpu_task`	63	245 GiB	1× NVIDIA A10G
`v100_x1_task`	7	48 GiB	1× NVIDIA V100
`g6e_xlarge_task`	4	32 GiB	1× NVIDIA L40S

Note: Larger instances may take 5+ minutes to start.

Workflows & SDK

Developer Tools

Workflow User Interface

Testing

Latch Console Dashboard

Ready-to-Use Workflows

Development and Debugging

Setup

Example Usage

`latch develop` Sync Behavior

Advanced Notes for Nextflow Users

Choosing Instance Size

Workflows & SDK

Developer Tools

Workflow User Interface

Testing

Latch Console Dashboard

Ready-to-Use Workflows

​Setup

​Example Usage

​latch develop Sync Behavior

​Advanced Notes for Nextflow Users

​Choosing Instance Size

Setup

Example Usage

`latch develop` Sync Behavior

Advanced Notes for Nextflow Users

Choosing Instance Size