Available in
latch >= 2.65.7
latch develop
command lets you build your workflow image and drop into an interactive shell within that container. From there, you can:
- Run individual tasks or workflows directly
- Check installed packages and dependencies
- Test imports, data access, and other environment-specific behavior
Setup
- Install
latch
version2.65.7
or later. - Start a development shell for your workflow by using the following commands:
Example Usage
On your local machine, create test scripts first, then run them from within thelatch develop
interactive shell to validate your workflow.
Below is a sample test script for the nf-core/atac-seq
workflow. This script runs nf_nf_core_atacseq function with multiple input samples stored in public S3 buckets:
nf-core/atacseq Test Example
Inside latch-develop shell
If you are using local files while testing with latch develop, the workflow function signature still requires a LatchFile object. Pass the local file path to LatchFile, for example
LatchFile("/root/path/to/local_test_file.fastq.gz")
latch develop
Sync Behavior
latch develop
syncs files from your local workflow directory using rsync. Files outside this directory are not synced. If you need test data, place it in a subfolder inside your workflow directory.- Updated or new local files overwrite matching files in the container. All code changes must be made locally. Edits made inside the latch develop container are not saved back to your machine and may be overwritten during sync.
- Deleted local files are not removed from the container.
- If you change your Dockerfile, run
latch register --staging
to rebuild the image, then runlatch develop
again to enter the updated environment.
Advanced Notes for Nextflow Users
1. The `initialize` task will always fail in `latch develop`
1. The `initialize` task will always fail in `latch develop`
In a normal Latch production run, the Full GitHub repository example here.
initialize
task provisions a shared POSIX filesystem volume (OFS) and returns an identifier (usually passed as pvc_name
) so downstream Nextflow tasks can read and write to a common mount. This works because the task is running inside Latch’s execution environment, which has access to the internal dispatcher service that provisions storage and Kubernetes infrastructure to mount the volumeWhen you run latch develop
, none of those services exist — you’re just inside an interactive shell in the workflow container. As a result, any attempt to call initialize will fail.The solution is to either run the runtime_task
directly using ""
as the value of pvc_name
or adding a conditional if
check in the initialize
task as below:2. Update `latch.config` default executor
2. Update `latch.config` default executor
In your Full GitHub repository example here
latch.config
, set the executor to local
in debug mode so tasks can run without k8s. This ensures you can still test task logic inside latch develop
without depending on infrastructure that only exists in production.3. Use a compatible Nextflow base image
3. Use a compatible Nextflow base image
In the workflow’s Dockerfile, ensure that you are using either
latch-base-nextflow
versions >= v3.0.5
or >= v2.5.6
(See example)4. Use the Python environment with Latch installed
4. Use the Python environment with Latch installed
When running tasks, the Python interpreter must have the latch SDK. If your container uses another environment (e.g., mamba/conda), it may:
- Not include
latch
package - Override $PATH so the wrong Python is used
- Use
/usr/local/bin/python3
(default in Latch base images) - Install extra packages into this env instead of creating a separate one
- Check with:
5. Reduce resource requests
5. Reduce resource requests
Make sure that your process config requests fewer resources than normal. This is to avoid running into issues where processes won’t run because they request more resources than are available in the
latch develop
’s instance.6. Optionally, lower Java CPU usage
6. Optionally, lower Java CPU usage
If necessary, modify the
-XX:ActiveProcessorCount
in your Nextflow command in the entrypoint.py
file to cap Java’s perceived cores.(See example here)Choosing Instance Size
This feature is available in
latch
versions 2.59.0
and later.--instance-size
flag to run latch develop on a specific instance type. This is useful when debugging code that depends on certain hardware specs.
Example:
Instance Size | Cores | RAM | GPU |
---|---|---|---|
small_task | 2 | 4 GiB | – |
medium_task | 30 | 100 GiB | – |
large_task | 90 | 170 GiB | – |
small_gpu_task | 7 | 30 GiB | 1× NVIDIA T4 |
large_gpu_task | 63 | 245 GiB | 1× NVIDIA A10G |
v100_x1_task | 7 | 48 GiB | 1× NVIDIA V100 |
g6e_xlarge_task | 4 | 32 GiB | 1× NVIDIA L40S |