Environments for Individual Tasks

Different tasks in a workflow may need different sets of dependencies. Creating a single shared environment can be problematic as the some part of the workflow image will be unused in each task and slow down that task’s startup proportionally to the size of the extraneous chunk. Different dependencies might also need different system package versions in which case installing them together might be impractical.

Instead, consider defining an individual environment for each task using the optional dockerfile parameter in the task definition. Include only the dependencies that each specific task needs.

The value passed to dockerfile must be str literal (meaning no variables or expressions, just a value). If the path is relative, it will be resolved against the package root.

# assemble/__init__.py

from pathlib import Path

from latch import small_task
from latch.types import LatchFile

# Path relative to the task directory

@small_task(dockerfile="wf/assemble/Dockerfile")
def assembly_task(read1: LatchFile, read2: LatchFile) -> LatchFile:
    ...
    return LatchFile(str(sam_file), "latch:///covid_assembly.sam")

# sam_blaster/__init__.py

from pathlib import Path

from latch import small_task
from latch.types import LatchFile

@small_task(dockerfile="wf/sam_blaster/Dockerfile")
def sam_blaster(sam: LatchFile) -> LatchFile:
    ...
    return LatchFile(blasted_sam, f"latch:///{blasted_sam.name}")

Dockerfiles can be organized as follows:

Dockerfile
version
wf
├── __init__.py
├── assemble
│   ├── Dockerfile
│   └── __init__.py
└── sam_blaster
    ├── Dockerfile
    └── __init__.py

The root directory used when building the images is always the workflow directory.

​

Limitations

latch develop uses the Dockerfile in the workflow directory and not any of the individual Dockerfiles