The workflow environment is encapsulated in a Docker container, which is created from a recipe defined in a text document named Dockerfile. Latch provides several baseline environments which each latch workflow inherits from. In most cases, modifying the
Dockefile manually is unnecessary, so Latch will automatically generate one using conventional dependency lists and heuristics. To use a handwritten Dockerfile, run the eject command.
Automatic Dockerfile Generation
Below is the list of files used when auto-generating Dockerfiles.
If auto-generation does not cover your use case, please open a suggestion on GitHub.
Dependencies from a
requirements.txt file will be automatically installed using
pip install --requirement.
pyproject.toml files are not supported.
Any script in an
environment.R file will be automatically executed when the workflow is built. This is intended for installing dependencies but there are no actual limits on what the script does.
Currently only R 4.0.0 is supported.
Note that some R packages may have system dependencies that need to be installed using APT or another method. These packages will list these dependencies in their documentation. Missing dependencies will cause crashes during workflow build or when using the packages.
Dependencies from a
system-requirements.txt text document will be automatically installed using
apt-get install --yes
Environment variables from an
.env text document will be automatically set in the workflow environment.
Example of Auto-generated Dockerfile
The following Dockerfile is generated in the
subprocess template (using
latch init --template subprocess --dockerfile example_workflow):
# latch base image + dependencies for latch SDK --- removing these will break the workflow
run pip install latch==2.12.1
run mkdir /opt/latch
# install system requirements
copy system-requirements.txt /opt/latch/system-requirements.txt
run apt-get update --yes && xargs apt-get install --yes </opt/latch/system-requirements.txt
# copy all code from package (use .dockerignore to skip files)
copy . /root/
# set environment variables
# latch internal tagging system + expected root directory --- changing these lines will break the workflow
env FLYTE_INTERNAL_IMAGE $tag
Note on Python Requirements
The order of python requirement installation is as follows
Consequently, a package specified in the
requirements.txt file will overwrite a previous install of the same packaged installed by the
The auto-generated Dockerfile can be saved to the workflow root using
latch dockerfile <path to workflow root>. Subsequent
latch register and
latch develop commands will use the saved version. This also disables automatic generation so no dependency files will be used and changes in these files will not have any effect.
To start with a custom Dockerfile, the
--dockerfile option for
latch init can be used.
This can be used to switch to a more complicated handwritten Dockerfile or to debug any issues with auto-generation. Removing the Dockerfile will re-enable automatic generation.
If you use ejection because auto-generation does not cover your use case, please open a suggestion on GitHub.
By default, all files in the workflow root directory are included in the workflow build. Any unnecessary files will increase the resulting workflow container image size and increase registration and startup time proportional to their size.
To exclude files from the build use a
.dockerignore. Files can be specified one at a time or using glob patterns.
.dockerignore includes files auto-generated by Latch.
GPU Task Limitations
Commands that require certain kernel capabilities will fail with “Permission denied” in GPU tasks (
large-gpu-task). This includes
chroot among others.