> ## Documentation Index
> Fetch the complete documentation index at: https://wiki.latch.bio/llms.txt
> Use this file to discover all available pages before exploring further.

# Workflow Environment

Workflow code is rarely free of dependencies. It may require python or system packages or make use of environment variables. For example, a task that downloads compressed reference data from AWS S3 will need the `aws-cli` and `unzip` [APT](https://en.wikipedia.org/wiki/APT_\(software\)) packages, then use the `pyyaml` python package to read the included metadata.

The workflow environment is encapsulated in [a Docker container](https://en.wikipedia.org/wiki/Docker_\(software\)), which is created from a recipe defined in [a Dockerfile](https://docs.docker.com/engine/reference/builder/).

Latch provides automatic Dockerfile generation via the `latch dockerfile` command. You can pass a set of requirements files (detailed below) to this command to configure the generated Dockerfile to install specific dependencies in the environment.

### Python: `requirements.txt`

Dependencies from a [`requirements.txt` file](https://pip.pypa.io/en/stable/reference/requirements-file-format/) can be automatically installed using `pip`. In order to enable this, pass the path to your `requirements.txt` file to the `latch dockerfile` command using the `-p/--pip-requirements` option.

<Accordion title="requirements.txt">
  ```
  boto3==1.20.24
  boto3-stubs[s3,sts,sns,ses,logs]
  kubernetes awscli==1.22.24
  ```
</Accordion>

<Accordion title="Command">
  ```shell theme={null}
  latch dockerfile -p requirements.txt .
  ```
</Accordion>

<Accordion title="Generated Docker Commands">
  ```Dockerfile theme={null}
  copy requirements.txt /opt/latch/requirements.txt
  run pip install --requirement /opt/latch/requirements.txt
  ```
</Accordion>

### Python: `setup.py`, PEP-621 `pyproject.toml`

Workflows with a package specification in a [`setup.py` file](https://docs.python.org/3/distutils/setupscript.html) or a [PEP-621 compliant `pyproject.toml` file](https://peps.python.org/pep-0621/) can be automatically installed using `pip`. Pass the path to either the `setup.py` or the `pyproject.toml` using the `-i/--pyproject` option.

[Poetry `pyproject.toml` files](https://python-poetry.org/docs/pyproject/) are not supported.

<Accordion title="setup.py">
  ```python theme={null}
  from setuptools import setup

  setup(
      name='alphafold',
      version='2.2.3',
      author='DeepMind',
      ...
  )
  ```
</Accordion>

<Accordion title="Command">
  ```shell theme={null}
  latch dockerfile -i setup.py .
  ```
</Accordion>

<Accordion title="Generated Docker Commands">
  ```Dockerfile theme={null}
  copy . /root/
  run pip install --editable /root/
  ```
</Accordion>

### Conda `environment.yaml`

The [Conda](https://docs.conda.io/en/latest/) environment in an [`environment.yaml` file](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#create-env-file-manually) can be automatically installed using `mamba env create --file` with latest [mamba](https://mamba.readthedocs.io/en/latest/user_guide/mamba.html) installed via [Miniforge](https://github.com/conda-forge/miniforge). This environment will be used by default.

To enable, pass the path to the environment file using the `-c/--conda-env` option.

<Accordion title="Example File">
  ````yaml name: workflow channels: - conda-forge - defaults dependencies: - theme={null}
  python=3.7 - bwakit=0.7.17 variables: reference: ~/covid19 ```
  </Accordion>

  <Accordion title="Command">
  ```shell
  latch dockerfile -c environment.yaml .
  ````
</Accordion>

<Accordion title="Generated Docker Commands">
  ```Dockerfile theme={null}
  # Install Mambaforge
  run apt-get update --yes && \
      apt-get install --yes curl git && \
      curl \
          --location \
          --fail \
          --remote-name \
          https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh && \
      `# Docs for -b and -p flags: https://docs.anaconda.com/anaconda/install/silent-mode/#linux-macos` \
      bash Miniforge3-Linux-x86_64.sh -b -p /opt/conda -u && \
      rm Miniforge3-Linux-x86_64.sh

  # Set conda PATH
  env PATH=/opt/conda/bin:$PATH
  run conda config --set auto_activate_base false

  # Build conda environment
  copy environment.yml /opt/latch/environment.yaml
  run mamba env create \
      --file /opt/latch/environment.yaml \
      --name workflow
  env PATH=/opt/conda/envs/workflow/bin:$PATH
  ```
</Accordion>

### R: `environment.R`

A script in an `environment.R` file can be automatically executed when the Dockerfile is built. This is intended for installing dependencies but there are no actual limits on what the script does. The script is executed using [`rig`](https://github.com/r-lib/rig). By default, this uses the latest `R` version, though you can change this by editing the `run rig add release` line (shown below).

To enable this, pass in the path to the R file using the `-r/--r-env` flag.

Note that some R packages may have system dependencies that need to be installed using APT or another method. These packages will list these dependencies in their documentation. Missing dependencies will cause crashes during workflow build or when using the packages.

<Accordion title="environment.R">
  ```R theme={null}
  install.packages("RCurl")
  install.packages("BiocManager")

  BiocManager::install("S4Vectors")
  ```
</Accordion>

<Accordion title="Command">
  ```shell theme={null}
  latch dockerfile -r environment.R .
  ```
</Accordion>

<Accordion title="Generated Docker Commands">
  ```Dockerfile theme={null}
  # Install rig the R installation manager
  run \
      curl \
          --location \
          --fail \
          --remote-name \
          https://github.com/r-lib/rig/releases/download/latest/rig-linux-latest.tar.gz && \
      tar \
          --extract \
          --gunzip \
          --file rig-linux-latest.tar.gz \
          --directory /usr/local/ && \
      rm rig-linux-latest.tar.gz

  # Install R
  run rig add release # Change to any R version

  # Install R dependencies
  copy version /opt/latch/environment.R
  run Rscript /opt/latch/environment.R

  ```
</Accordion>

### System: APT

A text file containing `apt` dependencies can also be installed by default. Each line of the file must contain an apt package (with format consistent with what is specified [here](https://linux.die.net/man/8/apt-get)). This can be enabled by passing the path to the file with the `-a/--apt-requirements` option.

<Accordion title="system-requirements.txt">
  ```

  autoconf
  samtools

  ```
</Accordion>

<Accordion title="Command">
  ```shell theme={null}
  latch dockerfile -a system-requirements.txt .
  ```
</Accordion>

<Accordion title="Generated Docker Commands">
  ```Dockerfile theme={null}
  copy system-requirements.txt /opt/latch/system-requirements.txt
  run apt-get update --yes && \
      xargs apt-get install --yes < /opt/latch/system-requirements.txt
  ```
</Accordion>

### Environment Variables

Environment variables contained in a file can be automatically added to the workflow environment. Pass the path of the file containing the environment variables using the `-d/--direnv` option.

<Accordion title=".env">
  ```
  BOWTIE2_INDEXES=reference
  PATH="/root/bowtie2:$PATH"
  ```
</Accordion>

<Accordion title="Command">
  ```shell theme={null}
  latch dockerfile -d .env .
  ```
</Accordion>

<Accordion title="Generated Docker Commands">
  ```Dockerfile theme={null}
  env BOWTIE2_INDEXES="reference"
  env PATH="/root/bowtie2:$PATH"
  ```
</Accordion>

## Example of Auto-generated Dockerfile

The following Dockerfile is generated in the `subprocess` template (using `latch init --template subprocess --dockerfile example_workflow`):

```Dockerfile theme={null}
# latch base image + dependencies for latch SDK --- removing these will break the workflow
from 812206152185.dkr.ecr.us-west-2.amazonaws.com/latch-base:ace9-main
run pip install latch==2.12.1
run mkdir /opt/latch

# install system requirements
copy system-requirements.txt /opt/latch/system-requirements.txt
run apt-get update --yes && xargs apt-get install --yes </opt/latch/system-requirements.txt

# copy all code from package (use .dockerignore to skip files)
copy . /root/

# set environment variables
env BOWTIE2_INDEXES=reference

# latch internal tagging system + expected root directory --- changing these lines will break the workflow
arg tag
env FLYTE_INTERNAL_IMAGE $tag
workdir /root
```

## Note on Python Requirements

The order of python requirement installation is as follows

1. `conda`
2. `setup.py` / `pyproject.toml`
3. `requirements.txt`

Consequently, a package specified in the `requirements.txt` file will overwrite a previous install of the same packaged installed by the `conda` environment.

## Excluding Files

By default, all files in the workflow root directory are included in the workflow build. Any unnecessary files will increase the resulting workflow container image size and increase registration and startup time proportional to their size.

To exclude files from the build use [a `.dockerignore`.](https://docs.docker.com/engine/reference/builder/#dockerignore-file) Files can be specified one at a time or using glob patterns.

The default `.dockerignore` includes files auto-generated by Latch.

## GPU Task Limitations

Commands that require certain [kernel capabilities](https://man7.org/linux/man-pages/man7/capabilities.7.html) will fail with "Permission denied" in GPU tasks (`small-gpu-task`, `large-gpu-task`, `v100_x*_task`, and `g6e_*_task`). This includes `mount` and `chroot` among others.