latch init
will use OFS as an underlying storage. Follow the Nextflow Tutorial in order to generate a Nextflow project on Latch.
To configure your filesystem, you can change the initialize
method in wf/entrypoint.py
file.
initialize
step of the workflow will provision a shared filesystem. Here you can configure which filesystem you want to use for your workflow.http://nf-dispatcher-service.flyte.svc.cluster.local/provision-storage-ofs
to provision OFS storage or use http://nf-dispatcher-service.flyte.svc.cluster.local/provision-storage-efs
to provision EFS storage.
storage_expiration_hours
option specifies when to clean up the data in your storage. Set to 0
hours to delete storage after execution completes. Set this parameter to a non-zero value to keep storage for relaunching.
version
option specifies the version of Nextflow integration to use. For most cases, it should always be set to 2
for new workflows unless you are running a legacy workflow built on version 1
of Nextflow integration.fs_size_tb
(OFS only) - approximate expected size in TBs for the filesystem. OFS requires you to specify the filesystem size ahead of time to provision more memory for the task. See OFS task memory requirement for more details.fs_size_tb
parameter you can specify an approximate storage size of your filesystem which will change the memory requirement for each workflow task. The following are the memory requirements for file system size:
Filesystem Size(TB) | Additional Task Memory Request(GB) |
---|---|
0-1 | 2 |
2-10 | 3 |
11-20 | 4 |
21-30 | 5 |
31-40 | 6 |
41-50 | 7 |
EFS | ObjectiveFS | |
---|---|---|
Storage($/GB/month) | 0.30 | 0.023 |
Throughput Reads ($/GB/month) | 0.03 | N/A |
Throughput Writes ($/GB/month) | 0.06 | N/A |
Mount Cost($/mount/hr) | N/A | 0.18 |
RAM cost($/GiB/hr) | N/A | 0.009972 |
EFS | ObjectiveFS | |
---|---|---|
Sequential Read(MB/s) | 92.55 | 122.23 |
Sequential Write(MB/s) | 124.20 | 125.07 |
Random Read(MB/s) | 58.39 | 77.79 |
Random Write(MB/s) | 73.69 | 87.90 |
Staging to workdir from LData(MB/s) | 212.31 | 188.40 |
Writing to LData from workdir(MB/s) | 246.64 | 208.74 |
EFS | ObjectiveFS | |
---|---|---|
Cost | +++ | + |
Performance | +++ | ++ |
Execution time | + | ++ |