LatchFile
/ LatchDir
and LPath
are the two ways to work with remote files on Latch.LPath
is the newer option, designed to mimic Python’s pathlib.Path
for more intuitive and ergonomic file handling.LatchFile
/ LatchDir
remain fully supported and appear in many workflow examples, but we recommend using LPath
for new workflows.See LPath API for details.test.txt
without touching foo.txt
.
local_path
will always be the absolute path on the task’s machine where the
file has been copied to (the machine that your code is running on).
remote_path
will be a remote object URL with s3
or latch
as its host.
There are cases when we would want
to access these local_path
and remote_path
attributes directly:
- Specifying the remote destination of a returned directory (eg. in the above return statement).
- Manually fetching additional files from s3 similar to a passed file’s remote source.
- Using the Latch SDK to list other files similar to a passed file (eg.
latch ls latch:///foo
)
Using Globs to Move Groups of Files
Often times logic is needed to move groups of files together based on a shared pattern. For instance, you may wish to return all files that end with afastq.gz
extension after a
trimming
task has been run.
To do this in the SDK, you can leverage the file_glob
function to construct
lists of LatchFile
s defined by a pattern.
The class of allowed patterns are defined as
globs. It is likely you’ve
already used globs in the terminal by using wildcard characters in common
commands, eg. ls *.txt
.
The second argument must be a valid latch URL pointing to a directory. This will
be the remote location of returned LatchFile
constructed with this utility.
In this example, all files ending with .fastq.gz
in the working directory of
the task will be returned to the latch:///fastqc_outputs
directory:
latch:///
URLs
Recall that URLs (Uniform Resource Locators) describe the location of an object
on the internet.
A simplified representation of a URL string syntax can be denoted as:
https://google.com
and s3://my-bucket/dna.fa
are both valid descriptions of
objects, a webpage or a fasta file.
When referencing files stored within LatchBio’s managed filesystem (called
LatchData) we must use the latch
scheme to appropriately resolve objects to
the appropriate account.
For instance, latch:///foo.txt
might meant two entirely different things in
the context of two different accounts. The resolution to retrieve the correct
object occurs based on the user that executed the workflow,
Some examples of valid latch URLs referencing objects in a user’s filesystem:
latch:///guide_design/off_targets.csv
latch:///foo.txt
Shared latch
URLs
Paths that are shared amongst accounts will bear the latch://shared/<path>
syntax.