Introduction

Data

Registry

Workflows & SDK

Pods

Plots

Administration

Resources

LatchBio

Documentation

API References

Blog

Website

Change Log

Sign Up

Contact Us

Log In

Import Registry Records in Latch SDK Workflows

Latch is a platform for you to store, analyze, and visualize multiomics data.

What is LatchBio?

Quickstart

Latch Data is a cloud based file storage system built for storing biological data.

Overview

What is Latch Data?

Basic Uploading and Downloading

Latch allows you to mount your own AWS S3 Buckets and use them the same as you would any data on Latch. All you need is to connect AWS account with Latch which will allow you to mount any buckets in that account onto Latch.

Mounting S3 Buckets

Latch allows you to mount your own GCP Buckets and use them the same as you would any data on Latch.

Mounting GCP Buckets

This tutorial explains how to get an API key from BaseSpace that you can use to automatically import your sequencing runs into Latch.

BaseSpace Downloader

Command Line Interface

Command Line Interface Data Upload/Download

Data Sharing

FastQC aims to provide a simple way to do some quality control checks on raw sequence data coming from high throughput sequencing pipelines. It provides a modular set of analyses which you can use to give a quick impression of whether your data has any problems of which you should be aware before doing any further analysis.

FastQC

CELLxGENE Explorer allows scientists to execute interactive analyses on a dataset to explore how patterns of gene expression are determined by environmental and genetic factors using an interactive speed no-code UI.

CELLxGENE

The Integrative Genomics Viewer (IGV) is a high-performance, easy-to-use, interactive tool for the visual exploration of genomic data. BAM, Fasta, etc. files can be opened in an IGV Browser natively within the Latch Platform. 

Using IGV on Latch

Learn how to set default viewer settings when opening files in IGV on Latch

Setting Viewer Defaults for IGV on Latch

The IGV browser comes with many hosted options for common Reference Genomes to view Alignment Files to.

IGV Hosted Reference Genomes

Connect your sample sheets, metadata, and analysis — all in one place.

What is Latch Registry?

Learn how to set up your registry to manage your data.

Registry Basics

Manually Create Single Records

Bulk Import Data Using a CSV

Bulk Link Sequencing Files to Existing Records

Benchling Integration

As you centralize your biological data in Latch Registry, you will often find that some records are related or even dependent on one another. Linked records offer a powerful way for you to define relationships between records from different tables, enhancing your data analysis and enabling new insights that may have been difficult to discover without linked records.

Connect Records Across Tables

Using Registry in a Workflow

Account Objects

Registry Projects

Table Objects

Record Objects

Latch's Workflow Manager comes with out-of-the-box community workflows and a Python SDK that allows uploading of custom workflows.

What are Latch Workflows?

CRISPResso2 is a software pipeline for the analysis of genome editing experiments. It is designed to enable rapid and intuitive interpretation of results produced by amplicon sequencing.

CRISPResso2

CAS-OFFinder is an algorithm that searches for potential off-target sites of Cas9 RNA-guided endonucleases. It is one of the most highly cited & consistent tools for this purpose.

CAS-OFFinder

[CRISPOR](http://crispor.org/) is a website that helps select and express CRISPR guide sequences, described in two papers ([Gen Biol 2016](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-016-1012-2) and [NAR 2018](https://academic.oup.com/nar/article/46/W1/W242/4995687)). In its default mode, the user pastes an input DNA sequence and chooses the genome.

CRISPOR

AlphaFold produces highly accurate protein structure predictions from amino acid sequences.

AlphaFold

Every workflow in Latch allows you to import a CSV containing values for its parameters. This is another way to import data when doing bulk runs of a workflow.

CSV Parameter Import

Workflows on Latch provide visibility into the resource usage of each task execution, enabling developers to easily debug and optimize their workflow.

Resource Monitoring

Launching Workflows Programmatically

Quick Start

In this demonstration, we will examine a workflow which sorts and assembles COVID sequencing data.

Authoring your Own Workflow

A workflow is an analysis that takes in some input, processes it in one or more steps and produces some output.

What is a Workflow?

When building a workflow with multiple tasks, it can be difficult to decide when to split larger tasks into smaller tasks. Some of the tradeoffs are listed below to guide this decision:

When to Split Up Tasks

When to split up a task into multiple tasks

Files and directories on Latch can be referred to in code or through the CLI using **Latch URLs**.

Latch URLs

Working with Remote Files

When working with bioinformatics workflows, we are often passing around large files or directories between our tasks. These files are usually located in cloud object stores and are copied to the file systems of the machines on which the task is scheduled.

LatchFile / LatchDir

Task executions produce logs, displayed on the Latch console to provide users visibility into their workflows. However, these logs tend to be terribly verbose. It's tedious to sift through piles of logs looking for useful signals; instead, important information, warnings, and errors should be prominently displayed. This is accomplished through the Latch SDK's new messaging feature.

Messages

When a workflow is executed and tasks are scheduled, the machines needed to run the task are provisioned automatically and managed for the user until task completion. Tasks can be annotated with the resources they are expected to consume (eg. CPU, RAM, GPU) at runtime and these requests will be fullfilled during the scheduling process.

Defining Cloud Resources

Caching allows workflow developers to reuse the results of previously run tasks to prevent wasted time and computation. This is helpful when running large batches of workflows with redundant inputs or when debugging errors in the middle of a workflow where upstream state can be reused.

Caching

In order to support the functionality of an `if-elif-else` clause within the body of a workflow, we introduce the method `create_conditional_section`. This method creates a new conditional section in a workflow, allowing a user to conditionally execute a task based on the value of a task result.

Conditional Sections

There are many pipelines in bioinformatics that require running a processing step in parallel and aggregating their outputs at the end for downstream analysis. A prominent example of this is bulk RNA-sequencing, where alignment is performed to produce transcript abundances per sample, and gene counts of all samples are subsequently merged. Having a single count matrix makes it convenient to use in downstream steps, such as differential gene expression analysis. Another example is performing FastQC on multiple samples and summarizing the results in a MultiQC report.

Map Task

Often a workflow can depend on _secret data_, such as an API key, to function correctly. To make storing and using secret data easy, the Latch SDK comes with special utilities that handle this securely.

Storing and Using Secrets

Versioning

When developing a workflow, it's helpful to run the task functions before executing the entire workflow in the cloud to debug the environment and logical issues. Since tasks run in a different environment than your local computer (namely, the one defined in your Dockerfile(s)), there may be discrepancies when running your code locally.To address this, the Latch SDK comes with a command that allows you to run tasks and debug your environment without having to reregister your workflow and run it through the UI.

Development and Debugging

We will walk through a quick end-to-end flow for testing and debugging a variant calling workflow to demonstrate how to use latch develop.

Learning through An Example

It is often desirable to be able to access a shell from within a running task of a workflow, to debug a misbehaving program or inspect some files for example.

Remote Execution

Tutorial

Execution Profiles

Caching and Resuming

Private Registries

About Page

Publishing Results

Parameter Validation

Updating the Entrypoint

Dependencies

Using GPU Accelerators

Shared Storage

Snakemake Workflow Cloud Compatibility

Caching Snakemake Tasks

In this guide, we will walk through how to upload a simple Snakemake workflow to Latch.

Debugging Snakemake

The following page outlines common problems with uploading Snakemake workflows and solutions.

Troubleshooting

Architecture Overview

Registering an Example Workflow

Parameter Encoding

Using Latch Storage

Using the Latch Executor Plugin

Metadata

The Entrypoint

Writing the Dockerfile

Workflows Automations

Example Data Addition Trigger

Data Addition Trigger

Example Interval Trigger

Interval Trigger

Access the scale of the cloud with the flexibility of your personal computer.

What are Latch Pods?

In this tutorial, we will walk through the basics of creating your first Pod.

Latch Pods Basics

Latch Pods provides direct access to all files stored on Latch Data.

Accessing Latch Data in Pod using Latch Data FUSE

Pods allow easy customization of auto shutdown interval when inactive.

Setting Up Auto Shutdown from Latch

Configure Pods for scheduled shutdown using systemd.

Configuring Auto Shutdown

Latch Pods make it easy to host any custom application, such as Dash Apps, RShiny, Streamlit, and more, on Latch.

Host a Custom App

Pod Templates provide an easy way to take a snapshot of a Pod’s dependencies and files, and save it as a template that is reusable in future Pods for your organization. 

Pod Templates

The Custom File Viewer enables scientists to click on files in Latch Data with a specific extension and view them in an application hosted on a Latch Pod without having to manually enter the pod and download the file.

Custom File Viewer

You may want to set up SSH access for Pod if you want to access Pod from your local terminal or open your favorite IDE, such as Visual Studio Code, inside a Pod.

Set up SSH Access

Follow our tutorial to use Visual Studio Code inside your Pod and navigate the filesystem on your favorite IDE.

Integration with VSCode

Integration with GitHub

Set up dashboards with interactive visualizations and data transformations for scientists to explore their data.

What are Latch Plots?

Learn how to bring data into a plotting layout and chart it.

Plotting a CSV

Latch Plots Basics

The Notebook View is where you bring in data, perform transformations in  Python, and visualize your data.

Develop your first Plot Notebook

Develop your First Plot Notebook

An overview of all components in a Plot notebook

Plot Notebook Components

Components of a Plot Notebook

Learn how reactivity works in Plot notebooks

Reactivity

Plots Reactivity

Create interactive and customizable plots.

Plotting Overview

Learn about the plot types supported in a plotting layout.

 Plot Types

Supported Plot Types

Latch Plots allows you to install and import any libraries to generate custom plots and display them in the notebook view.

Custom Plots in Python

Add Custom Plots from Any Library

Facet your plot based on a categorical column.

Faceting

Learn what options are available for customizing the appearance of your plots.

Appearance

Error Bars

Learn how to make a selection of points and use that selection to filter the source table.

Selecting Points

Learn how to filter the values displayed in a plot.

Filtering

The Data Transformation component functions similarly to a Python cell in a Jupyter Notebook, allowing users to input, modify, and execute Python code interactively.

Widgets can be used to create transforms that users can update based on user input. You're able to define a widget and use its user input in your analysis code.

Widgets

Learn how to view and use dataframe outputs from a transfrom.

Transform Outputs

Write descriptions using markdown within a transform.

Description Text

Learn how to use a table display to view data and filter it for use downstream.

Table Display/Filter

Learn how to add description text to your layout.

Text Display

How to connect to Plots runtime using SSH