Import Registry Records in Latch SDK Workflows

LatchBio

Latch is a platform for you to store, analyze, and visualize multiomics data.

What is LatchBio?

Quickstart

Latch Data is a cloud based file storage system built for storing biological data.

Overview

What is Latch Data?

Basic Uploading and Downloading

Latch allows you to mount your own AWS S3 Buckets and use them the same as you would any data on Latch. All you need is to connect your AWS account with Latch to mount buckets from that account.

Mounting S3 Buckets

Latch allows you to mount your own GCP Buckets and use them the same as you would any data on Latch.

Mounting GCP Buckets

This tutorial explains how to get an API key from BaseSpace that you can use to automatically import your sequencing runs into Latch.

BaseSpace Downloader

Command Line Interface

Command Line Interface Data Upload/Download

Data Sharing

FastQC aims to provide a simple way to do some quality control checks on raw sequence data coming from high throughput sequencing pipelines. It provides a modular set of analyses which you can use to give a quick impression of whether your data has any problems of which you should be aware before doing any further analysis.

FastQC

CELLxGENE Explorer allows scientists to execute interactive analyses on a dataset to explore how patterns of gene expression are determined by environmental and genetic factors using an interactive speed no-code UI.

CELLxGENE

Connect your sample sheets, metadata, and analysis — all in one place.

What is Latch Registry?

Learn how to set up your registry to manage your data.

Registry Basics

As you centralize your biological data in Latch Registry, you will often find that some records are related or even dependent on one another. Linked records offer a powerful way for you to define relationships between records from different tables, enhancing your data analysis and enabling new insights that may have been difficult to discover without linked records.

Connect Records Across Tables

Using Registry in a Workflow

Account Objects

Registry Projects

Table Objects

Record Objects

Latch's Workflow Manager comes with out-of-the-box community workflows and a Python SDK that allows uploading of custom workflows.

Workflows & SDK

What are Latch Workflows?

Every workflow in Latch allows you to import a CSV containing values for its parameters. This is another way to import data when doing bulk runs of a workflow.

CSV Parameter Import

Workflows on Latch provide visibility into the resource usage of each task execution, enabling developers to easily debug and optimize their workflow.

Resource Monitoring

Launching Workflows Programmatically

Quick Start

In this demonstration, we will examine a workflow which sorts and assembles COVID sequencing data.

Authoring your Own Workflow

Tutorial

Execution Profiles

Caching and Resuming

Private Registries

Dependencies

Using GPU Accelerators

Shared Storage

Debugging Nextflow

Snakemake Workflow Cloud Compatibility

In this guide, we will walk through how to upload a simple Snakemake workflow to Latch.

Debugging Snakemake

The following page outlines common problems with uploading Snakemake workflows and solutions.

Troubleshooting

Architecture Overview

Registering an Example Workflow

Parameter Encoding

Using Latch Storage

Using the Latch Executor Plugin

Metadata

The Entrypoint

Writing the Dockerfile

Workflows Automations

Example Data Addition Trigger

Data Addition Trigger

Example Interval Trigger

Interval Trigger

Access the scale of the cloud with the flexibility of your personal computer.

What are Latch Pods?

In this tutorial, we will walk through the basics of creating your first Pod.

Latch Pods Basics

Latch Pods provides direct access to all files stored on Latch Data.

Accessing Latch Data in Pod using Latch Data FUSE

Latch Pods make it easy to host any custom application, such as Dash Apps, RShiny, Streamlit, and more, on Latch.

Host a Custom App

Pod Templates provide an easy way to take a snapshot of a Pod’s dependencies and files, and save it as a template that is reusable in future Pods for your organization. 

Pod Templates

The Custom File Viewer enables scientists to click on files in Latch Data with a specific extension and view them in an application hosted on a Latch Pod without having to manually enter the pod and download the file.

Custom File Viewer

Set up dashboards with interactive visualizations and data transformations for scientists to explore their data.

What are Latch Plots?

Learn about roles and permissions for members of an organization.

Workspaces & Orgs

Workspaces & Orgs: An Overview

What are Latch Workspaces?

Learn how to create a new workspace on Latch

Creating a Workspace

Learn how to invite users to an existing workspace

Inviting Members to a Workspace

Learn about workspace roles and permissions

Workspace Roles

Learn about billing for Latch workspaces and how to add your credit card information.

Billing

Workspace Billing

Moving & Copying Data

Copying and Moving Data Between Workspaces

Workspace Pod Templates

Opening Pod Templates in Other Workspaces

Manage multiple workspaces under a single organization on Latch

What are Latch Organizations?

Organization Basics

Learn how to add and transfer workspaces to an organization on Latch

Adding Workspaces to an Organization

Learn how to invite members to the organization and accept invitations

Inviting Members to an Organization

Organization Roles

Transferring Credits

Transferring Credits to Workspaces

Configure access controls and authentication methods

Security Settings

Learn how to distribute analysis packages to customers

Analysis Packages

For Kit Providers: Analysis Packages

Learn how ownership differs in workspaces created by the Organization versus workspaces where Analysis Packages are unpacked in.

Workspace Ownership Models

Comparing Workspace Ownership: Organization vs. Analysis Packages

Customize the Latch platform to reflect your brand's unique look and feel.

White-labeling

White-label Your Customer Experience

Recommended image format and sizing for white-labeling.

White-Labeling Image Recommendations

White-labeling Recommended Image Format & Sizing

Frequently Asked Questions

This Privacy Policy describes how Latch Bio, Inc. (“we”, “us”, “our”, or the “Company”) handles personal information that we collect through our digital properties that link to this Privacy Policy, including our website (collectively, the “Service”), as well as through social media, our marketing activities, and other activities described in this Privacy Policy.

Privacy Policy

Latch Privacy Policy

Terms of Service

Latch Terms of Service

Introduction

Data

Registry

Pods

Plots

Administration

Resources

Documentation

API References

Blog

Website

Change Log

Sign Up

Contact Us

Log In

The Integrative Genomics Viewer (IGV) is a high-performance, easy-to-use, interactive tool for the visual exploration of genomic data. BAM, Fasta, etc. files can be opened in an IGV Browser natively within the Latch Platform. 

Using IGV on Latch

Learn how to set default viewer settings when opening files in IGV on Latch

Setting Viewer Defaults for IGV on Latch

The IGV browser comes with many hosted options for common Reference Genomes to view Alignment Files to.

IGV Hosted Reference Genomes

Manually Create Single Records

Bulk Import Data Using a CSV

Bulk Link Sequencing Files to Existing Records

Benchling Integration

CRISPResso2 is a software pipeline for the analysis of genome editing experiments. It is designed to enable rapid and intuitive interpretation of results produced by amplicon sequencing.

CRISPResso2

CAS-OFFinder is an algorithm that searches for potential off-target sites of Cas9 RNA-guided endonucleases. It is one of the most highly cited & consistent tools for this purpose.

CAS-OFFinder

[CRISPOR](http://crispor.org/) is a website that helps select and express CRISPR guide sequences, described in two papers ([Gen Biol 2016](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-016-1012-2) and [NAR 2018](https://academic.oup.com/nar/article/46/W1/W242/4995687)). In its default mode, the user pastes an input DNA sequence and chooses the genome.

CRISPOR

AlphaFold produces highly accurate protein structure predictions from amino acid sequences.

AlphaFold

A workflow is an analysis that takes in some input, processes it in one or more steps and produces some output.

What is a Workflow?

When building a workflow with multiple tasks, it can be difficult to decide when to split larger tasks into smaller tasks. Some of the tradeoffs are listed below to guide this decision:

When to Split Up Tasks

When to split up a task into multiple tasks

Files and directories on Latch can be referred to in code or through the CLI using **Latch URLs**.

Latch URLs

Working with Remote Files

When working with bioinformatics workflows, we are often passing around large files or directories between our tasks. These files are usually located in cloud object stores and are copied to the file systems of the machines on which the task is scheduled.

LatchFile / LatchDir

Task executions produce logs, displayed on the Latch console to provide users visibility into their workflows. However, these logs tend to be terribly verbose. It's tedious to sift through piles of logs looking for useful signals; instead, important information, warnings, and errors should be prominently displayed. This is accomplished through the Latch SDK's new messaging feature.

Messages

When a workflow is executed and tasks are scheduled, the machines needed to run the task are provisioned automatically and managed for the user until task completion. Tasks can be annotated with the resources they are expected to consume (eg. CPU, RAM, GPU) at runtime and these requests will be fullfilled during the scheduling process.

Defining Cloud Resources

Caching allows workflow developers to reuse the results of previously run tasks to prevent wasted time and computation. This is helpful when running large batches of workflows with redundant inputs or when debugging errors in the middle of a workflow where upstream state can be reused.

Caching

In order to support the functionality of an `if-elif-else` clause within the body of a workflow, we introduce the method `create_conditional_section`. This method creates a new conditional section in a workflow, allowing a user to conditionally execute a task based on the value of a task result.

Conditional Sections

There are many pipelines in bioinformatics that require running a processing step in parallel and aggregating their outputs at the end for downstream analysis. A prominent example of this is bulk RNA-sequencing, where alignment is performed to produce transcript abundances per sample, and gene counts of all samples are subsequently merged. Having a single count matrix makes it convenient to use in downstream steps, such as differential gene expression analysis. Another example is performing FastQC on multiple samples and summarizing the results in a MultiQC report.

Map Task

Often a workflow can depend on _secret data_, such as an API key, to function correctly. To make storing and using secret data easy, the Latch SDK comes with special utilities that handle this securely.

Storing and Using Secrets

Versioning

When developing a workflow, it's helpful to run the task functions before executing the entire workflow in the cloud to debug the environment and logical issues. Since tasks run in a different environment than your local computer (namely, the one defined in your Dockerfile(s)), there may be discrepancies when running your code locally.To address this, the Latch SDK comes with a command that allows you to run tasks and debug your environment without having to reregister your workflow and run it through the UI.

Development and Debugging

We will walk through a quick end-to-end flow for testing and debugging a variant calling workflow to demonstrate how to use latch develop.

Learning through An Example

It is often desirable to be able to access a shell from within a running task of a workflow, to debug a misbehaving program or inspect some files for example.