LatchBio

Platform

Documentation

API References

Blog

Sign Up

Log In

Import Registry Records in Latch SDK Workflows

Latch is a platform for you to store, analyze, and visualize multiomics data

Getting Started on Latch

Latch Data is a cloud based file storage system built for storing biological data.

Overview

Basic Uploading and Downloading

Latch allows you to mount your own AWS S3 Buckets and use them the same as you would any data on Latch. All you need is to connect AWS account with Latch which will allow you to mount any buckets in that account onto Latch.

Mounting S3 Buckets

Latch allows you to mount your own AWS GCP Buckets and use them the same as you would any data on Latch. 

Mounting GCP Buckets

This tutorial explains how to get an API key from BaseSpace that you can use to automatically import your sequencing runs into Latch.

BaseSpace Downloader

Command Line Interface

Command Line Interface Data Upload/Download

Data Sharing

FastQC aims to provide a simple way to do some quality control checks on raw sequence data coming from high throughput sequencing pipelines. It provides a modular set of analyses which you can use to give a quick impression of whether your data has any problems of which you should be aware before doing any further analysis.

FastQC

The Integrative Genomics Viewer (IGV) is a high-performance, easy-to-use, interactive tool for the visual exploration of genomic data. BAM, Fasta, etc. files can be opened in an IGV Browser natively within the Latch Platform. 

Using IGV on Latch

Learn how to set default viewer settings when opening files in IGV on Latch

Setting Viewer Defaults for IGV on Latch

The IGV browser comes with many hosted options for common Reference Genomes to view Alignment Files to. Here is a list of the options provided.

IGV Hosted Reference Genomes

All your sample sheets, all your metadata, all your analysis — in one place

Latch Registry

A project is a collection of tables. Let’s explore a few examples of how projects can be used:

Create a Project

What is a Project?

A table is a collection of records. Each record, also known as a row, contains a unique record name and a set of attributes.

Create a Table

Manually Create Single Records

Bulk Import Data Using a CSV

Bulk Link Sequencing Files to Existing Records

Modify Records in a Table

As you centralize your biological data in Latch Registry, you will often find that some records are related or even dependent on one another. Linked records offer a powerful way for you to define relationships between records from different tables, enhancing your data analysis and enabling new insights that may have been difficult to discover without linked records.

Connect Records Across Tables

Benchling Integration

Latch Workflows come with out-of-the-box community workflows and a Python SDK that allows uploading of custom workflows.

Bulk RNA Seq

End-to-end bulk RNA-seq analysis using LVBRS

Every workflow in Latch allows you to import a CSV containing values for its parameters. This is another way to import data when doing bulk runs of a workflow.

CSV Import

In this tutorial, we will walk through the basics of creating your first Pod.

Latch Pod Basics

Best Practices for Working with Latch Data inside Pods

Latch Pods provides direct access to all files stored on Latch Data.

Accessing Latch Data in a Pod

You may want to set up SSH access for Pod if you want to access Pod from your local terminal or open your favorite IDE, such as Visual Studio Code, inside a Pod.

Set up SSH Access

It is easy to use Visual Studio Code inside your Pod and navigate the filesystem on your favorite IDE.

Integration with VSCode

If you want to add a GitHub repository to Pod when Pod is created for the first time or commit to a GitHub repository in the future, you have to first set up your GitHub personal access token (PAT).

Integration with GitHub

Pods allow easy customization of auto shutdown interval when inactive.

Customize Auto Shutdown

Configure Pods for scheduled shutdown using systemd

Scheduled Shutdown

Latch Pods make it easy to host any custom application, such as Dash Apps, RShiny, Streamlit, and more, on Latch.

Host a Custom App

Pod Templates provide an easy way to take a snapshot of a Pod’s dependencies and files, and save it as a template that is reusable in future Pods for your organization. 

Pod Templates

Set up dashboards with interactive visualizations and data transformations for scientists to explore their data.

In this document, we walk through the core components that make up a Latch Plot layout.

Concepts

Workspaces allow you to collaborate with team members, manage access, share data and more

Learn how to create a new workspace on Latch

Creating a Workspace

Here we will walk through how to invite users to your workspace

Inviting Members to a Workspace

Learn how to manage roles and permissions for members of a workspace

Roles & Permissions

Frequently Asked Questions

This Privacy Policy describes how Latch Bio, Inc. (“we”, “us”, “our”, or the “Company”) handles personal information that we collect through our digital properties that link to this Privacy Policy, including our website (collectively, the “Service”), as well as through social media, our marketing activities, and other activities described in this Privacy Policy.

Privacy Policy

Latch Privacy Policy

Terms of Service

Latch Terms of Service

The Latch SDK is an open-source toolchain to define serverless bioinformatics workflows with plain Python and deploy associated no-code interfaces using single commands.

Introduction

What is the Latch SDK?

Quick Start

In this demonstration, we will examine a workflow which sorts and assembles COVID sequencing data.

Authoring your Own Workflow

A workflow is an analysis that takes in some input, processes it in one or more steps and produces some output.

What is a Workflow?

When building a workflow with multiple tasks, it can be difficult to decide when to split larger tasks into smaller tasks. Some of the tradeoffs are listed below to guide this decision:

When to Split Up Tasks

When to split up a task into multiple tasks

Files and directories on Latch can be referred to in code or through the CLI using **Latch URLs**.

Latch URLs

Working with Remote Files

When working with bioinformatics workflows, we are often passing around large files or directories between our tasks. These files are usually located in cloud object stores and are copied to the file systems of the machines on which the task is scheduled.

LatchFile / LatchDir

Task executions produce logs, displayed on the Latch console to provide users visibility into their workflows. However, these logs tend to be terribly verbose. It's tedious to sift through piles of logs looking for useful signals; instead, important information, warnings, and errors should be prominently displayed. This is accomplished through the Latch SDK's new messaging feature.

Messages

When a workflow is executed and tasks are scheduled, the machines needed to run the task are provisioned automatically and managed for the user until task completion. Tasks can be annotated with the resources they are expected to consume (eg. CPU, RAM, GPU) at runtime and these requests will be fullfilled during the scheduling process.

Defining Cloud Resources

Caching allows workflow developers to reuse the results of previously run tasks to prevent wasted time and computation. This is helpful when running large batches of workflows with redundant inputs or when debugging errors in the middle of a workflow where upstream state can be reused.

Caching

In order to support the functionality of an `if-elif-else` clause within the body of a workflow, we introduce the method `create_conditional_section`. This method creates a new conditional section in a workflow, allowing a user to conditionally execute a task based on the value of a task result.

Conditional Sections

There are many pipelines in bioinformatics that require running a processing step in parallel and aggregating their outputs at the end for downstream analysis. A prominent example of this is bulk RNA-sequencing, where alignment is performed to produce transcript abundances per sample, and gene counts of all samples are subsequently merged. Having a single count matrix makes it convenient to use in downstream steps, such as differential gene expression analysis. Another example is performing FastQC on multiple samples and summarizing the results in a MultiQC report.

Map Task

Often a workflow can depend on _secret data_, such as an API key, to function correctly. To make storing and using secret data easy, the Latch SDK comes with special utilities that handle this securely.

Storing and Using Secrets

When developing a workflow, it's helpful to run the task functions before executing the entire workflow in the cloud to debug the environment and logical issues. Since tasks run in a different environment than your local computer (namely, the one defined in your Dockerfile(s)), there may be discrepancies when running your code locally.To address this, the Latch SDK comes with a command that allows you to run tasks and debug your environment without having to reregister your workflow and run it through the UI.

Development and Debugging

We will walk through a quick end-to-end flow for testing and debugging a variant calling workflow to demonstrate how to use latch develop.

Learning through An Example

It is often desirable to be able to access a shell from within a running task of a workflow, to debug a misbehaving program or inspect some files for example.

Remote Execution

Snakemake Workflow Cloud Compatibility

Private Registries

In this guide, we will walk through how to upload a simple Snakemake workflow to Latch.

Tutorial

Debugging Snakemake

The following page outlines common problems with uploading Snakemake workflows and solutions.

Introduction

Data

Visualizations

Registry

Workflows

Workflow Helpers

Pods

Plots

Workspace Administration

Resources

Import Registry Records in Latch SDK Workflows