Complexity of scientific workflows

As a PhD student in the Breaker Lab at Yale, my research focus was building computational pipelines for the discovery of novel noncoding RNA motifs. This involved computationally-intensive sifting through terabytes of bacterial genomes looking for RNA motifs with certain structural homologies and gene associations. As the number of different tools I wanted to incorporate into my research pipelines grew, I quickly began dealing with the complexity of programming large, multi-step pipelines.

While iterating on a complex, multi-step workflow, minor matters like the naming of intermediate files and discrepancies between my laptop and HPC cluster compute environments can quickly become sources of major frustration. I never ended up finding Nextflow during this time period, but I became strongly interested in software engineering best practices. I was convinced there had to be better ways to address these challenges than bash spaghetti with python meatballs.

Moving to biotech, finding Nextflow

For my first role out of academia, I joined ProFound Therapeutics in 2021. ProFound was then a stealth-mode biotech startup in the Flagship Pioneering family of companies. I was their second computational hire, and my first project was to scale certain computational analyses to a massive collection of novel proteins they were studying. Before I started building pipelines, I took time to do a careful assessment of modern tooling for orchestrating complex scientific workflows.

There are a number of excellent bio-specific and general tech tools for building reproducible workflows, but Nextflow stood out to me for two key reasons:

• Nextflow had excellent portability, where the same pipeline logic could be executed on a local computer, in AWS Cloud, or in an HPC cluster.

• nf-core, a vibrant community of Nextflow users across the globe, were collaborating on building a collection of gold-standard, open-source bioinformatic pipelines for all kinds of relevant bioinformatic analyses.

After bringing my analysis of Nextflow's advantages to my computational lead, we decided to make Nextflow a core technology in our platform and were off to the races.... sort of.

Building with Seqera Platform

It turned out that there was a lot more to setting up a scalable bioinformatic platform than simply choosing to use Nextflow. We ran into headaches setting up our AWS Batch executors, setting up automations, and the need to train scientists who wanted to analyze their own data on the complexities of AWS.

Luckily, Seqera had recently come onto the block with a solution that seemed tailor-made to many of our biggest headaches. Seqera was founded by the original developers of Nextflow and their flagship product was a platform we could deploy into our own AWS account. Not only could it quickly deploy/modify some of the tricky parts of AWS infrastructure, Seqera Platform could set up automations, observability, and provide a user-friendly GUI for non-technical folks to run our Nextflow pipelines.

Thanks to my advocacy and the buy-in of my technical lead, we signed up with Seqera as one of their earliest commercial customers. The rock-solid combination of Nextflow and Seqera ended up fully living up to its promise and more as the Seqera team continued to add game-changing new features to both Nextflow and Seqera Platform over the nearly two years I worked with their team as a customer.

Super-scaling bioinformatics

For my next role after ProFound Tx, I joined GeneDx as a Senior Software Engineer on the bioinformatics platform team. The opportunity at GeneDx appealed to me for two reasons:

• With hundreds of patients' genetic sequencing tests passing through bioinformatics pipelines every day, the data quality and reliability of my team's work was going to be critically important from Day 1.

• I had the opportunity to act as technical lead for parts of the "Cloudflow" project: a major migration of bioinformatic pipelines from WDL running on-prem to Nextflow running in the cloud.

While the original migration plan involved setting up an in-house Nextflow orchestrator and building a variant-calling pipeline from scratch, we made two changes to the plan not long after I joined:

• Instead of building a Nextflow pipeline from scratch, we decided to try configuring the open-source nf-core/sarek pipeline as the starting point for our planned production pipeline.

• We started a proof-of-concept agreement with Seqera to explore using Seqera Platform as our pipeline orchestration solution.

With the combination of Seqera Platform and a gold-standard nf-core pipeline to build on, our small CloudFlow project team began delivering on project milestones at an incredible pace. Within a few short months of our change in direction, we had setup scalable bioinformatics infrastructure connected to Seqera Platform in not one, but two public clouds. We also developed proof-of-concept durable automations that handled the entire process from data coming off the sequencer to having processed genomic variants uploaded to our data portal.

What next?

Given the passion I've developed for Nextflow and Seqera products over the past 3+ years as a user and customer, it was a natural fit for me to join Seqera's community team when a position opened up. While the specifics of my role as developer advocate are still to-be-determined, here are a couple of topics that I'm incredibly passionate about that you'll likely hear me talk about in the coming months:

• Highlighting the benefits for biopharma and healthcare companies that choose to build on and contribute to open-source projects like nf-core pipelines.

• Expanding the range of scientific disciplines that are choosing to build sharable, reproducible pipelines using Nextflow.

• Diversifying the bioinformatics talent and leadership pool by offering training and support to individuals from marginalized and underrepresented communities.

• Bringing the power of modern software best practices like continuous integration and continuous deployments to complex data pipelines.

I'm thrilled to be getting more deeply involved in the incredibly vibrant open-source science communities that are built around Nextflow, nf-core, Seqera, and more! I'm also very much looking forward to meeting more of you in person and virtually over the coming months!

1) Nextflow adoption and impact is accelerating

In Evan Flodan's welcome address he shared some convincing data from telemetry and other sources showing that Nextflow is the fastest growing workflow management for Biology. One piece of data came from a recent biorxiv pre-print by the EuroFAANG and nf-core teams:

This chart is exciting to me for two reasons:

1. In comparison to download stats alone, scientific publications likely represent successful scientific output.

2. Publications are likely a lagging indicator. If the trend in Nextflow citations from 2022 to 2023 holds, we may be seeing a transformational shift in the adoption of containerized workflows for science.

To me, it seems like Nextflow's foundational hypothesis that better software enables better science is increasingly being validated by the numbers.

2) Imaging analysis has a strong and growing presence in the nf-core community.

Although the early years of Nextflow saw adoption most strongly for processing next-generation sequencing (NGS) data, there is a growing diversity in the scientific disciplines that have been taking advantage of the utility it provides. So as someone with a strong scientific interest in image analysis (see pycytominer), I was thrilled to see several talks focused specifically on new Nextflow pipelines designed for large-scale processing of various forms of imaging data. Pipelines were demoed focused on multiple imaging data modalities, including:

• Whole slide imaging of tissue sections (Parallelization of computer vision over large corpora of gigapixel biomedical images)

• Single-cell spatial omics (SpaFlow: A Nextflow pipeline for single cell phenotyping in spatial omics)

• Diffusion MRI Processing (Mining diamonds with wooden hammers : getting diffusion MRI processing into the age of steel)

The recently launched combinatorial fluorescence in-situ hybridization analysis pipeline nf-core/molkart also had a prominent role in Seqera Lab's demo of their new Data Studios feature:

3) Multiple companies are improving their Nextflow infrastructure offerings

As expected, Seqera Labs announced and demoed several new features for Seqera Platform including the previously-mentioned Data Studios. One exciting free-to-everyone feature is Seqera Containers, which provides a dead-simple interface for building on-demand, multi-architecture docker images for any combination of pypi and conda packages.

Beyond that, I was excited to see other companies expanding their Nextflow infrastructure tooling.

Memverge discussed their Memory Machine Cloud offering which can act as a control plane for launching Nextflow pipelines. It has the very clever capability for a spot instance to pause mid-processing and resume on different instance when the spot allocation is reclaimed. Memverge also discussed their effort into addressing the I/O bottleneck of many data-intensive bioinformatics pipelines with their development of the open-source JuiceFS file system.

Re-scale announced their support for an executor plugin that allows Nextflow to orchestrate tightly coupled jobs on Re-scale's cloud-based High-Performance Computing (HPC) or High-Throughput Computing (HTC) platforms.

Finally, Colby Ford from Tuple discussedahab, which can manage Nextflow, Snakemake, WDL and CWL-based in Kubernetes clusters deployed in Azure.

Conclusions

It's clear that the Nextflow ecosystem is rapidly evolving, with growing adoption, expanding use cases in imaging analysis, and significant advancements in infrastructure tooling. It's an exciting time to be part the Nextflow community!

Changelog:

• 2024-06-14 - Updated cover image to picture of me meeting up with nf-core collaborator Maxime Garcia.

Maintenance Burden

Over the past year I've ended up tinkering a great deal with the underlying deployment pattern of my website. This included:

• Migrating from Github Pages to Cloudflare Pages for content hosting

• Migrating my DNS configuration (and then separately my domain registration) from Google Domains to Cloudflare

• Setting up previews of development branches in Cloudflare

• Troubleshooting issues with all of the above.

These were all necessary and/or useful improvements, and I had a long list of additional things I wanted to improve further around setting up edge caching and improving load times, but none of these improvements resulted in me writing more content.

Activation Energy

Besides the maintenance burden I described above, the other reason that I didn't find myself writing a lot of content was the high-activation energy involved in adding new content. Each new article required:

• Creating a new branch in my Github repo

• Writing the content

• Making a commit

• Pushing the commit

• Creating a PR

• Previewing the PR build

• Merging the PR

While the version-control and CI/CD process provides substantial value within the context of building software, it felt like unnecessary overhead. And when I had an idea or thought I wanted to share, I would often get mentally blocked with all the little things that needed to happen.

In the end this high maintenance burden and high activation energy made it too hard to do the thing that my blog was intended to accomplish: provide a platform for me to share my ideas and interests.

Going with a Platform

In the end, I decided to replace my burdensome self-hosted publishing solution with a good commercial platform: Hashnode. I'm not going to do a breakdown of the features and rationale for choosing this platform in particular, but I do want to mention a bit of the "strategic" value of this decision because I think there are some valuable learnings here for teams building computational platforms in biopharma.

Choosing to use a good quality commercial platform to handle undifferentiated parts of your software stack can mean massive time savings from:

• building the features you need now

• building features that you don't know you need, but will need in the future

• visual tooling and an optimized user interface for routine tasks

That last bullet point is always particularly hard to justify building for an internal tool but can provide massive time savings when routine tasks become less burdensome, frustrating, or fully automated.

I've got a backlog of article ideas that I've been kicking around for the last year, and I expect this new platform-based approach makes it a lot easier to get those out into the world.

Based on the recommendation of multiple colleagues in the Flagship Pioneering informatics community, I recently listened to The Phoenix Project on audiobook. This novel focuses on a struggling IT department within the fictional company Parts Unlimited, and how the main character, Bill, turns things around by implementing the core principles of the DevOps movement. It is an engaging and educational read, and I highly recommend it to anyone interested in DevOps.

Although Bill's story is set in a medium-sized manufacturing company, I found many of the challenges he faced relatable as a computational biologist working in a small biotech startup. His frantic efforts to ensure integrity of critical data for his colleagues in HR and finance reminded me of the urgency I feel when delivering bioinformatic analyses to bench scientists who need them for their experiments. The novel inspired me to consider how I could apply DevOps principles to the specific problems I face at work.

As I started diving deeper into DevOps, I was somewhat surprised to discover that I already possessed all the core technical skills. I had simply lacked a framework to apply those technical skills in an integrated, synergistic way. That led to my decision to set up a blog focused on applying the principles of DevOps to specific problems faced by computational biologists, bioinformaticians, machine learning engineers and data scientists who work in the biotech and pharmaceutical industries.

Criteria for the blog

If I wanted to write a blog that would in part be focused on DevOps, I wanted to set up the website using that same set of principles I wanted to write about. The DevOps principles I wanted to apply to the blog were:

• Automate everything
• Use version control
• Use a CI/CD pipeline to deploy the site

I also had a few other criteria for the blog:

• Use a static site generator to make the blog easy to maintain.
• Be able to write new posts in markdown.
• Keep costs as low as possible, preferably free.
• Be able to use a custom domain name.
• Have sufficient flexibility to customize the look and feel of the site.

Static site generator

I decided to use Jekyll as the static site generator for the blog. Jekyll is a popular choice for static site generators, and is well supported by GitHub Pages. Because this was my first time using Jekyll and I don't believe in re-inventing the wheel, I decided to use a pre-built theme to get started. I chose Beautiful Jekyll because it is well documented, and has a clean, modern look.

Setting up the blog using CI/CD

I largely followed the instructions on the Beautiful Jekyll website to set up the blog, but I made a few modifications in-line with this blog's focus on DevOps and automation.

Setting up a DevContainer

Earlier this year, I led an effort at ProFound to set up standardized development environments. I'll write more about that effort in a future post, but I wanted to use the same approach for this blog. VsCode makes it trivially easy to set up a DevContainer for a project even for a project that uses a framework/language that you are not familiar with. Using the "Add Development Container Configuration Files" command, I was able to quickly generate a `devcontainer.json` file that would launch a Docker container with all the necessary dependencies for Beautiful Jekyll.

{ 
    "name": "Jekyll",
    "image": "mcr.microsoft.com/devcontainers/jekyll:0-buster",
    "features": { "ghcr.io/devcontainers/features/node:1": {} },
    "forwardPorts": [4000],
    "postCreateCommand": "bundle exec jekyll serve --watch" 
}

As soon as this file was saved, VSCode automatically prompted me to re-open the project in the container.

Setting up a Cloud IDE

One of the benefits of using a DevContainer is that you can use the same container to develop locally or in the cloud using Github Codespaces. Github Codespaces has a generous free tier, and is a great way to reduce the friction of setting up a new project. You can read more about how to set up a Github Codespace in the Github documentation.

Setting up a CI/CD pipeline

The original repo for Beautiful Jekyll included a simple GitHub Actions workflow While enabling the Github Pages feature in the repo settings, I found a template Github Actions pipeline that can be used to build and deploy the site instead:

# Sample workflow for building and deploying a Jekyll site to GitHub Pages
name: Deploy Jekyll site to Pages

on:
  push:
    branches: ["main"]
  workflow_dispatch:

# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
permissions:
  contents: read
  pages: write
  id-token: write

# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
# However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
concurrency:
  group: "pages"
  cancel-in-progress: false

jobs:
  # Build job
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Setup Ruby
        uses: ruby/setup-ruby@ee2113536afb7f793eed4ce60e8d3b26db912da4 # v1.127.0
        with:
          ruby-version: '3.1' # Not needed with a .ruby-version file
          bundler-cache: true # runs 'bundle install' and caches installed gems automatically
          cache-version: 0 # Increment this number if you need to re-download cached gems
      - name: Setup Pages
        id: pages
        uses: actions/configure-pages@v3
      - name: Build with Jekyll
        # Outputs to the './_site' directory by default
        run: bundle exec jekyll build --baseurl "${{ steps.pages.outputs.base_path }}"
        env:
          JEKYLL_ENV: production
      - name: Upload artifact
        # Automatically uploads an artifact from the './_site' directory by default
        uses: actions/upload-pages-artifact@v1

  # Deployment job
  deploy:
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    runs-on: ubuntu-latest
    needs: build
    steps:
      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v2

Setting up a custom domain

I wanted to use a custom domain name for the blog, so I followed the instructions on the Github Pages documentation to set up a custom domain. In order to use a custom domain, you need to create a CNAME record in your DNS settings that points to <username>.github.io. I did this manually in the console for Cloudflare, but I plan to integrate these settings into a Terraform configuration file that I'll build into this repo in the future. I could have waited until I had the Terraform setup ready to go too, but getting a minimum viable product up and running quickly is a key part of the DevOps philosophy.

Conclusion

I'm really happy with how with how simple it was to set up this blog using DevOps principles. It only took a few hours to set up the blog, and I have something simple and robust enough for me to write posts and iterate on with little-to-no overhead. I'm looking forward to writing more posts in the future, and I hope you'll join me on this journey!

References

• Beautiful Jekyll
• Jekyll
• Github Pages
• Github Actions

Image credits

Cover image credit by Nathan da Silva on Unsplash