Everyone from Eric Green at the NHGRI to David Dooling at Washington University’s Genome Center to Kevin Davies at Bio-IT World rightly acknowledges that the $1,000 genome is a misnomer if you decide to include the cost of computing, analyzing, storing, and querying the data. And if you don’t – well, you’ll be a touch over budget.

Nevertheless, recent innovations have changed the playing field. Just two years ago the debate was whether MAQ, the fastest algorithm at the time for mapping reads, was accurate enough based on its gapless strategy. Today, there are some electrifying algorithms out there based on the Burrows Wheeler transform that move 10 times the number of reads through the same compute pipelines. And where Burrows Wheeler doesn’t work – long reads like the kinds we’re seeing more and more – there are fantastic new approaches like GASSST (which we use to get a 7x increase in mapping speeds with arbitrarily long reads and arbitrarily large gaps).

I gave a talk at ABRF this year entitled, “The Bioinformatics Bottleneck,” where I challenged an audience full of bioinformaticians that if we keep getting stuck arguing about the fastest algorithm to map reads and whether to store image files, the industry would simply move on without us. (No, Steve Lincoln, I didn’t out-Steve you.) Illumina, Life Tech, Roche, and now PacBio are going to keep making machines. Beijing Genomics is going to keep buying them. In the heat of the moment I think I said something about not only throwing out the image files, but also throwing out the reads themselves – after all it’s the representation of variation, not the reads, that researchers and clinicians will ultimately care about. As late as March 2010 this was met with shock and even disdain; but today great companies like Complete Genomics and important projects like the Personal Genome Project are doing just that – providing us with the variant calls only; not the reads. Unless of course you want them – your choice, by the way, and may your disk drives spin forever.

Now that’s not to say that we don’t need to keep mapping. And so we do. GenomeQuest is on track to have mapped 100 billion reads in 2010 at year end – about 11 million an hour. Of course if our systems were 100% utilized – an impossible dream for any data center – the number would be far higher. And next year we expect it will be five times that at least. Mapping goes on, powered by innovations in software, and innovations in hardware as well. Earlier this year we announced a partnership with SGI that enabled us to build a purpose-designed architecture designed specifically for mapping. By ensuring reference data availability on every compute node, we minimize network traffic; through significant redundancy in both compute and head nodes we can ensure quality of service. By the end of 2011 we expect to be able to map 300 whole human genomes at 30x coverage per month, or 6,000 exomes per month.

The idea that so many genomes can move through a uniform, industry tested workflow is enchanting. And it sets our sights on the next realm of opportunity: analysis of thousands of genomes at the same time. Projects such as the 1,000 Genome Project (actually planning more like 2,000 – and do you really think Durbin and Altschuler will stop there?) are troves of undiscovered knowledge on the basis of disease. Being able to overlay that on top of your own 100 exome project provides critical information on background. So using the same technology, GenomeQuest has an exciting new suite of products for multi-genome analysis that are currently available for early access. As the year rolls on into 2011, these workflows for computing allelic frequency, performing large-scale tumor/normal studies, and asking population-scale questions across thousands of genomes will be further enhanced in close collaboration with our users.

The end game for healthcare is genomic medicine, enabled by the sequencing of individual patients and the large-scale comparison against populations – from phenotype to clinical presentation to genotype to treatment to response. That’s why we’ve partnered with Beth Israel Deaconess Medical Center – to work closely on the development of whole genome reports that can actually be usable by pathologists, like any other lab test. The idea is to map clinical actions to specific variations and present them in a way that a trained genomics physician can ultimately guide the course of treatment. There is a long road ahead and plenty of stakeholders involved, but being in the clinic in 2010 drives our thinking about the research and clinical development applications that exist today in pharma and academia.

Genomics is still a bit of the wild west – in a single day I can have a meeting with a pharma executive, a seminar on an FDA position-paper, a discussion with agbio researchers on the genomics of polyploid organisms, and a deep dive with a team of bioinformatics talking about word-based hashing algorithms. When genomics is in the clinic this’ll have to stop.

But in the meantime…

However, on the other hand, without broad use of new software to analyze this massive volume of data, this group reported an “information bottleneck” for researchers — exacerbated by a culture of free, fragmented software.

So what to do?

I, for one, am quite optimistic on the topic.

I think that NIH and market forces will resoundingly address the challenge.  I heard from Eric Green, director of NHGRI, that his team and the NIH see the bottleneck as a critical issue and they are taking steps to adjust their priorities accordingly.

And, while researchers in the past, may have made significant discoveries with free and varied software, I contend that the following market forces are encouraging the best of them and their organizations to substantially and positively raise their software aspirations:

• the data volumes in whole-genome projects simply demands a new generation of analysis methods, tools, and platforms — and providers are responding.
• more and more research happens in partnership, requiring collaboration software to share data, intermediate findings, and discoveries — again, solutions exist.
• more and more organizations will want to inject public studies directly into their research, remove silos, and create a “continuous/growing/queryable database of variant information” — available today.
• more and more research will be aimed at personalized medicine, diagnostic testing, and clinical applications – areas that call for strict and documented information exchange.
• bioinformatics teams and companies will utilize emerging sequence database systems to arm researchers with powerful new discovery applications. (In a conversation yesterday with George Church, he commented: “how could it be any other way” when I asked if he envisioned the industry moving to such systems.)

All told, motivated by these opportunities, I believe that we are entering an “Information Renaissance” — powered by  sequencing + software technology!

Keep in mind that the answers lie in the sequence data — and researchers will not stop until they get them.  And if high-quality, fair-priced software is required to get at them, so be it. Increasingly, great software is becoming a “must have” with “must have” rewards.

I’ve seen this happen in other professions as they turned to computers. Electrical engineers started with free or home-grown software to mostly capture schematic drawings.  Then chip manufacturing technology exploded and a $5B electronic design automation (EDA) software industry was born.  Soon software was synthesizing whole circuitry from simple specs, placing and routing millions of transistors on chips, simulating whole systems before manufacturing, even telling you how to test the chip after manufacturing – all “must have’s” that have co-powered Moore’s Law for over 20 years.

So, as exciting as sequencing performance has made Life Science research in the past decade, I suggest that a co-powered “Information Renaissance” will take us to new levels of discovery.  And we all very much look forward to it!

Moving data to the cloud remains the biggest obstacle to cloud adoption. The article makes the case for moving the computation to the data instead of vice versa. Presupposing however that people will be willing to move their data at least 1 time. Otherwise, “moving the computation to the data” is an argument for building and maintaining a local compute cluster, nearby the sequencing instrument.

At GQ we realize there is no “one size fits all”. We support a hosted, cloud based solution for customers with limited IT expertise or inclination. Or, for lab operations running multiple sequencing instruments, we can install the cloud locally so you benefit from the bandwidth of the Local Area Network connecting the instrument to computing. Either way, data moves only 1 time and GQ solves for scalability with its algorithms and data model.

A recent commentary in GenomeWeb “Considering a Cloud? Cost isn’t everything…” citing the paper “The Real Cost of a CPU Hour” illustrates the confusion. The paper benchmarks HPC on a dedicated cluster versus bare metal resources on Amazon EC2. The conclusion is foregone since commodity EC2 network can’t keep up with a tuned applications running on a compute cluster with a high-speed network.

A more informative title for the blogger and the article would be: “Considering Infrastructure-As-A-Service? Beware if your application requires Message Passing Interface and therefore High-Speed Network”.

Fortunately for large scale Sequence Data Management the predominant application mode is “embarrassingly parallel” and therefore HPC are not needed, except maybe on the Web Server where response time is critical. The statistics in the article show that  Embarrassingly Parallel applications run acceptably well on Amazon EC2, adding less than 5% overhead to the computations.

So what?

GenomeQuest is now for bioinformatics and computational biologists (we call them developers for short). These are people who prefer to write code in Unix, and prefer awk, perl, and sed to Firefox, Internet Explorer, Safari, or Chrome.

So why is that important?

With no up-front investment, developers can use the GQ API to write large-scale sequence comparison applications for the cloud without regard for the details of the computing and the reference data. And, they can publish their applications to the GenomeQuest Web Application, so research biologist customers can use and reuse them.

Its helpful to think of the cloud as a layered architecture. The VC Bessemer provides a nice definition of this layering.

• Infrastructure-as-a-Service: Web-services applications framework for provisioning computers, networks, and storage resources without regard to the actual hardware topology. Examples of IaaS are Amazon EC2 and SGI

• Platform-as-a-Service:  Web-services application framework for provisioning data, algorithm, and analysis services and developing new applications. Leveraging all the benefits of the IaaS layer, the GenomeQuest Developer API Framework provides services for managing, comparing, and mining sequence databases without regard to the underlying computers and storage.

• Software-as-a-Service: A finished Web-appliction end-to-end solution for a given end-user application. Examples of this in the Life Science space include the Ingenuity IPA product and the GenomeQuest 6.3 SDM platform.

To my knowledge, we have the industry’s only cloud-based “PaaS”. With a few commands from a remote desktop, developers can upload data, apply algorithms on hundreds of cores while accessing terabytes of well managed sequence data. In the future, we intend to leverage the extraordinary economies of scale offered by IaaS providers such as Amazon.

If you’re curious about the API’s, take a look here.

Dave Dooling at PolITiGenomics finds two good reasons for considering cloud options: when organizations have peak demands for compute power and when limitations on space/power/cooling preclude building a system in-house. These are two good reasons, but hardly enough to justify all the cloud computing hype.

So, what’s the argument for cloud computing?

Unlike computing which gets cheaper every year, people cost more every year. So, it makes sense to evaluate the annual software development and maintenance costs, the cost of managing the reference databases; integrating and maintaining new applications, the productivity of the end-users and how to change the ratio of end-user-to-support-programmer from 2-to-1 to 10-to-1 or 20-to-1. Cloud computing defined as “Infrastructure” (computers, networks, and storage) doesn’t alleviate these costs.

A well written technical blog ‘MassGenomics‘ written by Dan Koboldt illustrates why. Dan says “What’s more, with the advent of next-generation sequencing, I hate to tell you, but people are going to be reporting a lot of false positives.  I guarantee it.  So when you filter all of the variants, you might actually remove the ones you’re looking for.”

Its easy to see why researchers are not enthusiastic about tabular reports. They want to get into the data on their own, without intermediaries, and they want software to facilitate that, not be in the way.