Installation instructions, a quickstart guide, and full documentation on the latest stable release can be found at Read the Docs.
Easily mastered, the Python user API for defining and running workflows is built on one core class. Also, everything is open source under the Apache License.
Toil workflows support arbitrary worker and leader failure, with strong check-pointing that always allows resumption.
Caching, fine grained, per task, resource requirement specifications, and support for the AWS spot market mean workflows can be executed with little waste.
To demonstrate Toil, we processed over 20,000 RNA-seq samples to create a consistent meta-analysis of five datasets free of computational batch effects that we make freely available. Nearly all the samples were analysed in under four days using a commercial cloud cluster of 32,000 preemptable cores. Read our paper.
Built for the cloud
Develop and test on your laptop, then deploy on Microsoft Azure, Amazon Web Services (including the spot market), Google Compute Engine, OpenStack, or on an individual machine.
Build a workflow on your laptop, then scale to the cloud and run it concurrently on hundreds of nodes and thousands of cores with ease. We've tested it with 32,000 preemptable cores so far, but Toil can handle more.
Toil plays nice with databases and services, such as Apache Spark. Service clusters can be created quickly and easily integrated with a Toil workflow, with precisely defined start and end times that fits with the flow of other jobs in the workflow. Read more.
Common Workflow Language
Common Workflow Language is an emerging standard for writing portable workflows. Toil has nearly full support for the stable v1.0 CWL specification. Read more.
Declarative and dynamic workflows
Workflows can be declared statically, but new jobs can be added dynamically during execution within any existing job, allowing arbitrarily complex workflow graphs with millions of jobs within them.
Complete file and stream management
Temporary and persistent file management that abstracts the details of the underlying file system, providing a uniform interface regardless of environment. Supports both atomic file transfer and streaming interfaces, and user data encryption.