Skip to content

TaskTide Logo

TaskTide

Website Maven Central Documentation API Reference DOI CI CD License OpenSSF Scorecard

TaskTide is a modular Workflow Orchestration Engine designed for Containerized, Cloud, HPC, Grid, and Edge Computing workloads. It enables the execution of ETL-style workflows and arbitrary Data Application as task collections.

By modelling Workflow, and Execution States as first-class orchestration entities. TaskTide provides its users with task registration, real-time workflow introspection, and lifecycle influence as they are actively consumed across distributed compute resources.

TaskTide ships as a lightweight, configurable solution for workflow orchestration. That decouples Workflow Orhcestration logic from Infrastructure Specific backends. Supporting its deployment across different environments like HPC, Cloud, and Containerized. In additon to Relational (ex Postgres, Maria, MySQL, Microsoft, Oracle etc), Non-Relational (ex MongoDB, CouchDB, Oracle etc) database management systems, and Embedded databases (SQLite, RocksDB). Supporting a diverse sets of operational needs for data application deployment.


TaskTide database backed producer-consumer operations


🧑‍💻 Getting Started

An installation guide catering for different uses is provided here. Backend database configurations should follow provider recommendations, since Jakara NoSQL brings in NoSQL support, and JPA-Hibernate using Hikari Connection Pool brings in SQL. The use for each database type for TaskTide is documented here for NoSQL, here for SQL, and here for ItemStore for embedded databases like SQLite. and RocksDB.

How TaskTide should run can be configured based on parameters in a TaskTide configuration file and command-line arguments described here.


💻 Running TaskTide

As a Java application TaskTide requires Java+17.

# Run TaskTide container image
docker container run --rm \
    bkenna/tasktide:latest \
        < CLI: Manager | Engine | Web-Api > \
            < CLI Options: >


# -- OR --- Run the required client with provided configs
tasktide \
  < CLI: Manager | Engine | API > \
    < CLI Options: >


# --- OR --- Run using parameters from TaskTide config file
tasktide


🧰 TaskTide Resources


🚀 TaskTide Features

  • 🛠️ Pilot Job Execution Model: Tasks can be dynamically scheduled and executed inside long-running jobs. Providing users a set of operational dials for influence over task lifecycles, introspection, and monitoring in real-time.

  • 🔄 ETL-Friendly: Tasks are treated as extraction, transformation, and loading scripts/programs.

  • Flaticon database Backend Agnostic: Works with document (e.g. MongoDB), embedded (e.g. RocksDB, SQLite), key-value (e.g. Redis), and relational (e.g Postgres) stores.

  • 📦 Flexible Workload Packaging: Supports containerized workloads alongside native executables and scripts, enabling diverse packaging mechanisms for data applications.

  • ⛴️ Portable Worloads: As generic orchestration software. TaskTide gives you portability across different environments.

  • 🔀 Nested Workflow Modeling : Compose tasks into hierarchical workflows using a flexible domain model. Providing operational structure to data application deployment that is robust and transparent.

  • 🧪 Tested: Built with CI/CD, Docker support, and integration tests across database types.


🧱 Architecture

  • Core Lib – Defines the stateful task and workflow data structure described here.

  • Engine Lib – Defines the task processing and tracking logic for WorkItems and their tasks described here.

  • Web API – Defines Jakarta-WS REST API, and an embedded Jetty-WebServer described here.

  • Mutex - Defines ItemStore mechanisms for acquiring a mutex on the configured RocksDB/SQLite database described here.

  • ItemStore - Defines an interface for configuring TaskTide with embedded databases (RocksDB/SQLite) described here.

  • Parser - Defines a configurable command-line argument tree for TaskTide described here.

  • Client Application – Provides access and services for workflow deployments and persistence described here.



TaskTide Architecture

TaskTide