Skip to main content

Concepts

This page outlines the main concepts in Coflux.

Projects​

A Coflux server can host multiple projects. Data for each project is isolated from other projects, and orchestration is handled by a dedicated process for each project.

You should use a separate project when:

  1. Data needs to be kept separate for reasons of security or privacy.
  2. Throughput is
  3. There's a logical separation of concerns.

Workspaces​

A individual project can contain multiple workspaces. All workspaces within a projects are controlled by the same orchestration process, and some level of separation is provided between workspaces, but workspace inheritance allows controlled data sharing. Workspaces might be mapped to deployment environments (e.g., production, staging, development), or separated further - for example a workspace per customer in a production environment, or a workspace per developer in a development environment. Or even more granular separation is possible - for example using temporary workspaces which correspond with a Git branch, to work on fixing a bug or building a new feature.

Workspace inheritance​

By default there is isolation between workspaces within a project - for example, workflows, runs, results are separated. But workspaces can be arranged into a hierarchy. This allows:

  1. Cached (or memoised) results to be inherited from parent workspaces.
  2. Steps to be re-run in a 'child' workspace.

For example, a development workspace can inherit from a production workspace, allowing you to re-run whole workflows, or specific steps within a workflow, in a development workspace, experimenting with changes to the code without having to re-run the whole workflow from scratch. When working with a team on a shared project, you might choose to set up separate workspaces for each engineer, or even create workspaces temporarily to work on specific features.

This makes it easier to diagnose issues that arise in a production workspace by retrying individual steps locally, and trying out code changes safely.

Agents​

An agent is a process that hosts modules (collections workflows/tasks). An agent connects to the server and is associated with a specific project and workspace. The agent waits for commands from the server telling it to execute specific tasks, and the agent monitors and reports progress of these executions back to the server.

This model of having agents connect to the server provides flexibility over where and how agents are run. During development an agent can run locally on a laptop, restarting automatically as code changes are made. Or multiple agents can run in the cloud, or on dedicated machines - or a combination. An agent can be started with specific environment variables associated with the deployment environment (e.g., production access keys).

Workflows​

A workflow is defined in a module, in code. Additionally, tasks can be defined, and called from workflows (or other tasks).

Workflows and tasks are collectively referred to as targets, although workflows are really just special forms of tasks, from which runs can be started. You can think of the distinction between workflows and tasks a bit like the distinction between public and private functions in a module.

Workflows need to be registered with a project and workspace so that they appear in the UI. This can be done explicitly (e.g., for a production workspace as part of a build process), or automatically by an agent when it starts/restarts (using the --register or --dev flag).

Runs​

When a workflow is submitted, this initiates a run. A run is made up of steps, which each correspond to a target to be executed. The target (a workflow or task) can call other tasks, which cause those to scheduled as steps. Each step has at least one associated execution. Steps can be retried (manually or automatically), which will lead to multiple executions being associated with the step.

Assets

Executions can 'persist' assets (files or directories) so that they can be shared with other executions. A persisted asset is given a reference, which must be passed to other executions so that it can be 'restored'.