Bacalhau has recently introduced different job types in v1.1, providing more control and flexibility over the orchestration and scheduling of those jobs - depending on their type.

Despite the differences in job types, all jobs benefit from core functionalities provided by Bacalhau, including:

1. Node selection - the appropriate nodes are selected based on several criteria, including resource availability, priority and feedback from the nodes.

2. Job monitoring - jobs are monitored to ensure they complete, and that they stay in a healthy state.

3. Retries - within limits, Bacalhau will retry certain jobs a set number of times should it fail to complete successfully when requested.

Batch Jobs

Batch jobs are executed on demand, running on a specified number of Bacalhau nodes. These jobs either run until completion or until they reach a timeout. They are designed to carry out a single, discrete task before finishing. This is the only queueable job type.

Ideal for intermittent yet intensive data dives, for instance performing computation over large datasets before publishing the response. This approach eliminates the continuous processing overhead, focusing on specific, in-depth investigations and computation.

Ops Jobs

Similar to batch jobs, ops jobs have a broader reach. They are executed on all nodes that align with the job specification, but otherwise behave like batch jobs.

Ops jobs are perfect for urgent investigations, granting direct access to logs on host machines, where previously you may have had to wait for the logs to arrive at a central location before being able to query them. They can also be used for delivering configuration files for other systems should you wish to deploy an update to many machines at once.

Daemon Jobs

Daemon jobs run continuously on all nodes that meet the criteria given in the job specification. Should any new compute nodes join the cluster after the job was started, and should they meet the criteria, the job will be scheduled to run on that node too.

A good application of daemon jobs is to handle continuously generated data on every compute node. This might be from edge devices like sensors, or cameras, or from logs where they are generated. The data can then be aggregated and compressed them before sending it onwards. For logs, the aggregated data can be relayed at regular intervals to platforms like Kafka or Kinesis, or directly to other logging services with edge devices potentially delivering results via MQTT.

Service Jobs

Service jobs run continuously on a specified number of nodes that meet the criteria given in the job specification. Bacalhau's orchestrator selects the optimal nodes to run the job, and continuously monitors its health, performance. If required, it will reschedule on other nodes.

This job type is good for long-running consumers such as streaming or queuing services, or real-time event listeners.

A Constraint represents a condition that must be met for a compute node to be eligible to run a given job. Operators have the flexibility to manually define node labels when initiating a node using the bacalhau serve command. Additionally, Bacalhau boasts features like automatic resource detection and dynamic labeling, further enhancing its capability.

By defining constraints, you can ensure that jobs are scheduled on nodes that have the necessary requirements or conditions.

Constraint Parameters:

1. Key: The name of the attribute or property to check on the compute node. This could be anything from a specific hardware feature, operating system version, or any other node property.

2. Operator: Determines the kind of comparison to be made against the Key's value, which can be:

   1. in: Checks if the Key's value exists within the provided list of values.

   2. notin: Ensures the Key's value doesn't match any in the provided list of values.

   3. exists: Verifies that a value for the specified Key is present, regardless of its actual value.

   4. !: Confirms the absence of the specified Key. i.e DoesNotExist

   5. gt: Assesses if the Key's value is greater than the provided value.

   6. lt: Assesses if the Key's value is less than the provided value.

   7. = & ==: Both are used to compare the Key's value for an exact match with the provided value.

   8. !=: Ensures the Key's value is not the same as the provided value.

3. Values (optional): A list of values that the node attribute, specified by the Key, is compared against using the Operator. This is not needed for operators like exists or !.

Example:

Consider a scenario where a job should only run on nodes with a GPU and an operating system version greater than 2.0. The constraints for such a requirement might look like:

In this example, the first constraint checks if the node has a GPU, the second constraint ensures the OS is linux, and deployed in eu-west-1 or eu-west-2`.

Notes:

1. Constraints are evaluated as a logical AND, meaning all constraints must be satisfied for a node to be eligible.

2. Using too many specific constraints can lead to a job not being scheduled if no nodes satisfy all the conditions.

3. It's essential to balance the specificity of constraints with the broader needs and resources available in the cluster.

A Job represents a discrete unit of work that can be scheduled and executed. It carries all the necessary information to define the nature of the work, how it should be executed, and the resources it requires.

Type: batch
Count: 1
Priority: 50
Meta:
  version: "1.2.5"
Labels:
  project: "my-project"
Constraints:
  - Key: Architecture
    Operator: '='
    Values:
      - arm64
  - Key: region
    Operator: '='
    Values:
      - us-west-2
Tasks:
  #...

job Parameters

• Name (string : <optional>): A logical name to refer to the job. Defaults to job ID.

• Namespace (string: "default"): The namespace in which the job is running. ClientID is used as a namespace in the public demo network.

• Type (string: <required>): The type of the job, such as batch, ops, daemon or service. You can learn more about the supported jobs types in the Job Types guide.

• Priority (int: 0): Determines the scheduling priority.

• Count (int: <required): Number of replicas to be scheduled. This is only applicable for jobs of type batch and service.

• Meta (Meta : nil): Arbitrary metadata associated with the job.

• Labels (Label[] : nil): Arbitrary labels associated with the job for filtering purposes.

• Constraints (Constraint[] : nil): These are selectors which must be true for a compute node to run this job.

• Tasks (Task[] : <required>):: Task associated with the job, which defines a unit of work within the job. Today we are only supporting single task per job, but with future plans to extend this.

Server-Generated Parameters

The following parameters are generated by the server and should not be set directly.

• ID (string): A unique identifier assigned to this job. It's auto-generated by the server and should not be set directly. Used for distinguishing between jobs with similar names.

• State (State): Represents the current state of the job.

• Version (int): A monotonically increasing version number incremented on job specification update.

• Revision (int): A monotonically increasing revision number incremented on each update to the job's state or specification.

• CreateTime (int): Timestamp of job creation.

• ModifyTime (int): Timestamp of last job modification.

An InputSource defines where and how to retrieve specific artifacts needed for a Task, such as files or data, and where to mount them within the task's context. This ensures the necessary data is present before the task's execution begins.

Bacalhau's InputSource natively supports fetching data from remote sources like S3 and IPFS and can also mount local directories. It is intended to be flexible for future expansion.

InputSource Parameters:

• Source (SpecConfig : <required>): Specifies the origin of the artifact, which could be a URL, an S3 bucket, or other locations.

• Alias (string: <optional>): An optional identifier for this input source. It's particularly useful for dynamic operations within a task, such as dynamically importing data in WebAssembly using an alias.

• Target (string: <required>): Defines the path inside the task's environment where the retrieved artifact should be mounted or stored. This ensures that the task can access the data during its execution.

Usage Examples

InputSources:
  - Source:
      Type: s3
      Params:
        Bucket: my_bucket
        Region: us-west-1
    Target: /my_s3_data
  - Source:
      Type: localDirectory
      Params:
        SourcePath: /path/to/local/directory
        ReadWrite: true
    Target: /my_local_data

In this example, the first input source fetches data from an S3 bucket and mounts it at /my_s3_data within the task. The second input source mounts a local directory at /my_local_data and allows the task to read and write data to it.

Overview

This documentation introduces templating support for bacalhau job run, providing users with the ability to dynamically inject variables into their job specifications. This feature is particularly useful when running multiple jobs with varying parameters such as DuckDB query, S3 buckets, prefixes, and time ranges without the need to edit each job specification file manually.

Motivation

The motivation behind this feature arises from the need to streamline the process of preparing and running multiple jobs with different configurations. Rather than manually editing job specs for each run, users can leverage placeholders and pass actual values at runtime.

Templating Implementation

The templating functionality in Bacalhau is built upon the Go text/template package. This powerful library offers a wide range of features for manipulating and formatting text based on template definitions and input variables.

For more detailed information about the Go text/template library and its syntax, please refer to the official documentation: Go text/template Package.

Usage Examples

Sample Job Spec (job.yaml)

Name: docker job
Type: batch
Count: 1
Tasks:
  - Name: main
    Engine:
      Type: docker
      Params:
        Image: ubuntu:latest
        Entrypoint:
          - /bin/bash
        Parameters:
          - -c
          - echo {{.greeting}} {{.name}}

Running with Templating:

bacalhau job run job.yaml --template-vars "greeting=Hello,name=World"

Defining Flag Multiple Times:

bacalhau job run job.yaml --template-vars "greeting=Hello" --template-vars "name=World"

Disabling Templating:

bacalhau job run job.yaml --no-template

Using Environment Variables:

You can also use environment variables for templating:

export greeting=Hello
export name=World
bacalhau job run job.yaml --template-envs "*"

Passing A Subset of Environment Variables:

bacalhau job run job.yaml --template-envs "greeting|name"

Dry Run to Preview Templated Spec:

To preview the final templated job spec without actually submitting the job, you can use the --dry-run flag:

bacalhau job run job.yaml --template-vars "greeting=Hello,name=World" --dry-run

This will output the processed job specification, showing you how the placeholders have been replaced with the provided values.

More Examples

Query Live Logs

Name: Live logs processing
Type: ops
Tasks:
  - Name: main
    Engine:
      Type: docker
      Params:
        Image: expanso/nginx-access-log-processor:1.0.0
        Parameters:
          - --query
          - {{.query}}
          - --start-time
          - {{or (index . "start-time") ""}}
          - --end-time
          - {{or (index . "end-time") ""}}
    InputSources:
      - Target: /logs
        Source:
          Type: localDirectory
          Params:
            SourcePath: /data/log-orchestration/logs

This is an ops job that runs on all nodes that match the job selection criteria. It accepts duckdb query variable, and two optional start-time and end-time variables to define the time range for the query.

To run this job, you can use the following command:

bacalhau job run job.yaml \
  -V "query=SELECT status FROM logs WHERE status LIKE '5__'" \
  -V "start-time=-5m"

Query S3 Logs

Name: S3 logs processing
Type: batch
Count: 1
Tasks:
  - Name: main
    Engine:
      Type: docker
      Params:
        Image: expanso/nginx-access-log-processor:1.0.0
        Parameters:
          - --query
          - {{.query}}
    InputSources:
      - Target: /logs
        Source:
          Type: s3
          Params:
            Bucket: {{.AccessLogBucket}}
            Key: {{.AccessLogPrefix}}
            Filter: {{or (index . "AccessLogPattern") ".*"}}
            Region: {{.AWSRegion}}

This is a batch job that runs on a single node. It accepts duckdb query variable, and four other variables to define the S3 bucket, prefix, pattern for the logs and the AWS region.

To run this job, you can use the following command:

bacalhau job run job.yaml  \
    -V "AccessLogBucket=my-bucket" \
    -V "AWSRegion=us-east-1" \
    -V "AccessLogPrefix=2023-11-19-*"  \
    -V "AccessLogPattern=^[10-12].*"

A ResultPath denotes a specific location within a Task that contains meaningful output or results. By specifying a ResultPath, you can pinpoint which files or directories are essential and should be retained or published after the task's execution.

ResultPath Parameters:

1. Name: A descriptive label or identifier for the result, allowing for easier referencing and understanding of the output's nature or significance.

2. Path: Specifies the exact location, either a file or a directory, within the task's environment where the result or output is stored. This ensures that after the task completes, the critical data at this path can be accessed, retained, or published as necessary.

The Resources provides a structured way to detail the computational resources a Task requires. By specifying these requirements, you ensure that the task is scheduled on a node with adequate resources, optimizing performance and avoiding potential issues linked to resource constraints.

Resources Parameters:

1. CPU (string: <optional>): Defines the CPU resources required for the task. Units can be specified in cores (e.g., 2 for 2 CPU cores) or in milliCPU units (e.g., 250m or 0.25 for 250 milliCPU units). For instance, if you have half a CPU core, you can represent it as 500m or 0.5.

2. Memory (string: <optional>): Highlights the amount of RAM needed for the task. You can specify the memory in various units such as:

   • Kb for Kilobytes

   • Mb for Megabytes

   • Gb for Gigabytes

   • Tb for Terabytes

3. Disk (string: <optional>): States the disk storage space needed for the task. Similarly, the disk space can be expressed in units like Gb for Gigabytes, Mb for Megabytes, and so on. As an example, 10Gb indicates 10 Gigabytes of storage space.

4. GPU (string: <optional>): Denotes the number of GPU units required. For example, 2 signifies the requirement of 2 GPU units. This is crucial for tasks involving heavy computational processes, machine learning models, or tasks that leverage GPU acceleration.

The Network object offers a method to specify the networking requirements of a Task. It defines the scope and constraints of the network connectivity based on the demands of the task.

Network Parameters:

1. Type (string: "None"): Indicates the network configuration's nature. There are several network modes available:

   • None: This mode implies that the task does not necessitate any networking capabilities.

   • Full: Specifies that the task mandates unrestricted, raw IP networking without any imposed filters.

   • HTTP: This mode constrains the task to only require HTTP networking with specific domains. In this model:

     • The job specifier puts forward a job, stipulating the domain(s) it intends to communicate with.

     • The compute provider assesses the inherent risk of the job based on these domains and bids accordingly.

     • At runtime, the network traffic remains strictly confined to the designated domain(s).

The primary risks for the compute provider center around possible violations of its terms, its hosting provider's terms, or even prevailing laws in its jurisdiction. This encompasses issues such as unauthorized access or distribution of illicit content and potential cyber-attacks.

Conversely, the job specifier's primary risk involves operating in a paid environment. External entities might seek to exploit this environment, for instance, through a compromised package download that initiates a crypto mining operation, depleting the allocated, prepaid job time. By limiting traffic strictly to the pre-specified domains, the potential for such cyber threats diminishes considerably.

While a compute provider might impose its limits through other means, having domains declared upfront allows it to selectively bid on jobs that it can execute without issues, improving the user experience for job specifiers.

1. Domains (string[]: <optional>): A list of domain strings, relevant primarily when the Type is set to HTTP. It dictates the specific domains the task can communicate with over HTTP.

Understanding and utilizing these configurations aptly can ensure that tasks are executed in an environment that aligns with their networking requirements, bolstering efficiency and security.

The Labels block within a Job specification plays a crucial role in Bacalhau, serving as a mechanism for filtering jobs. By attaching specific labels to jobs, users can quickly and effectively filter and manage jobs via both the Command Line Interface (CLI) and Application Programming Interface (API) based on various criteria.

Labels Parameters

Labels are essentially key-value pairs attached to jobs, allowing for detailed categorizations and filtrations. Each label consists of a Key and a Value. These labels can be filtered using operators to pinpoint specific jobs fitting certain criteria.

Filtering Operators

Jobs can be filtered using the following operators:

1. in: Checks if the key's value matches any within a specified list of values.

2. notin: Validates that the key's value isn’t within a provided list of values.

3. exists: Checks for the presence of a specified key, regardless of its value.

4. !: Validates the absence of a specified key. (i.e., DoesNotExist)

5. gt: Checks if the key's value is greater than a specified value.

6. lt: Checks if the key's value is less than a specified value.

7. = & ==: Used for exact match comparisons between the key’s value and a specified value.

8. !=: Validates that the key’s value doesn't match a specified value.

Example Usage

Filter jobs with a label whose key is "environment" and value is "development":

Filter jobs with a label whose key is "version" and value is greater than "2.0":

Filter jobs with a label "project" existing:

Filter jobs without a "project" label:

Practical Applications

• Job Management: Enables efficient management of jobs by categorizing them based on distinct attributes or criteria.

• Automation: Facilitates the automation of job deployment and management processes by allowing scripts and tools to target specific categories of jobs.

• Monitoring & Analytics: Enhances monitoring and analytics by grouping jobs into meaningful categories, allowing for detailed insights and analysis.

Conclusion

The Labels block is instrumental in the enhanced management, filtering, and operation of jobs within Bacalhau. By understanding and utilizing the available operators and label parameters effectively, users can optimize their workflow, automate processes, and achieve detailed insights into their jobs.

When running a node, you can choose which jobs you want to run by using configuration options, environment variables or flags to specify a job selection policy.

setting-up/networking-instructions/networking.md

Job selection probes

If you want more control over making the decision to take on jobs, you can use the --job-selection-probe-exec and --job-selection-probe-http flags.

These are external programs that are passed the following data structure so that they can make a decision about whether or not to take on a job:

{
  "node_id": "XXX",
  "job_id": "XXX",
  "spec": {
    "engine": "docker",
    "verifier": "ipfs",
    "job_spec_vm": {
      "image": "ubuntu:latest",
      "entrypoint": ["cat", "/file.txt"]
    },
    "inputs": [{
      "engine": "ipfs",
      "cid": "XXX",
      "path": "/file.txt"
    }]
  }
}

The exec probe is a script to run that will be given the job data on stdin, and must exit with status code 0 if the job should be run.

The http probe is a URL to POST the job data to. The job will be rejected if the HTTP request returns a non-positive status code (e.g. >= 400).

If the HTTP response is a JSON blob, it should match the following schema and will be used to respond to the bid directly:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "shouldBid": {
      "description": "If the job should be accepted",
      "type": "boolean"
    },
    "shouldWait": {
      "description": "If the node should wait for an async response that will come later. `shouldBid` will be ignored",
      "type": "boolean",
      "default": false,
    },
    "reason": {
      "description": "Human-readable string explaining why the job should be accepted or rejected, or why the wait is required",
      "type": "string"
    }
  },
  "required": [
    "shouldBid",
    "reason"
  ]
}

For example, the following response will reject the job:

{
  "shouldBid": false,
  "reason": "The job did not pass this specific validation: ...",
}

If the HTTP response is not a JSON blob, the content is ignored and any non-error status code will accept the job.

SpecConfig provides a unified structure to specify configurations for various components in Bacalhau, including engines, publishers, and input sources. Its flexible design allows seamless integration with multiple systems like Docker, WebAssembly (Wasm), AWS S3, and local directories, among others.

SpecConfig Parameters

1. Type (string : <required>): Specifies the type of the configuration. Examples include docker and wasm for execution engines, S3 for input sources and publishers, etc.

2. Params (map[string]any : <optional>): A set of key-value pairs that provide the specific configurations for the chosen type. The keys and values are flexible and depend on the Type. For instance, parameters for a Docker engine might include image name and version, while an S3 publisher would require configurations like the bucket name and AWS region. If not provided, it defaults to nil.

Usage Examples

Here are a few hypothetical examples to demonstrate how you might define SpecConfig for different components:

Docker Engine

Full Docker spec can be found .

S3 Publisher

Local Directory Input Source

Remember, the exact keys and values in the Params map will vary depending on the specific requirements of the component being configured. Always refer to the individual component's documentation to understand the available parameters.

In both the Job and Task specifications within Bacalhau, the Meta block is a versatile element used to attach arbitrary metadata. This metadata isn't utilized for filtering or categorizing jobs; there's a separate block specifically designated for that purpose. Instead, the Meta block is instrumental for embedding additional information for operators or external systems, enhancing clarity and context.

Meta Parameters in Job and Task Specs

The Meta block is comprised of key-value pairs, with both keys and values being strings. These pairs aren't constrained by a predefined structure, offering flexibility for users to annotate jobs and tasks with diverse metadata.

User-Defined Metadata

Users can incorporate any arbitrary key-value pairs to convey descriptive information or context about the job or task.

Example:

1. project: Identifies the associated project.

2. version: Specifies the version of the application or service.

3. owner: Names the responsible team or individual.

4. environment: Indicates the stage in the development lifecycle.

Auto-Generated Metadata by Bacalhau

Beyond user-defined metadata, Bacalhau automatically injects specific metadata keys for identification and security purposes.

Bacalhau Auto-Generated Keys:

1. bacalhau.org/requester.id: A unique identifier for the orchestrator that handled the job.

2. bacalhau.org/requester.publicKey: The public key of the requester, aiding in security and validation.

3. bacalhau.org/client.id: The ID for the client submitting the job, enhancing traceability.

Example:

Implications and Utility

1. Identification: The metadata aids in uniquely identifying jobs and tasks, connecting them to their originators and executors.

2. Context Enhancement: Metadata can supplement jobs and tasks with additional data, offering insights and context that aren't captured by standard parameters.

3. Security Enhancement: Auto-generated keys like the requester's public key contribute to the secure handling and execution of jobs and tasks.

The Timeouts object provides a mechanism to impose timing constraints on specific task operations, particularly execution. By setting these timeouts, users can ensure tasks don't run indefinitely and align them with intended durations.

Timeouts Parameters:

• ExecutionTimeout (int: <optional>): Defines the maximum duration (in seconds) that a task is permitted to run. A value of zero indicates that there's no set timeout. This could be particularly useful for tasks that function as daemons and are designed to run indefinitely.

Utilizing the Timeouts judiciously helps in managing resource utilization and ensures tasks adhere to expected timelines, thereby enhancing the efficiency and predictability of job executions.

State Structure Specification

Within Bacalhau, the State structure is designed to represent the status or state of an object (like a ), coupled with a human-readable message for added context. Below is a breakdown of the structure:

State Parameters

1. StateType (T : <required>): Represents the current state of the object. This is a generic parameter that will take on a specific value from a set of defined state types for the object in question. For jobs, this will be one of the values.

2. Message (string : <optional>): A human-readable message giving more context about the current state. Particularly useful for states like Failed to provide insight into the nature of any error.

Job State Types

When State is used for a job, the StateType can be one of the following:

1. Pending: This indicates that the job is submitted but is not yet scheduled for execution.

2. Running: The job is scheduled and is currently undergoing execution.

3. Completed: This state signifies that a job has successfully executed its task. Only applicable for batch jobs.

4. Failed: A state indicating that the job encountered errors and couldn't successfully complete.

5. JobStateTypeStopped: The job has been intentionally halted by the user before its natural completion.

The inclusion of the Message field can offer detailed insights, especially in states like Failed, aiding in error comprehension and debugging.

A Task signifies a distinct unit of work within the broader context of a Job. It defines the specifics of how the task should be executed, where the results should be published, what environment variables are needed, among other configurations

Task Parameters

1. Name (string : <required>): A unique identifier representing the name of the task.

2. Engine ( : required): Configures the execution engine for the task, such as or .

3. Publisher ( : optional): Specifies where the results of the task should be published, such as and publishers. Only applicable for tasks of type batch and ops.

4. Env (map[string]string : optional): A set of environment variables for the driver.

5. Meta ( : optional): Allows association of arbitrary metadata with this task.

6. InputSources ([] : optional): Lists remote artifacts that should be downloaded before task execution and mounted within the task, such as from or .

7. ResultPaths ([] : optional): Indicates volumes within the task that should be included in the published result. Only applicable for tasks of type batch and ops.

8. Resources ( : optional): Details the resources that this task requires.

9. Network ( : optional): Configurations related to the networking aspects of the task.

10. Timeouts ( : optional): Configurations concerning any timeouts associated with the task.