1 of 100

v1.3.1

Welcome to Bacalhau Docs

This is an older version of Bacalhau. For the latest version, go to this link.

What is Bacalhau?

Bacalhau is a platform for fast, cost efficient, and secure computation by running jobs where the data is generated and stored. With Bacalhau, you can streamline your existing workflows without the need of extensive rewriting by running arbitrary Docker containers and WebAssembly (wasm) images as tasks. This architecture is also referred to as Compute Over Data (or CoD). Bacalhau was coined from the Portuguese word for salted Cod fish.

Bacalhau seeks to transform data processing for large-scale datasets to improve cost and efficiency, and to open up data processing to larger audiences. Our goals is to create an open, collaborative compute ecosystem that enables unparalleled collaboration. We (Expanso.io) offer a demo network so you can try out jobs without even installing. Give it a shot!

Why Bacalhau?

⚡️ Bacalhau simplifies the process of managing compute jobs by providing a unified platform for managing jobs across different regions, clouds, and edge devices.

🤝 Bacalhau provides reliable and network-partition resistant orchestration, ensuring that your jobs will complete even if there are network disruptions.

🚨 Bacalhau provides a complete and permanent audit log of exactly what happened, so you can be confident that your jobs are being executed securely.

🔐 You can run private workloads to reduce the chance of leaking private information or inadvertently sharing your data outside of your organization.

💸 Bacalhau reduces ingress/egress costs since jobs are processed closer to the source.

🤓 You can mount your data anywhere on your machine, and Bacalhau will be able to run against that data.

💥 You can integrate with services running on nodes to run a jobs, such as on DuckDB.

📚 Bacalhau operates at scale over parallel jobs. You can batch process petabytes (quadrillion bytes) of data.

How it works

Bacalhau concists of a peer-to-peer network of nodes that enables decentralized communication between computers. The network consists of two types of nodes:

Requester Node: responsible for handling user requests, discovering and ranking compute nodes, forwarding jobs to compute nodes, and monitoring the job lifecycle.

Compute Node: responsible for executing jobs and producing results. Different compute nodes can be used for different types of jobs, depending on their capabilities and resources.

For a more detailed tutorial, check out our Getting Started Tutorial.

The goal of the Bacalhau project is to make it easy to perform distributed computation next to where the data resides. In order to do this, first you need to ingest some data.

Data ingestion

Data is identified by its content identifier (CID) and can be accessed by anyone who knows the CID. Here are some options that can help you mount your data:

Copy data from a URL to public storage
Pin Data to public storage
Copy Data from S3 Bucket to public storage

The options are not limited to the above mentioned. You can mount your data anywhere on your machine, and Bacalhau will be able to run against that data

Security in Bacalhau

All workloads run under restricted Docker or WASM permissions on the node. Additionally, you can use existing (locked down) binaries that are pre-installed through Pluggable Executors.

Best practices in 12-factor apps is to use environment variables to store sensitive data such as access tokens, API keys, or passwords. These variables can be accessed by Bacalhau at runtime and are not visible to anyone who has access to the code or the server.

Alternatively, you can pre-provision credentials to the nodes and access those on a node by node basis.

Finally, endpoints (such as vaults) can also be used to provide secure access to Bacalhau. This way, the client can authenticate with Bacalhau using the token without exposing their credentials.

Workloads Bacalhau is best suited for

Bacalhau can be used for a variety of data processing workloads, including machine learning, data analytics, and scientific computing. It is well-suited for workloads that require processing large amounts of data in a distributed and parallelized manner.

Use Cases

Once you have more than 10 devices generating or storing around 100GB of data, you're likely to face challenges with processing that data efficiently. Traditional computing approaches may struggle to handle such large volumes, and that's where distributed computing solutions like Bacalhau can be extremely useful. Bacalhau can be used in various industries, including security, web serving, financial services, IoT, Edge, Fog, and multi-cloud. Bacalhau shines when it comes to data-intensive applications like data engineering, model training, model inference, molecular dynamics, etc.

Here are some example tutorials on how you can process your data with Bacalhau:

Stable Diffusion AI
Generate Realistic Images using StyleGAN3 and Bacalhau
Object Detection with YOLOv5 on Bacalhau
Running Genomics on Bacalhau
Training Pytorch Model with Bacalhau

For more tutorials, visit our example page

Community

Bacalhau has a very friendly community and we are always happy to help you get started:

GitHub Discussions – ask anything about the project, give feedback or answer questions that will help other users.
Join the Slack Community and go to #bacalhau channel – it is the easiest way engage with other members in the community and get help.
Contributing – learn how to contribute to the Bacalhau project.

Next Steps

👉 Continue with Bacalhau Getting Started guide to learn how to install and run a job with the Bacalhau client.

👉 Or jump directly to try out the different Examples that showcases Bacalhau abilities.

Getting Started

How Bacalhau Works

In this tutorial we will go over the components and the architecture of Bacalhau. You will learn how it is built, what components are used, how you could interact and how you could use Bacalhau.

Chapter 1 - Architecture

Bacalhau is a peer-to-peer network of nodes that enables decentralized communication between computers. The network consists of two types of nodes, which can communicate with each other.

The requester and compute nodes together form a p2p network and use gossiping to discover each other, share information about node capabilities, available resources and health status. Bacalhau is a peer-to-peer network of nodes that enables decentralized communication between computers.

Requester Node: responsible for handling user requests, discovering and ranking compute nodes, forwarding jobs to compute nodes, and monitoring the job lifecycle.

Compute Node: responsible for executing jobs and producing results. Different compute nodes can be used for different types of jobs, depending on their capabilities and resources.

To interact with the Bacalhau network, users can use the Bacalhau CLI (command-line interface) to send requests to a requester node in the network. These requests are sent using the JSON format over HTTP, a widely-used protocol for transmitting data over the internet. Bacalhau's architecture involves two main sections which are the core components and interfaces.

Components overview

Core Components

The core components are responsible for handling requests and connecting different nodes. The network includes two different components:

Requester node

In the Bacalhau network, the requester node is responsible for handling requests from clients using JSON over HTTP. This node serves as the main custodian of jobs that are submitted to it. When a job is submitted to a requester node, it selects compute nodes that are capable and suitable to execute the job, and coordinates the job execution.

Compute node

In the Bacalhau network, it is the compute node that is responsible for determining whether it can execute a job or not. This model allows for a more decentralized approach to job orchestration as the network will function properly even if the requester nodes have stale view of the network, or if concurrent requesters are allocating jobs to the same compute nodes. Once the compute node has run the job and produced results, it will publish the results to a remote destination as specified in the job specification (e.g. S3), and notify the requester of the job completion. The compute node has a collection of named executors, storage sources, and publishers, and it will choose the most appropriate ones based on the job specifications.

Interfaces

The interfaces handle the distribution, execution, storage and publishing of jobs. In the following all the different components are described and their respective protocols are shown.

Transport

The transport interface is responsible for sending messages about jobs that are created, accepted, and executed to other compute nodes. It also manages the identity of individual Bacalhau nodes to ensure that messages are only delivered to authorized nodes, which improves network security. To achieve this, the transport interface uses a protocol, which is a point-to-point scheduling protocol that runs securely and is used to distribute job messages efficiently to other nodes on the network. This is our upgrade to previous handlers as it ensures that messages are delivered to the right nodes without causing network congestion, thereby making communication between nodes more scalable and efficient.

Executor

The executor is a critical component of the Bacalhau network that handles the execution of jobs and ensures that the storage used by the job is local. One of its main responsibilities is to present the input and output storage volumes into the job when it is run. The executor performs two primary functions: presenting the storage volumes in a format that is suitable for the executor and running the job. When the job is completed, the executor will merge the stdout, stderr and named output volumes into a results folder that is then published to a remote location. Overall, the executor plays a crucial role in the Bacalhau network by ensuring that jobs are executed properly, and their results are published accurately.

Storage Provider

In a peer-to-peer network like Bacalhau, storage providers play a crucial role in presenting an upstream storage source. There can be different storage providers available in the network, each with its own way of manifesting the CID (Content IDentifier) to the executor. For instance, there can be a POSIX storage provider that presents the CID as a POSIX filesystem, or a library storage provider that streams the contents of the CID via a library call. Therefore, the storage providers and Executor implementations are loosely coupled, allowing the POSIX and library storage providers to be used across multiple executors, wherever it is deemed appropriate.

Publisher

The publisher is responsible for uploading the final results of a job to a remote location where clients can access them, such as S3 or IPFS.

Chapter 2 - Job cycle

Job preparation

Advanced job preparation

Job Submission

Bacalhau provides an interface to interact with the server via a REST API. Bacalhau uses 127.0.0.1 as the localhost and 1234 as the port by default.

Bacalhau Docker CLI commands

Bacalhau WASM CLI commands

Job Acceptance

When a job is submitted to a requester node, it selects compute nodes that are capable and suitable to execute the job, and communicate with them directly. The compute node has a collection of named executors, storage sources, and publishers, and it will choose the most appropriate ones based on the job specifications.

Job execution

Results publishing

When the Compute node completes the job, it publishes the results to S3's remote storage, IPFS.

Chapter 3 - Returning Information

The Bacalhau client receives updates on the task execution status and results. A user can access the results and manage tasks through the command line interface.

Get Job Results

To Get the results of a job you can run the following command.

One can choose from a wide range of flags, from which a few are shown below.

Describe a Job

To describe a specific job, inserting the ID to the CLI or API gives back an overview of the job.

List of Jobs

If you run more then one job or you want to find a specific job ID

Job Executions

To list executions follow the following commands.

Chapter 4 - Monitoring and Management

Stop a Job

Job History

Job Logs

Installation

Install the Bacalhau CLI

In this tutorial, you'll learn how to install and run a job with the Bacalhau client using the Bacalhau CLI or Docker.

Step 1 - Install the Bacalhau Client

The Bacalhau client is a command-line interface (CLI) that allows you to submit jobs to the Bacalhau. The client is available for Linux, macOS, and Windows. You can also run the Bacalhau client in a Docker container.

Step 1.1 - Install the Bacalhau CLI

Step 1.2 - Verify the Installation

To verify installation and check the version of the client and server, use the version command. To run a Bacalhau client command with Docker, prefix it with docker run ghcr.io/bacalhau-project/bacalhau:latest.

If you're wondering which server is being used, the Bacalhau Project has a demo network that's shared with the community. This network allows you to familiarize with Bacalhau's capabilities and launch jobs from your computer without maintaining a compute cluster on your own.

Step 2 - Submit a Hello World job

To submit a job in Bacalhau, we will use the bacalhau docker run command. The command runs a job using the Docker executor on the node. Let's take a quick look at its syntax:

Let's take a look at the results of the command execution in the terminal:

After the above command is run, the job is submitted to the public network, which processes the job and Bacalhau prints out the related job id:

The job_id above is shown in its full form. For convenience, you can use the shortened version, in this case: 9d20bbad.

Let's take a look at the results of the command execution in the terminal:

Step 3 - Checking the State of your Jobs

After having deployed the job, we now can use the CLI for the interaction with the network. The jobs were sent to the public demo network, where it was processed and we can call the following functions. The job_id will differ for every submission.

Step 3.1 - Job status:

You can check the status of the job using bacalhau list command adding the --id-filter flag and specifying your job id.

Let's take a look at the results of the command execution in the terminal:

When it says Completed, that means the job is done, and we can get the results.

Step 3.2 - Job information:

You can find out more information about your job by using bacalhau describe.

Let's take a look at the results of the command execution in the terminal:

This outputs all information about the job, including stdout, stderr, where the job was scheduled, and so on.

Step 3.3 - Job download:

You can download your job results directly by using bacalhau get.

This results in

In the command below, we created a directory called myfolder and download our job output to be stored in that directory.

After the download has finished you should see the following contents in the results directory.

Step 4 - Viewing your Job Output

That should print out the string Hello World.

Step 5 - Where to go next?

Here are few resources that provide a deeper dive into running jobs with Bacalhau:

Support

Docker Workload Onboarding

How to use docker containers with Bacalhau

Docker Workloads

Bacalhau executes jobs by running them within containers. Bacalhau employs a syntax closely resembling Docker, allowing you to utilize the same containers. The key distinction lies in how input and output data are transmitted to the container via IPFS, enabling scalability on a global level.

This section describes how to migrate a workload based on a Docker container into a format that will work with the Bacalhau client.

Requirements

Here are few things to note before getting started:

Container Registry: Ensure that the container is published to a public container registry that is accessible from the Bacalhau network.
Architecture Compatibility: Bacalhau supports only images that match the host node's architecture. Typically, most nodes run on linux/amd64, so containers in arm64 format are not able to run.
Input Flags: The --input ipfs://... flag supports only directories and does not support CID subpaths. The --input https://... flag supports only single files and does not support URL directories. The --input s3://... flag supports S3 keys and prefixes. For example, s3://bucket/logs-2023-04* includes all logs for April 2023.

Note: Only about a third of examples have their containers here. If you can't find one, feel free to contact the team.

Runtime Restrictions

To help provide a safe, secure network for all users, we add the following runtime restrictions:

Limited Ingress/Egress Networking:

Data Passing with Docker Volumes:

A job includes the concept of input and output volumes, and the Docker executor implements support for these. This means you can specify your CIDs, URLs, and/or S3 objects as input paths and also write results to an output volume. This can be seen in the following example:

The above example demonstrates an input volume flag -i s3://mybucket/logs-2023-04*, which mounts all S3 objects in bucket mybucket with logs-2023-04 prefix within the docker container at location /input (root).

Output volumes are mounted to the Docker container at the location specified. In the example above, any content written to /output_folder will be made available within the apples folder in the job results CID.

Once the job has run on the executor, the contents of stdout and stderr will be added to any named output volumes the job has used (in this case apples), and all those entities will be packaged into the results folder which is then published to a remote location by the publisher.

Onboarding Your Workload

Step 1 - Read Data From Your Directory

If you need to pass data into your container you will do this through a Docker volume. You'll need to modify your code to read from a local directory.

We make the assumption that you are reading from a directory called /inputs, which is set as the default.

Step 2 - Write Data to the Your Directory

If you need to return data from your container you will do this through a Docker volume. You'll need to modify your code to write to a local directory.

We make the assumption that you are writing to a directory called /outputs, which is set as the default.

Step 3 - Build and Push Your Image To a Registry

Most Bacalhau nodes are of an x86_64 architecture, therefore containers should be built for x86_64 systems.

For example:

Step 4 - Test Your Container

To test your docker image locally, you'll need to execute the following command, changing the environment variables as necessary:

Let's see what each command will be used for:

Exports the current working directory of the host system to the LOCAL_INPUT_DIR variable. This variable will be used for binding a volume and transferring data into the container.

Exports the current working directory of the host system to the LOCAL_OUTPUT_DIR variable. Similarly, this variable will be used for binding a volume and transferring data from the container.

Creates an array of commands CMD that will be executed inside the container. In this case, it is a simple command executing 'ls' in the /inputs directory and writing text to the /outputs/stdout file.

Launches a Docker container using the specified variables and commands. It binds volumes to facilitate data exchange between the host and the container.

For example:

The result of the commands' execution is shown below:

Step 5 - Upload the Input Data

Data is identified by its content identifier (CID) and can be accessed by anyone who knows the CID. You can use either of these methods to upload your data:

You can choose to

You can mount your data anywhere on your machine, and Bacalhau will be able to run against that data

Step 6 - Run the Workload on Bacalhau

To launch your workload in a Docker container, using the specified image and working with input data specified via IPFS CID, run the following command.

To check the status of your job, run the following command.

To get more information on your job, you can run the following command.

To download your job, run.

To put this all together into one would look like the following.

This outputs the following.

The --input flag does not support CID subpaths for ipfs:// content.

Alternatively, you can run your workload with a publicly accessible http(s) URL, which will download the data temporarily into your public storage:

The --input flag does not support URL directories.

Troubleshooting

If you run into this compute error while running your docker image

This can often be resolved by re-tagging your docker image

Support

WebAssembly (WASM) Workloads

Bacalhau supports running programs that are compiled to WebAssembly (WASM). With the Bacalhau client, you can upload WASM programs, retrieve data from public storage, read and write data, receive program arguments, and access environment variables.

Prerequisites and Limitations

Supported WebAssembly System Interface (WASI) Bacalhau can run compiled WASM programs that expect the WebAssembly System Interface (WASI) Snapshot 1. Through this interface, WebAssembly programs can access data, environment variables, and program arguments.
Networking Restrictions All ingress/egress networking is disabled – you won't be able to pull data/code/weights etc. from an external source. WASM jobs can say what data they need using URLs or CIDs (Content IDentifier) and can then access the data by reading from the filesystem.
Single-Threading There is no multi-threading as WASI does not expose any interface for it.

Onboarding Your Workload

Step 1: Replace network operations with filesystem reads and writes

If your program typically involves reading from and writing to network endpoints, follow these steps to adapt it for Bacalhau:

Replace Network Operations: Instead of making HTTP requests to external servers (e.g., example.com), modify your program to read data from the local filesystem.
Input Data Handling: Specify the input data location in Bacalhau using the --input flag when running the job. For instance, if your program used to fetch data from example.com, read from the /inputs folder locally, and provide the URL as input when executing the Bacalhau job. For example, --input http://example.com.
Output Handling: Adjust your program to output results to standard output (stdout) or standard error (stderr) pipes. Alternatively, you can write results to the filesystem, typically into an output mount. In the case of WASM jobs, a default folder at /outputs is available, ensuring that data written there will persist after the job concludes.

By making these adjustments, you can effectively transition your program to operate within the Bacalhau environment, utilizing filesystem operations instead of traditional network interactions.

You can specify additional or different output mounts using the -o flag.

Step 2: Configure your compiler to output WASI-compliant WebAssembly

You will need to compile your program to WebAssembly that expects WASI. Check the instructions for your compiler to see how to do this.

For example, Rust users can specify the wasm32-wasi target to rustup and cargo to get programs compiled for WASI WebAssembly. See the Rust example for more information on this.

Step 3: Upload the input data

Data is identified by its content identifier (CID) and can be accessed by anyone who knows the CID. You can use either of these methods to upload your data:

Copy data from a URL to public storage
Pin Data to public storage
Copy Data from S3 Bucket to public storage.

You can mount your data anywhere on your machine, and Bacalhau will be able to run against that data

Step 4: Run your program

You can run a WebAssembly program on Bacalhau using the bacalhau wasm run command.

bacalhau wasm run

Run Locally Compiled Program:

If your program is locally compiled, specify it as an argument. For instance, running the following command will upload and execute the main.wasm program:

bacalhau wasm run main.wasm

The program you specify will be uploaded to a Bacalhau storage node and will be publicly available.

Alternative Program Specification:

You can use a Content IDentifier (CID) for a specific WebAssembly program.

bacalhau wasm run Qmajb9T3jBdMSp7xh2JruNrqg3hniCnM6EUVsBocARPJRQ

Input Data Specification:

Make sure to specify any input data using --input flag.

bacalhau wasm run --input http://example.com

This ensures the necessary data is available for the program's execution.

Program arguments

You can give the WASM program arguments by specifying them after the program path or CID. If the WASM program is already compiled and located in the current directory, you can run it by adding arguments after the file name:

bacalhau wasm run echo.wasm hello world

For a specific WebAssembly program, run:

bacalhau wasm run Qmajb9T3jBdMSp7xh2JruNrqg3hniCnM6EUVsBocARPJRQ hello world

Write your program to use program arguments to specify input and output paths. This makes your program more flexible in handling different configurations of input and output volumes.

For example, instead of hard-coding your program to read from /inputs/data.txt, accept a program argument that should contain the path and then specify the path as an argument to bacalhau wasm run:

bacalhau wasm run prog.wasm /inputs/data.txt

Your language of choice should contain a standard way of reading program arguments that will work with WASI.

Environment variables

You can also specify environment variables using the -e flag.

$ bacalhau wasm run prog.wasm -e HELLO=world

Examples

See the Rust example for a workload that leverages WebAssembly support.

Support

If you have questions or need support or guidance, please reach out to the Bacalhau team via Slack (#general channel).

Hardware Setup

Different jobs may require different amounts of resources to execute. Some jobs may have specific hardware requirements, such as GPU. This page describes how to specify hardware requirements for your job.

Please bear in mind that each executor is implemented independently and these docs might be slightly out of date. Double check the man page for the executor you are using with bacalhau [executor] --help.

Docker Executor

The following table describes how to specify hardware requirements for the Docker executor.

How it Works

When you specify hardware requirements, the job will be offered out to the network to see if there are any nodes that can satisfy the requirements. If there are, the job will be scheduled on the node and the executor will be started.

GPU Setup

Bacalhau supports GPU workloads. Learn how to run a job using GPU workloads with the Bacalhau client.

Prerequisites

The Bacalhau network must have an executor node with a GPU exposed
Your container must include the CUDA runtime (cudart) and must be compatible with the CUDA version running on the node

Usage

Use following command to see available resources amount:

To submit a request for a job that requires more than the standard set of resources, add the --cpu and --memory flags. For example, for a job that requires 2 CPU cores and 4Gb of RAM, use --cpu=2 --memory=4Gb, e.g.:

To submit a GPU job request, use the --gpu flag under the docker run command to select the number of GPUs your job requires. For example:

Limitations

The following limitations currently exist within Bacalhau.

Maximum CPU and memory limits depend on the participants in the network
For GPU:
1. NVIDIA, Intel or AMD GPUs only
2. Only the Docker Executor supports GPUs

Setting Up

Running a Node

Job selection policy

When running a node, you can choose which jobs you want to run by using configuration options, environment variables or flags to specify a job selection policy.

Config property

serve flag

Default value

Meaning

Node.Compute.JobSelection.Locality

--job-selection-data-locality

Anywhere

Only accept jobs that reference data we have locally ("local") or anywhere ("anywhere").

Node.Compute.JobSelection.ProbeExec

--job-selection-probe-exec

unused

Use the result of an external program to decide if we should take on the job.

Node.Compute.JobSelection.ProbeHttp

--job-selection-probe-http

unused

Use the result of a HTTP POST to decide if we should take on the job.

Node.Compute.JobSelection.RejectStatelessJobs

--job-selection-reject-stateless

False

Node.Compute.JobSelection.AcceptNetworkedJobs

--job-selection-accept-networked

False

Job selection probes

If you want more control over making the decision to take on jobs, you can use the --job-selection-probe-exec and --job-selection-probe-http flags.

These are external programs that are passed the following data structure so that they can make a decision about whether or not to take on a job:

{
  "node_id": "XXX",
  "job_id": "XXX",
  "spec": {
    "engine": "docker",
    "verifier": "ipfs",
    "job_spec_vm": {
      "image": "ubuntu:latest",
      "entrypoint": ["cat", "/file.txt"]
    },
    "inputs": [{
      "engine": "ipfs",
      "cid": "XXX",
      "path": "/file.txt"
    }]
  }
}

The exec probe is a script to run that will be given the job data on stdin, and must exit with status code 0 if the job should be run.

The http probe is a URL to POST the job data to. The job will be rejected if the HTTP request returns a non-positive status code (e.g. >= 400).

If the HTTP response is a JSON blob, it should match the following schema and will be used to respond to the bid directly:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "shouldBid": {
      "description": "If the job should be accepted",
      "type": "boolean"
    },
    "shouldWait": {
      "description": "If the node should wait for an async response that will come later. `shouldBid` will be ignored",
      "type": "boolean",
      "default": false,
    },
    "reason": {
      "description": "Human-readable string explaining why the job should be accepted or rejected, or why the wait is required",
      "type": "string"
    }
  },
  "required": [
    "shouldBid",
    "reason"
  ]
}

For example, the following response will reject the job:

{
  "shouldBid": false,
  "reason": "The job did not pass this specific validation: ...",
}

If the HTTP response is not a JSON blob, the content is ignored and any non-error status code will accept the job.

Configuration Overview

How to configure your Bacalhau node.

Bacalhau employs the viper and cobra libraries for configuration management. Users can configure their Bacalhau node through a combination of command-line flags, environment variables, and the dedicated configuration file.

The Bacalhau Repo

Bacalhau manages its configuration, metadata, and internal state within a specialized repository named .bacalhau. Serving as the heart of the Bacalhau node, this repository holds the data and settings that determine node behavior. It's located on the filesystem, and by default, Bacalhau initializes this repository at $HOME/.bacalhau, where $HOME is the home directory of the user running the bacalhau process.

To customize this location, users can:

Set the BACALHAU_DIR environment variable to specify their desired path.
Utilize the --repo command line flag to specify their desired path.

Upon executing a Bacalhau command for the first time, the system will initialize the .bacalhau repository. If such a repository already exists, Bacalhau will seamlessly access its contents.

Structure of a Newly Initialized .bacalhau Repository

Below is the structure of a freshly initialized `.bacalhau` repository:

$ tree ~/.bacalhau
├── QmdGUjsMHEgtAfdtw7U62yPEcAZFtA33tKMsczLToegZtv-compute/
│   ├── executions.db
│   └── jobStats.json
├── QmdGUjsMHEgtAfdtw7U62yPEcAZFtA33tKMsczLToegZtv-requester/
│   └── jobs.db
├── config.yaml
├── executor_storages/
├── libp2p_private_key
├── plugins/
├── repo.version
└── user_id.pem

This repository comprises four directories and seven files:

Files

user_id.pem:
- This file houses the Bacalhau node user's cryptographic private key, used for signing requests sent to a Requester Node.
- Format: PEM.
repo.version:
- Indicates the version of the Bacalhau node's repository.
- Format: JSON, e.g., {"Version":1}.
libp2p_private_key:
- Stores the Bacalhau node's libp2p private key, essential for its network identity. The NodeID of a Bacalhau node is derived from this key.
- Format: Base64 encoded RSA private key.
config.yaml:
- Contains configuration settings for the Bacalhau node.
- Format: YAML.
update.json:
- A file containing the date/time when the last version check was made.
- Format: JSON, e.g., {"LastCheck":"2024-01-24T11:06:14.631816Z"}
tokens.json:
- A file containing the tokens obtained through authenticating with bacalhau clusters.

Directories

QmdGUjsMHEgtAfdtw7U62yPEcAZFtA33tKMsczLToegZtv-compute:
- Contains the BoltDB executions.db database, which aids the Compute node in state persistence. Additionally, the jobStats.json file records the Compute Node's completed jobs tally.
- Note: The segment QmdGUjsMHEgtAfdtw7U62yPEcAZFtA33tKMsczLToegZtv is a unique NodeID for each Bacalhau node, derived from the libp2p_private_key.
QmdGUjsMHEgtAfdtw7U62yPEcAZFtA33tKMsczLToegZtv-requester:
- Contains the BoltDB jobs.db database for the Requester node's state persistence.
- Note: NodeID derivation is similar to the Compute directory.
executor_storages:
- Storage for data handled by Bacalhau storage drivers.
plugins:
- Houses binaries that allow the Compute node to execute specific tasks.
- Note: This feature is currently experimental and isn't active during standard node operations.

Configuring a Bacalhau Node

Within a .bacalhau repository, a config.yaml file may be present. This file serves as the configuration source for the bacalhau node and adheres to the YAML format.

Although the config.yaml file is optional, its presence allows Bacalhau to load custom configurations; otherwise, Bacalhau is configured with built-in default values, environment variables and command line flags.

Modifications to the config.yaml file will not be dynamically loaded by the Bacalhau node. A restart of the node is required for any changes to take effect. Bacalhau determines its configuration based on the following precedence order, with each item superseding the subsequent:

Command-line Flag
Environment Variable
Config File
Defaults

Relationship Between `config.yaml` and Bacalhau Environment Variables

Bacalhau establishes a direct relationship between the value-bearing keys within the config.yaml file and corresponding environment variables. For these keys that have no further sub-keys, the environment variable name is constructed by capitalizing each segment of the key, and then joining them with underscores, prefixed with BACALHAU_.

For example, a YAML key with the path Node.IPFS.Connect translates to the environment variable BACALHAU_NODE_IPFS_CONNECT and is represented in a file like:

Node:
    IPFS:
        Connect: value

There is no corresponding environment variable for either Node or Node.IPFS. Config values may also have other environment variables that set them for simplicity or to maintain backwards compatibility.

Environments

Bacalhau leverages the BACALHAU_ENVIRONMENT environment variable to determine the specific environment configuration when initializing a repository. Notably, if a .bacalhau repository has already been initialized, the BACALHAU_ENVIRONMENT setting will be ignored.
By default, if the BACALHAU_ENVIRONMENT variable is not explicitly set by the user, Bacalhau will adopt the production environment settings.
Below is a breakdown of the configurations associated with each environment:
1. Production (public network)
- Environment Variable: BACALHAU_ENVIRONMENT=production
- Configurations:
  - Node.ClientAPI.Host: "bootstrap.production.bacalhau.org"
  - Node.Client.API.Host: 1234
  - ...other configurations specific to this environment...
2. Staging (staging network)
- Environment Variable: BACALHAU_ENVIRONMENT=staging
- Configurations:
  - Node.ClientAPI.Host: "bootstrap.staging.bacalhau.org"
  - Node.Client.API.Host: 1234
  - ...other configurations specific to this environment...
3. Development (development network)
- Environment Variable: BACALHAU_ENVIRONMENT=development
- Configurations:
  - Node.ClientAPI.Host: "bootstrap.development.bacalhau.org"
  - Node.Client.API.Host: 1234
  - ...other configurations specific to this environment...
4. Local (private or local networks)
- Environment Variable: BACALHAU_ENVIRONMENT=local
- Configurations:
  - Node.ClientAPI.Host: "0.0.0.0"
  - Node.Client.API.Host: 1234
  - ...other configurations specific to this environment...
Note: The above configurations provided for each environment are not exhaustive. Consult the specific environment documentation for a comprehensive list of configurations.

Usage Examples

How to initialize a Bacalhau Server for a local private network

$ env BACALHAU_ENVIRONMENT=local ./bin/darwin_arm64/bacalhau serve
INF pkg/repo/fs.go:187 > Initializing repo at '/Users/frrist/.bacalhau' for environment 'local'

How to initialize a Bacalhau Server with a custom repo path

$ bacalhau --repo=/path/to/repo serve
INF pkg/repo/fs.go:187 > Initializing repo at '/path/to/repo' for environment 'production'

$ export BACALHAU_DIR=/path/to/repo
$ bacalhau serve
INF pkg/repo/fs.go:187 > Initializing repo at '/path/to/repo' for environment 'production'

How to start a Bacalhau Server with DEBUG logs

$ env LOG_LEVEL=debug ./bin/darwin_arm64/bacalhau serve
DBG pkg/system/environment.go:53 > Defaulting to production environment: os.Args: [./bin/darwin_arm64/bacalhau serve]

Configuring Transport Level Security

How to configure TLS for the requester node APIs

By default, the requester node APIs used by the Bacalhau CLI are accessible over HTTP, but it is possible to configure it to use Transport Level Security (TLS) so that they are accessible over HTTPS instead. There are several ways to obtain the necessary certificates and keys, and Bacalhau supports obtaining them via ACME and Certificate Authorities or even self-signing them.

Once configured, you must ensure that instead of using http://IP:PORT you use https://IP:PORT to access the Bacalhau API

Getting a certificate from Let's Encrypt with ACME

Alternatively, you may set these options via the environment variable, BACALHAU_AUTO_TLS. If you are using a configuration file, you can set the values inNode.ServerAPI.TLS.AutoCert instead.

As a result of the Lets Encrypt verification step, it is necessary for the server to be able to handle requests on port 443. This typically requires elevated privileges, and rather than obtain these through a privileged account (such as root), you should instead use setcap to grant the executable the right to bind to ports <1024.

A cache of ACME data is held in the config repository, by default ~/.bacalhau/autocert-cache, and this will be used to manage renewals to avoid rate limits.

Getting a certificate from a Certificate Authority

Obtaining a TLS certificate from a Certificate Authority (CA) without using the Automated Certificate Management Environment (ACME) protocol involves a manual process that typically requires the following steps:

Choose a Certificate Authority: First, you need to select a trusted Certificate Authority that issues TLS certificates. Popular CAs include DigiCert, GlobalSign, Comodo (now Sectigo), and others. You may also consider whether you want a free or paid certificate, as CAs offer different pricing models.
Generate a Certificate Signing Request (CSR): A CSR is a text file containing information about your organization and the domain for which you need the certificate. You can generate a CSR using various tools or directly on your web server. Typically, this involves providing details such as your organization's name, common name (your domain name), location, and other relevant information.
Submit the CSR: Access your chosen CA's website and locate their certificate issuance or order page. You'll typically find an option to "Submit CSR" or a similar option. Paste the contents of your CSR into the provided text box.
Verify Domain Ownership: The CA will usually require you to verify that you own the domain for which you're requesting the certificate. They may send an email to one of the standard domain-related email addresses (e.g., admin@yourdomain.com, webmaster@yourdomain.com). Follow the instructions in the email to confirm domain ownership.
Complete Additional Verification: Depending on the CA's policies and the type of certificate you're requesting (e.g., Extended Validation or EV certificates), you may need to provide additional documentation to verify your organization's identity. This can include legal documents or phone calls from the CA to confirm your request.
Payment and Processing: If you're obtaining a paid certificate, you'll need to make the payment at this stage. Once the CA has received your payment and completed the verification process, they will issue the TLS certificate.

Once you have obtained your certificates, you will need to put two files in a location that bacalhau can read them. You need the server certificate, often called something like server.cert or server.cert.pem, and the server key which is often called something like server.key or server.key.pem.

Once you have these two files available, you must start bacalhau serve which two new flags. These are tlscert and tlskey flags, whose arguments should point to the relevant file. An example of how it is used is:

Alternatively, you may set these options via the environment variables, BACALHAU_TLS_CERT and BACALHAU_TLS_KEY. If you are using a configuration file, you can set the values inNode.ServerAPI.TLS.ServerCertificate and Node.ServerAPI.TLS.ServerKey instead.

Self-signed certificates

Once you have generated the necessary files, the steps are much like above, you must start bacalhau serve which two new flags. These are tlscert and tlskey flags, whose arguments should point to the relevant file. An example of how it is used is:

Alternatively, you may set these options via the environment variables, BACALHAU_TLS_CERT and BACALHAU_TLS_KEY. If you are using a configuration file, you can set the values inNode.ServerAPI.TLS.ServerCertificate and Node.ServerAPI.TLS.ServerKey instead.

If you use self-signed certificates, it is unlikely that any clients will be able to verify the certificate when connecting to the Bacalhau APIs. There are three options available to work around this problem:

Provide a CA certificate file of trusted certificate authorities, which many software libraries support in addition to system authorities.
Install the CA certificate file in the system keychain of each machine that needs access to the Bacalhau APIs.
Instruct the software library you are using not to verify HTTPS requests.

Observability

Logging

Logging in Bacalhau outputs in human-friendly format to stderr at INFO level by default, but this can be changed by two environment variables:

LOG_LEVEL - Can be one of trace, debug, error, warn or fatal to output more or fewer logging messages as required
LOG_TYPE - Can be one of the following values:
- default - output logs to stderr in a human-friendly format
- json - log messages outputted to stdout in JSON format
- combined - log JSON formatted messages to stdout and human-friendly format to stderr

Log statements should include the relevant trace, span and job ID so it can be tracked back to the work being performed.

Metrics

Bacalhau produces a number of different metrics including those around the libp2p resource manager (rcmgr), performance of the requester HTTP API and the number of jobs accepted/completed/received.

Tracing

Traces are produced for all major pieces of work when processing a job, although the naming of some spans is still being worked on. You can find relevant traces covering working on a job by searching for the jobid attribute.

Viewing

The metrics and traces can easily be forwarded to a variety of different services as we use OpenTelemetry, such as Honeycomb or Datadog.

To view the data locally, or simply to not use a SaaS offering, you can start up Jaeger and Prometheus placing these three files into a directory then running docker compose start while running Bacalhau with the OTEL_EXPORTER_OTLP_PROTOCOL=grpc and OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 environment variables.

Running a Compute Node Using Docker

Good news everyone! You can now run your Bacalhau-IPFS stack in Docker.

This page describes several ways in which to operate Bacalhau. You can choose the method that best suits your needs. The methods are:

Pre-Prerequisites

This guide works best on a Linux machine. If you're trying to run this on a Mac, you may encounter issues. Remember that network host mode doesn't work.

Connect to the Public Bacalhau Network Using Docker

This method is appropriate for those who:

Provide compute resources to the public Bacalhau network

This is not appropriate for:

Testing and development
Running a private network

Prerequisites

(Optional) Start a Public IPFS Node

This will start a local IPFS node and connect it to the public DHT. If you already have an IPFS node running, then you can skip this step.

Some notes about this command:

It wipes the $(pwd)/ipfs directory to make sure you have a clean slate
It runs the IPFS container in the specified Docker network
It exposes the IPFS API port to the world on port 4002, to avoid clashes with Bacalhau
It exposes the admin RPC API to the local host only, on port 5001
We are not specifying or removing the bootstrap nodes, so it will default to connecting to public machines

Start a Public Bacalhau Node

Bacalhau consists of two parts: a "requester" that is responsible for operating the API and managing jobs, and a "compute" element that is responsible for executing jobs. In a public context, you'd typically just run a compute node, and allow the public requesters to handle the traffic.

Notes about the command:

It runs the Bacalhau container in "host" mode. This means that the container will use the same network as the host.
It uses the root user, which is the default system user that has access to the Docker socket on a Mac. You may need to change this to suit your environment.
It mounts the Docker Socket
It mounts the /tmp directory
It exposes the Bacalhau API ports to the world
The container version should match that of the current release
The IPFS connect string points to the RPC port of the IPFS node in Docker. Because Bacalhau is running in the same network, it can use DNS to find the IPFS container IP. If you're running your own node, replace it
The --node-type flag is set to compute because we only want to run a compute node
The --labels flag is used to set a human-readable label for the node, and so we can run jobs on our machine later
We specify the --peer env flag so that it uses the environment specified by BACALHAU_ENVIRONMENT=production and therefore connects to the public network peers

There are several ways to ensure that the Bacalhau compute node is connected to the network.

First, check that the Bacalhau libp2p port is open and connected. On Linux you can run lsof and it should look something like this:

Note the three established connections at the bottom. These are the production bootstrap nodes that Bacalhau is now connected to.

You can also check that the node is connected by listing the current network peers and grepping for your IP address or node ID. The node ID can be obtained from the Bacalhau logs. It will look something like this:

Finally, submit a job with the label you specified when you ran the compute node. If this label is unique, there should be only one node with this label. The job should succeed. Run the following:

If instead, your job fails with the following error, it means that the compute node is not connected to the network:

Run a Private Bacalhau Network Using Docker (Insecure)

This method is insecure. It does not lock down the IPFS node. Anyone connected to your network can access the IPFS node and read/write data. This is not recommended for production use.

This method is appropriate for:

Testing and development
Evaluating the Bacalhau platform before scaling jobs via the public network

This method is useful for testing and development. It's easier to use because it doesn't require a secret IPFS swarm key -- this is essentially an authentication token that allows you to connect to the node.

This method is not appropriate for:

Secure, private use
Production use

Prerequisites

Start a Local IPFS Node (Insecure)

To run an insecure, private node, you need to initialize your IPFS configuration by removing all of the default public bootstrap nodes. Then we run the node in the normal way, without the special LIBP2P_FORCE_PNET flag that checks for a secure private connection.

Some notes about this command:

It wipes the $(pwd)/ipfs directory to make sure you have a clean slate
It removes the default bootstrap nodes
It runs the IPFS container in the specified Docker network
It exposes the IPFS API port to the local host only, to prevent accidentally exposing the IPFS node, on 4002, to avoid clashes with Bacalhau
It exposes the admin RPC API to the local host only, on port 5001

Start a Private Bacalhau Node

Notes about the command:

It runs the Bacalhau container in the specified Docker network
It uses the root user, which is the default system user that has access to the Docker socket on a Mac. You may need to change this to suit your environment
It mounts the Docker Socket
It mounts the /tmp directory and specifies this as the location where Bacalhau will write temporary execution data (BACALHAU_NODE_COMPUTESTORAGEPATH)
It exposes the Bacalhau API ports to the local host only, to prevent accidentally exposing the API to the public internet
The container version should match that of the Bacalhau installed on your system
The IPFS connect string points to the RPC port of the IPFS node. Because Bacalhau is running in the same network, it can use DNS to find the IPFS container IP.
The --node-type flag is set to requester,compute because we want to run both a requester and a compute node

Run a Job on the Private Network

Now it's time to run a job. Recall that you exposed the Bacalhau API on the default ports to the local host only. So you'll need to use the --api-host flag to tell Bacalhau where to find the API. Everything else is a standard part of the Bacalhau CLI.

The job should succeed. Run it again but this time capture the job ID to make it easier to retrieve the results.

Retrieve the Results on the Private Network (Insecure)

To retrieve the results using the Bacalhau CLI, you need to know the p2p swarm multiaddress of the IPFS node because you don't want to connect to the public global IPFS network. To do that you can run the IPFS id command (and parse to remove the trub at the bottom of the barrel):

Note that the command above changes the reported port from 4001 to 4002. This is because the IPFS node is running on port 4002, but the IPFS id command reports the port as 4001.

Now get the results:

Alternatively, you can use the Docker container, mount the results volume, and change the --api-host to the name of the Bacalhau container and the --ipfs-swarm-addrs back to port 4001:

Run a Private Bacalhau Network Using Docker (Secure)

Running a private secure network is useful in a range of scenarios, including:

Running a private network for a private project

You need two things. A private IPFS node to store data and a Bacalhau node to execute over that data. To keep the nodes private you need to tell the nodes to shush and use a secret key. This is a bit harder to use, and a bit more involved than the insecure version.

Prerequisites

Start a Private IPFS Node (Secure)

First, you need to bootstrap a new IPFS cluster for your own private use. This consists of a process of generating a swarm key, removing any bootstrap nodes, and then starting the IPFS node.

Some notes about this command:

It wipes the $(pwd)/ipfs directory to make sure you have a clean slate
It generates a new swarm key -- this is the token that is required to connect to this node
It removes the default bootstrap nodes
It runs the IPFS container in the specified Docker network
It exposes the IPFS API port to the local host only, to prevent accidentally exposing the IPFS node, on 4002, to avoid clashes with Bacalhau
It exposes the admin RPC API to the local host only, on port 5001

Start a Private Bacalhau Node (Secure)

Run a Job on the Private Network (Secure)

Retrieve the Results on the Private Network (Secure)

The same process as above can be used to retrieve results from the IPFS node as long as the Bacalhau get command has access to the IPFS swarm key.

Running the Bacalhau binary from outside of Docker:

Alternatively, you can use the Docker container, mount the results volume, and change the --api-host to the name of the Bacalhau container and the --ipfs-swarm-addrs back to port 4001:

Common Prerequisites

Create a New Docker Network

Without this, inter-container DNS will not work, and internet access may not work either.

Double check that this network can access the internet (so Bacalhau can call external URLs).

This should be successful. If it is not, then please troubleshoot your docker networking. For example, on my Mac, I had to totally uninstall Docker, restart the computer, and then reinstall Docker. Then it worked. Also check https://docs.docker.com/desktop/troubleshoot/known-issues/. Apparently "ping from inside a container to the Internet does not work as expected.". No idea what that means. How do you break ping?

Test that the IPFS Node is Working

You can now browse the IPFS web UI at http://127.0.0.1:5001/webui.

Test that the Bacalhau Node is Working

Ensure that the Bacalhau logs (docker logs bacalhau) have no errors.

Check that your Bacalhau installation is the same version:

The versions should match. Alternatively, you can use the Docker container:

Perform a list command to ensure you can connect to the Bacalhau API.

It should return empty.

Authenticate with docker hub

Should you wish to authenticate with Docker Hub when pulling images, you can do so by specifying credentials as environment variables wherever your compute node is running.

Workload Onboarding

This directory contains examples relating to performing common tasks with Bacalhau.

Container

Python

R (language)

Data Ingestion

Jobs

Networking Instructions

Examples

Case Studies

Data Engineering

This directory contains examples relating to data engineering workloads. The goal is to provide a range of examples that show you how to work with Bacalhau in a variety of use cases.

Model Inference

Stable Diffusion Checkpoint Inference

Model Training

Molecular Dynamics

How Bacalhau Works

In this tutorial we will go over the components and the architecture of Bacalhau. You will learn how it is built, what components are used, how you could interact and how you could use Bacalhau.

Chapter 1 - Architecture

Bacalhau is a peer-to-peer network of nodes that enables decentralized communication between computers. The network consists of two types of nodes, which can communicate with each other.

Requester Node: responsible for handling user requests, discovering and ranking compute nodes, forwarding jobs to compute nodes, and monitoring the job lifecycle.

Compute Node: responsible for executing jobs and producing results. Different compute nodes can be used for different types of jobs, depending on their capabilities and resources.

Components overview

Core Components

The core components are responsible for handling requests and connecting different nodes. The network includes two different components:

Requester node

Compute node

Interfaces

The interfaces handle the distribution, execution, storage and publishing of jobs. In the following all the different components are described and their respective protocols are shown.

Transport

Executor

Storage Provider

Publisher

The publisher is responsible for uploading the final results of a job to a remote location where clients can access them, such as S3 or IPFS.

Chapter 2 - Job cycle

Job preparation

You can create jobs in the Bacalhau network using various introduced in version 1.2. Each job may need specific variables, resource requirements and data details that are described in the .

Advanced job preparation

Prepare data with Bacalhau by , or . Mount data anywhere for Bacalhau to run against. Refer to , , and Source Specifications for data source usage.

Optimize workflows without completely redesigning them. Run arbitrary tasks using Docker containers and WebAssembly images. Follow the Onboarding guides for and workloads.

Explore GPU workload support with Bacalhau. Learn how to run using the Bacalhau client in the GPU Workloads section. Integrate Python applications with Bacalhau using the .

For node operation, refer to the section for configuring and running a Bacalhau node. If you prefer an isolated environment, explore the for performing tasks without connecting to the main Bacalhau network.

Job Submission

You should use the Bacalhau client to send a task to the network. The client transmits the job information to the Bacalhau network via established protocols and interfaces. Jobs submitted via the Bacalhau CLI are forwarded to a Bacalhau network node at via port 1234 by default. This Bacalhau node will act as the requester node for the duration of the job lifecycle.

Bacalhau provides an interface to interact with the server via a REST API. Bacalhau uses 127.0.0.1 as the localhost and 1234 as the port by default.

bacalhau create [flags]

You can use the command with to create a job in Bacalhau using JSON and YAML formats.

Endpoint: `PUT /api/v1/orchestrator/jobs`

You can use to submit a new job for execution.

You can use the bacalhau docker run to start a job in a Docker container. Below, you can see an excerpt of the commands:

Bacalhau Docker CLI commands

bacalhau docker run [flags] IMAGE[:TAG|@DIGEST] [COMMAND] [ARG...]

Flags:
    --concurrency int                  How many nodes should run the job (default 1)
    --cpu string                       Job CPU cores (e.g. 500m, 2, 8).
    --disk string                      Job Disk requirement (e.g. 500Gb, 2Tb, 8Tb).
    --do-not-track                     When true the job will not be tracked(?) TODO BETTER DEFINITION
    --domain stringArray               Domain(s) that the job needs to access (for HTTP networking)
    --download                         Should we download the results once the job is complete?
    --download-timeout-secs duration   Timeout duration for IPFS downloads. (default 5m0s)
    --dry-run                          Do not submit the job, but instead print out what will be submitted
    --entrypoint strings               Override the default ENTRYPOINT of the image
-e, --env strings                      The environment variables to supply to the job (e.g. --env FOO=bar --env BAR=baz)
-f, --follow                           When specified will follow the output from the job as it runs
-g, --gettimeout int                   Timeout for getting the results of a job in --wait (default 10)
    --gpu string                       Job GPU requirement (e.g. 1, 2, 8).
-h, --help                             help for run
    --id-only                          Print out only the Job ID on successful submission.
-i, --input storage                    Mount URIs as inputs to the job. Can be specified multiple times. Format: src=URI,dst=PATH[,opt=key=value]
                                    Examples:
                                    # Mount IPFS CID to /inputs directory
                                    -i ipfs://QmeZRGhe4PmjctYVSVHuEiA9oSXnqmYa4kQubSHgWbjv72
                                    # Mount S3 object to a specific path
                                    -i s3://bucket/key,dst=/my/input/path
                                    # Mount S3 object with specific endpoint and region
                                    -i src=s3://bucket/key,dst=/my/input/path,opt=endpoint=https://s3.example.com,opt=region=us-east-1
    --ipfs-connect string              The ipfs host multiaddress to connect to, otherwise an in-process IPFS node will be created if not set.
    --ipfs-serve-path string           path local Ipfs node will persist data to
    --ipfs-swarm-addrs strings         IPFS multiaddress to connect the in-process IPFS node to - cannot be used with --ipfs-connect. (default [/ip4/35.245.161.250/tcp/4001/p2p/12D3KooWAQpZzf3qiNxpwizXeArGjft98ZBoMNgVNNpoWtKAvtYH,/ip4/35.245.161.250/udp/4001/quic/p2p/12D3KooWAQpZzf3qiNxpwizXeArGjft98ZBoMNgVNNpoWtKAvtYH,/ip4/34.86.254.26/tcp/4001/p2p/12D3KooWLfFBjDo8dFe1Q4kSm8inKjPeHzmLBkQ1QAjTHocAUazK,/ip4/34.86.254.26/udp/4001/quic/p2p/12D3KooWLfFBjDo8dFe1Q4kSm8inKjPeHzmLBkQ1QAjTHocAUazK,/ip4/35.245.215.155/tcp/4001/p2p/12D3KooWH3rxmhLUrpzg81KAwUuXXuqeGt4qyWRniunb5ipjemFF,/ip4/35.245.215.155/udp/4001/quic/p2p/12D3KooWH3rxmhLUrpzg81KAwUuXXuqeGt4qyWRniunb5ipjemFF,/ip4/34.145.201.224/tcp/4001/p2p/12D3KooWBCBZnXnNbjxqqxu2oygPdLGseEbfMbFhrkDTRjUNnZYf,/ip4/34.145.201.224/udp/4001/quic/p2p/12D3KooWBCBZnXnNbjxqqxu2oygPdLGseEbfMbFhrkDTRjUNnZYf,/ip4/35.245.41.51/tcp/4001/p2p/12D3KooWJM8j97yoDTb7B9xV1WpBXakT4Zof3aMgFuSQQH56rCXa,/ip4/35.245.41.51/udp/4001/quic/p2p/12D3KooWJM8j97yoDTb7B9xV1WpBXakT4Zof3aMgFuSQQH56rCXa])
    --ipfs-swarm-key string            Optional IPFS swarm key required to connect to a private IPFS swarm
-l, --labels strings                   List of labels for the job. Enter multiple in the format '-l a -l 2'. All characters not matching /a-zA-Z0-9_:|-/ and all emojis will be stripped.
    --memory string                    Job Memory requirement (e.g. 500Mb, 2Gb, 8Gb).
    --network network-type             Networking capability required by the job. None, HTTP, or Full (default None)
    --node-details                     Print out details of all nodes (overridden by --id-only).
-o, --output strings                   name:path of the output data volumes. 'outputs:/outputs' is always added unless '/outputs' is mapped to a different name. (default [outputs:/outputs])
    --output-dir string                Directory to write the output to.
    --private-internal-ipfs            Whether the in-process IPFS node should auto-discover other nodes, including the public IPFS network - cannot be used with --ipfs-connect. Use "--private-internal-ipfs=false" to disable. To persist a local Ipfs node, set BACALHAU_SERVE_IPFS_PATH to a valid path. (default true)
-p, --publisher publisher              Where to publish the result of the job (default ipfs)
    --raw                              Download raw result CIDs instead of merging multiple CIDs into a single result
-s, --selector string                  Selector (label query) to filter nodes on which this job can be executed, supports '=', '==', and '!='.(e.g. -s key1=value1,key2=value2). Matching objects must satisfy all of the specified label constraints.
    --target all|any                   Whether to target the minimum number of matching nodes ("any") (default) or all matching nodes ("all") (default any)
    --timeout int                      Job execution timeout in seconds (e.g. 300 for 5 minutes)
    --wait                             Wait for the job to finish. Use --wait=false to return as soon as the job is submitted. (default true)
    --wait-timeout-secs int            When using --wait, how many seconds to wait for the job to complete before giving up. (default 600)
-w, --workdir string                   Working directory inside the container. Overrides the working directory shipped with the image (e.g. via WORKDIR in Dockerfile).

You can also use the bacalhau wasm run to run a job compiled into the (WASM) format. Below, you can find an excerpt of the commands in the Bacalhau CLI:

Bacalhau WASM CLI commands

bacalhau wasm run {cid-of-wasm | } [--entry-point ] [wasm-args ...] [flags]

Flags:
    --concurrency int                  How many nodes should run the job (default 1)
    --cpu string                       Job CPU cores (e.g. 500m, 2, 8).
    --disk string                      Job Disk requirement (e.g. 500Gb, 2Tb, 8Tb).
    --do-not-track                     When true the job will not be tracked(?) TODO BETTER DEFINITION
    --domain stringArray               Domain(s) that the job needs to access (for HTTP networking)
    --download                         Should we download the results once the job is complete?
    --download-timeout-secs duration   Timeout duration for IPFS downloads. (default 5m0s)
    --dry-run                          Do not submit the job, but instead print out what will be submitted
    --entry-point string               The name of the WASM function in the entry module to call. This should be a zero-parameter zero-result function that
                                will execute the job. (default "_start")
-e, --env strings                      The environment variables to supply to the job (e.g. --env FOO=bar --env BAR=baz)
-f, --follow                           When specified will follow the output from the job as it runs
-g, --gettimeout int                   Timeout for getting the results of a job in --wait (default 10)
    --gpu string                       Job GPU requirement (e.g. 1, 2, 8).
-h, --help                             help for run
    --id-only                          Print out only the Job ID on successful submission.
-U, --import-module-urls url           URL of the WASM modules to import from a URL source. URL accept any valid URL supported by the 'wget' command, and supports both HTTP and HTTPS.
-I, --import-module-volumes cid:path   CID:path of the WASM modules to import from IPFS, if you need to set the path of the mounted data.
-i, --input storage                    Mount URIs as inputs to the job. Can be specified multiple times. Format: src=URI,dst=PATH[,opt=key=value]
                                        Examples:
                                        # Mount IPFS CID to /inputs directory
                                        -i ipfs://QmeZRGhe4PmjctYVSVHuEiA9oSXnqmYa4kQubSHgWbjv72
                                        # Mount S3 object to a specific path
                                        -i s3://bucket/key,dst=/my/input/path
                                        # Mount S3 object with specific endpoint and region
                                        -i src=s3://bucket/key,dst=/my/input/path,opt=endpoint=https://s3.example.com,opt=region=us-east-1
    --ipfs-connect string              The ipfs host multiaddress to connect to, otherwise an in-process IPFS node will be created if not set.
    --ipfs-serve-path string           path local Ipfs node will persist data to
    --ipfs-swarm-addrs strings         IPFS multiaddress to connect the in-process IPFS node to - cannot be used with --ipfs-connect. (default [/ip4/35.245.161.250/tcp/4001/p2p/12D3KooWAQpZzf3qiNxpwizXeArGjft98ZBoMNgVNNpoWtKAvtYH,/ip4/35.245.161.250/udp/4001/quic/p2p/12D3KooWAQpZzf3qiNxpwizXeArGjft98ZBoMNgVNNpoWtKAvtYH,/ip4/34.86.254.26/tcp/4001/p2p/12D3KooWLfFBjDo8dFe1Q4kSm8inKjPeHzmLBkQ1QAjTHocAUazK,/ip4/34.86.254.26/udp/4001/quic/p2p/12D3KooWLfFBjDo8dFe1Q4kSm8inKjPeHzmLBkQ1QAjTHocAUazK,/ip4/35.245.215.155/tcp/4001/p2p/12D3KooWH3rxmhLUrpzg81KAwUuXXuqeGt4qyWRniunb5ipjemFF,/ip4/35.245.215.155/udp/4001/quic/p2p/12D3KooWH3rxmhLUrpzg81KAwUuXXuqeGt4qyWRniunb5ipjemFF,/ip4/34.145.201.224/tcp/4001/p2p/12D3KooWBCBZnXnNbjxqqxu2oygPdLGseEbfMbFhrkDTRjUNnZYf,/ip4/34.145.201.224/udp/4001/quic/p2p/12D3KooWBCBZnXnNbjxqqxu2oygPdLGseEbfMbFhrkDTRjUNnZYf,/ip4/35.245.41.51/tcp/4001/p2p/12D3KooWJM8j97yoDTb7B9xV1WpBXakT4Zof3aMgFuSQQH56rCXa,/ip4/35.245.41.51/udp/4001/quic/p2p/12D3KooWJM8j97yoDTb7B9xV1WpBXakT4Zof3aMgFuSQQH56rCXa])
    --ipfs-swarm-key string            Optional IPFS swarm key required to connect to a private IPFS swarm
-l, --labels strings                   List of labels for the job. Enter multiple in the format '-l a -l 2'. All characters not matching /a-zA-Z0-9_:|-/ and all emojis will be stripped.
    --memory string                    Job Memory requirement (e.g. 500Mb, 2Gb, 8Gb).
    --network network-type             Networking capability required by the job. None, HTTP, or Full (default None)
    --node-details                     Print out details of all nodes (overridden by --id-only).
-o, --output strings                   name:path of the output data volumes. 'outputs:/outputs' is always added unless '/outputs' is mapped to a different name. (default [outputs:/outputs])
    --output-dir string                Directory to write the output to.
    --private-internal-ipfs            Whether the in-process IPFS node should auto-discover other nodes, including the public IPFS network - cannot be used with --ipfs-connect. Use "--private-internal-ipfs=false" to disable. To persist a local Ipfs node, set BACALHAU_SERVE_IPFS_PATH to a valid path. (default true)
-p, --publisher publisher              Where to publish the result of the job (default ipfs)
    --raw                              Download raw result CIDs instead of merging multiple CIDs into a single result
-s, --selector string                  Selector (label query) to filter nodes on which this job can be executed, supports '=', '==', and '!='.(e.g. -s key1=value1,key2=value2). Matching objects must satisfy all of the specified label constraints.
    --target all|any                   Whether to target the minimum number of matching nodes ("any") (default) or all matching nodes ("all") (default any)
    --timeout int                      Job execution timeout in seconds (e.g. 300 for 5 minutes)
    --wait                             Wait for the job to finish. Use --wait=false to return as soon as the job is submitted. (default true)
    --wait-timeout-secs int            When using --wait, how many seconds to wait for the job to complete before giving up. (default 600)

Job Acceptance

Job execution

The selected compute node receives the job and starts its execution inside a container. The container can use different executors to work with the data and perform the necessary actions. A job can use the docker executor, WASM executor or a library storage volumes. Use to view the parameters to configure the Docker Engine. If you want tasks to be executed in a WebAssembly environment, pay attention to .

Results publishing

When the Compute node completes the job, it publishes the results to S3's remote storage, IPFS.

Bacalhau's seamless integration with IPFS ensures that users have a decentralized option for publishing their task results, enhancing accessibility and resilience while reducing dependence on a single point of failure. View to get the detailed information.

Bacalhau's S3 Publisher provides users with a secure and efficient method to publish task results to any S3-compatible storage service. This publisher supports not just AWS S3, but other S3-compatible services offered by cloud providers like Google Cloud Storage and Azure Blob Storage, as well as open-source options like MinIO. View to get the detailed information.

Chapter 3 - Returning Information

The Bacalhau client receives updates on the task execution status and results. A user can access the results and manage tasks through the command line interface.

Get Job Results

To Get the results of a job you can run the following command.

bacalhau get [id] [flags]

One can choose from a wide range of flags, from which a few are shown below.

Usage:
  bacalhau get [id] [flags]

Flags:
      --download-timeout-secs duration   Timeout duration for IPFS downloads. (default 5m0s)
  -h, --help                             help for get
      --ipfs-connect string              The ipfs host multiaddress to connect to, otherwise an in-process IPFS node will be created if not set.
      --ipfs-serve-path string           path local Ipfs node will persist data to
      --ipfs-swarm-addrs strings         IPFS multiaddress to connect the in-process IPFS node to - cannot be used with --ipfs-connect. (default [/ip4/35.245.161.250/tcp/4001/p2p/12D3KooWAQpZzf3qiNxpwizXeArGjft98ZBoMNgVNNpoWtKAvtYH,/ip4/35.245.161.250/udp/4001/quic/p2p/12D3KooWAQpZzf3qiNxpwizXeArGjft98ZBoMNgVNNpoWtKAvtYH,/ip4/34.86.254.26/tcp/4001/p2p/12D3KooWLfFBjDo8dFe1Q4kSm8inKjPeHzmLBkQ1QAjTHocAUazK,/ip4/34.86.254.26/udp/4001/quic/p2p/12D3KooWLfFBjDo8dFe1Q4kSm8inKjPeHzmLBkQ1QAjTHocAUazK,/ip4/35.245.215.155/tcp/4001/p2p/12D3KooWH3rxmhLUrpzg81KAwUuXXuqeGt4qyWRniunb5ipjemFF,/ip4/35.245.215.155/udp/4001/quic/p2p/12D3KooWH3rxmhLUrpzg81KAwUuXXuqeGt4qyWRniunb5ipjemFF,/ip4/34.145.201.224/tcp/4001/p2p/12D3KooWBCBZnXnNbjxqqxu2oygPdLGseEbfMbFhrkDTRjUNnZYf,/ip4/34.145.201.224/udp/4001/quic/p2p/12D3KooWBCBZnXnNbjxqqxu2oygPdLGseEbfMbFhrkDTRjUNnZYf,/ip4/35.245.41.51/tcp/4001/p2p/12D3KooWJM8j97yoDTb7B9xV1WpBXakT4Zof3aMgFuSQQH56rCXa,/ip4/35.245.41.51/udp/4001/quic/p2p/12D3KooWJM8j97yoDTb7B9xV1WpBXakT4Zof3aMgFuSQQH56rCXa])
      --ipfs-swarm-key string            Optional IPFS swarm key required to connect to a private IPFS swarm
      --output-dir string                Directory to write the output to.
      --private-internal-ipfs            Whether the in-process IPFS node should auto-discover other nodes, including the public IPFS network - cannot be used with --ipfs-connect. Use "--private-internal-ipfs=false" to disable. To persist a local Ipfs node, set BACALHAU_SERVE_IPFS_PATH to a valid path. (default true)
      --raw                              Download raw result CIDs instead of merging multiple CIDs into a single result

Describe a Job

To describe a specific job, inserting the ID to the CLI or API gives back an overview of the job.

bacalhau describe [id] [flags]

You can use the command with to get a full description of a job in yaml format.

Endpoint: `GET /api/v1/orchestrator/jobs/:jobID`

You can use to retrieve the specification and current status of a particular job.

List of Jobs

If you run more then one job or you want to find a specific job ID

bacalhau list [flags]

You can use the command with to list jobs on the network in yaml format.

Endpoint: `GET /api/v1/orchestrator/jobs`

You can use to retrieve a list of jobs.

Job Executions

To list executions follow the following commands.

bacalhau job executions [id] [flags]

You can use the command with to list all executions associated with a job, identified by its ID, in yaml format.

Endpoint: `GET /api/v1/orchestrator/jobs/:jobID/executions`

You can use to retrieve all executions for a particular job.

Chapter 4 - Monitoring and Management

The Bacalhau client provides the user with tools to monitor and manage the execution of jobs. You can get information about status, progress and decide on next steps. View the if you want to know the node's health, capabilities, and deployed Bacalhau version. To get information about the status and characteristics of the nodes in the cluster use .

Stop a Job

bacalhau cancel [id] [flags]

You can use the command with to cancel a job that was previously submitted and stop it running if it has not yet completed.

Endpoint: `DELETE /api/v1/orchestrator/jobs/:jobID`

You can use to terminate a specific job asynchronously.

Job History

bacalhau job history [id] [flags]

You can use the command with to enumerate the historical events related to a job, identified by its ID.

Endpoint: `GET /api/v1/orchestrator/jobs/:jobID/history`

You can use to retrieve historical events for a specific job.

Job Logs

bacalhau logs [flags] [id]

You can use this to retrieve the log output (stdout, and stderr) from a job. If the job is still running it is possible to follow the logs after the previously generated logs are retrieved.

To familiarize yourself with all the commands used in Bacalhau, please view

Running a Compute Node Using Docker

Good news everyone! You can now run your Bacalhau-IPFS stack in Docker.

This page describes several ways in which to operate Bacalhau. You can choose the method that best suits your needs. The methods are:

Pre-Prerequisites

This guide works best on a Linux machine. If you're trying to run this on a Mac, you may encounter issues. Remember that network host mode doesn't work.
You need to have Docker installed. If you don't have it, you can .

Connect to the Public Bacalhau Network Using Docker

This method is appropriate for those who:

Provide compute resources to the public Bacalhau network

This is not appropriate for:

Testing and development
Running a private network

Prerequisites

(Optional) Start a Public IPFS Node

This will start a local IPFS node and connect it to the public DHT. If you already have an IPFS node running, then you can skip this step.

Some notes about this command:

It wipes the $(pwd)/ipfs directory to make sure you have a clean slate
It runs the IPFS container in the specified Docker network
It exposes the IPFS API port to the world on port 4002, to avoid clashes with Bacalhau
It exposes the admin RPC API to the local host only, on port 5001
We are not specifying or removing the bootstrap nodes, so it will default to connecting to public machines

# Wipe the current ipfs directory if it exists
rm -rf $(pwd)/ipfs && mkdir $(pwd)/ipfs
# Start the IPFS node
docker run \
    -d --network bacalhau-network --name ipfs_host \
    -v $(pwd)/staging:/export -v $(pwd)/ipfs:/data/ipfs \
    -p 4002:4001 -p 4002:4001/udp \
    -p 127.0.0.1:5001:5001 \
    ipfs/kubo:latest

You can now .

Start a Public Bacalhau Node

Notes about the command:

It runs the Bacalhau container in "host" mode. This means that the container will use the same network as the host.
It uses the root user, which is the default system user that has access to the Docker socket on a Mac. You may need to change this to suit your environment.
It mounts the Docker Socket
It mounts the /tmp directory
It exposes the Bacalhau API ports to the world
The container version should match that of the current release
The IPFS connect string points to the RPC port of the IPFS node in Docker. Because Bacalhau is running in the same network, it can use DNS to find the IPFS container IP. If you're running your own node, replace it
The --node-type flag is set to compute because we only want to run a compute node
The --labels flag is used to set a human-readable label for the node, and so we can run jobs on our machine later
We specify the --peer env flag so that it uses the environment specified by BACALHAU_ENVIRONMENT=production and therefore connects to the public network peers

sudo docker run \
    -d --rm --name bacalhau \
    --net host \
    --env BACALHAU_ENVIRONMENT=production \
    -u root \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /tmp:/tmp \
    ghcr.io/bacalhau-project/bacalhau:latest \
    serve \
        --ipfs-connect /dns4/localhost/tcp/5001 \
        --node-type compute \
        --labels "owner=docs-quick-start" \
        --private-internal-ipfs=false \
        --peer env

There are several ways to ensure that the Bacalhau compute node is connected to the network.

First, check that the Bacalhau libp2p port is open and connected. On Linux you can run lsof and it should look something like this:

❯ sudo lsof -i :1235

COMMAND      PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
bacalhau 1284922 root    3u  IPv4 134301303      0t0  TCP *:1235 (LISTEN)
bacalhau 1284922 root    7u  IPv4 134301307      0t0  UDP *:1235
bacalhau 1284922 root    8u  IPv6 134301308      0t0  TCP *:1235 (LISTEN)
bacalhau 1284922 root    9u  IPv6 134301309      0t0  UDP *:1235
bacalhau 1284922 root   12u  IPv4 134303799      0t0  TCP phil-ethereum-node.europe-west2-c.c.bacalhau-development.internal:1235->191.115.245.35.bc.googleusercontent.com:1235 (ESTABLISHED)
bacalhau 1284922 root   13u  IPv4 134302914      0t0  TCP phil-ethereum-node.europe-west2-c.c.bacalhau-development.internal:1235->251.61.245.35.bc.googleusercontent.com:1235 (ESTABLISHED)
bacalhau 1284922 root   14u  IPv4 134302917      0t0  TCP phil-ethereum-node.europe-west2-c.c.bacalhau-development.internal:1235->239.251.245.35.bc.googleusercontent.com:1235 (ESTABLISHED)

Note the three established connections at the bottom. These are the production bootstrap nodes that Bacalhau is now connected to.

❯ curl -s bootstrap.production.bacalhau.org:1234/peers | jq | grep -A 10 QmaEpsWj4Gw31tBZZ6yagS9ZRSfT8oPgqFuqbvffWJrba5

    "ID": "QmaEpsWj4Gw31tBZZ6yagS9ZRSfT8oPgqFuqbvffWJrba5",
    "Addrs": [
      "/ip4/35.197.229.139/tcp/1235",
      "/ip4/10.154.0.4/tcp/1235",
      "/ip4/127.0.0.1/tcp/1235",
      "/ip4/10.154.0.4/udp/1235/quic",
      "/ip4/127.0.0.1/udp/1235/quic",
      "/ip6/::1/tcp/1235",
      "/ip6/::1/udp/1235/quic"
    ]
  }

Finally, submit a job with the label you specified when you ran the compute node. If this label is unique, there should be only one node with this label. The job should succeed. Run the following:

bacalhau docker run --input=http://example.org/index.html --selector owner=docs-quick-start ghcr.io/bacalhau-project/examples/upload:v1

If instead, your job fails with the following error, it means that the compute node is not connected to the network:

Error: failed to submit job: publicapi: after posting request: error starting job: not enough nodes to run job. requested: 1, available: 0

Run a Private Bacalhau Network Using Docker (Insecure)

This method is insecure. It does not lock down the IPFS node. Anyone connected to your network can access the IPFS node and read/write data. This is not recommended for production use.

This method is appropriate for:

Testing and development
Evaluating the Bacalhau platform before scaling jobs via the public network

This method is not appropriate for:

Secure, private use
Production use

Prerequisites

Start a Local IPFS Node (Insecure)

Some notes about this command:

It wipes the $(pwd)/ipfs directory to make sure you have a clean slate
It removes the default bootstrap nodes
It runs the IPFS container in the specified Docker network
It exposes the IPFS API port to the local host only, to prevent accidentally exposing the IPFS node, on 4002, to avoid clashes with Bacalhau
It exposes the admin RPC API to the local host only, on port 5001

# Wipe the current ipfs directory if it exists
rm -rf $(pwd)/ipfs && mkdir $(pwd)/ipfs
# Remove the bootstrap nodes
docker run -t -v $(pwd)/staging:/export -v $(pwd)/ipfs:/data/ipfs ipfs/kubo:latest bootstrap rm --all
# Start the IPFS node
docker run \
    -d --network bacalhau-network --name ipfs_host \
    -v $(pwd)/staging:/export -v $(pwd)/ipfs:/data/ipfs \
    -p 127.0.0.1:4002:4001 -p 127.0.0.1:4002:4001/udp \
    -p 127.0.0.1:8080:8080 -p 127.0.0.1:5001:5001 \
    ipfs/kubo:latest

You can now .

Start a Private Bacalhau Node

Notes about the command:

It runs the Bacalhau container in the specified Docker network
It uses the root user, which is the default system user that has access to the Docker socket on a Mac. You may need to change this to suit your environment
It mounts the Docker Socket
It mounts the /tmp directory and specifies this as the location where Bacalhau will write temporary execution data (BACALHAU_NODE_COMPUTESTORAGEPATH)
It exposes the Bacalhau API ports to the local host only, to prevent accidentally exposing the API to the public internet
The container version should match that of the Bacalhau installed on your system
The IPFS connect string points to the RPC port of the IPFS node. Because Bacalhau is running in the same network, it can use DNS to find the IPFS container IP.
The --node-type flag is set to requester,compute because we want to run both a requester and a compute node

docker run \
    -d --network bacalhau-network --name bacalhau \
    -u root \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /tmp:/tmp \
    -e BACALHAU_NODE_COMPUTESTORAGEPATH=/tmp \
    -p 127.0.0.1:1234:1234 -p 127.0.0.1:1235:1235 -p 127.0.0.1:1235:1235/udp \
    ghcr.io/bacalhau-project/bacalhau:latest \
    serve \
        --ipfs-connect /dns4/ipfs_host/tcp/5001 \
        --node-type requester,compute

You can now .

Run a Job on the Private Network

bacalhau docker run --api-host=localhost --input=http://example.org/index.html ghcr.io/bacalhau-project/examples/upload:v1

The job should succeed. Run it again but this time capture the job ID to make it easier to retrieve the results.

export JOB_ID=$(bacalhau docker run --api-host=localhost --input=http://example.org/index.html --wait --id-only ghcr.io/bacalhau-project/examples/upload:v1)

Retrieve the Results on the Private Network (Insecure)

export SWARM_ID=$(docker run -t --rm --network=bacalhau-network ipfs/kubo:latest --api=/dns4/ipfs_host/tcp/5001 id -f="<id>" | tail -n 1)
export SWARM_ADDR=/ip4/127.0.0.1/tcp/4002/p2p/$SWARM_ID

Note that the command above changes the reported port from 4001 to 4002. This is because the IPFS node is running on port 4002, but the IPFS id command reports the port as 4001.

Now get the results:

rm -rf results && mkdir results && \
    bacalhau --api-host=localhost --ipfs-swarm-addrs=$SWARM_ADDR get --output-dir=results $JOB_ID

Alternatively, you can use the Docker container, mount the results volume, and change the --api-host to the name of the Bacalhau container and the --ipfs-swarm-addrs back to port 4001:

rm -rf results && mkdir results && \
docker run -t --rm --network=bacalhau-network \
    -v $(pwd)/results:/results \
    ghcr.io/bacalhau-project/bacalhau:latest \
    get --api-host=bacalhau --ipfs-swarm-addrs=/dns4/bacalhau/tcp/4001/p2p/$SWARM_ID --output-dir=/results $JOB_ID

Run a Private Bacalhau Network Using Docker (Secure)

Running a private secure network is useful in a range of scenarios, including:

Running a private network for a private project

Prerequisites

Start a Private IPFS Node (Secure)

Private IPFS nodes are experimental. for more information.

First, you need to bootstrap a new IPFS cluster for your own private use. This consists of a process of generating a swarm key, removing any bootstrap nodes, and then starting the IPFS node.

Some notes about this command:

It wipes the $(pwd)/ipfs directory to make sure you have a clean slate
It generates a new swarm key -- this is the token that is required to connect to this node
It removes the default bootstrap nodes
It runs the IPFS container in the specified Docker network
It exposes the IPFS API port to the local host only, to prevent accidentally exposing the IPFS node, on 4002, to avoid clashes with Bacalhau
It exposes the admin RPC API to the local host only, on port 5001

# Wipe the current ipfs directory if it exists
rm -rf $(pwd)/ipfs && mkdir $(pwd)/ipfs
# Create a new swarm key -- a secret key that will be used to bootstrap the private network
echo -e "/key/swarm/psk/1.0.0/\n/base16/\n`tr -dc 'a-f0-9' < /dev/urandom | head -c64`" > $(pwd)/ipfs/swarm.key
# Remove the bootstrap nodes
docker run -t -v $(pwd)/staging:/export -v $(pwd)/ipfs:/data/ipfs ipfs/kubo:latest bootstrap rm --all
# Start the IPFS node
docker run \
    -d --network bacalhau-network --name ipfs_host \
    -e LIBP2P_FORCE_PNET=1 \
    -v $(pwd)/staging:/export -v $(pwd)/ipfs:/data/ipfs \
    -p 127.0.0.1:4002:4001 -p 127.0.0.1:4002:4001/udp \
    -p 127.0.0.1:8080:8080 -p 127.0.0.1:5001:5001 \
    ipfs/kubo:latest

Start a Private Bacalhau Node (Secure)

The instructions to run a secure private Bacalhau network are the same as the insecure version, .

Run a Job on the Private Network (Secure)

The instructions to run a job are the same as the insecure version, .

Retrieve the Results on the Private Network (Secure)

The same process as above can be used to retrieve results from the IPFS node as long as the Bacalhau get command has access to the IPFS swarm key.

Running the Bacalhau binary from outside of Docker:

bacalhau --api-host=localhost --ipfs-swarm-addrs=$SWARM_ADDR --ipfs-swarm-key=$(pwd)/ipfs/swarm.key get $JOB_ID

Alternatively, you can use the Docker container, mount the results volume, and change the --api-host to the name of the Bacalhau container and the --ipfs-swarm-addrs back to port 4001:

mkdir results && \
docker run -t --rm --network=bacalhau-network \
    -v $(pwd)/results:/results \
    -v $(pwd)/ipfs:/ipfs \
    ghcr.io/bacalhau-project/bacalhau:latest \
    get --api-host=bacalhau --ipfs-swarm-addrs=/dns4/bacalhau/tcp/4001/p2p/$SWARM_ID --ipfs-swarm-key=/ipfs/swarm.key --output-dir=/results $JOB_ID

Common Prerequisites

Create a New Docker Network

Without this, inter-container DNS will not work, and internet access may not work either.

docker network create --driver bridge bacalhau-network

Double check that this network can access the internet (so Bacalhau can call external URLs).

docker run --rm --network bacalhau-network alpine ping -c 2 bacalhau.org

Test that the IPFS Node is Working

You can now browse the IPFS web UI at http://127.0.0.1:5001/webui.

Test that the Bacalhau Node is Working

Ensure that the Bacalhau logs (docker logs bacalhau) have no errors.

Check that your Bacalhau installation is the same version:

bacalhau --api-host=localhost version

The versions should match. Alternatively, you can use the Docker container:

docker run --network=bacalhau-network --rm ghcr.io/bacalhau-project/bacalhau:latest --api-host=bacalhau version

Perform a list command to ensure you can connect to the Bacalhau API.

bacalhau --api-host=localhost list

It should return empty.

Authenticate with docker hub

If you are retrieving and running images from you may encounter issues with rate-limiting. Docker provides higher limits when authenticated, the size of the limit is based on the type of your account.

Should you wish to authenticate with Docker Hub when pulling images, you can do so by specifying credentials as environment variables wherever your compute node is running.

Environment variable

Description

Currently, this authentication is only available (and required) by the

Building and Running Custom Python Container

Introduction

In this tutorial example, we will walk you through building your own Python container and running the container on Bacalhau.

Prerequisites

To get started, you need to install the Bacalhau client, see more information here

1. Sample Recommendation Dataset

We will be using a simple recommendation script that, when given a movie ID, recommends other movies based on user ratings. Assuming you want recommendations for the movie 'Toy Story' (1995), it will suggest movies from similar categories:

Recommendations for Toy Story (1995):
1  :  Toy Story (1995)
58  :  Postino, Il (The Postman) (1994)
3159  :  Fantasia 2000 (1999)
359  :  I Like It Like That (1994)
756  :  Carmen Miranda: Bananas Is My Business (1994)
618  :  Two Much (1996)
48  :  Pocahontas (1995)
2695  :  Boys, The (1997)
2923  :  Citizen's Band (a.k.a. Handle with Care) (1977)
688  :  Operation Dumbo Drop (1995)

Downloading the dataset

Download Movielens1M dataset from this link https://files.grouplens.org/datasets/movielens/ml-1m.zip

wget https://files.grouplens.org/datasets/movielens/ml-1m.zip

In this example, we’ll be using 2 files from the MovieLens 1M dataset: ratings.dat and movies.dat. After the dataset is downloaded, extract the zip and place ratings.dat and movies.dat into a folder called input:

# Extracting the downloaded zip file
unzip ml-1m.zip

#moving  ratings.dat and movies.dat into a folder called 'input'
mkdir input; mv ml-1m/movies.dat ml-1m/ratings.dat input/

The structure of the input directory should be

input
├── movies.dat
└── ratings.dat

Installing Dependencies

To create a requirements.txt for the Python libraries we’ll be using, create:

# content of the requirements.txt
numpy
pandas

To install the dependencies, run:

pip install -r requirements.txt

Writing the Script

Create a new file called similar-movies.py and in it paste the following script

# content of the similar-movies.py

# Imports
import numpy as np
import pandas as pd
import argparse
from distutils.dir_util import mkpath
import warnings
warnings.filterwarnings("ignore")
# Read the files with pandas
data = pd.io.parsers.read_csv('input/ratings.dat',
names=['user_id', 'movie_id', 'rating', 'time'],
engine='python', delimiter='::', encoding='latin-1')
movie_data = pd.io.parsers.read_csv('input/movies.dat',
names=['movie_id', 'title', 'genre'],
engine='python', delimiter='::', encoding='latin-1')

# Create the ratings matrix of shape (m×u) with rows as movies and columns as users

ratings_mat = np.ndarray(
shape=((np.max(data.movie_id.values)), np.max(data.user_id.values)),
dtype=np.uint8)
ratings_mat[data.movie_id.values-1, data.user_id.values-1] = data.rating.values

# Normalise matrix (subtract mean off)

normalised_mat = ratings_mat - np.asarray([(np.mean(ratings_mat, 1))]).T

# Compute SVD

normalised_mat = ratings_mat - np.matrix(np.mean(ratings_mat, 1)).T
cov_mat = np.cov(normalised_mat)
evals, evecs = np.linalg.eig(cov_mat)

# Calculate cosine similarity, sort by most similar, and return the top N.

def top_cosine_similarity(data, movie_id, top_n=10):

index = movie_id - 1
# Movie id starts from 1

movie_row = data[index, :]
magnitude = np.sqrt(np.einsum('ij, ij -> i', data, data))
similarity = np.dot(movie_row, data.T) / (magnitude[index] * magnitude)
sort_indexes = np.argsort(-similarity)
return sort_indexes[:top_n]

# Helper function to print top N similar movies
def print_similar_movies(movie_data, movie_id, top_indexes):
print('Recommendations for {0}: \n'.format(
movie_data[movie_data.movie_id == movie_id].title.values[0]))
for id in top_indexes + 1:
print(str(id),' : ',movie_data[movie_data.movie_id == id].title.values[0])


parser = argparse.ArgumentParser(description='Personal information')
parser.add_argument('--k', dest='k', type=int, help='principal components to represent the movies',default=50)
parser.add_argument('--id', dest='id', type=int, help='Id of the movie',default=1)
parser.add_argument('--n', dest='n', type=int, help='No of recommendations',default=10)

args = parser.parse_args()
k = args.k
movie_id = args.id # Grab an id from movies.dat
top_n = args.n

# k = 50
# # Grab an id from movies.dat
# movie_id = 1
# top_n = 10

sliced = evecs[:, :k] # representative data
top_indexes = top_cosine_similarity(sliced, movie_id, top_n)
print_similar_movies(movie_data, movie_id, top_indexes)

What the similar-movies.py script does

Read the files with pandas. The code uses Pandas to read data from the files ratings.dat and movies.dat.
Create the ratings matrix of shape (m×u) with rows as movies and columns as user
Normalise matrix (subtract mean off). The ratings matrix is normalized by subtracting the mean off.
Compute SVD: a singular value decomposition (SVD) of the normalized ratings matrix is performed.
Calculate cosine similarity, sort by most similar, and return the top N.
Select k principal components to represent the movies, a movie_id to find recommendations, and print the top_n results.

For further reading on how the script works, go to Simple Movie Recommender Using SVD | Alyssa

Running the Script

Running the script similar-movies.py using the default values:

python similar-movies.py

You can also use other flags to set your own values.

2. Setting Up Docker

We will create a Dockerfile and add the desired configuration to the file. These commands specify how the image will be built, and what extra requirements will be included.

FROM python:3.8
ADD similar-movies.py .
ADD /input input
COPY ./requirements.txt /requirements.txt
RUN pip install -r requirements.txt

We will use the python:3.8 docker image and add our script similar-movies.py to copy the script to the docker image, similarly, we also add the dataset directory and also the requirements, after that run the command to install the dependencies in the image

The final folder structure will look like this:

├── Dockerfile
├── input
│   ├── movies.dat
│   └── ratings.dat
├── requirements.txt
└── similar-movies.py

See more information on how to containerize your script/app here

Build the container

We will run docker build command to build the container:

docker build -t <hub-user>/<repo-name>:<tag> .

Before running the command replace:

hub-user with your docker hub username, If you don’t have a docker hub account follow these instructions to create a docker account, and use the username of the account you created

repo-name with the name of the container, you can name it anything you want

tag this is not required, but you can use the latest tag

In our case:

docker build -t jsace/python-similar-movies .

Push the container

Next, upload the image to the registry. This can be done by using the Docker hub username, repo name or tag.

docker push <hub-user>/<repo-name>:<tag>

In our case:

docker push jsace/python-similar-movies

3. Running a Bacalhau Job

After the repo image has been pushed to Docker Hub, we can now use the container for running on Bacalhau. You can submit a Bacalhau job by running your container on Bacalhau with default or custom parameters.

Running the Container with Default Parameters

To submit a Bacalhau job by running your container on Bacalhau with default parameters, run the following Bacalhau command:

export JOB_ID=$(bacalhau docker run \
    --id-only \
    --wait \
    jsace/python-similar-movies \
    -- python similar-movies.py)

Structure of the command

bacalhau docker run: call to Bacalhau
jsace/python-similar-movies: the name and of the docker image we are using
-- python similar-movies.py: execute the Python script

When a job is submitted, Bacalhau prints out the related job_id. We store that in an environment variable so that we can reuse it later on.

Running the Container with Custom Parameters

To submit a Bacalhau job by running your container on Bacalhau with custom parameters, run the following Bacalhau command:

bacalhau docker run \
    jsace/python-similar-movies \
    -- python similar-movies.py --k 50 --id 10 --n 10

Structure of the command

bacalhau docker run: call to Bacalhau
jsace/python-similar-movies: the name of the docker image we are using
-- python similar-movies.py --k 50 --id 10 --n 10: execute the python script. The script will use Singular Value Decomposition (SVD) and cosine similarity to find 10 movies most similar to the one with identifier 10, using 50 principal components.

4. Checking the State of your Jobs

Job status: You can check the status of the job using bacalhau list.

bacalhau list --id-filter ${JOB_ID}

When it says Published or Completed, that means the job is done, and we can get the results.

Job information: You can find out more information about your job by using bacalhau describe.

bacalhau describe ${JOB_ID}

Job download: You can download your job results directly by using bacalhau get. Alternatively, you can choose to create a directory to store your results. In the command below, we created a directory (results) and downloaded our job output to be stored in that directory.

rm -rf results && mkdir -p results
bacalhau get $JOB_ID --output-dir results

5. Viewing your Job Output

To view the file, run the following command:

cat results/stdout # displays the contents of the file

Support

If you have questions or need support or guidance, please reach out to the Bacalhau team via Slack (#general channel).