Bacalhau Docs
GithubSlackBlogEnterprise
v1.6.x
  • Documentation
  • Use Cases
  • CLI & API
  • References
  • Community
v1.6.x
  • Welcome
  • Getting Started
    • How Bacalhau Works
    • Getting Started
      • Step 1: Install the Bacalhau CLI
      • Step 2: Running Your Own Job
      • Step 3: Checking on the Status of Your Job
    • Creating Your Own Bacalhau Network
      • Setting Up a Cluster on Amazon Web Services (AWS) with Terraform πŸš€
      • Setting Up a Cluster on Google Cloud Platform (GCP) With Terraform πŸš€
      • Setting Up a Cluster on Azure with Terraform πŸš€
    • Hardware Setup
    • Container Onboarding
      • Docker Workloads
      • WebAssembly (Wasm) Workloads
  • Setting Up
    • Running Nodes
      • Node Onboarding
      • GPU Installation
      • Job selection policy
      • Access Management
      • Node persistence
      • Configuring Your Input Sources
      • Configuring Transport Level Security
      • Limits and Timeouts
      • Test Network Locally
      • Bacalhau WebUI
      • Private IPFS Network Setup
    • Workload Onboarding
      • Container
        • Docker Workload Onboarding
        • WebAssembly (Wasm) Workloads
        • Bacalhau Docker Image
        • How To Work With Custom Containers in Bacalhau
      • Python
        • Building and Running Custom Python Container
        • Running Pandas on Bacalhau
        • Running a Python Script
        • Running Jupyter Notebooks on Bacalhau
        • Scripting Bacalhau with Python
      • R (language)
        • Building and Running your Custom R Containers on Bacalhau
        • Running a Simple R Script on Bacalhau
      • Run CUDA programs on Bacalhau
      • Running a Prolog Script
      • Reading Data from Multiple S3 Buckets using Bacalhau
      • Running Rust programs as WebAssembly (WASM)
      • Generate Synthetic Data using Sparkov Data Generation technique
    • Networking Instructions
      • Accessing the Internet from Jobs
      • Utilizing NATS.io within Bacalhau
    • GPU Workloads Setup
    • Automatic Update Checking
    • Marketplace Deployments
      • Google Cloud Marketplace
    • Inter-Nodes TLS
  • Guides
    • Configuration Management
    • Write a config.yaml
    • Write a SpecConfig
    • Using Labels and Constraints
  • Examples
    • Table of Contents for Bacalhau Examples
    • Data Engineering
      • Using Bacalhau with DuckDB
      • Ethereum Blockchain Analysis with Ethereum-ETL and Bacalhau
      • Convert CSV To Parquet Or Avro
      • Simple Image Processing
      • Oceanography - Data Conversion
      • Video Processing
      • Bacalhau and BigQuery
    • Data Ingestion
      • Copy Data from URL to Public Storage
      • Pinning Data
      • Running a Job over S3 data
    • Model Inference
      • EasyOCR (Optical Character Recognition) on Bacalhau
      • Running Inference on Dolly 2.0 Model with Hugging Face
      • Speech Recognition using Whisper
      • Stable Diffusion on a GPU
      • Stable Diffusion on a CPU
      • Object Detection with YOLOv5 on Bacalhau
      • Generate Realistic Images using StyleGAN3 and Bacalhau
      • Stable Diffusion Checkpoint Inference
      • Running Inference on a Model stored on S3
    • Model Training
      • Training Pytorch Model with Bacalhau
      • Training Tensorflow Model
      • Stable Diffusion Dreambooth (Finetuning)
    • Molecular Dynamics
      • Running BIDS Apps on Bacalhau
      • Coresets On Bacalhau
      • Genomics Data Generation
      • Gromacs for Analysis
      • Molecular Simulation with OpenMM and Bacalhau
    • Systems Engineering
      • Ad-hoc log query using DuckDB
  • References
    • Jobs Guide
      • Job Specification
        • Job Types
        • Task Specification
          • Engines
            • Docker Engine Specification
            • WebAssembly (WASM) Engine Specification
          • Publishers
            • IPFS Publisher Specification
            • Local Publisher Specification
            • S3 Publisher Specification
          • Sources
            • IPFS Source Specification
            • Local Source Specification
            • S3 Source Specification
            • URL Source Specification
          • Network Specification
          • Input Source Specification
          • Resources Specification
          • ResultPath Specification
        • Constraint Specification
        • Labels Specification
        • Meta Specification
      • Job Templates
      • Queuing & Timeouts
        • Job Queuing
        • Timeouts Specification
      • Job Results
        • State
    • CLI Guide
      • Single CLI commands
        • Agent
          • Agent Overview
          • Agent Alive
          • Agent Node
          • Agent Version
        • Config
          • Config Overview
          • Config Auto-Resources
          • Config Default
          • Config List
          • Config Set
        • Job
          • Job Overview
          • Job Describe
          • Job Executions
          • Job History
          • Job List
          • Job Logs
          • Job Run
          • Job Stop
        • Node
          • Node Overview
          • Node Approve
          • Node Delete
          • Node List
          • Node Describe
          • Node Reject
      • Command Migration
    • API Guide
      • Bacalhau API overview
      • Best Practices
      • Agent Endpoint
      • Orchestrator Endpoint
      • Migration API
    • Node Management
    • Authentication & Authorization
    • Database Integration
    • Debugging
      • Debugging Failed Jobs
      • Debugging Locally
    • Running Locally In Devstack
    • Setting up Dev Environment
  • Help & FAQ
    • Bacalhau FAQs
    • Glossary
    • Release Notes
      • v1.5.0 Release Notes
      • v1.4.0 Release Notes
  • Integrations
    • Apache Airflow Provider for Bacalhau
    • Lilypad
    • Bacalhau Python SDK
    • Observability for WebAssembly Workloads
  • Community
    • Social Media
    • Style Guide
    • Ways to Contribute
Powered by GitBook
LogoLogo

Use Cases

  • Distributed ETL
  • Edge ML
  • Distributed Data Warehousing
  • Fleet Management

About Us

  • Who we are
  • What we value

News & Blog

  • Blog

Get Support

  • Request Enterprise Solutions

Expanso (2025). All Rights Reserved.

On this page
  • Introduction
  • TLDR
  • Prerequisites
  • Start Initial Orchestrator Node
  • Create and Set Up a Token
  • Create and Connect Compute Node
  • Submitting Jobs
  • Publishers and Sources Configuration
  • Best Practices for Production Use Cases

Was this helpful?

Export as PDF
  1. Getting Started

Creating Your Own Bacalhau Network

PreviousStep 3: Checking on the Status of Your JobNextSetting Up a Cluster on Amazon Web Services (AWS) with Terraform πŸš€

Was this helpful?

Introduction

While we () offer a public network for testing out Bacalhau workloads, your best bet is to create your own private network so you can securely run private workloads.

If you are familiar with running clusters with other clustered orchestration systems (Kubernetes, Mesosphere, DataBricks/Spark, Cloudera/Hadoop, Nomad, etc), we think you will be pleasantly surprised how easy it is to set up your first Bacalhau cluster!

This tutorial describes the process of creating your own private network from multiple nodes, configuring the nodes and running demo jobs.​

TLDR

  1. for setting up a Bacalhau cluster. Hang onto this URL, you'll need it for all the nodes you set up (both orchestrator and compute nodes).

  2. Start the :

bacalhau serve --orchestrator 
  1. Install Bacalhau on each node you want to run your compute on (use the same URL)

  2. Run the server on the

bacalhau serve --compue --api-host <ORCHESTRATOR_IP>
  1. Done! You can run an example, like:

bacalhau docker run apline echo hello --api-host <ORCHESTRATOR_IP>

Prerequisites

  1. Prepare the hosts on which the nodes are going to be set up. They could be:

    1. Physical Hosts

    2. Cloud VMs (, , or any other provider)

    3. Local Hypervisor VMs

    4. An executor (Bash, , , etc)

  2. on each host

  3. Ensure that all nodes are connected to the same network and that the necessary ports are open for communication between them. This will require bidirectional communication on port 4222 between all nodes, and port 1234 to the orchestrator node.

Ensure your nodes have an internet connection in case you have to download or upload any data (docker images, input data, results). This is not required but most jobs require some form of network connectivity (even just for downloading containers).

Start Initial Orchestrator Node

The Bacalhau network consists of nodes of two types: compute and orchestrator. Compute Node is responsible for executing jobs and producing results. Orchestrator Node is responsible for handling user requests, forwarding jobs to compute nodes and monitoring the job lifecycle.

The first step is to start up the initial Orchestrator node. This node will connect to nothing but will listen for connections.

Create and Set Up a Token

When you set up a network for the first time, any compute node can join. This can be dangerous - malicious nodes could join your network and "see" jobs being executed, bid on jobs to try to capture them, etc. We don't recommend this!

We recommend adding a Compute Token to the orchestrator. This will mean only nodes that join the network with the token will be allowed to join.

While it is not REQUIRED to run with a compute token to join, we highly recommend it!

Let's use the uuidgen tool to create our token, then add it to the Bacalhau configuration and run the orchestrator node:

# Create token and write it into the 'my_token' file
$ uuidgen
2EE91AD9-89B5-46CC-86B6-B0E76A3F763F

# On the orchestrator machine, add token to the Bacalhau configuration
bacalhau config set orchestrator.auth.token="2EE91AD9-89B5-46CC-86B6-B0E76A3F763F"

Now start (or restart) your orchestrator node like usual.

#Start the Orchestrator node
bacalhau serve --orchestrator

This will produce output similar to this, indicating that the node is up and running:

17:27:42.273 | INF cmd/cli/serve/serve.go:102 > Config loaded from: [/home/username/.bacalhau/config.yaml], and with data-dir /home/username/.bacalhau
17:27:42.322 | INF cmd/cli/serve/serve.go:228 > Starting bacalhau...
17:27:42.405 | WRN pkg/nats/logger.go:49 > Filestore [KV_node_v1] Stream state too short (0 bytes) [Server:n-0f29f45c-c894-4f8f-8a0a-8f2f1f64d96d]
17:27:42.479 | INF cmd/cli/serve/serve.go:300 > bacalhau node running [address:0.0.0.0:1234] [compute_enabled:false] [name:n-0f29f45c-c894-4f8f-8a0a-8f2f1f64d96d] [orchestrator_address:0.0.0.0:4222] [orchestrator_enabled:true] [webui_enabled:true]

Your orchestrator now requires a token to join. Awesome!

Create and Connect Compute Node

Now let's start a compute node on it and connect to the orchestrator node. You'll use the same token

#Add token to the Bacalhau configuration
bacalhau config set compute.auth.token="2EE91AD9-89B5-46CC-86B6-B0E76A3F763F"

Then execute the serve command to connect to the orchestrator node:you created earlier, but add it to a different configuration setting.

bacalhau serve --сompute -c API.Host=<IP-of-Orchestrator> 

This will produce output similar to this, indicating that the node is up and running:

# formatting has been adjusted for better readability
16:23:33.386 | INF cmd/cli/serve/serve.go:256 > bacalhau node running 
[address:0.0.0.0:1235] 
[capacity:"{CPU: 1.40, Memory: 2.9 GB, Disk: 13 GB, GPU: 0}"]
[compute_enabled:true] [engines:["docker","wasm"]]
[name:n-7a510a5b-86de-41db-846f-8c6a24b67482] [orchestrator_enabled:false]
[orchestrators:["127.0.0.1","0.0.0.0"]] [publishers:["local","noop"]]
[storages:["urldownload","inline"]] [webui_enabled:false]

To ensure that the nodes are connected to the network, run the following command, specifying the public IP of the orchestrator node:

bacalhau -c API.Host=<IP-of-Orchestrator> node list

This will produce output similar to this, indicating that the nodes belong to the same network:

 ID          TYPE       STATUS    LABELS                                              CPU     MEMORY      DISK         GPU  
 n-7a510a5b  Compute              Architecture=amd64 Operating-System=linux           0.8 /   1.5 GB /    12.3 GB /    0 /      
 n-b2ab8483  Requester  APPROVED  Architecture=amd64 Operating-System=linux           0.8     1.5 GB      12.3 GB      0

You have your first network up and running!

Submitting Jobs

You can submit your jobs using the bacalhau docker run, bacalhau wasm run and bacalhau job run commands. For example submit a hello-world job:

bacalhau docker run alpine echo hello -c API.Host=<IP-of-Orchestrator> 
Job successfully submitted. Job ID: j-5be2a5b2-567e-4f57-ac9e-8816e47ebeff
Checking job status... (Enter Ctrl+C to exit at any time, your job will continue running):

 TIME          EXEC. ID    TOPIC            EVENT         
 16:34:16.467              Submission       Job submitted 
 16:34:16.484  e-1e9dca31  Scheduling       Requested execution on n-d41eeae7 
 16:34:16.550  e-1e9dca31  Execution        Running 
 16:34:17.506  e-1e9dca31  Execution        Completed successfully 
                                             
To get more details about the run, execute:
	bacalhau job describe j-5be2a5b2-567e-4f57-ac9e-8816e47ebeff

To get more details about the run executions, execute:
	bacalhau job executions j-5be2a5b2-567e-4f57-ac9e-8816e47ebeff

If you would like to avoid adding the API.Host you can configure your client's default by either setting an environment variable:

export BACALHAU_API_HOST=<IP-address-of-orchestrator>

Or adding it to your config:

bacalhau config set API.Host=<IP-address-of-orchestrator>

Publishers and Sources Configuration

By default only local publisher and URL & local sources are available on the compute node. Out of the box Bacalhau also supports:

The following describes how to configure the appropriate sources and publishers:

Your chosen publisher can be set for your Bacalhau compute nodes declaratively or imperatively using either configuration yaml file:

Publisher:
  Type: "s3"
  Params:
    Bucket: "my-task-results"
    Key: "task123/result.tar.gz"
    Endpoint: "https://s3.us-west-2.amazonaws.com"

Or within your imperative job execution commands:

bacalhau docker run -p s3://bucket/key,opt=endpoint=http://s3.example.com,opt=region=us-east-1 ubuntu …
InputSources:
  - Source:
      Type: "s3"
      Params:
        Bucket: "my-bucket"
        Key: "data/"
        Endpoint: "https://storage.googleapis.com"
  - Target: "/data"
Publisher:
  Type: ipfs

Or within your imperative job execution commands:

bacalhau docker run --publisher ipfs ubuntu ...
InputSources:
  - Source:
      Type: "ipfs"
      Params:
        CID: "QmY7Yh4UquoXHLPFo2XbhXkhBvFoPwmQUSa92pxnxjY3fZ"
  - Target: "/data"

Or imperative format:

bacalhau docker run --input QmY7Yh4UquoXHLPFo2XbhXkhBvFoPwmQUSa92pxnxjY3fZ:/data ...

Bacalhau allows to publish job results directly to the compute node. Please note that this method is not a reliable storage option and is recommended to be used mainly for introductory purposes.

Publisher:
  Type: local

Or within your imperative job execution commands:

bacalhau docker run --publisher local ubuntu ...
bacalhau config set Compute.AllowListedLocalPaths=/etc/config:rw,/etc/*.conf:ro

Further, the path to local data in declarative or imperative form must be specified in the job. Declarative example of the local input source:

InputSources:
  - Source:
      Type: "localDirectory"
      Params:
        SourcePath: "/etc/config"
        ReadWrite: true
    Target: "/config"

Imperative example of the local input source:

bacalhau docker run --input file:///etc/config:/config ubuntu ...

Best Practices for Production Use Cases

When using a private cluster in production, here are a few considerations to note.

  1. Ensure you are running the Bacalhau agents with limited permissions. This enhances security and reduces the risk of unauthorized access to critical system resources.

  2. Ensure separation of concerns in your cloud deployments by mounting the Bacalhau repository on a non-boot disk. This prevents instability on shutdown or restarts and improves performance within your host instances.

If you want to run workloads, ensure that is installed.​

Though S3 is typically associated with Amazon Web Services, the S3 publisher is compatible with any S3 compatible blob store service. This includes , , , , and many others.

To set up you need to specify environment variables such as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, populating a credentials file to be located on your compute node, i.e. ~/.aws/credentials, or creating an for your compute nodes if you are utilizing cloud instances.

S3 compatible publishers can also be used as for your jobs, with a similar configuration.

By default, bacalhau does not connect or create its own IPFS network. Consider creating your network and connect to it using the .

can be set for your Bacalhau compute nodes declaratively or imperatively using either configuration yaml file:

Data pinned to the IPFS network can be used as . To do this, you will need to specify the CID in declarative:

can be set for your Bacalhau compute nodes declaratively or imperatively using configuration yaml file:

The allows Bacalhau jobs to access files and directories that are already present on the compute node. To allow jobs to access local files when starting a node, the Compute.AllowListedLocalPaths configuration key should be used, specifying the path to the data and access mode :rw for Read-Write access or :ro for Read-Only (used by default). For example:

Utilize a service file to manage the Bacalhau process, ensuring the correct user is specified and consistently used. Here’s a

Create an authentication file for your clients. A can ease the process of maintaining secure data transmission within your network. With this, clients can authenticate themselves, and you can limit the Bacalhau API endpoints unauthorized users have access to.

For many other common questions, we recommend checking out the .

That's all folks! πŸŽ‰ Please contact us on #bacalhau channel for questions and feedback!

Docker
Docker Engine
AWS S3
GCP storage
Azure Blob
Oracle Cloud
S3 publisher
IAM role
input sources
IPFS publisher
input source
Local publisher
Local input source
sample service file
dedicated authentication file or policy
Bacalhau FAQ
Slack
Expanso
Download the CLI
AWS
GCP
Azure
Docker Containers
WASM
Install Bacalhau
Orchestrator node
Compute node
own private IPFS
appropriate flags