In this tutorial you are setting up your own network
Bacalhau allows you to create your own private network so you can securely run private workloads without the risks inherent in working on public nodes or inadvertently distributing data outside your organization.
This tutorial describes the process of creating your own private network from multiple nodes, configuring the nodes, and running demo jobs.
Install Bacalhau curl -sL https://get.bacalhau.org/install.sh | bash on every host
Start the Requester node: bacalhau serve --node-type requester
Copy and paste the command it outputs under the "To connect a compute node to this orchestrator, run the following command in your shell" line to other hosts
Copy and paste the environment variables it outputs under the "To connect to this node from the client, run the following commands in your shell" line to a client machine
Done! Run sample hello-world command on the client machine bacalhau docker run apline echo hello
Prepare the hosts on which the nodes are going to be set up. They could be:
Physical Hosts
Local Hypervisor VMs
Install Bacalhau on each host
Ensure that all nodes are connected to the same network and that the necessary ports are open for communication between them.
Ensure your nodes have an internet connection in case you have to download or upload any data (docker images, input data, results)
Ensure that Docker Engine is installed in case you are going to run Docker Workloads
Bacalhau is designed to be versatile in its deployment, capable of running on various environments: physical hosts, virtual machines or cloud instances. Its resource requirements are modest, ensuring compatibility with a wide range of hardware configurations. However, for certain workloads, such as machine learning, it's advisable to consider hardware configurations optimized for computational tasks, including GPUs.
The Bacalhau network consists of nodes of two types: compute and requester. Compute Node is responsible for executing jobs and producing results. Requester Node is responsible for handling user requests, forwarding jobs to compute nodes and monitoring the job lifecycle.
The first step is to start up the initial Requester node. This node will connect to nothing but will listen for connections.
Start by creating a secure token. This token will be used for authentication between the orchestrator and compute nodes during their communications. Any string can be used as a token, preferably not easy to guess or bruteforce. In addition, new authentication methods will be introduced in future releases.
Let's use the uuidgen tool to create our token, then add it to the Bacalhau configuration and run the requester node:
This will produce output similar to this, indicating that the node is up and running:
Note that for security reasons, the output of the command contains the localhost 127.0.0.1 address instead of your real IP. To connect to this node, you should replace it with your real public IP address yourself. The method for obtaining your public IP address may vary depending on the type of instance you're using. Windows and Linux instances can be queried for their public IP using the following command:
If you are using a cloud deployment, you can find your public IP through their console, e.g. AWS and Google Cloud
Now let's move to another host from the preconditions, start a compute node on it and connect to the requester node. Here you will also need to add the same token to the configuration as on the requester.
Then execute the serve command to connect to the requester node:
This will produce output similar to this, indicating that the node is up and running:
To ensure that the nodes are connected to the network, run the following command, specifying the public IP of the requester node:
This will produce output similar to this, indicating that the nodes belong to the same network:
To connect to the requester node find the following lines in the requester node logs:
The exact commands list will be different for each node and is outputted by the bacalhau serve command.
Note that by default such command contains 127.0.0.1 or 0.0.0.0 instead of actual public IP. Make sure to replace it before executing the command.
Now you can submit your jobs using the bacalhau docker run, bacalhau wasm run and bacalhau job run commands. For example submit a hello-world job bacalhau docker run alpine echo hello:
You will be able to see the job execution logs on the compute node:
By default, IPFS & Local publishers and URL & IPFS sources are available on the compute node. The following describes how to configure the appropriate sources and publishers:
To set up S3 publisher you need to specify environment variables such as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, populating a credentials file to be located on your compute node, i.e. ~/.aws/credentials, or creating an IAM role for your compute nodes if you are utilizing cloud instances.
Your chosen publisher can be set for your Bacalhau compute nodes declaratively or imperatively using either configuration yaml file:
Or within your imperative job execution commands:
S3 compatible publishers can also be used as input sources for your jobs, with a similar configuration.
By default, bacalhau creates its own in-process IPFS node that will attempt to discover other IPFS nodes, including public nodes, on its own. If you specify the --private-internal-ipfs flag when starting the node, the node will not attempt to discover other nodes. Note, that such an IPFS node exists only with the compute node and will be shut down along with it. Alternatively, you can create your own private IPFS network and connect to it using the appropriate flags.
IPFS publisher can be set for your Bacalhau compute nodes declaratively or imperatively using either configuration yaml file:
Or within your imperative job execution commands:
Data pinned to the IPFS network can be used as input source. To do this, you will need to specify the CID in declarative:
Or imperative format:
Bacalhau allows to publish job results directly to the compute node. Please note that this method is not a reliable storage option and is recommended to be used mainly for introductory purposes.
Local publisher can be set for your Bacalhau compute nodes declaratively or imperatively using configuration yaml file:
Or within your imperative job execution commands:
The Local input source allows Bacalhau jobs to access files and directories that are already present on the compute node. To allow jobs to access local files when starting a node, the --allow-listed-local-paths flag should be used, specifying the path to the data and access mode :rw for Read-Write access or :ro for Read-Only (used by default). For example:
Further, the path to local data in declarative or imperative form must be specified in the job. Declarative example of the local input source:
Imperative example of the local input source:
Your private cluster can be quickly set up for testing packaged jobs and tweaking data processing pipelines. However, when using a private cluster in production, here are a few considerations to note.
Ensure you are running the Bacalhau process from a dedicated system user with limited permissions. This enhances security and reduces the risk of unauthorized access to critical system resources. If you are using an orchestrator such as Terraform, utilize a service file to manage the Bacalhau process, ensuring the correct user is specified and consistently used. Here’s a sample service file
Create an authentication file for your clients. A dedicated authentication file or policy can ease the process of maintaining secure data transmission within your network. With this, clients can authenticate themselves, and you can limit the Bacalhau API endpoints unauthorized users have access to.
Consistency is a key consideration when deploying decentralized tools such as Bacalhau. You can use an installation script to affix a specific version of Bacalhau or specify deployment actions, ensuring that each host instance has all the necessary resources for efficient operations.
Ensure separation of concerns in your cloud deployments by mounting the Bacalhau repository on a separate non-boot disk. This prevents instability on shutdown or restarts and improves performance within your host instances.
That's all folks! 🎉 Please contact us on Slack #bacalhau channel for questions and feedback!