Release Notes for Bacalhau v.1.4.0
Embedded libp2p/IPFS deprecation and migration to NATS.
CLI command updates.
Updates to job spec v2, while deprecating job spec v1.
Job queuing was extended with new job timeouts.
Improved error reporting.
Introduction of Node Manager.
We migrated to NATS already in v.1.3.0. (read more here) and will deprecate IPFS/libp2p in v.1.4.0. natively. If you want to migrate to NATS, please make sure to read these docs on the process.
In version 1.4.0 of Bacalhau, all legacy commands will be removed. Here’s a breakdown of the old commands and their new equivalents:
For some commands there are actions required to migrate to Bacalhau v.1.4.0. In your network. In the following view these actions are specified.
Special Attention to create , validate and describe Commands
If a user tries to use a legacy command, an error message will guide them to the correct new command. For example:
This error depends on the version you are running. There might also appear a failed request warning.
In 1.3.2, we released limited queuing functionality on Compute nodes that would allow a Job to be scheduled on a Compute node if it expected that it would be able to start the job in a reasonable time, and that there wasn’t another node better suited to running it elsewhere. Though a useful enhancement of Job delegation across the network, we feel this isn’t the most optimal path for determining which nodes can execute which Jobs at which time. To that end, we’re introducing a Queuing system in the Requester nodes of a Bacalhau network.
From 1.4.0, if a Job is submitted to a Bacalhau network, but no Compute node has the capacity to either execute, or prepare to execute the Job, the Requester node which received the Job will store it internally and either send it to a Compute node for processing, or until the Job timeout has elapsed. With this change, networks with heavy utilization should see a marked increase in the successful completion of Jobs. Fore more information about this go to this guide.
In the new version of Bacalhau the errors given to users were improved to give more granular feedback on what went wrong and to improve debugging. This makes errors more concise and faster to debug.
In Bacalhau 1.4.0, we’re introducing the Node Manager. This feature simplifies node operations, providing a clear view of all compute nodes and their status. You can approve, deny, or delete nodes as needed, making management straightforward. Heartbeats from nodes keep the Node Manager updated on their connectivity, enhancing overall stability and performance. For more information on this topic, check out the blog post about our release notes for a previous version (v.1.3.1).
Users who are not prepared for the changes in CLI behavior and job specification definitions are advised to remain on Bacalhau v1.3.1. This version continues to support the legacy commands and job specifications. Users can maintain their own private Bacalhau cluster using v1.3.1.
When users are ready to transition to the new CLI behavior and job specification requirements, they can upgrade to Bacalhau v1.4.
Old Commands | New Commands |
---|---|
Old Command | New Command | Action Required | |
---|---|---|---|
bacalhau create
bacalhau job run
bacalhau cancel
bacalhau job stop
bacalhau list
bacalhau job list
bacalhau logs
bacalhau job logs
bacalhau get
bacalhau job get
bacalhau describe
bacalhau job describe
bacalhau id
bacalhau agent node
bacalhau validate
bacalhau job validate
create Command
create accepts a v1beta1 job spec.
job run accepts the current job spec.
Users must update their job specifications to align with the new job run requirements.
describe Command
describe returns a v1beta1 job spec and its corresponding state in YAML format.
job describe provides columnar data detailing various parts of the job.
Users should expect a different output format with job describe compared to describe.
validate Command
validate validates a v1beta1 job spec.
job validate validates the current job spec.
v1beta1 job specs will not be considered valid when passed to the job validate command.
Major configuration management update
Major WebUI update: improved job and nodes monitoring and management
Improved error reporting
Improved job progress visibility
Cross-version compatibility: full support for v1.4.0
The approach to the Bacalhau configuration management has been significantly redesigned. The changes are described in more details here. Another notable change is that the default endpoint is deprecated. So now in order to connect to the public demo network, the address bootstrap.production.bacalhau.org
will need to be manually set in the api.host
key:
or via --config
flag when executing a command:
The integrated web interface has been completely revamped to offer a more intuitive user experience:
Added dark interface theme
Added detailed view for jobs with real time log streaming mode
Added ability to stop a job via WebUI
Added detailed view for nodes
The error return logic has been redesigned in the new version:
Certain error messages have been redesigned and shortened
The color of the error text has been changed to a more prominent - red
Correct HTTP status codes now are used. For example, when requesting a non-existent job, the response is returned with the code 404
instead of 500
, as in v1.4.0
and earlier
The dynamic output of job status to the console has been redesigned:
More informative format, indicating important job execution details that were previously not displayed
Improved progress visibility for jobs with multiple executions
Added --follow
flag which allows tracking job logs right after job start
v.1.5.0
is fully compatible with v1.4.0
between any node types: Compute - Orchestrator and Client - Orchestrator, which provides seamless upgrade experience.
There have been some changes made to how bacalhau handles configuration:
The bacalhau repo ~/.bacalhau
no longer contains a config file.
Bacalhau no longer looks in the repo ~/.bacalhau
for a config file.
Bacalhau never writes a config file to disk unless instructed to by a user.
A config file is not required to operate bacalhau.
Bacalhau searches for a default config file. The location is OS dependent:
Linux: ~/.config/bacalhau/config.yaml
OSX: ~/.config/Application\ Support/bacalhau/config.yaml
Windows: $AppData\bacalhau\config.yaml
. Usually this is something like C:\Users\username\bacalhau\config.yaml
Bacalhau no longer relies on the ~/.bacalhau
directory for configuration and only creates a config file when instructed. While not required, it will look for a default config file in OS-specific locations.
To view the configuration that bacalhau will receive when a command is executed against it, users can run the bacalhau config list command. Users who wish to see bacalhau’s config represented as YAML may run bacalhau config list --output=yaml
.
As described above, bacalhau still has the concept of a default config file, which for the sake of simplify we’ll say lives in ~/.config/bacalhau/config.yaml
. There are two ways this file can be modified:
A text editor vim ~/.config/bacalhau/config.yaml
.
The bacalhau config set command.
The --config
(or -c
) flag allows flexible configuration of bacalhau through various methods. You can use this flag multiple times to combine different configuration sources. To specify a config file to bacalhau, users may use the --config
flag, passing a path to a config file for bacalhau to use. When this flag is provided bacalhau will not search for a default config, and will instead use the configuration provided to it by the --config
flag.
In Bacalhau, configuration keys are structured identifiers used to configure and customize the behavior of the application. They represent specific settings that control various aspects of Bacalhau's functionality, such as network parameters, API endpoints, node operations, and user interface options. The configuration file is organized in a tree-like structure using nested mappings (dictionaries) in YAML format. Each level of indentation represents a deeper level in the hierarchy.
Example: part of the config file
In this YAML configuration file:
Top-Level Keys (Categories): API
, Orchestrator
Sub-Level Keys (Subcategories): Under API
, we have Host
and Port
; under Orchestrator
we have Host
, Port
and NodeManager
Leaf Nodes (Settings): Host
, Port
, NameProvider
, DataDir
, DisconnectTimeout
— these contain the actual configuration values.
Config keys use dot notation to represent the path from the root of the configuration hierarchy down to a specific leaf node. Each segment in the key corresponds to a level in the hierarchy. Syntax is Category.Subcategory(s)...LeafNode
config set
, config list
and --config
The bacalhau config list
returns all keys and their corresponding value. The bacalhau config set
command accepts a key and a value to set it to. The --config
flag accepts a key and a value that will be applied to bacalhau when it runs.
API Host
Using bacalhau config set
in the Default Config File:Run bacalhau config list
to find the appropriate key
Run the bacalhau config set
command
Observe how bacalhau config list
reflects the new setting
Observe the change has been reflected in the default config file
How to Modify the API Host Using bacalhau config set
in a Custom Config File
Run the config set command with a flag
Observe the created config file
Observe the default config and output of bacalhau config list
does not reflect this change.
How to Start Bacalhau With a Custom Config File
--config
Flag The --config
(or -c
) flag allows flexible configuration of bacalhau through various methods. You can use this flag multiple times to combine different configuration sources.
or using the short form:
YAML Config Files: Specify paths to YAML configuration files. Example:
Key-Value Pairs: Set specific configuration values using dot notation. Example:
Boolean Flags: Enable boolean options by specifying the key alone. Example:
When multiple configuration options are provided, they are applied in the following order of precedence (highest to lowest):
Command-line key-value pairs and boolean flags
YAML configuration files
Default values
Within each category, options specified later override earlier ones.
Using a single config file:
Merging multiple config files:
Overriding specific values:
Combining file and multiple overrides:
In the last example, WebUI.Enabled
will be set to true
, API.Host
will be 192.168.1.5
, and other values will be loaded from config.yaml
if present.
Remember, later options override earlier ones, allowing for flexible configuration management.
bacalhau completion
CommandThe bacalhau completion
command will generate shell completion for your shell. You can use the command like:
After running the above command, commands like bacalhau config set
and bacalhau --config
will have autocompletion for all possible configuration values along with their descriptions
If you have questions or need support or guidance, please reach out to the Bacalhau team via Slack (#general channel).