Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Release Notes for Bacalhau v.1.4.0
Embedded libp2p/IPFS deprecation and migration to NATS.
CLI command updates.
Updates to job spec v2, while deprecating job spec v1.
Job queuing was extended with new job timeouts.
Improved error reporting.
Introduction of Node Manager.
We migrated to NATS already in v.1.3.0. (read more here) and will deprecate IPFS/libp2p in v.1.4.0. natively. If you want to migrate to NATS, please make sure to read these docs on the process.
In version 1.4.0 of Bacalhau, all legacy commands will be removed. Here’s a breakdown of the old commands and their new equivalents:
bacalhau create
bacalhau job run
bacalhau cancel
bacalhau job stop
bacalhau list
bacalhau job list
bacalhau logs
bacalhau job logs
bacalhau get
bacalhau job get
bacalhau describe
bacalhau job describe
bacalhau id
bacalhau agent node
bacalhau validate
bacalhau job validate
For some commands there are actions required to migrate to Bacalhau v.1.4.0. In your network. In the following view these actions are specified.
Special Attention to create , validate and describe Commands
create Command
create accepts a v1beta1 job spec.
job run accepts the current job spec.
Users must update their job specifications to align with the new job run requirements.
describe Command
describe returns a v1beta1 job spec and its corresponding state in YAML format.
job describe provides columnar data detailing various parts of the job.
Users should expect a different output format with job describe compared to describe.
validate Command
validate validates a v1beta1 job spec.
job validate validates the current job spec.
v1beta1 job specs will not be considered valid when passed to the job validate command.
If a user tries to use a legacy command, an error message will guide them to the correct new command. For example:
This error depends on the version you are running. There might also appear a failed request warning.
In 1.3.2, we released limited queuing functionality on Compute nodes that would allow a Job to be scheduled on a Compute node if it expected that it would be able to start the job in a reasonable time, and that there wasn’t another node better suited to running it elsewhere. Though a useful enhancement of Job delegation across the network, we feel this isn’t the most optimal path for determining which nodes can execute which Jobs at which time. To that end, we’re introducing a Queuing system in the Requester nodes of a Bacalhau network.
From 1.4.0, if a Job is submitted to a Bacalhau network, but no Compute node has the capacity to either execute, or prepare to execute the Job, the Requester node which received the Job will store it internally and either send it to a Compute node for processing, or until the Job timeout has elapsed. With this change, networks with heavy utilization should see a marked increase in the successful completion of Jobs. Fore more information about this go to this guide.
In the new version of Bacalhau the errors given to users were improved to give more granular feedback on what went wrong and to improve debugging. This makes errors more concise and faster to debug.
In Bacalhau 1.4.0, we’re introducing the Node Manager. This feature simplifies node operations, providing a clear view of all compute nodes and their status. You can approve, deny, or delete nodes as needed, making management straightforward. Heartbeats from nodes keep the Node Manager updated on their connectivity, enhancing overall stability and performance. For more information on this topic, check out the blog post about our release notes for a previous version (v.1.3.1).
Users who are not prepared for the changes in CLI behavior and job specification definitions are advised to remain on Bacalhau v1.3.1. This version continues to support the legacy commands and job specifications. Users can maintain their own private Bacalhau cluster using v1.3.1.
When users are ready to transition to the new CLI behavior and job specification requirements, they can upgrade to Bacalhau v1.4.
Major configuration management update
Major WebUI update: improved job and nodes monitoring and management
Improved error reporting
Improved job progress visibility
Cross-version compatibility: full support for v1.4.0
or via --config flag when executing a command:
The integrated web interface has been completely revamped to offer a more intuitive user experience:
Added dark interface theme
Added detailed view for jobs with real time log streaming mode
Added ability to stop a job via WebUI
Added detailed view for nodes
The error return logic has been redesigned in the new version:
Certain error messages have been redesigned and shortened
The color of the error text has been changed to a more prominent - red
Correct HTTP status codes now are used. For example, when requesting a non-existent job, the response is returned with the code 404 instead of 500, as in v1.4.0 and earlier
The dynamic output of job status to the console has been redesigned:
More informative format, indicating important job execution details that were previously not displayed
Improved progress visibility for jobs with multiple executions
Added --follow flag which allows tracking job logs right after job start
v.1.5.0 is fully compatible with v1.4.0 between any node types: Compute - Orchestrator and Client - Orchestrator, which provides seamless upgrade experience.
The approach to the Bacalhau configuration management has been significantly redesigned. The changes are described in more details . Another notable change is that the default endpoint is deprecated. So now in order to connect to the public demo network, the address bootstrap.production.bacalhau.org will need to be manually set in the api.host key:
Find Bacalhau on your favorite social media platform
Bacalhau has a presence on all major social media networks, so you can follow us on your favorite sites.
The Bacalhau and Expanso YouTube channels are home to a wealth of information about the Bacalhau project — everything from developer demos, educational videos, events, etc. You can explore and learn more about Bacalhau.
Our blog contains product updates, company news, and educational content on how you can leverage Bacalhau and the Compute over Data ecosystem Bacalhau Blog. Also, you can subscribe to our blog and receive the latest news straight to your inbox.
You can follow our X accounts @BacalhauProject and @ExpansoIO to get Bacalhau news, product updates, etc in tweet-sized bites.
Connect with us on LinkedIn to stay informed about the latest news, industry trends, and exclusive job opportunities!
If you have questions or need support or guidance, please reach out to the Bacalhau team via Slack (#bacalhau channel)
If you have any hints, tips or suggestions to add, check out the different ways to contribute to Bacalhau.
This guide explains things to keep in mind when writing for Bacalhau’s documentation.
The Bacalhau documentation aims to provide information on how to transform data processing for large-scale datasets to improve cost and efficiency. Our purpose is to open data processing to larger audiences, and our documentation set reflects this intention.
The primary objective of the Bacalhau documentation style guide is to help authors write clear and accurate information while facilitating consistency across all documentation. In this way, we can collectively create valuable learning materials that help developers learn and process data with Bacalhau.
Our goal is to use conversational tone that is natural, and friendly towards the reader and also ensure that the document's content is simple and easy to follow.
As we write for a large global audience, we aim for a universally accessible voice.
Maintain a friendly, informal tone, but focus on being clear and concise in a knowledgeable manner.
Avoid the use of slangs and colloquial language.
Avoid offensive language in all forms, and toward all identities and cultures.
Write in the second person (e.g. You can…), present tense to guide the reader to their intended outcome.
When writing consider that many users are not native English speakers.
Use languages that encourages readers and walk them through the steps to achieve the outcome they’re looking for.
The goal is for readers to leave with a feeling of accomplishment and satisfaction while gaining information efficiently and solving their problems in a way that helps them thrive with Bacalhau.
Our documentation should reflect our purpose, giving the appropriate amount of technical detail and clarity needed in a way that is palatable to all audiences. As best as we can, our documentation should reflect industry-standard practices and inspire trust in our voice and project. To do this,
Ensure that your word choice does not reflect an assumption of the reader’s knowledge level (e.g, using easy, simple, quick, etc) or exclude the detailed explanations or background needed to be successful.
Provide the commands needed to be successful with explanations of the ‘why’ behind the command is preferred when appropriate, with the goal of the reader both gaining their expected outcome and learning in the process.
To ensure clarity, start by briefly specifying the context of the current topic.
Share abbreviations and acronyms with at least one reference to its full name or title on the page to inspire deeper learning into Compute.
In non CLI-based references, capitalize Bacalhau, Docker, Compute and other program/project names.
Test each code snippet and example, walking through each step to ensure accuracy as it's written.
With these guidelines, we can provide a comprehensive set of documentation that is clear, actionable, and helpful.
The Bacalhau documentation set should include articles and tutorials that have a consistent structure, which includes an introduction and the procedural steps necessary for a reader to get to their expected outcome.
The specific structure depends on the type of documentation you are writing. On general note, the documentation should include: an introduction, a conclusion, and any prerequisites necessary for a reader to get started.
Most of the tutorials and examples we publish are procedural, which walk the reader through accomplishing a task step-by-step. The structure for a procedural documentation should be:
If the documentation is conceptual:
In this way, readers can learn and hop to pertinent information as they need efficiently, and find answers when they need.
Our documentation is written in Markdown markup language. The following rules explain how we organize and structure our writing.
All titles should follow title case capitalization structure.
Titles should be an H1 header. Introduction, prerequisites, steps, and conclusion should all have H2 headers.
All pages in the document should have a front-matter description which briefly summarizes the contents of a webpage.
Use Code Block formatting option for the code blocks. It should be used for:
Commands the reader needs to execute to complete the tutorial
Files and scripts
Terminal output
Interactive dialogues that are in text
Do not include the command prompt ($ or #) in the code block
Use a Code formatting option for inline code examples. It should be used for:
Command names, like bacalhau docker run
File names and paths, like /inputs
Example URLs, like http://your_domain
Command flag like --gpu
The hints include:
info: Use to add some supplementary information to a section or point that could benefit from some highlighting to draw the reader’s attention.
success: Use to add some guidance on how to carry out a step and show achievements.
warning: Use to make the reader aware they need to be careful when acting on some advice.
danger: Use to indicate that there are dangers or consequences associated with some information or steps.
Here’s an example of an Info hint:
Info hints are great for showing general information, or providing tips and tricks.
Bold text should be used for:
list item
Emphasis on words, project names etc
Italics should be used in inline list. For example (e.g, easy, simple, quick, etc)
If have questions or need support or guidance, please reach out to the Bacalhau team via Slack (#bacalhau channel)
If you have any hints, tips or suggestions to add, check out the different ways to contribute to Bacalhau.
We are excited to announce the release of Bacalhau v1.6.0, introducing a new communication architecture that significantly improves the reliability and resilience of distributed compute networks.
At the heart of this release is the new messaging protocol, a complete redesign of node communication that brings significant improvements to network reliability:
Key Benefits
Self-Healing Network: Compute nodes and orchestrators automatically reconnect and sync after network interruptions
Offline-First Operation: Compute nodes can start and operate even when disconnected from the orchestrator
Automatic State Recovery: When nodes reconnect, they automatically share all missed job execution information and results
Zero Data Loss: Ensures no job execution data or results are lost during network disruptions
Seamless Recovery: Network interruptions are handled transparently without requiring manual intervention
Technical Improvements
Reliable Message Delivery: Ordered, at-least-once message delivery between nodes
Automatic Recovery: Built-in failure detection and recovery mechanisms
Connection Health Monitoring: Proactive health checks and connection management
Event-Based Architecture: Decoupled event processing from message delivery
Efficient Checkpointing: Maintains system state for reliable recovery
Backward Compatibility: Maintains compatibility with v1.5 orchestrators
Direct Result Downloads: Download job results directly from the interface
Simplified Configuration: Automatic request routing eliminates manual IP configuration
Improved Architecture: Streamlined backend setup while maintaining security
Reverse Proxy Support: Added capability to run orchestrator behind a reverse proxy
Agent Configuration: New bacalhau agent config command to inspect agent configuration
TLS Support: Added TLS encryption support for NATS communication
Better Logging: Implemented more human-readable logging patterns
Bacalhau v1.6.0 maintains backward compatibility while introducing the new BMP:
Compute nodes maintain compatibility with v1.5 orchestrators, and vice versa
Support for re-handshake from legacy clients
We're excited for you to experience the enhanced reliability and resilience provided by the BMP in Bacalhau v1.6.0. This release represents a significant architectural advancement in making distributed computing more robust and dependable.
If you want to contribute to Bacalhau and the Compute ecosystem, here is a list of things that need help and instructions for getting started.
We appreciate any help that would improve Bacalhau. If you need more guidance, reach out to us on Slack (#bacalhau channel) .
You can contribute to the Bacalhau code by opening an issue with a bug report, requesting a feature, suggesting changes, and others. You can also clone the Bacalhau code repository and submit a pull request for an open issue.
With lots of code comes the need for lots of good documentation! If writing technical documentation is your area, or you love writing, we greatly value your contribution!
Before contributing to the Bacalhau documentation, please read through our Style guide and check out the Bacalhau documentation.
For questions, feedback, or answers to questions that help other users, use GitHub Discussions.
Bacalhau has an extensive list of examples and tutorials that showcase its abilities. However, this is just the tip of the iceberg; there are more use cases where Bacalhau can be implemented. If you have a specific idea you want to try out with Bacalhau, we encourage you to do that and build anything that you think is missing.
If you are interested in contributing, please email contribute@bacalhau.org for more information. Include your interests so we can ensure you get to work on something fun and valuable.
This guide explains what to keep in mind when contributing to Bacalhau documentation. While the grammar, formatting, and style guide outlines the rules to follow, this guide helps you structure your writing properly and choose the correct tone for your audience.
The purpose of a walkthrough is to instruct the user how to do something. You do not need to convince the reader of something or explain a concept. Walkthroughs are a list of steps the reader must follow to achieve a process or function.
Most of the Bacalhau documentation project documentation falls under the Walkthrough category. Walkthroughs are short, have a neutral tone, and instruct the reader on achieving a particular process or function. The instructions contain steps about where to go, what to type, and the actions to perform on the User Interface. There is little to no conceptual information within walkthroughs.
The scope for some of the components in a walkthrough are as follows:
Audience
General
Easy for anyone to read with minimal effort.
Formality
Neutral
Slang is restricted, but standard casual expressions are allowed.
Domain
Technical
Acronyms and tech-specific language are used and expected.
Tone
Neutral
Writing contains little to no emotion.
Intent
Instruct
Instruct the reader how to do something.
The end goal of a walkthrough is for the reader to achieve a very particular function. Installing Bacalhau is an example. Following this walkthrough doesn’t teach the reader much about working with the decentralized web or what Bacalhau is. Still, the user would have the Bacalhau Desktop application installed on the computer by the end.
Walkthroughs tend to be short because they cover one particular function or process. They typically take between two and ten minutes to read. Most of the content in a walkthrough is a numbered list. Images and GIFs can help the reader understand what they should be doing.
When a walkthrough is converted into a video, that video should be at most five minutes.
Walkthroughs are split into three major sections:
What you are about to do.
The steps you need to do.
Summary of what you just did and potential next steps.
Conceptual articles
Articles are written with the intent to inform and explain something. These articles don’t contain any steps or actions that the reader has to perform right now.
These articles are vastly different in tone when compared to walkthroughs. Some topics and concepts can be challenging to understand, so creative writing and interesting diagrams are highly sought-after for these articles. Whatever writers can do to make a subject more understandable, the better.
Article goals
Use the following goals when writing conceptual articles:
Audience
Knowledgeable
Requires a certain amount of focus to understand.
Formality
Neutral
Slang is restricted, but standard casual expressions are allowed.
Domain
Any
Usually technical but depends on the article.
Tone
Confident and friendly
The reader must feel confident that the writer knows what they’re talking about.
Intent
Describe
Tell the reader why something does the thing that it does, or why it exists.
Article structure
Articles are separated into five major sections:
Introduction to the thing we’re about to explain.
What the thing is.
Why it’s essential.
What other topics it relates to.
Summary review of what we just read.
Tutorials
When writing a tutorial, you’re teaching a reader how to achieve a complex end-goal. Tutorials are a mix of walkthroughs and conceptual articles. Most tutorials will span several pages and contain multiple walkthroughs within them.
Take the hypothetical tutorial Get up and running with Bacalhau, for example. This tutorial will likely have the following pages:
A brief introduction to what Bacalhau is.
Choose and install a command line client.
Understanding storage deals.
Import and store a file.
Pages 1 and 3 are conceptual articles, describing particular design patterns and ideas to the reader. All the other pages are walkthroughs instructing the user how to perform one specific action.
When designing a tutorial, keep in mind the walkthroughs and articles that already exist, and note down any additional content items that would need to be completed before creating the tutorial.
Here are some language-specific rules that the Bacalhau documentation follows. If you use a writing service like Grammarly, most of these rules are turned on by default.
American English
While Bacalhau is a global project, the fact is that American English is the most commonly used style of English used today. With that in mind, when writing content for the Bacalhau project, use American English spelling. The basic rules for converting other styles of English into American English are:
Swap the s for a z in words like categorize and pluralize.
Remove the u from words like color and honor.
Swap tre for ter in words like center.
The Oxford comma
In a list of three or more items, follow each item except the last with a comma ,:
One, two, three, and four.
One, two, three and four.
Henry, Elizabeth, and George.
Henry, Elizabeth and George.
References to Bacalhau
As a proper noun, the name “Bacalhau” (capitalized) should be used only to refer to the overarching project, to the protocol, or to the project’s canonical network:
Bacalhau has attracted contributors from around the globe!
The Bacalhau ecosystem is thriving! I love contributing to Bacalhau's documentation!
Consistency in the usage of these terms helps keep these various concepts distinct.
Acronyms
If you have to use an acronym, spell the full phrase first and include the acronym in parentheses () the first time it is used in each document.
Virtual Machine (VM), Compute over Data (CoD).
How the Markdown syntax looks, and code formatting rules to follow.
Syntax
The Bacalhau Docs project follows the GitHub Flavoured Markdown syntax for markdown. This way, all articles display properly within GitHub itself.
Rules
We use the rules set out in the VSCode Markdownlint extension. You can import these rules into any text editor like Vim or Sublime. All rules are listed within the Markdownlint repository.
We highly recommend installing VSCode with the Markdownlint extension to help with your writing. The extension shows warnings within your markdown whenever your copy doesn’t conform to a rule.