Skip to main content

Pinning Data

If you have data that you want to make available to your Bacalhau jobs (or other people), you can pin it using a pinning service like Web3.Storage, Estuary, etc. Pinning services store data on behalf of users. The pinning provider is essentially guaranteeing that your data will be available if someone knows the CID. Most pinning services offer you a free tier, so you can try them out without spending any money.

Web3.Storage​

This example will demonstrate how to pin data using Web3.Storage. Web3.Storage is a pinning service that is built on top of IPFS and Filecoin. It is free to use for small amounts of data and has a generous free tier. You can find more information about Web3.Storage here.

  • First you need to create an account (if you don't have one already).
  • Next, sign in and browse to the Create API Key page. Follow the instructions to create an API key. Once created, you will need to copy the API key to your clipboard.

Ways to pin using web3.storage​

  1. Pin a local file using their test client: To test that your API key is working, use web3.storage's test client.

You can now see (or upload) your file via the web3.storage account page.

warning

Note that you shouldn't share your API key with anyone. Delete this API key once you have finished with this example.

  1. Pin a local file via curl: You can also pin a file via curl. Please view the API documentation to see all available commands. This example submits a single file to be pinned.
export TOKEN=YOUR_API_KEY
echo hello world > foo.txt
curl -X POST https://api.web3.storage/upload -H "Authorization: Bearer ${TOKEN}" -H "X-NAME: foo.txt" -d @foo.txt
  1. Pin multiple local files via Node.JS: Web3.Storage has a node.js library to interact with their API. The following example requires node.js to be installed. The following code uses a docker container. The javascript code is located on their website or on github.

First, create some files to upload.

%%writefile nodejs/test1.txt
First test file

Then run the following command, which uses the environmental variable TOKEN to authenticate with the API.

export TOKEN=YOUR_API_KEY
docker run --rm --env TOKEN=$TOKEN -v $PWD/nodejs:/nodejs node:18-alpine ash -c 'cd /nodejs && npm install && node put-files.js --token=$TOKEN test1.txt test2.txt'

The response will return the CID of the file, which can now be used as an input to Bacalhau.

  1. Pin a file from a URL via Curl: You can use curl to download a file and then re-upload it to web3.storage. For example:
export TOKEN=YOUR_API_KEY
curl -o train-images-idx3-ubyte.gz http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
curl -X POST https://api.web3.storage/upload -H "Authorization: Bearer ${TOKEN}" -H "X-NAME: train-images-idx3-ubyte.gz" -d @train-images-idx3-ubyte.gz
  1. Pin a file from a URL via Node.JS: You can combine the node.js example above with a wget to then upload it to web3.storage.
docker run --rm --env TOKEN=$TOKEN -v $PWD/nodejs:/nodejs node:18-alpine ash -c 'cd /nodejs && wget http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz && npm install && node put-files.js --token=$TOKEN train-images-idx3-ubyte.gz'

Estuary​

This example shows you how to pin data using estuary.

Ways to pin using Esturay​

  1. Pin a local file via the Esturay UI: You can browse to the Estuary UI to upload a file via your web browser.
tip

Due to the way Estuary batches files for pinning, it may take some time before your file is accessible/listable.

  1. Pin a local file via Curl: Please view the API documentation to see all available commands. This example submits a single file to be pinned.
export TOKEN=YOUR_API_KEY
echo hello world > foo.txt
curl -X POST https://upload.estuary.tech/content/add -H "Authorization: Bearer ${TOKEN}" -H "Content-Type: multipart/form-data" -F "data=@foo.txt"

The response will return the CID of the file.

View pinned files​

If the upload was successful, you can view the file via your estuary account page. Alternatively, you can obtain this information from the CLI:

curl -X GET -H "Authorization: Bearer ${TOKEN}" https://api.estuary.tech/content/list