How to Use Azure BLOB and AWS S3 with JuiceFS as a Shared storage

Introduction 

In this blog we will explore how to use Azure Blob and AWS S3 object storage with JuiceFS. This is a low-cost shared storage solution to meet the various product applications of machine learning, AI, and big data. 

What is JuiceFS? 

JuiceFS is a large scale, high performance and low-cost shared storage solution which supports local storage and all major cloud object storage e.g., Azure Blob, AWS s3 and GCS. JuiceFS is fully compatible with Hadoop, POSIX and cloud native container orchestration solutions. It offers outstanding performance, encryption, and compression features.  

JuiceFS is an open-source tool and can be installed on Windows, Linux or MacOS. Users can also benefit from the cloud hosted services of JuiceFS which offer free subscription for up to 1 TB storage with 10 mount points. In case a company or a user requires more storage, they can opt for the $0.02GB option. 

Now that we have briefly explored JuiceFS, let’s move onto discuss the cloud hosted services  

Components 

The following are the primary components used for the purpose of this blog.  

1. JuiceFS cloud hosted account  
2. AWS S3 bucket  
3. Azure storage account 
4. An Ubuntu VM 

Create Azure BLOB Shared File System  

1. Login to the JuiceFS console and click on create Filesystem.  

2. We Enter the Filesystem (FS) name and the region for your storage account.

3. Now create the storage account in the West US 2 region and copy the access keys. 

4. Log in to the node where you want to mount this shared storage volume and run the following command to download JuiceFS client components.  

curl -L https://juicefs.com/static/juicefs -o juicefs && chmod +x juicefs 

5. Run the following command to start the mount process 

sudo ./juicefs mount azdemofs /jfs 

This command will ask for the following information. 

Token for JuiceFS File System (can be retrieved from JuiceFS portal) 

Azure Storage Account Name  

Azure Storage Account Access Key 

Enter these details and hit enter, at this moment your shared Volume is ready to use. Validate it by running the df -h command.  

6. Mount JuiceFS at boot 

• Copy juicefs file into /sbin/ as mount.juicefs  
• Update fstab file with following  

<META-URL>    <MOUNTPOINT>       juicefs     _netdev[,<MOUNT-OPTIONS>]     0  0 

e.g: JuiceFS:azdemofs/jfs       juicefs     _netdev,max-uploads=50,writeback,cache-size=2048     0 0du -h 

Now that we can mount the same volume to multiple nodes to share the data across nodes, let’s explore how to use AWS S3 bucket with JuiceFS as a shared volume.  

Using Amazon S3 bucket with JuiceFS 

1. Login to JuiceFS console and create a new File System. Enter the name and select the region where we want to create the S3 bucket.The S3 bucket will be created with the same name as your filesystem.  

2. Now we need to create an IAM user with full access to S3 and get the access and secret id to be used while mounting this file system.  

3. To create the IAM user login to AWS console and navigate to IAM and create new user.  

Enter the username and select the access key. Select the programmatic access option to ensure that the user is not logged into the AWS console.  

4. Select S3Full Access group 

5. Create user 

6. On the next screen, copy the access and secret key and save it in a safe place.  

7. Now log in to the node where you want to mount the volume and execute the following command:  

curl -L https://juicefs.com/static/juicefs -o juicefs && chmod +x juicefs && sudo  

./juicefs mount juicefsdemos3 /jfs 

8. Verified volume is mounted. 

Conclusion 

JucieFS offers shared storage needs for various applications. It also offers support from multiple object storage providers giving users significant flexibility. If you have any questions or queries regarding JuiceFS then please contact us using the comment box below! We will be happy to help!