Introducing Kadalu Storage - Part 1

Posted on Jan 18, 2022 by Aravinda Vishwanathapura
9 minutes read.

Kadalu Storage is an opinionated distributed Storage solution focused on providing scalable and easy to use Volumes for Cloud, On-Premise, Kubernetes, or Raspberry Pis.

Kadalu Storage uses the core filesystem layer from the GlusterFS project. Kadalu Moana project provides a modern Storage manager on top of the GlusterFS core filesystem layer.

History

After facing many challenges with Glusterd based Volume management, we experimented to integrate the GlusterFS core filesystem with Kubernetes without using Glusterd. This experiment helped us to understand that some of Gluster’s capabilities are limited by the Management layer and not by the filesystem itself.

Rewriting the management layer also helped us to fix the user experience issues without worrying about backward compatibility.

Highlights of Kadalu Storage

(Warning: The list of features is huge)

  • Multiple Pools support - With Gluster, managing multiple Storage pools is not possible. Kadalu Storage introduced multiple Storage pool support.

  • No mesh network between the Storage nodes. Upon node reboot, handshake happens only with the Manager node.

  • Inbuilt Monitoring - All the Status CLIs provide rich metrics and the Storage manager provides ready-to-use Prometheus compatible metrics URL. If users are using a non-Prometheus based monitoring solution then it also provides the metrics in JSON format.

  • Persistent Storage units'(Bricks) Ports - Port is assigned on Kadalu Volume creation and it will be persisted throughout the life cycle of the Volume. It is also possible to manually assign the required ports to these Storage units.

  • Volume Create CLI enhancements - Kadalu Storage has eliminated many confusions and has made Volume create commands more intuitive.

  • Auto Starting the Volumes on Create - Kadalu Volumes are automatically started on creation by default. Override by adding the --no-start flag to the Volume create command.

  • ReST APIs and SDKs - Kadalu Storage provides ReST APIs for all the Storage management operations. Programming language SDKs are available for Python and Crystal. In future, more languages will be added to the list.

  • User management - Kadalu Storage CLI operations are not limited to the Storage nodes. You can install Kadalu CLI in a remote machine and interact with the Storage pools. Kadalu Storage adds support for creating Storage admins, maintainers, viewers, and Clients.

  • Automatic Pool creation and adding nodes to the Pool - With Kadalu Storage, you can directly jump to creating the Volume without creating a Pool or adding nodes to the Pool. Use --auto-create-pool and --auto-add-nodes to volume create command to do the needful. Automatic doesn’t mean insecure, it does all the required validations.

  • Importing a Volume - While migrating a Volume from Gluster or migrating a Volume by moving disks to the new set of nodes, Kadalu Storage provides an easy option to re-create the Volume with the same Volume ID as it was before.

  • Multiple node addresses - Kadalu Storage provides more control to specify a node address to use for public communication or internal activities like Heal.

  • Kadalu Storage can co-exist with GlusterFS - Even though Kadalu Storage uses the core file system from Gluster, it can co-exist with Gluster. Kadalu Storage made sure that no ports or work directory will clash.

  • ZFS as first-class support - Utilities to create Storage units using ZFS, Snapshot support and many configurations to make it usable with ZFS.

  • Snapshot CLI enhancements - Usability improvements to make intuitive to the users.

  • File and directory Snapshots using Xfs reflinks

  • Geosync - New feature to sync files from main Volume to a backup volume. This uses Path-based Geo-replication instead of depending on GFID.

  • [Developers] Template-based Volfile generation - This allows developers to integrate and test the new Xlator by writing a YAML file.

  • Secure communication between nodes - Uses token-based authentication to communicate with other nodes.

  • Showing Size of Volumes in info command - No need to mount the Volume to see the utilization and total capacity.

  • Automated Tests - Tests may not be a feature of Storage but that gives the confidence to use the Storage in production.

  • The control plane can be externally hosted in Cloud - Which allows users to control the Storage pools across the data centres.

  • Migration tool to import the Volumes from the Gluster Cluster

  • [Developers] Easy to add a new feature - Get started working on a feature in a day or two if you are familiar with Python/Ruby.

  • Set Volume options while creating a Volume

  • Effective Quota management

  • Import/Export Storage Pool configurations

  • Multiple access modes: Fuse/NFS/Samba

We couldn’t add all the features to the list, we will plan to write about each feature in detail in our future blog posts. The highlight of today’s blog post is Volume create CLI enhancements.

Do you remember the Twitter post about finding the three differences? This blog post will help you find those three differences If you haven’t found them yet.

Spot the three differences

Clear Separation of replica or disperse groups in a Volume

For example, look at the two commands below. Which one is more clear?

kadalu volume create PROD/vol1 replica 3 \
    server1.example.com:/storages/s1     \
    server2.example.com:/storages/s2     \
    server3.example.com:/storages/s3     \
    server4.example.com:/storages/s4     \
    server5.example.com:/storages/s5     \
    server6.example.com:/storages/s6
kadalu volume create PROD/vol1       \
    replica                          \
    server1.example.com:/storages/s1 \
    server2.example.com:/storages/s2 \
    server3.example.com:/storages/s3 \
    replica                          \
    server4.example.com:/storages/s4 \
    server5.example.com:/storages/s5 \
    server6.example.com:/storages/s6

Users coming from Gluster may find the first syntax familiar. But the second one gives more clarity about which storage units together form a replica group. Also specifying the replica count is not necessary.

Don’t worry! Both are valid syntax in Kadalu Storage; we are not removing one in favour of the other. Your earlier scripts will continue to work.

An alternate word for replica - Mirror

ZFS users are familiar with mirror keywords instead of a replica. We got this covered by adding a mirror alias to replica.

kadalu volume create PROD/vol1       \
    mirror                           \
    server1.example.com:/storages/s1 \
    server2.example.com:/storages/s2 \
    server3.example.com:/storages/s3 \
    mirror                           \
    server4.example.com:/storages/s4 \
    server5.example.com:/storages/s5 \
    server6.example.com:/storages/s6

Changes to Arbiter syntax

Arbiter Volume is a type of Kadalu Storage Volume that uses the third storage unit in the replica/mirror group for storing only metadata. We added support for the arbiter prefix just before specifying the Arbiter storage unit.

kadalu volume create PROD/vol1               \
    mirror                                   \
    server1.example.com:/storages/s1         \
    server2.example.com:/storages/s2         \
    arbiter server3.example.com:/storages/s3 \
    mirror                                   \
    server4.example.com:/storages/s4         \
    server5.example.com:/storages/s5         \
    arbiter server6.example.com:/storages/s6

or Volume Type itself as Arbiter

kadalu volume create PROD/vol1       \
    arbiter                          \
    server1.example.com:/storages/s1 \
    server2.example.com:/storages/s2 \
    server3.example.com:/storages/s3 \
    arbiter                          \
    server4.example.com:/storages/s4 \
    server5.example.com:/storages/s5 \
    server6.example.com:/storages/s6

Again the old syntax of specifying replica 2 arbiter 1 is supported for those who are coming from Gluster.

Raw devices

Can Kadalu Storage accept raw devices instead of a mounted directory as storage units? Yes. It also exposes a few options to control the filesystem creation.

kadalu volume create PROD/vol1   \
    mirror                       \
    server1.example.com:/dev/vhd \
    server2.example.com:/dev/vhd \
    server3.example.com:/dev/vhd \
    mirror                       \
    server4.example.com:/dev/vhd \
    server5.example.com:/dev/vhd \
    server6.example.com:/dev/vhd

By default, the xfs filesystem is created without LVM. Use below additional options to customize.

  • --fs-type=xfs|ext4|zfs (Default is xfs)

  • --fs-options=…​

  • --use-lvm (Only applicable for ext4 and xfs fs type)

Auto start Volumes on Create

Why do you need an additional step to start a Volume after it is created? We fixed it. Kadalu Storage manager automatically starts the Volume on creation. Use the --no-start flag if this behaviour is not required. Stop and Start commands are available for future maintenance activities.

***

We will cover another feature from the list next week. Read this Quickstart guide to get started. Please feel free to open issues if something is not working or you have suggestions.

Please join the Kadalu community meeting happening on Jan 20th, 2022 4 pm-5 pm IST | UTC 10:30-11:30 am. Bridge Link: https://meet.google.com/jtp-kvsh-ggu

Join the Kadalu team as we plan, design and develop new features. We are available on our Slack instance, https://kadalu.slack.com.

Aravinda Vishwanathapura

Kadalu Maintainer

Website| Twitter| Github

© 2022 Kadalu Community(https://github.com/kadalu). All Rights Reserved.