K3s add node. pub on the server, and paste them on a new line into ~/.
K3s add node. Steps To Reproduce: Create 3 node HA etcd, reboot a node. Flannel options can only be set on server nodes, and must be identical on all servers in the cluster. For the other, set the hostname to knode2 and assign an IP of 192. Easy Configuration Via Cloud-Init You can turn a standard k3OS image into a configured system during boot time. Step 1: Prepare K3S kubelet Adding Roles To Existing Servers . An example implementation of AWX on single node K3s using AWX Operator, with easy-to-use simplified configuration with ownership of data and passwords. k3s Adding/Removing Nodes. 20. Check them all out! Date URL Part 2022-12-07 K3s cluster updates Updates to the cluster build 2022-03-29 Kubernetes GUIs Exploring Kubernetes GUIs 2022-03-11 K3s single-node cluster for noobs Deploying K3s Introduction I’m starting a new job in the next few days that will require me to learn Kubernetes (often stylized as k8s). Jun 27, 2023 · This command will fetch the convenience script from K3S and install this first node as control-pane, etcd and master node. K3s comes with Rancher's Local Path Provisioner and this enables the ability to create persistent volume claims out of the box using local storage on the respective node. sh. To install K3S on master node you need to invoke command from root user: $ curl -sfL https://get. In this article, I will show you how to deploy a three-node K3s cluster on Ubuntu nodes that are created using Terraform and a local KVM libvirt provider. For one, set the hostname to knode1 and assign an IP of 192. Out of the box, K3s ships with the local path storage provisioner, which uses storage local to the node. On the dedicated server that you want to connect to the existing cluster, install the K3S agent by running the following command. They create these requirements for etcd quorum. admin-user. Provision master node. nano /etc/hosts. In order to add additional nodes, you update the original cluster. Describe the bug: The bug consists while trying to add node access to NVIDIA Gpu. Flannel is a lightweight provider of layer 3 network fabric that implements the Kubernetes Container Network Interface (CNI). Multiple variables can be employed to extend the configurability of this installation. You can use this functionality to ensure that specific Pods only run on nodes with certain isolation, security, or regulatory properties. For more information please reference the official documentation here. You can add more nodes later. The worker nodes are defined as The most easy approach ist to just add a label to each node and use this as selector force Apr 17, 2018 · But if you really want a HA cluster please follow the documentation's minimal requirements and use 3 nodes for masters. I already added the vip to the primary node and it is now pingable, when I attempt to run the script below it fails. toml. Jul 10, 2020 · If you are only using k3s, consider adding an alias command. The suggested way is to create your own namespace to label a kubernetes node by simply reproducing the kubelet way. The command syntax for joining a worker node to cluster is: kubeadm join [api-server-endpoint] [flags] Dec 9, 2023 · To join an agent node, we don’t have to specify the cluster configuration, but we still need the cluster token. Before you begin You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. None of these will uninstall k3s though. 1 Problem The role of agent nodes displayed in this command "kubectl get nodes" is "<none>" Here is the output of the command "kubectl get nodes" Feature Is there any way to modify this role string "<none>" Nov 14, 2023 · Step 4: Deploying Any Application Using Kubectl, K3s, And Your Ubuntu server. Get manifests / repo Nov 13, 2019 · Worker node (k3s-2) with private ip of 10. You signed out in another tab or window. For example the list of firewall ports hwich are required on your cluster. At this point, you have a three-node K3s cluster that runs the control plane and etcd components in a highly available mode. sudo k3s kubectl create -f dashboard. On a previous post we saw how ridiculously easy is to bootstrap a k3s cluster on a Raspberry Pi but what do we need to do to join new worker nodes to the cluster? The installation of a new node to make it join to an existing cluster is as follows: # curl -sfL https://get. kubectl taint nodes yasin node-role. Note that server nodes are schedulable by default. Apr 5, 2024 · In this case a single node will be configured as master node. ssh/authorized_keys on the agent. 1. So my use cases are: Jun 17, 2020 · You signed in with another tab or window. You can switch between clusters via kubectl config use-context command. Dec 23, 2023 · I would like to add a new server node to ensure HA if the server nodes goes down. Aug 21, 2020 · Repeat these steps in node-2 and node-3 to launch additional servers. Make sure that you change the hostname for each VM. txt. Verify Installation. 0+k3s2 kubernetes02. Note that my cluster is made of VMs. start dashboard locally. Jan 12, 2020 · ENV CentOS 7. Bring cluster back up on second node (everything seems fine) Use kubectl to delete the failed node (which sits in NotReady state) Run k3s-uninstall. Easy cluster configuration: We can easily set up a single node or multi-node cluster, without any complexity. Next, install the K3s using the following command. Add this line. If you want to know the details for a specific node, use this: kubectl label --list nodes node_name. . Authenticate the CLI - doctl auth init. Taints are the opposite -- they allow a node to repel a set of pods. 52. 25. Mar 26, 2020 · I am trying to add a k3s worker node to a microk8s master node. The kubeadm join command is used to bootstrap a Kubernetes worker node or an additional control plane node, and join it to the cluster. We can check the status of the node with: sudo kubectl get nodes. Our friends at K3D created a useful guide for that. The tool is written in Go and is cross-compiled for Linux, Windows, MacOS and even on Raspberry Pi. In case of having more than one etcd you can add them all in the same environment Mar 26, 2024 · Adding labels to nodes allows you to target Pods for scheduling on specific nodes or groups of nodes. Hey guys, sorry I am very new to kubernetes but trying hard to get better. For example ,if you want to add the control-plane role to a dedicated etcd node, you can remove the --disable-apiserver --disable-controller-manager --disable-scheduler flags from the systemd unit or config file, and restart the service. With your single-node K3s cluster ready, let’s deploy and run an application on K3s. Now, let's install k3s. create a folder, where the files of the k3s-node-0 WSL2 instace will be saved. It is what is commonly referred to as a CNI Plugin. The Flannel Options. If you want storage that can be shared across multiple nodes if you are running them, you will need to use a different solution which the NFS storage provisioner Mar 14, 2021 · By Techno Tim 1 min read. Oct 21, 2020 · I would like to create small cluster with one master and 3 agent nodes, but in my opinion master node should be in separate server so it have resources only for itself. Install doctl. k3d sets the newly created cluster as active. In this video I show how to add an additional node to an existing cluster. 130 k3s-worker02 2) Add Firewall rules on Master and Worker Nodes. The two options only add labels and/or taints at registration time, so they can only be added once and not changed after that again by running K3s commands. With this additional node added, we can perform our final test: bring down the node0! Jan 26, 2021 · I've created simple config file to override maxPods value (default 110):. 17. Etcd3, MySQL, and Postgres are also supported. kubectl drain <node-name> --ignore-daemonsets --delete-local-data. 168. Boot your first worker node and ssh into it: ssh pi@knode1. What worked for me was doing the following: Login to the k3s server. domain. Note that it will attempt to rejoin if you restart the service though. You can check the status of the service with the below command: 1. Mar 11, 2022 · Hey! Listen! This post is part of a series on my journey into K3s. image rancher/k3s:v1. I am using ksup and kubevip. May 24, 2023 · K3s Version: v1. You can list Kubernetes node details along with their labels in this fashion: kubectl get nodes --show-labels. 2) Edit instance group for nodes (Only if you are using Oct 12, 2021 · On top of everything that has already been answered since k8S 1. From the Rancher menu (the hamburger icon at to Oct 3, 2020 · Details I wanted to use the k3s cluster as part of GitLab CI for which I need to connent the cluster to GitLab which needs an external IP. Accessible over HTTPS from remote host; All data will be stored under /data; Fixed (configurable) passwords for AWX and PostgreSQL; Fixed (configurable) versions of AWX Jul 21, 2019 · There are multiple features related here: In the next release, users can choose which new nodes would be used automatically by Longhorn and which directory they would use as the default data path. Prometheus is an open-source monitoring and alerting tool that collects and stores time-series data, while Grafana is a popular data visualization platform that Dec 19, 2020 · Deploy the admin-user configuration: (if you’re doing this from your dev machine, remove sudo k3s and just use kubectl) 1. Apr 7, 2022 · Adding and removing nodes: We can easily add and remove nodes from a K3s cluster by executing one-line commands. This is a sample option I used to build a cluster with one master node and one worker node and limit their ram usage to 1 GB. Building a Multi-architecture CPU Kubernetes cluster is easier than you think with k3s . Every cluster will consist of one or more containers: - 1 (or more) server node container (k3s) - (optionally) 1 loadbalancer container as the entrypoint to the cluster (nginx) - (optionally) 1 (or more) agent node containers (k3s) Mar 29, 2024 · Collaborator. To set up a single-node K3s cluster, run the Rancher server installation command on just one node instead of two nodes. 1. In case firewall is enabled on your RHEL systems, then allow following ports in firewall on the master node. K3s part 2: Create a single node cluster. Once again I will be leveraging my terraform code that I have mentioned and provided in previous posts. Colors are used purely for visual aid. The token parameter can also be used to set a deterministic token when adding nodes. If using RaspberryPi for a worker node, make sure you have. Apr 22, 2021 · Remove label from a node. =bar" --memory string Memory limit imposed on the node [From docker] -n, --network strings Add node to (another) runtime network Aug 29, 2019 · Setup the DigitalOcean CLI. Master node installation. k3s. As VPN connection between AWS and Oracle Cloud needed and every time worker node has to cross ocean (probably) to reach master node. To start testing with k3s, keep it simple, only create one node. Mar 25, 2021 · Summary. 7 K3S v1. When empty, this token will be generated automatically for further use. In order to remove nodes, remove the node information from the nodes list in the original cluster. Create a new k3s node in docker. Below is an example showing how to add labels and a taint: Sep 28, 2022 · Bring up a new cluster. The only part we are interested in in that guide is the “Configure containerd” section. PS C:\ > C:\wsl2\ mkdir C:\wsl2\k3s - node -0. Roles can be added to existing dedicated nodes by restarting K3s with the disable flags removed. Run --cluster-reset on second node. apiVersion: kubelet. Nov 27, 2023 · 192. Oct 24, 2020 · Adding a master node. This article focuses on the minimal manual May 21, 2022 · 0. Default Endpoint Fallback Containerd has an implicit "default endpoint" for all registries. Place the binary in /usr/local/bin on each air-gapped node and ensure it is executable. 16, the kubernetes. In configuration file the server IP is on localhost how to generate an external IP to use the clus The [token] is the actual token to write, as generated by k3s token generate. get bearer token. We will need the IP and username of our virtual machine to SSH into and install Kubernetes. If you have not tainted the server nodes and will be running workloads on them, please ensure you also create the registries. sh on first node to start clean. Actual behavior: k3s is in a crush loop and doesnt rejoin the cluster. Note Node affinity is a property of Pods that attracts them to a set of nodes (either as a preference or a hard requirement). It adds support for sqlite3 as the default storage backend. This should result in an id_rsa private key and an id_rsa. kubectl label node <nodename> <labelname>-. K3s is a fully conformant production-ready Kubernetes distribution with the following changes: It is packaged as a single binary. See The ability to specify the default data path on the host #582 and Mark nodes with/without certain label as disabled/enabled by default #583 . On every master node they run the etcd which works very close to masters. Aug 3, 2021 · Today, I had a few people ping me about someone who was frustrated configuring a K3s cluster to enable NFS storage provisioner. Feb 20, 2021 · Before you add arm64 nodes to your cluster you must consider whether you want to specifically exclude incompatible Pods from running on these nodes with node affinity rules, or if you want to exclude all Pods by default and specifically allow the ones you know work on arm64 with taints and tolerations. In this step, you will install the latest version of K3s on your Ubuntu machine. Make sure you set the following values, replacing the examples provided: INSTALL_K3S_VERSION: This should match the version when you installed K3S on the control node. If you want to start a cluster with extra worker nodes, then extend the creation command like: k3d cluster create --servers 3 --agents 5. If no token is given, a random one will be generated. Expected behavior: Rebooted node should rejoin the cluster and sync with the quorum. As you can notice, we use the same command but with a minus sign with the label name. The labels are in form of key-value pair. Dec 7, 2020 · Server. K3s is a lightweight, production-grade Kubernetes distribution that simplifies the setup process by coming pre-configured with DNS, networking, and other tools out of the box. You add master replicas by. Add an additional node (everything seems fine) Shut down nodes. Note that if you are planning to create a cluster with more than one VM, each VM should have a unique hostname. We’ll also Jul 18, 2022 · nice, I noticed that as long as the new IP I changed has working network connectivity, simply reboot the k3s single node, after reboot, all pod started working, it did not work previously because the new IP I changed actually did not have network connectivity because Ubuntu 20. I am going to be adding a few more nodes in the coming days so if I see it again I will update this. as the very end of your /boot/cmdline. You can also do the same on the K3S worker nodes. Note! If your k3s_endpoint is an IP address, you don’t have to write anything in /etc/hosts and skip this step. Tolerations are applied to pods. Sep 12, 2021 · K3s is a lightweight Kubernetes deployment by Rancher that is fully compliant, yet also compact enough to run on development boxes and edge devices. k8s. When I log into the failed node and look at the logs I see this error: domain='domain. At this point you have installed k3s, check whether the newly deployed node is in Ready state using the following command. $ k3sup install --ip <IP> --user ubuntu --k3s-extra-args "--cluster-init". io/ prefixes are reserved for Kubernetes core components. A token in secure format, including the cluster CA hash, will be written to stdout. A few options to check. Step#2. Dec 21, 2020 · role=server: sets the role for the node to be a control plane. Follow the steps below to set up the necessary components. k3d cluster create three-node-cluster --agents 3. May 9, 2020 · Worker Node and Master Nodes communication is very critical for Kubernetes cluster. de Ready <none> 93s v1. But I can't see in k3s documentation --disable-agent anymore, and I read that it is buggy so they removed it, so I am wondering how can I have only server setup on one node and Jan 10, 2020 · Step 4: Join new Kubernetes Worker Node a Cluster. When running the k3s server command on these nodes, you must set the datastore-endpoint parameter so that K3s knows how to connect to the external datastore. io/ and k8s. The simplest method is using the following command. If we want to use it as worker we should run. NAME STATUS ROLES AGE VERSION. Apr 18, 2022 · I'm going to deploy a Kubernetes cluster using Rancher's K3s distribution on top of LXC containers. Dec 22, 2023 · Step 1— Installing K3s. Jul 10, 2022 · Select k3d as provide and give name to the cluster and specify how many mater/worker nodes you want. io/v1beta1 kind: KubeletConfiguration maxPods: 250 And then we have to pass this config file as an argument during K3s installation (I recommend to specify absolute pathname): Mar 10, 2020 · Grab some SD cards for the two worker nodes and install Raspbian on each. config. Our foundation will be Ubuntu for ARM, then we’ll add k3s, and then join it to our cluster. 1) First drain the node. 156. hostname='k3s-master02'. All nodes in this example are running Ubuntu 20. It wraps Kubernetes and other components in a single, simple launcher. journalctl --no-pager -u k3s &> k3s-$(hostname). If you want k3s to stop and uninstall, that requires external action. For example: Jul 12, 2021 · After checking the configuration you can also check the current logs and status of your K3S deployment of your master node using the following commands: # Get k3s server logs systemctl status k3s # Full logs journalctl-u k3s # Save logs into a file journalctl-u k3s > logs. using a kube-up script with the following flags: KUBE_REPLICATE_EXISTING_MASTER=true - to create a replica of an existing master. Now that we have our VMs ready, let’s install Kubernetes on them. That is for removing the node from the etcd cluster without deleting it. 110 k3s-master 192. 04. Synopsis¶. All you need is ssh access and the k3sup binary to get kubectl access immediately. as we can see there is one node with the name of yasin and the role is master. . io/master-. logand attach the resulting files? Create a new WSL2 instance using the K3S image and K3S as a server, in the current user context. Copy the contents of id_rsa. change the working directory to c:\wsl2. journalctl -u k3s-agent. admin-user-role. yml file containing deployment service and load balancer configurations. This section describes the roles for etcd nodes, controlplane nodes, and worker nodes in Kubernetes, and how the roles work together in a cluster. So, Go to your new node and open the /etc/hosts file using nano. If you have an idea that would be awesome ! Thanks !! During that process, it grabbed the node that was having issues, upgraded it, then it connected just fine as a worker. 0. io | sh -. As I said, the best way would be just to ping them. Create a new cluster. Additional context / logs: k3sup 🚀 (said 'ketchup') k3sup is a light-weight utility to get from zero to KUBECONFIG with k3s on any local or remote VM. yasin Ready master 11d v1. You can deploy it anywhere you like, just follow the documentation. Feb 19, 2024 · Longhorn provides a straightforward method for installing the iSCSI driver and NFSv4 directly on all nodes. Cluster Configuration: 1 server , 2 agents. Installing open-iscsi. yml. run sudo kubectl delete node <name-of-node> (No need for sudo if your kubeconfig already has 644 as its write permissions) If the server IP changed - Login to the node. kubectl drain <node-name>. Trying to configure the config. Dec 21, 2020 · Nodes are supposed to be removed from the etcd cluster when you remove them from the kubernetes cluster, so I'm curious to see if there are any logs showing how that failed. 0+k3s2 Deploying a Service For a first test, some NGINX instances are being deployed to the cluster with the following manifest: Sep 17, 2021 · K3d cluster create k3d cluster create¶. de Ready control-plane,master 18m v1. kubernetes. 0+k3s. Author. I set up a pi cluster thanks to Jeff Geerling videos, but I wonder how to add a node to a running cluster. Mar 15, 2021 · This will tell K3S that it won’t use the embedded DB and instead to use the one provided in that address. We have two main options when installing K3s. Feb 28, 2023 · Adding Nodes to the Cluster. For both examples, assume that a HA K3s cluster with embedded etcd has been installed on 3 nodes. sudo k3s kubectl -n kubernetes-dashboard create token admin-user. Tolerations allow scheduling but don't guarantee scheduling: the scheduler also evaluates other Aug 24, 2023 · This page shows how to assign a Kubernetes Pod to a particular node using Node Affinity in a Kubernetes cluster. Flannel Options. 19. K3s agents can be configured with the options --node-label and --node-taint which adds a label and taint to the kubelet. May 4, 2021 · 3 min read | by Jordi Prats. Abbreviated notes are below for generic hosts, and detailed instructions for DigitalOcean droplets. 8 ; Establish connection between VM's. Use the following syntax to remove a label from a node: bash. I found something about kubeadm tokens but I can’t find specific documentation for k3s. Run k3d cluster create command to spin up multi node clusters. 04 network configuration issue. If you use labels for node isolation, choose label keys that the kubelet cannot modify. Create a Virtual Machine in Proxmox. In the case of k3s I will still deploy 3 nodes so that I can demonstrate adding addtional nodes to the cluster. In this configuration, each agent node is registered to the same server node. txt file. Login to your server as your sudo-enabled user (in this tutorial, it will be sammy) using the following command if using password-based login: ssh sammy @ your_server_ip. The first step is to configure one (or more) manager nodes. First, get the name of the master. All the usernames for VMs are ubuntu by default. List your VMs: doctl compute droplet ls --format Name,PublicIPv4,Status. Each k3s server is configured with: Feb 9, 2021 · $ sudo k3s kubectl get node NAME STATUS ROLES AGE VERSION kubernetes01. On the k8s master node , using an add-node command gives the details how a worker should join itself: Nov 11, 2020 · After rebooting one of the arm64 masters it will not rejoin the cluster. io. Node(s) CPU architecture, OS, and Version: Ubuntu 20. Run multipass ls and take note of IP of master-node . Ensure that you retain a copy of this token, as it is required Nov 30, 2021 · Hit enter at each of the prompts. Prerequisites . yml -f dashboard. We can use a script or install it from a binary file. sudo kubectl get nodes. I went with the latter on our EKS clusters Jun 29, 2022 · So im not sure if this is needed if ever just an agent changes its IP. and in case you want to explicitly define the listening ports of your Traefik instance you should add the following arguments (adjusting the ports to your liking): Aug 22, 2021 · Step 3: Configure K3S to use nvidia-container-runtime. Reload to refresh your session. 120 k3s-worker01 192. K3s embedded sqlite database is used in this case. 8+k3s1. Jun 14, 2021 · I still intend to spin 2 KVM guests for the cluster, one acting as a main one with 2 k3s server nodes, and the other one backup one in case if the main one is unavailable with 1 k3s server node, resulting in a total of 3 k3s server nodes (with some amount of agent nodes). K3s is a highly available, certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances. Your typical application deployment will use a . If you just want it gone, use kubectl delete node. In my experience, it's easier to create your droplets using the UI and then to manage them through the CLI. For example to remove the label color: blue from worker-2 node, we will use: bash. Adding K8s sauce with k3sup. May 23, 2018 · Adding a new master replica After you have created an HA-compatible cluster, you can add master replicas to it. Download the K3s install script at get. cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1. It’s quick and simple. We are passing the --k3s-extra-args "--cluster-init Oct 13, 2022 · How to read node labels in Kubernetes. Create a new k3s cluster with containerized nodes (k3s in docker). In this tutorial I shared the steps to add a worker (previously known as minnion) node to an existing Kubernetes cluster. As this link. The default backend for Flannel is vxlan. You might have to ignore daemonsets and local-data in the machine. Create a hostPath backed persistent volume claim and a pod to utilize it: You can boot up with k3s available in under 10 seconds, with fast cluster scaling. You basically just need to execute kubeadm join command to join the node to the cluster but you must account for multiple pre-requisites. For a high availability set up, it is often recommended to use 3 master nodes, but a single node will be fine for testing. yaml file on each server as well. local'. If you do not Download the K3s binary from the releases page, matching the same version used to get the airgap images. In K3S you have one or more master nodes and one or more worker nodes, but the manager nodes can also run workloads. 51. Tolerations allow the scheduler to schedule pods with matching taints. pub on the server, and paste them on a new line into ~/. 3-k3s2: specifies the K3s image to be used. kubectl get nodes. K3s also makes it fairly painless to join new workers to the cluster. In this video we’ll build a Raspberry Pi 4 with an ARM CPU and add it to our existing x86 x64 amd64 CPU Kubernetes cluster. K3s bootstrap data: If you are using K3s in an HA configuration with an external SQL datastore, and your server (control-plane) nodes were not started with the --token CLI flag, you will no longer be able to add additional K3s servers to the cluster without specifying the token. A K3s user can manipulate Kubernetes resources by calling the K3s API on the server node. Jun 17, 2019 · 4. Simplified & Secure K3s is packaged as a single <70MB binary that reduces the dependencies and steps needed to install, run and auto-update a production Kubernetes cluster. Lines show the traffic flow between components. curl -sfL https://get. 1 x86_64 GNU/Linux. yml file with any additional nodes and specify their role in the Kubernetes cluster. We should now tell K3S to use nvidia-container-runtime (which is a kind of plugin of containerd) on the containerd of our node. Install the K3S agent. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. # Get k3s worker Mar 3, 2016 · List the nodes and get the <node-name> you want to drain or (remove from cluster) kubectl get nodes. The most important thing is to check if the machines can connect with each other. sudo reboot. Create an API key in the UI with Read and Write access. Agents are simply the nodes that gets added to the master node; they can be another master node or a worker node. The output of this command should be saved, as the secret portion of the token cannot be shown again. in Windows Terminal open a PowerShell instance. You can add the explorer to the cluster May 31, 2022 · A server node is defined as a Raspberry Pi which runs the k3s server. RKE supports adding/removing nodes for worker and controlplane hosts. Below we cover a simple example. KUBE_GCE_ZONE=zone - zone where the master replica will run. However, installing Rancher on a single-node cluster can be useful if you want to save resources by using a single node in the short term, while preserving a high-availability migration path. You switched accounts on another tab or window. Check Journalctl for errors. As seen here, we checked from the different points of view to ensure our new control plane node is running correctly. Adding nodes from on-prem to a cloud provider or from different cloud provider will make lots of issues from network perspective. Place the install script anywhere on each air-gapped node, and name it install. 4. Oct 24, 2020 · In K3s terms, a master node is called the server and the rest of the nodes are called agents. This diagram is applicable to Kubernetes clusters launched with Rancher using RKE. service -n 300 -xn. Jul 17, 2022 · View of the nodes from kubectl. 13. tmpl file seems to not work properly and the node does not get resource access to Nov 14, 2020 · After the virtual machine is up, ssh onto it and change its hostname: sudo hostnamectl set-hostname k3smaster. For example: k3d cluster create two-node-cluster --agents 2. pub public key being created. K3s is a lightweight Kubernetes distribution that is designed for use in production environments with limited resources, making it an ideal choice for small-scale and edge deployments. nx es bn yy ma gk dh oz gl ay