PlatformProduct updates

K3S installation

We use Kubernetes, an open-source container orchestration system to manage our applications.

At the moment the only Kubernetes distribution supported is K3s (click here for the official documentation) by Suse Linux for both on-premise and cloud deployments.

Why K3s?

Kubernetes has been widely adopted in modern software development as it offers a powerful, portable and open-source platform that automates the management of containerized applications.

When setting up a Kubernetes environment, it comes in two flavours: vanilla Kubernetes and managed Kubernetes. With vanilla Kubernetes, a software development team has to pull the Kubernetes source code binaries, follow the code path, and build the environment on the machine. On the other hand, managed Kubernetes comes pre-compiled and pre-configured with tools that improve features to enhance a certain focus area, such as storage, security, deployment, monitoring, etc. Managed Kubernetes versions are also known as Kubernetes distributions. Some popular Kubernetes distributions are Rancher, Red Hat OpenShift, Mirantis, VMware Tanzu, EKS, GKE and AKS.

Kubernetes distributions can have different components that may cause applications that work in one distribution to not necessarily work or even crash into another. Some of the most important components that differ between distributions are:

  • Container Runtime: The container runtime is the software that is responsible for running containers. Each Kubernetes Distribution may offer support for different Container Runtimes. Some popular container runtimes include Docker, CRI-O, Apache Mesos, CoreOS, rkt, Canonical LXC and frakti among others.

  • Storage: Storage is important for Kubernetes applications as it offers a way to persist this data. Kubernetes’ Container Storage Interface (CSI) allows third-party vendors to easily create storage solutions for containerized applications. Some Kubernetes Distributions build their own storage solutions while others integrate with existing third party solutions. Popular storage solutions for Kubernetes include: Amazon ElasticBlock Storage (EBS), GlusterFS, Portworx, Rook, OpenEBS among others.

  • Networking: Kubernetes applications are typically broken down into container-based microservices which are hosted in different PODs, running in different machines. Networking implementations allow for the seamless communication and interaction between different containerized components. Networking in Kubernetes is a herculean task, and each distribution may rely on a networking solution to facilitate communication between pods, services and the internet. Popular networking implementations include Flannel, Weave Net, Calico and Canal among others.

In order to offer our customers a better and more seamless experience while configuring, running, upgrading and troubleshooting our products while also avoiding compatibility issues between different distributions we decided to officially support ONLY ONE Kubernetes distribution: K3s. The main reasons for choosing K3s are:

  1. Costs — K3s is 100% open source and there’s no need to pay for any expensive licenses.

  2. Less setup overhead — a lot of time is saved when setting up a new environment because you don’t need to go through a lengthy process of acquiring extra licenses based on how many CPU cores you have. Also, K3s can be installed using only one command.

  3. It supports many Linux distros — K3s supports popular Linux distributions including open source ones, it can also run both on-premise and in the cloud (AWS, Azure, GCP).

  4. It’s fast and lightweight — K3s is packaged as a single <100MB binary and its lightweight architecture makes it faster than stock Kubernetes for the workloads that it runs.

  5. Easy to update — Thanks to its reduced dependencies.

  6. Batteries included — CRI, CNI, service load balancer, and ingress controller are included.

  7. Smaller attack surface — Thanks to its small size and reduced amount of dependencies.

  8. Certified — K3s is an official CNCF project that delivers a powerful certified Kubernetes distribution.

  9. Flexible — you can run K3s using single-node or multi-node cluster setup.

K3s and Antivirus

Antivirus software adds an unknown variable to the existing complexity of Kubernetes. Most of them have not yet kept up with newer technologies such as Kubernetes and have not reached a CNCF certified status. In environments where antivirus software had been enabled, there has seen issues stemming from interfering actions from such software. As an example, there have been incidents where the antivirus software had pruned files in the ContainerD filesystem incorrectly, causing the ContainerD mounts to go corrupt and causing data loss.

Issues resulting from third-party tools, such as antivirus and intrusion detection software, interfering with ContainerD or other necessary system calls are deemed resolved should disabling such tools restore functionality.

Also, having Antivirus software running in the nodes can result in poor performance. Some antivirus software may continue to impact even if all paths our products rely on are excluded. Whilst performing scanning and inspection of files, disk I/O and CPU usage can increase, thus slowing our product performance. In some cases, this reduction in performance can be dramatic and can even render our products impossible to use.

Lastly, where a customer environment has deviated from our supported configurations, we always reserve the right to recommend the customer to revert to a supported configuration to resolve the reported issue.

Secure mode K3s

If the cluster has been deployed before January 2023, most probably it will come across this issue at some point.

If cluster runs a pod that uses a public image that hasn’t yet been copied to our public repo, error ErrImagePull with the message Failed to pull image will arise.

To address this issue this configuration file needs to be added, 

mirrors:
  "*":
    endpoint:
      - "https://images.master.k3s.getvisibility.com"
    rewrite:
      "(.*)": "gv-public/$1"
  "images.master.k3s.getvisibility.com":
    endpoint:
      - "https://images.master.k3s.getvisibility.com"

This file instructs K3s to rewrite all images (except images.master.k3s.getvisibility.com) and redirect them to Getvisibility’s public repository.

Requirements

The minimum requirement for the Kubernetes cluster is a single node (1 virtual machine) with the following specs:

EDC / Synergy
DDS / DSPM
Ultimate

CPU cores

8

16

20

Memory

32GB

64GB

80GB

Storage

500GB min 32M inodes

600GB min 39M inodes

700GB min 32M inodes

Please also refer to Estimate hardware capacity needs.

Installation

Ensure the following items are in place and configured:

  • Domain Name Service (DNS) with public name resolution enabled

  • Network Time Protocol (NTP)

  • Software Update Service - access to a network-based repository for software update packages.

  • Fixed private IPv4 address

  • Unique static hostname

For details on how to configure Rancher behind a proxy, refer to the official Rancher documentation and ensure the network settings mentioned above are applied.

If using proxy, please run this before using starting k3s.sh installation:

export http_proxy="$PROXY_IP"
export https_proxy="$PROXY_IP"
no_proxy="$NODE_IP,localhost,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local"

When running k3s.sh script you need to provide product name in a form of PRODUCT_NAME argument. This will instruct the installer to test your current environment against product requirements, which are different.

Allowed product names are:

  • synergy

  • focus

  • dspm

  • enterprise

  • ultimate

Capitalization of the name is important. If you provide name that cannot be recognized or if you don't provide product name at all, the script will default to PRODUCT_NAME="dspm".

curl -sfL https://assets.master.k3s.getvisibility.com/k3s/k3s.sh | \
INSTALL_K3S_VERSION="v1.26.10+k3s1" K3S_KUBECONFIG_MODE="644" PRODUCT_NAME=enterprise sh -s - server --node-name=local-01

This is just a regular k3s installation command when you want to install Getvisibility Enterprise product.

Be cautious when skipping built-in checks - we built them for the product to achieve optimal performance with minimal required maintenance.

Argument
Description

SKIP_PRECHECK=true

to skip all built in checks

SKIP_SYSTEM_CHECKS=true

to skip hardware checks

SKIP_NETWORK_CHECKS=true

to skip connectivity checks

ONLY_PRECHECK=true

will run precheck only and stop after that

The installer, depending on the combination of arguments provided, will perform a set of actions before returning back to the command line.

Provided not all the checks have been skipped, it is possible for the installer to abort the installation process. If that happens please review the output paying special attention to any WARN messages. Should you have any concerns or questions please cotact the Support with the result screen attached.

Run the kubectl registration command:

The command below is just an example, it will not work during deployment. For direct customers, Customer Support Team will provide the registration command, otherwise you should have been provided registration command in the Welcome Email.

kubectl apply -f https://....k3s.getvisibility.com/v3/import/dxslsxcf84....yaml

For security reasons the registration command can be used only a single time, the command becomes invalid after the first use. In case you need to run it again you must contact the support team for a new registration command.

Monitor the progress of the installation: 

watch -c "kubectl get deployments -A"

  • The K3s deployment is complete when elements of all the deployments (coredns, local-path-provisioner, metrics-server, traefik and cattle-cluster-agent) show at least "1" as "AVAILABLE"

  • In case of errors you can inspect the logs of a pod using kubectl logs , e.g. 

 kubectl logs cattle-cluster-agent-d96d648d8-wjvl9 -n cattle-system

K3s support matrix

Please note that we don’t use Docker as the container runtime, instead we use containerd.

Network settings

Your network should be configured to allow the following public urls to be accessible over port 443 (HTTPS) and HTTPS traffic is bypassed (NOT intercepted):

https://assets.master.k3s.getvisibility.com (Custom K3s installation files)
https://images.master.k3s.getvisibility.com (Private Docker registry)
https://charts.master.k3s.getvisibility.com (Private Helm registry)
https://prod-eu-west-1-starport-layer-bucket.s3.eu-west-1.amazonaws.com (Docker registry AWS CDN)
https://rpm.rancher.io (Rancher RPM repo for configuring SELinux packages on RHEL or CentOS)
https://api.master.k3s.getvisibility.com (Private API server)
https://rancher.master.k3s.getvisibility.com (Rancher management server)
https://rancher.$RESELLER_NAME.k3s.getvisibility.com (Rancher management server, where $RESELLER_NAME is Getvisibility for direct customers)

For more details on how to configure Rancher behind a proxy Configuring Rancher and Fleet agent to run behind an HTTP proxy

Rancher might be trying to reach to git.rancher.io since it’s a default hard-coded repository, but we have our own private repo with all our charts. So it's ok to block it as we can't disable it.

PreviousUnderstand installation requirementsNextConfiguring a HA K3s cluster

Last updated

Was this helpful?