Soo broken

Kubenretes (K3s) - Adding a private insecure registry

Published On: 2020-06-25, Reading Time: 4 minutes


Introduction

K3s is a minimalistic kubernetes platform created by Rancher. It uses SQLite instead of etcd and provides a powerfull platform with builtin service Loadbalancer.

I have settled on using k3s for my home server, where I also do some development, I needed to run a local registry to test my artifacts and as part of Continuous Integration.

Warning an insecure registry is not recommended in most cases. It exposes your registry to trivial man-in-the-middle (MITM) attacks. Only use this solution for isolated testing or in a tightly controlled, air-gapped environment.

Creating a registry.

  1. Create deployment, we will use a non-persistent volume for now. configure it to listen on port 5000, registry images will be stored in /var/lib/registry-storage. This will allow you to keep registry state after system restart.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-registry-v0
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    version: v0
spec:
  replicas: 1
  selector:
    k8s-app: kube-registry
    version: v0
  template:
    metadata:
      labels:
        k8s-app: kube-registry
        version: v0
    spec:
      containers:
      - name: registry
        image: registry:2
        resources:
          limits:
            cpu: 100m
            memory: 200Mi
        env:
        - name: REGISTRY_HTTP_ADDR
          value: :5000
        - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
          value: /var/lib/registry
        volumeMounts:
        - name: image-store
          mountPath: /var/lib/registry
        ports:
        - containerPort: 5000
          name: registry
          protocol: TCP
      volumes:
      - name: image-store
        hostPath:
          path: /var/lib/registry-storage
        #emptyDir: {}


2. Create a service to expose your registry, with a type: LoadBalancer. This will expose port 5000 on localhost

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
apiVersion: v1
kind: Service
metadata:
  name: kube-registry
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    kubernetes.io/name: "KubeRegistry"
spec:
  selector:
    k8s-app: kube-registry
  ports:
  - name: registry
    port: 5000
    targetPort: 5000
    protocol: TCP
  type: LoadBalancer

Configure k3s to use your new registry

  1. Create /etc/rancher/k3s/registries.yaml file on each k3s node, If you have more than one node Make sure to taint, pods to run only on node.
  "<your-host>:5000":
    endpoint:
      - "http://<your-host>:5000"

Configure docker client to access an insecure registry.

  1. Linux create file /etc/docker/daemon.json (for Mac and windows go Docker/Preferences/Docker Engine)
{
  "insecure-registries" : ["myregistrydomain.com:5000"]
}

Your are all set. time to fill in your registry with images.

Test your registry

  1. Pull test docker image
$ docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
Digest: sha256:95cf004f559831017cdf4628aaf1bb30133677be8702a8c5f2994629f637a209
Status: Image is up to date for busybox:latest
docker.io/library/busybox:latest
  1. change the image tag
docker tag busybox:latest <your-host>:5000/test:latest
  1. push to your k3s registry
$ docker push core:5000/test:latest
The push refers to repository [core:5000/test]
1be74353c3d0: Pushed
latest: digest: sha256:fd4a8673d0344c3a7f427fe4440d4b8dfd4fa59cfabbd9098f9eb0cb4ba905d0 size: 527
  1. Create a test kubernetes resource
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
apiVersion: v1
kind: Pod
metadata:
  name: test-registry-image
  namespace: playground
spec:
  containers:
  - name: busybox
    image: core:5000/test:latest
    command: ["sh"]
    args: ["-c", "while [ 1 ]; do echo 'Hello World' && sleep 10; done"]

    resources:
      limits:
        cpu: 100m
        memory: 50Mi
  1. Create a namespace
$ kubectl create namespace playground
namespace/playground created
  1. Apply your test resource and see results
 $ kubectl apply -f test-registry.yaml
pod/test-registry-image created
$ kubectl -n playground get pods
NAME                  READY   STATUS    RESTARTS   AGE
test-registry-image   1/1     Running   0          7s

Note Status running is a good sign, Kubernetes was hable to pull the image and run the container.

  1. Confirm is running by tailing the console
 $ kubectl -n playground logs -f test-registry-image
Hello World
Hello World
Hello World
Hello World
^C

Taking it further

This pattern with some small changes can be used in large production systems a few things that should be improved

  • Block registry access
  • Replace the LoadBalancer with an Nginx ingress or an AWS ALB or GCE.
  • Use NFS or cloud alternatives.
  • Add SSL
  • Authentication.

Edits

7/5/2020 Corrected example pod to use image from local:5000 registry and renamed test pod to test-registry-image