Kubenretes (K3s) - Adding a private insecure registry

Introduction

K3s is a minimalistic kubernetes platform created by Rancher. It uses SQLite instead of etcd and provides a powerfull platform with builtin service Loadbalancer.

I have settled on using k3s for my home server, where I also do some development, I needed to run a local registry to test my artifacts and as part of Continuous Integration.

Warning an insecure registry is not recommended in most cases. It exposes your registry to trivial man-in-the-middle (MITM) attacks. Only use this solution for isolated testing or in a tightly controlled, air-gapped environment.

Creating a registry.

Create deployment, we will use a non-persistent volume for now. configure it to listen on port 5000, registry images will be stored in /var/lib/registry-storage. This will allow you to keep registry state after system restart.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-registry-v0
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    version: v0
spec:
  replicas: 1
  selector:
    k8s-app: kube-registry
    version: v0
  template:
    metadata:
      labels:
        k8s-app: kube-registry
        version: v0
    spec:
      containers:
      - name: registry
        image: registry:2
        resources:
          limits:
            cpu: 100m
            memory: 200Mi
        env:
        - name: REGISTRY_HTTP_ADDR
          value: :5000
        - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
          value: /var/lib/registry
        volumeMounts:
        - name: image-store
          mountPath: /var/lib/registry
        ports:
        - containerPort: 5000
          name: registry
          protocol: TCP
      volumes:
      - name: image-store
        hostPath:
          path: /var/lib/registry-storage
        #emptyDir: {}

2. Create a service to expose your registry, with a type: LoadBalancer. This will expose port 5000 on localhost

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
apiVersion: v1
kind: Service
metadata:
  name: kube-registry
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    kubernetes.io/name: "KubeRegistry"
spec:
  selector:
    k8s-app: kube-registry
  ports:
  - name: registry
    port: 5000
    targetPort: 5000
    protocol: TCP
  type: LoadBalancer

Configure k3s to use your new registry

Create /etc/rancher/k3s/registries.yaml file on each k3s node, If you have more than one node Make sure to taint, pods to run only on node.

  "<your-host>:5000":
    endpoint:
      - "http://<your-host>:5000"

Configure docker client to access an insecure registry.

Linux create file /etc/docker/daemon.json (for Mac and windows go Docker/Preferences/Docker Engine)

{
  "insecure-registries" : ["myregistrydomain.com:5000"]
}

Your are all set. time to fill in your registry with images.

Test your registry

Pull test docker image

$ docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
Digest: sha256:95cf004f559831017cdf4628aaf1bb30133677be8702a8c5f2994629f637a209
Status: Image is up to date for busybox:latest
docker.io/library/busybox:latest

change the image tag

docker tag busybox:latest <your-host>:5000/test:latest

push to your k3s registry

$ docker push core:5000/test:latest
The push refers to repository [core:5000/test]
1be74353c3d0: Pushed
latest: digest: sha256:fd4a8673d0344c3a7f427fe4440d4b8dfd4fa59cfabbd9098f9eb0cb4ba905d0 size: 527

Create a test kubernetes resource

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
apiVersion: v1
kind: Pod
metadata:
  name: test-registry-image
  namespace: playground
spec:
  containers:
  - name: busybox
    image: core:5000/test:latest
    command: ["sh"]
    args: ["-c", "while [ 1 ]; do echo 'Hello World' && sleep 10; done"]

    resources:
      limits:
        cpu: 100m
        memory: 50Mi

Create a namespace

$ kubectl create namespace playground
namespace/playground created

Apply your test resource and see results

 $ kubectl apply -f test-registry.yaml
pod/test-registry-image created

$ kubectl -n playground get pods
NAME                  READY   STATUS    RESTARTS   AGE
test-registry-image   1/1     Running   0          7s

Note Status running is a good sign, Kubernetes was hable to pull the image and run the container.

Confirm is running by tailing the console

 $ kubectl -n playground logs -f test-registry-image
Hello World
Hello World
Hello World
Hello World
^C

Taking it further

This pattern with some small changes can be used in large production systems a few things that should be improved

Block registry access
Replace the LoadBalancer with an Nginx ingress or an AWS ALB or GCE.
Use NFS or cloud alternatives.
Add SSL
Authentication.

Edits

7/5/2020 Corrected example pod to use image from local:5000 registry and renamed test pod to test-registry-image

Soo broken