Altinity · sunsingerus · Jan 24, 2019 · Jan 22, 2019 · Jan 22, 2019 · Jan 22, 2019
diff --git a/.gitignore b/.gitignore
@@ -10,3 +10,6 @@
 
 # Output of the go coverage tool, specifically when used with LiteIDE
 *.out
+
+# IDE folders
+.idea
diff --git a/docs/introduction.md b/docs/introduction.md
@@ -0,0 +1,33 @@
+# ClickHouse Operator Introduction
+
+## Prerequisites
+You may need to have the following items in order to have ClickHouse installation in k8s
+
+1. Persistent Volumes
+1. Zookeeper
+
+### Persistent Volumes
+ClickHouse needs disk space to keep data. Kubernetes provides [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)
+for this purpose.
+As it is stated
+> A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator.
+
+This means that we have to do some homework in order to provide Persistent Volumes to ClickHouse installation.
+PVs can be provided by:
+
+1. system administrator, who is in charge of k8s installation, can prepare required number of PVs
+1. Persistent Volume Provisioner, which may be set up in k8s installation in order to [provision volumes dynamically](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/)
+
+When ClickHouse required some disk storage, in places [Persistent Volume Claim](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims)
+which specifies desired storage class and size. Each Persistent Volume has class assigned and size provisioned.
+So the main bond between software and disk to be provisioned is [Storage Class](https://kubernetes.io/docs/concepts/storage/storage-classes/).
+
+### Zookeeper
+
+In case we'd like to have [data replication](https://clickhouse.yandex/docs/en/operations/table_engines/replication/) in ClickHouse,
+we need to have [Zookeeper](https://zookeeper.apache.org/) instance accessible by ClickHouse.
+There is no requirement to have Zookeeper instance dedicated to serve ClickHouse replication, we just need to have access to running Zookeeper.
+However, in case we'd like to have high-available ClickHouse installation, we need to have Zookeeper cluster of at least 3 nodes.
+So, we can either use
+1. Already existing Zookeeper instance, or
+1. Setup our own Zookeeper - in most cases inside the same k8s installation.
diff --git a/docs/zookeeper_setup.md b/docs/zookeeper_setup.md
@@ -0,0 +1,130 @@
+# Task: Setup Zookeeper
+
+We are going to setup Zookeeper in k8s environment.
+This document assumes k8s cluster already setup and `kubectl` has access to it.
+
+What we'll need here:
+1. Create k8s components:
+  * [Service](https://kubernetes.io/docs/concepts/services-networking/service/) - used to provide central access point to Zookeeper
+  * [Headless Service](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services) - used to provide DNS namings
+  * [Disruption Balance](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/) - used to specify max number of offline pods
+  * [Storage Class](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/) - used to specify storage class to be used by Zookeeper for data storage
+  * [Stateful Set](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) - used to manage and scale sets of pods    
+
+ All files are located in [manifests/zookeeper](../manifests/zookeeper) folder
+
+ ## Zookeeper Service
+ This service provides DNS name for client access to all Zookeeper nodes.
+ Create service.
+ ```bash
+ kubectl apply -f 01-service-client-access.yaml
+ ```
+ Should have as a result
+ ```text
+service/zookeeper created
+```
+
+## Zookeeper Headless Service
+This service provides DNS names for all Zookeeper nodes
+Create service.
+```bash
+kubectl apply -f 02-headless-service.yaml
+```
+Should have as a result
+```text
+service/zookeeper-nodes created
+```
+
+## Disruption Budget
+Disruption Budget instructs k8s on how many offline Zookeeper nodes can be at any time
+Create budget.
+```bash
+kubectl apply -f 03-pod-disruption-budget.yaml
+``` 
+Should have as a result
+```text
+poddisruptionbudget.policy/zookeeper-pod-distribution-budget created
+```
+
+## Storage Class
+This part is not that straightforward and may require communication with k8s instance administrator.
+
+First of all, we need to deside, whether Zookeeper would use [Persistent Volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) as a storage or just stick to more simple [Volume](https://kubernetes.io/docs/concepts/storage/volumes) (In doc [emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) type is used)
+
+In case we'd prefer to stick with simpler solution and go with [Volume of type emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir), we are done here and need to adjust [StatefulSet config](../manifests/zookeeper/05-stateful-set.yaml) as described in next [Stateful Set unit](#stateful-set). Just move to [it](#stateful-set).
+
+In case we'd prefer to go with [Persistent Volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) storage, some additional steps have to be done.
+
+Shortly, [Storage Class](https://kubernetes.io/docs/concepts/storage/storage-classes/) is used to bind together [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/),
+which are created either by k8s admin manually or automatically by [Provisioner](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/). In any case, Persistent Volumes are provided externally to an application to be deployed into k8s. So, this application has to know **Storage Class Name** to ask for from the k8s in application's claim for new persistent volume - [Persistent Volume Claim](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims).
+This **Storage Class Name** should be asked from k8s admin and written as application's **Persistent Volume Claim** `.spec.volumeClaimTemplates.storageClassName` parameter in [05-stateful-set.yaml](../manifests/zookeeper/05-stateful-set.yaml). 
+
+## Stateful Set
+Edit [05-stateful-set.yaml](../manifests/zookeeper/05-stateful-set.yaml) according to your Storage Preferences.
+
+In case we'd go with [Volume of type emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir), ensure `.spec.template.spec.containers.volumes` is in place and look like the following:
+```yaml
+      volumes:
+      - name: datadir-volume
+        emptyDir:
+          medium: "" #accepted values:  empty str (means node's default medium) or Memory
+          sizeLimit: 1Gi
+```
+and ensure `.spec.volumeClaimTemplates` is commented.
+
+In case we'd go with **Persistent Volume** storage, ensure `.spec.template.spec.containers.volumes` is commented and ensure `.spec.volumeClaimTemplates` is uncommented.
+```yaml
+  volumeClaimTemplates:
+  - metadata:
+      name: datadir-volume
+    spec:
+      accessModes:
+        - ReadWriteOnce
+      resources:
+        requests:
+          storage: 1Gi
+## storageClassName has to be coordinated with k8s admin and has to be created as a `kind: StorageClass` resource
+      storageClassName: storageclass-zookeeper
+```
+and ensure **storageClassName** (`storageclass-zookeeper` in this example) is specified correctly, as described in [Storgae Class](#storage-class) section
+
+As `.yaml` file is ready, just apply it with `kubectl`
+```bash
+kubectl apply -f 05-stateful-set.yaml
+```
+Should have as a result
+```text
+statefulset.apps/zookeeper-node created
+```
+
+Now we can explore Zookeeper cluster deployed in k8s:
+
+```bash
+kubectl get pod
+```
+
+```text
+NAME                  READY   STATUS    RESTARTS   AGE
+zookeeper-node-0      1/1     Running   0          9m2s
+zookeeper-node-1      1/1     Running   0          9m2s
+zookeeper-node-2      1/1     Running   0          9m2s
+```
+
+```bash
+kubectl get service
+```
+
+```text
+NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
+zookeeper                   ClusterIP   10.108.36.44   <none>        2181/TCP                     168m
+zookeeper-nodes             ClusterIP   None           <none>        2888/TCP,3888/TCP            31m
+```
+
+```bash
+kubectl get statefulset
+```
+
+```text
+NAME                READY   AGE
+zookeeper-node      3/3     10m
+```
diff --git a/manifests/clickhouseinstallation-object.yaml → ...erator/clickhouseinstallation-object.yaml b/manifests/clickhouseinstallation-object.yaml → ...erator/clickhouseinstallation-object.yaml
diff --git a/manifests/custom-resource-definitions.yaml → ...operator/custom-resource-definitions.yaml b/manifests/custom-resource-definitions.yaml → ...operator/custom-resource-definitions.yaml
diff --git a/manifests/zookeeper/01-service-client-access.yaml b/manifests/zookeeper/01-service-client-access.yaml
@@ -0,0 +1,13 @@
+# Setup Service to provide access to Zookeeper for clients
+apiVersion: v1
+kind: Service
+metadata:
+  name: zookeeper
+  labels:
+    app: zookeeper
+spec:
+  ports:
+  - port: 2181
+    name: client
+  selector:
+    app: zookeeper
diff --git a/manifests/zookeeper/02-headless-service.yaml b/manifests/zookeeper/02-headless-service.yaml
@@ -0,0 +1,16 @@
+# Setup Headless Service for StatefulSet
+apiVersion: v1
+kind: Service
+metadata:
+  name: zookeeper-nodes
+  labels:
+    app: zookeeper
+spec:
+  ports:
+  - port: 2888
+    name: server
+  - port: 3888
+    name: leader-election
+  clusterIP: None
+  selector:
+    app: zookeeper
diff --git a/manifests/zookeeper/03-pod-disruption-budget.yaml b/manifests/zookeeper/03-pod-disruption-budget.yaml
@@ -0,0 +1,10 @@
+# Setup max number of unavailable pods in StatefulSet
+apiVersion: policy/v1beta1
+kind: PodDisruptionBudget
+metadata:
+  name: zookeeper-pod-distribution-budget
+spec:
+  selector:
+    matchLabels:
+      app: zookeeper
+  maxUnavailable: 1
diff --git a/manifests/zookeeper/04-storageclass-zookeeper.yaml b/manifests/zookeeper/04-storageclass-zookeeper.yaml
@@ -0,0 +1,8 @@
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: storageclass-zookeeper
+provisioner: kubernetes.io/no-provisioner
+#volumeBindingMode: WaitForFirstConsumer
+volumeBindingMode: Immediate
+
diff --git a/manifests/zookeeper/05-stateful-set.yaml b/manifests/zookeeper/05-stateful-set.yaml
@@ -0,0 +1,115 @@
+# Setup Zookeeper StatefulSet
+# Possible params:
+# 1. replicas
+# 2. memory
+# 3. cpu
+# 4. storage
+# 5. storageClassName
+# 6. user to run app
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: zookeeper-node
+spec:
+  selector:
+    matchLabels:
+      app: zookeeper
+  serviceName: zookeeper-headless-service
+  replicas: 3
+  updateStrategy:
+    type: RollingUpdate
+  podManagementPolicy: Parallel
+  template:
+    metadata:
+      labels:
+        app: zookeeper
+    spec:
+      affinity:
+        podAntiAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            - labelSelector:
+                matchExpressions:
+                  - key: "app"
+                    operator: In
+                    values:
+                    - zookeeper
+              topologyKey: "kubernetes.io/hostname"
+      containers:
+      - name: kubernetes-zookeeper
+        imagePullPolicy: Always
+        image: "k8s.gcr.io/kubernetes-zookeeper:1.0-3.4.10"
+        resources:
+          requests:
+            memory: "1Gi"
+            cpu: "0.5"
+        ports:
+        - containerPort: 2181
+          name: client
+        - containerPort: 2888
+          name: server
+        - containerPort: 3888
+          name: leader-election
+        command:
+        - sh
+        - -c
+        - "start-zookeeper \
+          --servers=3 \
+          --data_dir=/var/lib/zookeeper/data \
+          --data_log_dir=/var/lib/zookeeper/data/log \
+          --conf_dir=/opt/zookeeper/conf \
+          --client_port=2181 \
+          --election_port=3888 \
+          --server_port=2888 \
+          --tick_time=2000 \
+          --init_limit=10 \
+          --sync_limit=5 \
+          --heap=512M \
+          --max_client_cnxns=60 \
+          --snap_retain_count=3 \
+          --purge_interval=12 \
+          --max_session_timeout=40000 \
+          --min_session_timeout=4000 \
+          --log_level=INFO"
+        readinessProbe:
+          exec:
+            command:
+            - sh
+            - -c
+            - "zookeeper-ready 2181"
+          initialDelaySeconds: 10
+          timeoutSeconds: 5
+        livenessProbe:
+          exec:
+            command:
+            - sh
+            - -c
+            - "zookeeper-ready 2181"
+          initialDelaySeconds: 10
+          timeoutSeconds: 5
+        volumeMounts:
+        - name: datadir-volume
+          mountPath: /var/lib/zookeeper
+      # Run as a non-privileged user
+      securityContext:
+        runAsUser: 1000
+        fsGroup: 1000
+## Mount either emptyDir via volume or PV of 'storageclass-zookeeper' storageClass
+## Uncomment what is required
+      volumes:
+      - name: datadir-volume
+        emptyDir:
+          medium: "" #accepted values:  empty str (means node's default medium) or Memory
+          sizeLimit: 1Gi
+## Mount either emptyDir via volume or PV of 'storageclass-zookeeper' storageClass
+## Uncomment what is required
+#  volumeClaimTemplates:
+#  - metadata:
+#      name: datadir-volume
+#    spec:
+#      accessModes:
+#        - ReadWriteOnce
+#      resources:
+#        requests:
+#          storage: 1Gi
+## storageClassName has to be coordinated with k8s admin and has to be created as a `kind: StorageClass` resource
+#      storageClassName: storageclass-zookeeper
-Original file line number
+Diff line change
@@ Expand Up / @@ -10,3 +10,6 @@ @@
     # Output of the go coverage tool, specifically when used with LiteIDE
     *.out
+    # IDE folders
+    .idea