add edgemark performance testing tool

Signed-off-by: wackxu <xushiwei5@huawei.com>

5 files changed, 214 insertions, 0 deletions

diff --git a/build/edgemark/Dockerfile b/build/edgemark/Dockerfile
new file mode 100644
index 000000000..8f4b85884
--- /dev/null
+++ b/build/edgemark/Dockerfile
@@ -0,0 +1,19 @@
+ARG BUILD_FROM=golang:1.16-alpine3.13

+

+FROM ${BUILD_FROM} AS builder

+

+ARG GO_LDFLAGS

+

+COPY . /go/src/github.com/kubeedge/kubeedge

+

+RUN apk --no-cache update && \

+apk --no-cache upgrade && \

+apk --no-cache add build-base linux-headers sqlite-dev binutils-gold && \

+CGO_ENABLED=1 GO111MODULE=off go build -v -o /usr/local/bin/edgemark -ldflags="${GO_LDFLAGS} -w -s -extldflags -static" \

+github.com/kubeedge/kubeedge/edge/cmd/edgemark

+

+FROM alpine:3.13

+

+COPY --from=builder /usr/local/bin/edgemark /usr/local/bin/edgemark

+

+ENTRYPOINT ["edgemark"]
\ No newline at end of file
diff --git a/build/edgemark/README.md b/build/edgemark/README.md
new file mode 100644
index 000000000..d74d5becf
--- /dev/null
+++ b/build/edgemark/README.md
@@ -0,0 +1,48 @@
+# Edgemark User Guide
+
+## Introduction
+
+Edgemark is a performance testing tool that inspired by
+[Kubemark](https://github.com/kubernetes/kubernetes/tree/master/cmd/kubemark)
+which allows users to run experiments on
+simulated clusters. The primary use case is scalability testing, as simulated
+clusters can be much bigger than the real ones. The objective is to expose
+problems with the KubeEdge cloud components CloudCore that appear only on bigger clusters.
+
+This document serves as a primer to understand what Edgemark is, what it is not,
+and how to use it.
+
+## Architecture
+
+On a very high level, Edgemark cluster consists of three parts: a real kubernetes master,
+KubeEdge CloudCore component and a set of “Hollow” Edge Nodes. Hollow Edge Node is registered
+with `HollowEdgeCore`, which pretends to be an ordinary EdgeCore, but does not create any real containers.
+`HollowEdgeCore` mocks runtime manager with Kubernetes `k8s.io/kubernetes/pkg/kubelet/cri/remote/fake/fake_runtime.go`,
+where most logic sits. Except for simulating runtime manager, other behaviors is the same as edgecore.
+
+![img](images/edgemark.jpg)
+
+Currently, Kubernetes master components run on a dedicated machine as pods that are
+created/managed by kubelet, which itself runs as either a systemd or a supervisord
+service on the master VM depending on the VM distro. Having a dedicated machine for the master
+has a slight advantage over running the master components on an external cluster,
+which is being able to completely isolate master resources from everything else.
+The CloudCore has multiple instances and requires multiple dedicated nodes to maintain high reliability.
+Usually In highly available configurations, a load balancer must sit in front of the CloudCore
+to correctly route requests to healthy CloudCore servers. And The HollowEdgeNodes
+on the other hand are run on an ‘external’ Kubernetes cluster
+as pods in an isolated namespace (named edgemark). This idea of using pods on a
+real cluster behave (or act) as nodes on the edgemark cluster lies at the heart of
+edgemark's design.
+
+## Requirements
+
+To run Edgemark, you need:
+
+1. A Kubernetes cluster (called `external cluster`) for running all your HollowEdgeNodes
+2. A Kubernetes cluster (called `edgemark cluster`)  the master for HollowEdgeNodes.
+3. Several dedicated nodes in `edgemark cluster` for deploy CloudCore, a load balancer
+   that expose CloudCore service for HollowEdgeNodes and the load balancer has to be directly
+   routable from HollowEdgeNodes.
+4. You also need access to a Docker repository that has the
+   container images for CloudCore, hollow-edge-node and node-problem-detector.
diff --git a/build/edgemark/edgemark_setup_guide.md b/build/edgemark/edgemark_setup_guide.md
new file mode 100644
index 000000000..fd0a48885
--- /dev/null
+++ b/build/edgemark/edgemark_setup_guide.md
@@ -0,0 +1,97 @@
+## Introduction
+This document serves to understand how to set up edgemark cluster given
+that a base cluster (to run hollow-edge-node pods) and
+separate master (to act as master for the hollow edge nodes) are already present.
+
+## Precondition
+You need edgemark master and external cluster to set up a edgemark cluster.
+
+The functions are as follows:
+
+- edgemark master: can be StandAlone or HA, used to be the edgemark cluster's master
+- external cluster: used to create hollow edge nodes for the edgemark cluster
+
+## Steps:
+1. Deploy CloudCore in edgemark cluster, refer to [CloudCore deploy](https://kubeedge.io/en/docs/setup/ha/) for more details.
+
+2. Build edgemark image
+
+If you want to build/use your own edgemark image, do as follows.
+
+- i. pull kubeedge code
+
+```
+cd $GOPATH/src/github.com/kubeedge
+git clone git@github.com:kubeedge/kubeedge.git
+```
+
+- ii. build edgemark image
+
+```
+cd $GOPATH/src/github.com/kubeedge/kubeedge
+make image WHAT=edgemark
+```
+
+Then you can get the image named `kubeedge/edgemark:{tag}` locally.
+
+3. Create hollow nodes in external cluster
+
+- i. create namespace and secret
+
+Copy edgemark master's `tokensecret` which is generated by CloudCore which is used to access CloudCore for edge node,
+and create it in namespace that will deploy hollow edge node in external cluster.
+
+first, save `tokensecret` in file from edgemark cluster,
+
+```
+kubectl get secret -nkubeedge tokensecret -oyaml > tokensecret.yaml
+```
+
+modify namespace in `tokensecret.yaml`
+```
+sed -i "s|namespace: .*|namespace: {ns}|g" tokensecret.yaml
+```
+
+create ns and tokensecret in external cluster
+
+```
+kubectl create ns edgemark
+
+kubectl create -f tokensecret.yaml
+```
+
+- ii. apply yaml to create hollow nodes
+
+You can use `hollow-edge-node_template.yaml` in the current directory.
+
+Note:
+
+- the parameters `{{numreplicas}}` means the number of hollow nodes in the edgemark cluster
+- the parameters `{{server}}` means the server address exposed for hollow nodes join in
+- the parameters `{{server}}`, `{{numreplicas}}`, `{{edgemark_image_registry}}` and `{{edgemark_image_tag}}` need to be filled in the template
+- your external cluster should have enough resources to be able to run `{{numreplicas}}` no. of hollow-node pods
+
+```
+kubectl create -f hollow-edge-node_template.yaml
+```
+
+Waiting for these hollow-node pods to be running. Then you can see these pods register as edgemark master's nodes.
+
+Finally, edgemark master and external cluster set up the edgemark cluster.
+
+
+4. Run performance testing with ClusterLoader2
+
+After set up the edgemark cluster, we can do our performance testing with ClusterLoader2.
+[ClusterLoader2](https://github.com/kubernetes/perf-tests/tree/master/clusterloader2) is
+an official K8s scalability and performance testing framework.
+refer to [ClusterLoader2 Getting started](https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/docs/GETTING_STARTED.md) for more details.
+we just need config `--provider=kubemark` when we run performance test, for example:
+
+```
+./clusterloader  --testconfig=config.yaml --provider=kubemark --kubeconfig=${HOME}/.kube/config --v=2
+```
+
+
+
+
diff --git a/build/edgemark/hollow_edge_node_template.yaml b/build/edgemark/hollow_edge_node_template.yaml
new file mode 100644
index 000000000..aa677ad94
--- /dev/null
+++ b/build/edgemark/hollow_edge_node_template.yaml
@@ -0,0 +1,50 @@
+kind: Deployment
+apiVersion: apps/v1
+metadata:
+  name: hollow-edge-node
+spec:
+  replicas: {{numreplicas}}
+  selector:
+    matchLabels:
+      app: hollow-edge-node
+  template:
+    metadata:
+      labels:
+        app: hollow-edge-node
+    spec:
+      containers:
+        - name: hollow-edgecore
+          image: {{edgemark_image_registry}}/edgemark:{{edgemark_image_tag}}
+          command:
+            - edgemark
+          args:
+            - --token=$(TOKEN)
+            - --name=$(NODE_NAME)
+            - --http-server=https://{{server}}:10002
+            - --websocket-server={{server}}:10000
+            - --alsologtostderr
+            - --v=2
+          env:
+            - name: NODE_NAME
+              valueFrom:
+                fieldRef:
+                  apiVersion: v1
+                  fieldPath: metadata.name
+            - name: TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: tokensecret
+                  key: tokendata
+          resources:
+            requests:
+              cpu: 20m
+              memory: 50M
+          securityContext:
+            privileged: true
+      tolerations:
+        - effect: NoExecute
+          key: node.kubernetes.io/unreachable
+          operator: Exists
+        - effect: NoExecute
+          key: node.kubernetes.io/not-ready
+          operator: Exists
diff --git a/build/edgemark/images/edgemark.jpg b/build/edgemark/images/edgemark.jpg
new file mode 100644
index 000000000..2dfe3dadf
--- /dev/null
+++ b/build/edgemark/images/edgemark.jpg