MetalLB is a load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols. MetalLB 是 K8S 集群(裸机)的负载平衡器实现,使用标准路由协议。

MetalLB is a young project. You should treat it as a beta system. The project maturity page explains what that implies.

Why

Kubernetes does not offer an implementation of network load-balancers (Services of type LoadBalancer) for bare metal clusters. The implementations of Network LB that Kubernetes does ship with are all glue code that calls out to various IaaS platforms (GCP, AWS, Azure…). If you’re not running on a supported IaaS platform (GCP, AWS, Azure…), LoadBalancers will remain in the “pending” state indefinitely when created

Kubernetes 没有为裸机集群提供网络负载均衡器(LoadBalancer 类型的服务)的实现。Kubernetes 所附带的网络负载均衡器的实现都是调用各种 IaaS 平台(GCP、AWS、Azure……)的粘合代码。如果您没有在受支持的 IaaS 平台(GCP、AWS、Azure……)上运行,那么负载平衡器在创建时将无限期地处于“挂起”状态。

Bare metal cluster operators are left with two lesser tools to bring user traffic into their clusters, “NodePort” and “externalIPs” services. Both of these options have significant downsides for production use, which makes bare metal clusters second class citizens in the Kubernetes ecosystem.

裸机集群运营商只剩下两个较小的工具来将用户流量引入集群,即“NodePort”和“externalIPs”服务。这两种选择对于生产使用都有显著的负面影响,这使得裸机 集群在 Kubernetes 生态系统中处于二等公民的地位。

MetalLB aims to redress this imbalance by offering a Network LB implementation that integrates with standard network equipment, so that external services on bare metal clusters also “just work” as much as possible.

MetalLB 的目标,是通过提供与标准网络设备集成的网络负载均衡器实现,来纠正这种不平衡,这样裸机集群上的外部服务也可以尽可能“正常工作”。

Requirements

MetalLB requires the following to function:

  • A Kubernetes cluster, running Kubernetes 1.9.0 or later, that does not already have network load-balancing functionality.
  • A cluster network configuration that can coexist with MetalLB.
  • Some IPv4 addresses for MetalLB to hand out.
  • Depending on the operating mode, you may need one or more routers capable of speaking BGP.

Deploying

Metallb 会在 Kubernetes 内运行,监控服务对象的变化,一旦察觉有新的 LoadBalancer 服务运行,并且没有可申请的负载均衡器之后,就会完成两部分的工作: 1. Metallb 将会在地址池中分配一个地址给服务;2. 根据不同配置,Metallb 会以二层(ARP/NDP)或者 BGP 的方式进行地址的广播。

Installation with Kubernetes manifests

To install MetalLB, simply apply the manifest:

kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/metallb.yaml

This will deploy MetalLB to your cluster, under the metallb-system namespace. The components in the manifest are:

  • The metallb-system/controller deployment. This is the cluster-wide controller that handles IP address assignments.
  • The metallb-system/speaker daemonset. This is the component that speaks the protocol(s) of your choice to make the services reachable.
  • Service accounts for the controller and speaker, along with the RBAC permissions that the components need to function.
[root@ec-c ~]# kubectl get pods -n metallb-system

NAME                          READY   STATUS    RESTARTS   AGE   IP               NODE 
controller-7cc9c87cfb-k8ccm   1/1     Running   0          3h   172.58.184.229   ec-f 
speaker-4nn9x                 1/1     Running   0          3h   172.16.1.64      ec-g 
speaker-vqgkj                 1/1     Running   0          3h   172.16.1.63      ec-f 
speaker-w4dhg                 1/1     Running   0          3h   172.16.1.62      ec-e 
speaker-z4rbg                 1/1     Running   0          3h   172.16.1.61      ec-d
## metallb.yaml 

apiVersion: v1
kind: Namespace
metadata:
  name: metallb-system
  labels:
    app: metallb
---

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: metallb-system
  name: controller
  labels:
    app: metallb
---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: metallb-system
  name: speaker
  labels:
    app: metallb

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metallb-system:controller
  labels:
    app: metallb
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
  resources: ["services/status"]
  verbs: ["update"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["create", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metallb-system:speaker
  labels:
    app: metallb
rules:
- apiGroups: [""]
  resources: ["services", "endpoints", "nodes"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: metallb-system
  name: config-watcher
  labels:
    app: metallb
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["create"]
---

## Role bindings
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metallb-system:controller
  labels:
    app: metallb
subjects:
- kind: ServiceAccount
  name: controller
  namespace: metallb-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: metallb-system:controller
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metallb-system:speaker
  labels:
    app: metallb
subjects:
- kind: ServiceAccount
  name: speaker
  namespace: metallb-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: metallb-system:speaker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: metallb-system
  name: config-watcher
  labels:
    app: metallb
subjects:
- kind: ServiceAccount
  name: controller
- kind: ServiceAccount
  name: speaker
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: config-watcher
---
apiVersion: apps/v1beta2
kind: DaemonSet
metadata:
  namespace: metallb-system
  name: speaker
  labels:
    app: metallb
    component: speaker
spec:
  selector:
    matchLabels:
      app: metallb
      component: speaker
  template:
    metadata:
      labels:
        app: metallb
        component: speaker
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "7472"
    spec:
      serviceAccountName: speaker
      terminationGracePeriodSeconds: 0
      hostNetwork: true
      containers:
      - name: speaker
        image: metallb/speaker:v0.7.3
        imagePullPolicy: IfNotPresent
        args:
        - --port=7472
        - --config=config
        env:
        - name: METALLB_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        ports:
        - name: monitoring
          containerPort: 7472
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
          
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop:
            - all
            add:
            - net_raw

---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  namespace: metallb-system
  name: controller
  labels:
    app: metallb
    component: controller
spec:
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: metallb
      component: controller
  template:
    metadata:
      labels:
        app: metallb
        component: controller
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "7472"
    spec:
      serviceAccountName: controller
      terminationGracePeriodSeconds: 0
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534 # nobody
      containers:
      - name: controller
        image: metallb/controller:v0.7.3
        imagePullPolicy: IfNotPresent
        args:
        - --port=7472
        - --config=config
        ports:
        - name: monitoring
          containerPort: 7472
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
          
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - all
          readOnlyRootFilesystem: true

---

Installation with Helm

Define and Deploy a Configmap

The installation manifest does not include a configuration file. MetalLB’s components will still start, but will remain idle until you define and deploy a configmap.

Layer 2 configuration

Layer 2 mode is the simplest to configure: in many cases, you don’t need any protocol-specific configuration, only IP addresses.

For example, the following configuration gives MetalLB control over IPs from 10.35.1.239 to 10.35.1.250, and configures Layer 2 mode:

## metallb-config.yaml 
---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.35.1.239-10.35.1.250
---
kubectl create -f metallb-config.yaml
BGP configuration

For a basic configuration featuring one BGP router and one IP address range, you need 4 pieces of information:

  • The router IP address that MetalLB should connect to,
  • The router’s AS number,
  • The AS number MetalLB should use,
  • An IP address range expressed as a CIDR prefix.

As an example, if you want to give MetalLB the range 192.168.10.0/24 and AS number 64500, and connect it to a router at 10.0.0.1 with AS number 64501, your configuration will look like:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    peers:
    - peer-address: 10.0.0.1
      peer-asn: 64501
      my-asn: 64500
    address-pools:
    - name: default
      protocol: bgp
      addresses:
      - 192.168.10.0/24
---

Test

## nginx-test.yaml
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1
        ports:
        - name: http
          containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer
---
kubectl create -f nginx-test.yaml 
[root@ec-c ~]# kubectl get pods -o wide
NAME                                  READY   STATUS    RESTARTS   AGE     IP               NODE   NOMINATED NODE   READINESS GATES
nginx-5cd9c7f879-jlgqq                1/1     Running   0          10m     172.58.89.216    ec-g   <none>           <none>
nginx-5cd9c7f879-rxk4r                1/1     Running   0          10m     172.58.184.230   ec-f   <none>           <none>
[root@ec-c ~]# kubectl exec -it nginx-5cd9c7f879-jlgqq bash
root@nginx-5cd9c7f879-jlgqq:/# 
root@nginx-5cd9c7f879-jlgqq:/usr/share/nginx/html# cat -n  index.html 
     1	<!DOCTYPE html>
     2	<html>
     3	<head>
     4	<title>Welcome to nginx!</title>
     5	<style>
     6	    body {
     7	        width: 35em;
     8	        margin: 0 auto;
     9	        font-family: Tahoma, Verdana, Arial, sans-serif;
    10	    }
    11	</style>
    12	</head>
    13	<body>
    14	<h1>Welcome to nginx!</h1>
    15	<p>If you see this page, the nginx web server is successfully installed and
    16	working. Further configuration is required.</p>
    17	
    18	<p>For online documentation and support please refer to
    19	<a href="http://nginx.org/">nginx.org</a>.<br/>
    20	Commercial support is available at
    21	<a href="http://nginx.com/">nginx.com</a>.</p>
    22	
    23	<p><em>Thank you for using nginx.</em></p>
    24	</body>
    25	</html>
# sed -i '14a <h3>on k8s worker-node [ec-g] </h3>' index.html

root@nginx-5cd9c7f879-jlgqq:/usr/share/nginx/html# sed -i '14a <h3>on k8s worker-node [ec-g] </h3>' index.html 
[root@ec-b metallb]# kubectl exec -it nginx-5cd9c7f879-rxk4r bash
root@nginx-5cd9c7f879-rxk4r:/# 
root@nginx-5cd9c7f879-rxk4r:/usr/share/nginx/html# sed -i '14a <h3>on k8s worker-node [ec-f] </h3>' index.html

root@nginx-5cd9c7f879-rxk4r:/usr/share/nginx/html# cat -n index.html 
     1	<!DOCTYPE html>
     2	<html>
     3	<head>
     4	<title>Welcome to nginx!</title>
     5	<style>
     6	    body {
     7	        width: 35em;
     8	        margin: 0 auto;
     9	        font-family: Tahoma, Verdana, Arial, sans-serif;
    10	    }
    11	</style>
    12	</head>
    13	<body>
    14	<h1>Welcome to nginx!</h1>
    15	<h3>on k8s worker-node [ec-f] </h3>
    16	<p>If you see this page, the nginx web server is successfully installed and
    17	working. Further configuration is required.</p>
    18	
    19	<p>For online documentation and support please refer to
    20	<a href="http://nginx.org/">nginx.org</a>.<br/>
    21	Commercial support is available at
    22	<a href="http://nginx.com/">nginx.com</a>.</p>
    23	
    24	<p><em>Thank you for using nginx.</em></p>
    25	</body>
    26	</html>

在浏览器中打开 EXTERNAL-IP

[root@ec-c ~]# kubectl get service
NAME                 TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
nginx                LoadBalancer   10.93.31.61    10.35.1.239   80:30446/TCP   19m

on worker-node [ec-f]
on worker-node [ec-g]

参考资料:
https://metallb.universe.tf/