我们知道JVM在docker容器环境中是无法正确检测到可用内存的,最近正好遇到了一个与之相关的问题,在此记录一下。

遇到问题的项目技术栈为JDK 8 + Spring Boot + Tomcat,部署在docker环境。项目运行过程中出现了java.lang.OutOfMemoryError: Java heap space异常,当时项目的部署文件如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: api-deployment
labels:
app: api
spec:
serviceName: api-app
replicas: 2
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
terminationGracePeriodSeconds: 30
containers:
- image: ...
imagePullPolicy: "Always"
name: api
ports:
- containerPort: 8080
livenessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 300
periodSeconds: 5
readinessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 60
periodSeconds: 5
securityContext:
capabilities:
add:
- SYS_PTRACE
envFrom:
- secretRef:
name: secret

问题应该出在k8s内存设置与JVM的配置这边,网上查询资料后发现tomcat可以通过CATALINA_OPTS环境变量来设置JVM参数,UseCGroupMemoryLimitForHeap 可以让JVM自动检测容器的可用内存,MaxRAMFraction 为容器内存和堆内存的比例,比如容器内存为2G,MaxRAMFraction为2,则最大堆内存为2G/2=1G,这里将MaxRAMFraction设置为2比较安全,设置了这两个参数后,JVM就能通过检测容器的内存来自动调整堆内存大小,不用再显示设置堆内存了。

更新后的配置文件里加了如下代码:

1
2
3
4
5
6
7
...
env:
- name: CATALINA_OPTS
value: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2"
resources:
requests:
memory: "512Mi"

项目运行一段时间后发现问题依旧,研究了下UseCGroupMemoryLimitForHeap参数,发现它是通过读取系统/sys/fs/cgroup/memory/memory.limit_in_bytes文件来检测内存的,登录到容器里查看了下该文件,发现里面是一个很大的值:9223372036854771712,等于没有内存限制,查了下资料发现这个字段是通过k8s文件中的resources->limits的这个属性来配置的,更新文件,加了如下代码:

1
2
limits:
memory: "2048Mi"

观察一段时间后内存就没再溢出,最终完整配置文件如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: api-deployment
labels:
app: api
spec:
serviceName: api-app
replicas: 2
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
terminationGracePeriodSeconds: 30
containers:
- image: ...
imagePullPolicy: "Always"
name: api
ports:
- containerPort: 8080
livenessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 300
periodSeconds: 5
readinessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 60
periodSeconds: 5
securityContext:
capabilities:
add:
- SYS_PTRACE
env:
- name: CATALINA_OPTS
value: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2"
envFrom:
- secretRef:
name: secret
resources:
requests:
memory: "512Mi"
limits:
memory: "2048Mi"

参考