Running your application on virtual machines is now a thing in the past and by now most organizations have either already switched to container-based environments to run their applications or are planning to switch to a containerized environment. This article will attempt to give some insight onto how the Java memory-based aspects should change when one is planning to move their Java application to containers.
Virtual Machine vs Container
The main difference between virtual machine and containers is that unlike in virtual machines where the number of virtual CPUs and virtual memory is completely defined, in containers these resource limits are defined by a Linux kernel feature called cgroups. This is why you see some of the Linux tools which provide system resource metrics such as ‘top’, ‘free’, ‘ps’ give output related to the host system (physical or virtual machine) rather than the container. Outputs of these tools get even more complicated to digest when you run them within a container which is running on a container orchestration platform such as AWS ECS or Kubernetes as in such cases multiple containers might be sharing a set of virtual machines.
Even for Java, the capability to read and understand container-based memory constraints were initially not there; but since Java 10, the support was introduced and the same support has been backported to Java 8 since 8u191. Let’s go into the details in the coming sections.
Java Ergonomics
Before going on to the details of the JDK’s support for running in containers, it is important to have an understanding of the Java ergonomics and some of the defaults related to Java. Please note that this article will be using JDK 11’s (the latest JDK release with long term support as of writing this article) documentation for this information. As per JDK 11 documentation, some of the important default selections are as follows.
- Garbage-First (G1) collector
- The maximum number of GC threads is limited by heap size and available CPU resources
- Initial heap size of 1/64 of physical memory
- Maximum heap size of 1/4 of physical memory
One of the main points to take away from this is the initial and maximum heap size. With the given facts an example would be as follows. If a Java application is running on a container with 1GB memory (with default memory parameters), the initial heap size would be 16MB and maximum allowed heap size would be 256MB. This shows the importance of having to look into memory tuning for JVM.
Java Improvements for Docker Containers
Container Support
As mentioned above, the JVM has been modified (from Java 10 and backported to JDK 8u191) to be aware that it is running in a Docker container and will extract container-specific configuration information instead of querying the operating system. The information being extracted is the number of CPUs and total memory that have been allocated to the container. This support is only available on Linux based platforms and is enabled by default. The relevant JVM flag for this is -XX:-UseContainerSupport.
If you want to check if the container support is enabled, simply run the below command in your container and you should notice the output as bool UseContainerSupport = true {product}
$ java -XX:+PrintFlagsFinal -version | grep UseContainerSupport
bool UseContainerSupport = true {product}
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)
Fine-Grained Control Over Heap Memory
Further to the above container support, JVM now has three new JVM options to allow Docker container users to gain more fine-grained control over the amount of system memory that will be used for the Java Heap as follows.
- -XX:InitialRAMPercentage
- -XX:MaxRAMPercentage
- -XX:MinRAMPercentage
Note that above parameters come in addition to the well known -Xmx and -Xms JVM parameters which can be used to set maximum and minimum heap size respectively. Some more details on the new parameters are as follows. The -XX:InitialRAMPercentage is used to calculate the initial heap size when -Xms is not set and both -XX:MaxRAMPercentage and -XX:MinRAMPercentage are used to calculate the maximumheap size (of course this is a bit counterintuitive) when -Xmxis not set. The exact formula for the calculation of these values is available for reference in OpenJDK source code.
Even though the percentage-based params give more flexibility, due to the room for variability and lack of control that comes with the percentage related parameters, most people argue that setting the heap size explicitly is the preferred option; but of course, this boils down to personal preference. No matter which way you prefer, it is always best to set the memory parameters as per your application’s needs and when it comes to deciding the heap memory you should consider JVM non-heap memory usage, the container memory capacity and auto-scaling rules as well if any.
JVM Internal Memory Usage and Native Memory Tracking
Java heap is the most discussed topic when it comes to Java memory. However, Java heap is not the only section in JVM which consumes memory and there are several parts which are not that much spoken of. However, when you are planning to run your application in a container, you should consider these non-heap JVM memory usage as well; so that it will help you set the memory limits (both container memory limit and heap memory limit) better.
JVM Native Memory Tracking (NMT) is a feature in Java Hotspot VM feature that tracks internal memory usage for a HotSpot JVM. Native Memory Tracking can be used with jcmd to track memory usage at different levels. NMT for Hotspot VM is turned off by default and can be enabled with the following JVM flag -XX:NativeMemoryTracking=[off | summary | detail]. The summary and details are two levels at which data will be collected. You can set this in your Dockerfile as follows alongside any other params. Note that enabling this will cause 5-10% performance overhead.
ENV JAVA_OPTS="-Xmx1024m XX:NativeMemoryTracking=details"
Furthermore, if you are using a container orchestration platform like AWS ECS, you can set it in the task definition under container definition as follows (sample only shows the environment section).
"containerDefinitions": [
{
"environment": [
{
"name": "JAVA_OPTS",
"value": "-Xmx1024m -XX:NativeMemoryTracking=detail "
}
]
}
]
Once NMT is enabled, you can execute jcmd command to get the output as follows.
$ jcmd 901 VM.native_memory summary scale=MB
901:
Native Memory Tracking:
Total: reserved=2013MB, committed=357MB
- Java Heap (reserved=384MB, committed=178MB)
(mmap: reserved=384MB, committed=178MB)
- Class (reserved=1088MB, committed=73MB)
(classes #14927)
( instance classes #14037, array classes #890)
(malloc=2MB #38039)
(mmap: reserved=1086MB, committed=71MB)
( Metadata: )
( reserved=62MB, committed=62MB)
( used=60MB)
( free=1MB)
( waste=0MB =0.00%)
( Class space:)
( reserved=1024MB, committed=9MB)
( used=9MB)
( free=1MB)
( waste=0MB =0.00%)
- Thread (reserved=240MB, committed=25MB)
(thread #238)
(stack: reserved=239MB, committed=24MB)
(malloc=1MB #1430)
- Code (reserved=243MB, committed=23MB)
(malloc=1MB #7348)
(mmap: reserved=242MB, committed=22MB)
- GC (reserved=2MB, committed=1MB)
(mmap: reserved=1MB, committed=1MB)
- Internal (reserved=1MB, committed=1MB)
(malloc=1MB #2883)
- Other (reserved=16MB, committed=16MB)
(malloc=16MB #57)
- Symbol (reserved=17MB, committed=17MB)
(malloc=15MB #181740)
(arena=2MB #1)
- Native Memory Tracking (reserved=4MB, committed=4MB)
(tracking overhead=4MB)
- Shared class space (reserved=17MB, committed=17MB)
(mmap: reserved=17MB, committed=17MB)
Thread Stack Size
A small note on the thread memory on the above output, this is the memory used by your thread stacks (number of threads into thread stack size). On x86 Solaris/Linux the thread stack size is 320k in the 32-bit VM and 1024k in the 64-bit VM. This can be tuned using -Xss property. Thread stack size can be checked using below command.
$ java -XX:+PrintFlagsFinal -version | grep ThreadStackSize
intx CompilerThreadStackSize = 0 {pd product}
intx ThreadStackSize = 1024 {pd product}
intx VMThreadStackSize = 1024 {pd product}
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)
Auto Scaling in Container Orchestration Platforms
Nowadays most of the organizations use container orchestration platforms to run and manage their containerized applications as it gives a lot of additional benefits including auto-scaling capability. Personally, for me, I have been using AWS ECS for some time and I’ll add some info on the some of the auto-scaling related parameters which define the auto-scaling strategy there in ECS below and I’m sure other orchestration platforms will more or less have the same set of parameters.
Number of Tasks
- Desired number of tasks – The value of the desired number of tasks is not constant as it will be dynamically changed by the orchestration platform (AWS ECS in this case) as per the auto-scaling factors.
- The minimum number of tasks – The lower boundary to which Service Auto Scaling can adjust your service’s desired count. Automatic task scaling policies you set cannot reduce the number of tasks below this number.
- The maximum number of tasks – The upper boundary to which Service Auto Scaling can adjust your service’s desired count. Automatic task scaling policies you set cannot increase the number of tasks above this number.
Auto Scaling Triggers
Auto-scaling trigger or policy looks at one of the ECS service metrics and triggers a scale-in and scale-out events based on the configured parameters. In ECS we can have multiple auto-scaling triggers based ECS service metrics such as ECS service average CPU utilization, ECS service average memory utilization and ALB request count per target. In each policy, there are the below set of parameters to tune how the auto-scaling events will trigger.
- Target value – This is the target for the selected metric. For example, if average CPU utilization is picked as the ECS service metric and if we would like to trigger auto-scaling if average CPU usage goes beyond 60%, then the target value would be set to 60.
- Scale-out cooldown time – A scale-out activity increases the number of your service’s tasks. The scale-out cooldown time is a parameter used to prevent excessively scale-out. For example, say the first scale-out event is triggered at time T=1 seconds when average CPU usage is more than 60%. However even though the scale-out event is triggered it might take some time for the new tasks to spawn up, initialize and shart sharing the load. Hence the average CPU usage might stay above 60% for some time for a few more seconds, say till T= 100. Now if we have the scale-out cooldown time as 300 seconds, after the first scale-out event, the orchestration platform will wait till another 300 seconds to pass before triggering another scale-out event.
- Scale-in cooldown time – A scale-in activity reduces the number of your service’s tasks. The scale-in cooldown period is used to block subsequent scale-in requests until it has expired. The intention is to scale in conservatively to protect your application’s availability. However, if another alarm triggers a scale-out activity during the cooldown period after a scale-in, Service Auto Scaling scales out your scalable target immediately.
It is vital that these parameters are set properly as per your application’s needs and performance tests will help you in finding the most suitable values. When it comes to scale-out cooldown time, one point to consider would be your application startup time. For example, say your application startup time is 15 seconds, then you should definitely set the scale-out cooldown time to a higher value than that so that when a scaling- occurs, you give enough time for the new instance to initialize and share of the load.
Conclusion
In summary,
- if you are using JDK 8u191, Java has the capability to understand that it is running on a container. (If you are running on a JDK prior to that you should seriously consider upgrading)
- always set memory limits for your application (either by percentage-based parameters or via explicit parameters which are preferred by many)
- understand both heap and no-heap memory usages of your application
- configure auto-scaling rules as per your application’s needs
Happy containerizing! 🙂
References
- https://en.wikipedia.org/wiki/Cgroups
- https://fabiokung.com/2014/03/13/memory-inside-linux-containers/
- https://docs.oracle.com/en/java/javase/11/gctuning/ergonomics.html#GUID-DB4CAE94-2041-4A16-90EC-6AE3D91EC1F1
- https://www.oracle.com/technetwork/java/javase/8u191-relnotes-5032181.html#JDK-8146115
- https://docs.oracle.com/en/java/javase/11/gctuning/ergonomics.html#GUID-DB4CAE94-2041-4A16-90EC-6AE3D91EC1F1
- https://stackoverflow.com/questions/54292282/clarification-of-meaning-new-jvm-memory-parameters-initialrampercentage-and-minr/54297753#54297753
- http://hg.openjdk.java.net/jdk-updates/jdk11u/file/a7f53869e42b/src/hotspot/share/runtime/arguments.cpp#l1750
- https://docs.oracle.com/javase/8/docs/technotes/guides/vm/nmt-8.html
- https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr006.html
- https://www.oracle.com/technetwork/java/hotspotfaq-138619.html#threads_oom
~ Rajind Ruparathna
Featured image credits: Markus Distelrath from Pixabay
Notebook 