Yourkit remote profiling with docker
Yourkit is the very commonly used profiler across performance teams for all java related profiling. Where the primary usage of this profiler is to profile the usecase on your local machine, there are situations where you need profile remotely running java severs. It becomes even difficult when the server is run inside docker container, because most of the times these containers are configured not to expose any port other than the designated ports like 8080
In such situation, the option is to profile the application on the docker container itself, and copy the snapshot onto local machine to analyze. Thankfully you do not need a license to profile an application, but you need license just to view/analyze the results, so you are saved from registering headache. In this document, we will see how to inflate yourkit onto docker container and get the snapshot.
I have used this frequently on XO containers, so I will use container names from XO.
first of all we need to find the container where the application is hosted.use docker ps command to get it.
docker -H <host> ps -fname=machine
This will give an output something like
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
c892579bed1f registry2.swarm.devfactory.com/crossover/xo-tomcat:95 "/scripts/docker-ent…" About an hour ago Up About an hour 10.69.11.214:20314->8080/tcp
From the output, you can easily find that the container id c892579bed1f is the one we need to use for further steps.
Once we have the docker containerid, the next step is to get into the container bash. we will use docker exec command to do the same.
docker -H dl6.<docker_host>.com exec -it c892579bed1f
this will open bash console to the container.
Next step is to download and inflate the yourkit binaries into the container. Run below command with appropriate binary url inside docker bash to do the same.
root@machine:/usr/local/tomcat# wget https://www.yourkit.com/download/YourKit-JavaProfiler-2018.04-b82.zip -P /tmp/ &&
unzip /tmp/YourKit-JavaProfiler-2018.04-b82.zip -d /usr/local &&
rm /tmp/YourKit-JavaProfiler-2018.04-b82.zip
This command will download the yourkit binaries on the machine, inflate the same to /usr/local directory and remove the binary downloaded to free up the space.
generates an output something like below
--2018-09-10 12:44:58-- https://www.yourkit.com/download/YourKit-JavaProfiler-2018.04-b82.zip
Resolving www.yourkit.com (www.yourkit.com)... 144.76.223.241
Connecting to www.yourkit.com (www.yourkit.com)|144.76.223.241|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 70990617 (68M) [application/zip]
Saving to: ‘/tmp/YourKit-JavaProfiler-2018.04-b82.zip’
YourKit-JavaProfiler-2018.04- 100%[=================================================>] 67.70M 717KB/s in 1m 52s
2018-09-10 12:46:51 (618 KB/s) - ‘/tmp/YourKit-JavaProfiler-2018.04-b82.zip’ saved [70990617/70990617]
Archive: /tmp/YourKit-JavaProfiler-2018.04-b82.zip
.
.
.
inflating: /usr/local/YourKit-JavaProfiler-2018.04/jre64/release
The next step is to attach the yourkit profiler agent to running JVM, you need to know the processid of running JVM, usually its 1 when its tomcat-only container.
Use attach.sh in <Yourkit installation directory>/bin
root@machine:/usr/local/tomcat# /usr/local/YourKit-JavaProfiler-2018.04/bin/attach.sh
Watch the output below, it shows all processes, and asks which pid you want to use, I entered 1 for pid, and used default port with just pressing Enter. You can also pass PID as first parameter
[YourKit Java Profiler 2018.04-b82] Log file: /root/.yjp/log/profiler-ui-362.log
Running JVMs:
Name | PID | Status |
---|---|---|
Tomcat | 1 | Profiler agent is not loaded. Ready for attach. |
Enter PID of the application you want to attach (0 to exit) and press Enter:
Not a number. Please re-enter.
1
Please specify comma-separated list of startup options, or press Enter for default options (recommended):
<<Enter>>
root@machine:/usr/local/tomcat#
Start profiling
Now our agent is attached to the process, and we need to start profiling. The utility yjp-controller-api-redist.jaris available in <Yourkit installation directory>/lib, because we will be using this utility frequently, lets change working dir. before invoking this utility.
root@machine:/usr/local/tomcat# cd /usr/local/YourKit-JavaProfiler-2018.04/lib
root@machine:/usr/local/YourKit-JavaProfiler-2018.04/lib# java -jar yjp-controller-api-redist.jar localhost 10001 start-cpu-sampling
Command yjp-controller-api-redist.jar has several options as listed below
Profling can be done in sampling mode or tracing mode. issue yjp-controller-api-redist.jar without any options
java -jar yjp-controller-api-redist.jar
YourKit Java Profiler 2018.04-b82 command line tools
Usage: java -jar yjp-controller-api-redist.jar <host> <port> <command>
where <command> is one of:
status
capture-memory-snapshot
capture-hprof-snapshot
capture-performance-snapshot
start-cpu-sampling
start-cpu-tracing
start-cpu-call-counting
stop-cpu-profiling
clear-cpu-data
start-alloc-recording-all
// record all objects
start-alloc-recording-adaptive [alloc-sampled]
// record all objects with size >= 4 KB, and only each 10th smaller object
stop-alloc-recording
clear-alloc-data
start-monitor-profiling
stop-monitor-profiling
clear-monitor-data
enable-stack-telemetry
disable-stack-telemetry
force-gc
clear-charts
Examples:
java -jar yjp-controller-api-redist.jar localhost 10001 capture-memory-snapshot
java -jar yjp-controller-api-redist.jar localhost 10001 start-cpu-sampling
Once the profiling is started, clean cpu data before executing your usecase
root@machine:/usr/local/YourKit-JavaProfiler-2018.04/lib# java -jar yjp-controller-api-redist.jar localhost 10001 clear-cpu-data
Now Execute your use-case and create snapshot
root@machine:/usr/local/YourKit-JavaProfiler-2018.04/lib# java -jar yjp-controller-api-redist.jar localhost 10001 capture-performance-snapshot
Snapshot captured: /root/Snapshots/Tomcat-2018-09-10.snapshot
root@machine:/usr/local/YourKit-JavaProfiler-2018.04/lib# java -jar yjp-controller-api-redist.jar localhost 10001 stop-cpu-profiling
CPU profiling stopped
You will see in the output that the snapshot is created at this location - /root/Snapshots/Tomcat-2018-09-10.snapshot
Copy snapshot
Use docker cp to copy the snapshot to your local machine - remember we have noted container id, we will have to use the same to copy. You can issue this command from separate shell to avoid exiting from current ptofiling session.
docker cp -H dl6.<docker_host>.com c892579bed1f:/root/Snapshots/Tomcat-2018-09-10.snapshot F:\yktsnaphosts
This will copy the snapshot to F:\yktsnaphosts\Tomcat-2018-09-10.snapshot, which you can open with your licensed yourkit version to analyze.
All Commands together
list docker containers
docker -H <docker-host> ps -fname=<container_name>
connect to docker container shell
docker -H <docker-host> exec -it <container-id> bash
download and inflate yourkit
wget https://www.yourkit.com/download/YourKit-JavaProfiler-2018.04-b82.zip -P /tmp/ &&
unzip /tmp/YourKit-JavaProfiler-2018.04-b82.zip -d /usr/local &&
rm /tmp/YourKit-JavaProfiler-2018.04-b82.zip
prepare for profiling
/usr/local/YourKit-JavaProfiler-2018.04/bin/attach.sh
cd /usr/local/YourKit-JavaProfiler-2018.04/lib
java -jar yjp-controller-api-redist.jar localhost 10001 start-cpu-sampling
clear cpu data
java -jar yjp-controller-api-redist.jar localhost 10001 clear-cpu-data
Do your activity
create new snapshot
java -jar yjp-controller-api-redist.jar localhost 10001 capture-performance-snapshot
Stop your profiling once you are done.
java -jar yjp-controller-api-redist.jar localhost 10001 stop-cpu-profiling
Capture snapshot (Execute this from your local machine)
docker cp -H <docker_host> <container_id>:/root/Snapshots/Tomcat-2018-09-10.snapshot <location_on_local_machine>
exit from docker shell
exit