Vote count:
0
All,
We have a configuration where behind the Load Balancer (LB) there are two web server running Apache. Each Web Server connects to a dedicated application server running Jboss 6.1.
From the LB we took off web server 1. That essentially took out app server 1. We stopped the Jboss instance in App Server 1, installed new version of our application and restarted it. Then we put web server 1 back on the LB. At this point we saw some CPU spikes on app server 1 that lasted about 30 min.
We followed the same process for web server 2/app server 2. When we put web server2 back onto the LB we noticed a huge spike on app server 2 which lasted about 2 hours.
I took some heap dump of app server 2 after putting web server 2 back to LB. Following are some logs for thread.
Any clue why the CPU spike is happening in app server 2?
Thank you all
"ajp-0.0.0.0-8009-116" daemon prio=5 tid=252 RUNNABLE
"ajp-0.0.0.0-8009-94" daemon prio=5 tid=230 RUNNABLE
"Finalizer" daemon prio=8 tid=3 WAITING
"ajp-0.0.0.0-8009-248" daemon prio=5 tid=413 RUNNABLE
"ajp-0.0.0.0-8009-189" daemon prio=5 tid=354 RUNNABLE
"ContainerBackgroundProcessor[StandardEngine[jboss.web]]" daemon prio=5 tid=51 TIMED_WAITING
"ajp-0.0.0.0-8009-192" daemon prio=5 tid=357 BLOCKED
"ajp-0.0.0.0-8009-24" daemon prio=5 tid=130 RUNNABLE
"ajp-0.0.0.0-8009-96" daemon prio=5 tid=232 RUNNABLE
"ajp-0.0.0.0-8009-69" daemon prio=5 tid=202 BLOCKED
"ajp-0.0.0.0-8009-237" daemon prio=5 tid=402 RUNNABLE
"Transaction Reaper" daemon prio=5 tid=49 TIMED_WAITING
"ajp-0.0.0.0-8009-229" daemon prio=5 tid=394 RUNNABLE
"ajp-0.0.0.0-8009-98" daemon prio=5 tid=234 RUNNABLE
"ajp-0.0.0.0-8009-66" daemon prio=5 tid=199 RUNNABLE
"ajp-0.0.0.0-8009-87" daemon prio=5 tid=223 RUNNABLE
"pool-8-thread-1" prio=5 tid=38 TIMED_WAITING
"HSQLDB Timer @425a6bfe" daemon prio=5 tid=65 TIMED_WAITING
Unknown thread "ajp-0.0.0.0-8009-225" daemon prio=5 tid=390 WAITING
"ajp-0.0.0.0-8009-122" daemon prio=5 tid=258 RUNNABLE
"AuthenticationCacheFlushThread" prio=5 tid=14 TIMED_WAITING
"ajp-0.0.0.0-8009-161" daemon prio=5 tid=326 RUNNABLE
"ajp-0.0.0.0-8009-136" daemon prio=5 tid=301 RUNNABLE
"RMI TCP Accept-1098" daemon prio=5 tid=23 RUNNABLE
"ajp-0.0.0.0-8009-203" daemon prio=5 tid=368 RUNNABLE
"ajp-0.0.0.0-8009-182" daemon prio=5 tid=347 RUNNABLE
"ajp-0.0.0.0-8009-85" daemon prio=5 tid=221 RUNNABLE
"ServerSocketRefresh" daemon prio=5 tid=18 WAITING
"ajp-0.0.0.0-8009-148" daemon prio=5 tid=313 RUNNABLE
"ajp-0.0.0.0-8009-99" daemon prio=5 tid=235 RUNNABLE
"ajp-0.0.0.0-8009-208" daemon prio=5 tid=373 RUNNABLE
"ajp-0.0.0.0-8009-210" daemon prio=5 tid=375 RUNNABLE
"ajp-0.0.0.0-8009-131" daemon prio=5 tid=296 RUNNABLE
"ajp-0.0.0.0-8009-Acceptor-0" daemon prio=5 tid=81 RUNNABLE
"ajp-0.0.0.0-8009-104" daemon prio=5 tid=240 RUNNABLE
"JBossQuartzScheduler_Worker-6" prio=5 tid=58 TIMED_WAITING
"ajp-0.0.0.0-8009-173" daemon prio=5 tid=338 RUNNABLE
"ajp-0.0.0.0-8009-218" daemon prio=5 tid=383 WAITING
"ajp-0.0.0.0-8009-48" daemon prio=5 tid=181 BLOCKED
"ajp-0.0.0.0-8009-263" daemon prio=5 tid=428 BLOCKED
"JBossQuartzScheduler_Worker-1" prio=5 tid=53 TIMED_WAITING
"ajp-0.0.0.0-8009-4" daemon prio=5 tid=110 RUNNABLE
"ajp-0.0.0.0-8009-177" daemon prio=5 tid=342 RUNNABLE
"Low Memory Detector" daemon prio=9 tid=7 RUNNABLE
"ajp-0.0.0.0-8009-262" daemon prio=5 tid=427 RUNNABLE
"ajp-0.0.0.0-8009-35" daemon prio=5 tid=156 RUNNABLE
"ajp-0.0.0.0-8009-141" daemon prio=5 tid=306 RUNNABLE
"http-0.0.0.0-8080-1" daemon prio=5 tid=82 WAITING
"ajp-0.0.0.0-8009-206" daemon prio=5 tid=371 RUNNABLE
"ajp-0.0.0.0-8009-81" daemon prio=5 tid=217 RUNNABLE
"ajp-0.0.0.0-8009-111" daemon prio=5 tid=247 RUNNABLE
Unknown thread "ajp-0.0.0.0-8009-106" daemon prio=5 tid=242 RUNNABLE
"Thread-0 (group:HornetQ-scheduled-threads-2074861536)" prio=5 tid=30 TIMED_WAITING
"ajp-0.0.0.0-8009-265" daemon prio=5 tid=430 RUNNABLE
"GC Daemon" daemon prio=2 tid=16 TIMED_WAITING
"ajp-0.0.0.0-8009-193" daemon prio=5 tid=358 RUNNABLE
"ajp-0.0.0.0-8009-25" daemon prio=5 tid=131 WAITING
"ajp-0.0.0.0-8009-39" daemon prio=5 tid=164 RUNNABLE
"ajp-0.0.0.0-8009-249" daemon prio=5 tid=414 RUNNABLE
"ajp-0.0.0.0-8009-132" daemon prio=5 tid=297 RUNNABLE
"Thread-3 (group:HornetQ-scheduled-threads-2074861536)" prio=5 tid=41 TIMED_WAITING
"ajp-0.0.0.0-8009-235" daemon prio=5 tid=400 RUNNABLE
"ajp-0.0.0.0-8009-78" daemon prio=5 tid=214 RUNNABLE
"DestroyJavaVM" prio=5 tid=11 RUNNABLE
"JBossQuartzScheduler_Worker-4" prio=5 tid=56 TIMED_WAITING
"ajp-0.0.0.0-8009-149" daemon prio=5 tid=314 RUNNABLE
"Old I/O server boss (channelId: 319085712, /0.0.0.0:5455)" prio=5 tid=40 RUNNABLE
"myScheduler-1" prio=5 tid=76 TIMED_WAITING
"ajp-0.0.0.0-8009-75" daemon prio=5 tid=211 RUNNABLE
"xnio-pool4-1" prio=5 tid=67 RUNNABLE
"ajp-0.0.0.0-8009-86" daemon prio=5 tid=222 RUNNABLE
"ajp-0.0.0.0-8009-115" daemon prio=5 tid=251 WAITING
"ajp-0.0.0.0-8009-162" daemon prio=5 tid=327 RUNNABLE
"ajp-0.0.0.0-8009-3" daemon prio=5 tid=109 RUNNABLE
"ajp-0.0.0.0-8009-217" daemon prio=5 tid=382 WAITING
"Thread-0 (group:HornetQ-Asynchronous-Persistent-Writes1010838859-1593335720)" daemon prio=5 tid=34 WAITING
"ajp-0.0.0.0-8009-183" daemon prio=5 tid=348 RUNNABLE
"ajp-0.0.0.0-8009-95" daemon prio=5 tid=231 RUNNABLE
"ajp-0.0.0.0-8009-188" daemon prio=5 tid=353 RUNNABLE
"ajp-0.0.0.0-8009-62" daemon prio=5 tid=195 RUNNABLE
"ajp-0.0.0.0-8009-174" daemon prio=5 tid=339 RUNNABLE
"ajp-0.0.0.0-8009-108" daemon prio=5 tid=244 RUNNABLE
"ajp-0.0.0.0-8009-209" daemon prio=5 tid=374 WAITING
"ajp-0.0.0.0-8009-130" daemon prio=5 tid=295 RUNNABLE
"ajp-0.0.0.0-8009-233" daemon prio=5 tid=398 RUNNABLE
"ajp-0.0.0.0-8009-271" daemon prio=5 tid=436 RUNNABLE
"Listener:4712" daemon prio=5 tid=46 RUNNABLE
"Thread-1 (group:HornetQ-server-threads1511239544-1522196272)" prio=5 tid=36 WAITING
"ajp-0.0.0.0-8009-169" daemon prio=5 tid=334 RUNNABLE
"ajp-0.0.0.0-8009-135" daemon prio=5 tid=300 RUNNABLE
"ajp-0.0.0.0-8009-227" daemon prio=5 tid=392 WAITING
"ajp-0.0.0.0-8009-107" daemon prio=5 tid=243 BLOCKED
"ajp-0.0.0.0-8009-178" daemon prio=5 tid=343 WAITING
"ajp-0.0.0.0-8009-171" daemon prio=5 tid=336 RUNNABLE
"ajp-0.0.0.0-8009-36" daemon prio=5 tid=157 RUNNABLE
"JBossQuartzScheduler_QuartzSchedulerThread" prio=5 tid=63 TIMED_WAITING
"ajp-0.0.0.0-8009-31" daemon prio=5 tid=137 RUNNABLE
"ajp-0.0.0.0-8009-211" daemon prio=5 tid=376 WAITING
"ajp-0.0.0.0-8009-216" daemon prio=5 tid=381 RUNNABLE
"AcceptorThread[ServerSocket[addr=/0.0.0.0,port=0,localport=3873]]" prio=5 tid=19 RUNNABLE
"ajp-0.0.0.0-8009-239" daemon prio=5 tid=404 WAITING
"ajp-0.0.0.0-8009-109" daemon prio=5 tid=245 RUNNABLE
"ajp-0.0.0.0-8009-80" daemon prio=5 tid=216 WAITING
"ajp-0.0.0.0-8009-140" daemon prio=5 tid=305 RUNNABLE
"ajp-0.0.0.0-8009-47" daemon prio=5 tid=180 RUNNABLE
"ajp-0.0.0.0-8009-207" daemon prio=5 tid=372 WAITING
Unknown thread "ajp-0.0.0.0-8009-133" daemon prio=5 tid=298 RUNNABLE
"JBossQuartzScheduler_Worker-8" prio=5 tid=60 TIMED_WAITING
"ajp-0.0.0.0-8009-26" daemon prio=5 tid=132 RUNNABLE
"RMI TCP Accept-1090" daemon prio=5 tid=20 RUNNABLE
"ajp-0.0.0.0-8009-251" daemon prio=5 tid=416 RUNNABLE
"ajp-0.0.0.0-8009-163" daemon prio=5 tid=328 RUNNABLE
"Old I/O server boss (channelId: 802289754, /0.0.0.0:5445)" prio=5 tid=39 RUNNABLE
"ajp-0.0.0.0-8009-232" daemon prio=5 tid=397 RUNNABLE
"ajp-0.0.0.0-8009-76" daemon prio=5 tid=212 RUNNABLE
"Transaction Reaper Worker 0" daemon prio=5 tid=50 WAITING
"ajp-0.0.0.0-8009-63" daemon prio=5 tid=196 RUNNABLE
"ajp-0.0.0.0-8009-215" daemon prio=5 tid=380 WAITING
"ajp-0.0.0.0-8009-100" daemon prio=5 tid=236 RUNNABLE
"C2 CompilerThread1" daemon prio=9 tid=6 RUNNABLE
"ajp-0.0.0.0-8009-185" daemon prio=5 tid=350 WAITING
"ajp-0.0.0.0-8009-71" daemon prio=5 tid=207 RUNNABLE
"ajp-0.0.0.0-8009-219" daemon prio=5 tid=384 RUNNABLE
"ajp-0.0.0.0-8009-153" daemon prio=5 tid=318 RUNNABLE
"ajp-0.0.0.0-8009-32" daemon prio=5 tid=138 RUNNABLE
"ajp-0.0.0.0-8009-245" daemon prio=5 tid=410 RUNNABLE
"xnio-pool4-3" prio=5 tid=69 RUNNABLE
"ajp-0.0.0.0-8009-194" daemon prio=5 tid=359 RUNNABLE
Unknown thread "ajp-0.0.0.0-8009-201" daemon prio=5 tid=366 WAITING
"ajp-0.0.0.0-8009-261" daemon prio=5 tid=426 RUNNABLE
"hornetq-expiry-reaper-thread" prio=3 tid=29 TIMED_WAITING
"ajp-0.0.0.0-8009-120" daemon prio=5 tid=256 RUNNABLE
"ajp-0.0.0.0-8009-155" daemon prio=5 tid=320 WAITING
"ajp-0.0.0.0-8009-247" daemon prio=5 tid=412 RUNNABLE
"ajp-0.0.0.0-8009-146" daemon prio=5 tid=311 RUNNABLE
"ServerSocketRefresh" daemon prio=5 tid=25 WAITING
"ajp-0.0.0.0-8009-257" daemon prio=5 tid=422 WAITING
"HDScanner" prio=5 tid=77 TIMED_WAITING
"ajp-0.0.0.0-8009-234" daemon prio=5 tid=399 WAITING
"hornetq-buffer-timeout" prio=5 tid=27 WAITING
"ajp-0.0.0.0-8009-117" daemon prio=5 tid=253 RUNNABLE
"hornetq-failure-check-thread" prio=5 tid=42 TIMED_WAITING
"ajp-0.0.0.0-8009-191" daemon prio=5 tid=356 RUNNABLE
"ajp-0.0.0.0-8009-164" daemon prio=5 tid=329 RUNNABLE
"ajp-0.0.0.0-8009-121" daemon prio=5 tid=257 RUNNABLE
"ajp-0.0.0.0-8009-172" daemon prio=5 tid=337 BLOCKED
"ajp-0.0.0.0-8009-246" daemon prio=5 tid=411 RUNNABLE
"ajp-0.0.0.0-8009-238" daemon prio=5 tid=403 WAITING
"ajp-0.0.0.0-8009-154" daemon prio=5 tid=319 RUNNABLE
"ajp-0.0.0.0-8009-70" daemon prio=5 tid=203 RUNNABLE
"ajp-0.0.0.0-8009-93" daemon prio=5 tid=229 BLOCKED
"ajp-0.0.0.0-8009-184" daemon prio=5 tid=349 RUNNABLE
"ajp-0.0.0.0-8009-270" daemon prio=5 tid=435 RUNNABLE
"Listener:4713" daemon prio=5 tid=47 RUNNABLE
"RMI Reaper" prio=5 tid=22 WAITING
"ajp-0.0.0.0-8009-226" daemon prio=5 tid=391 WAITING
"ajp-0.0.0.0-8009-34" daemon prio=5 tid=152 RUNNABLE
"ajp-0.0.0.0-8009-258" daemon prio=5 tid=423 WAITING
"ajp-0.0.0.0-8009-10" daemon prio=5 tid=116 WAITING
"pool-2-thread-1" prio=5 tid=43 RUNNABLE
"ajp-0.0.0.0-8009-220" daemon prio=5 tid=385 WAITING
"RMI Scheduler(0)" daemon prio=5 tid=84 TIMED_WAITING
"ajp-0.0.0.0-8009-33" daemon prio=5 tid=139 WAITING
"ajp-0.0.0.0-8009-77" daemon prio=5 tid=213 RUNNABLE
"ajp-0.0.0.0-8009-264" daemon prio=5 tid=429 RUNNABLE
"JBossQuartzScheduler_Worker-10" prio=5 tid=62 TIMED_WAITING
"ajp-0.0.0.0-8009-92" daemon prio=5 tid=228 RUNNABLE
"ajp-0.0.0.0-8009-202" daemon prio=5 tid=367 RUNNABLE
"Thread-298" daemon prio=5 tid=502 WAITING
"xnio-pool4-5" prio=5 tid=71 RUNNABLE
"MultiThreadedHttpConnectionManager cleanup" daemon prio=5 tid=682 WAITING
"ajp-0.0.0.0-8009-134" daemon prio=5 tid=299 RUNNABLE
"pool-1-thread-1" prio=5 tid=52 WAITING
"ajp-0.0.0.0-8009-156" daemon prio=5 tid=321 BLOCKED
"ajp-0.0.0.0-8009-84" daemon prio=5 tid=220 RUNNABLE
"ajp-0.0.0.0-8009-147" daemon prio=5 tid=312 RUNNABLE
"ajp-0.0.0.0-8009-37" daemon prio=5 tid=158 WAITING
"ajp-0.0.0.0-8009-72" daemon prio=5 tid=208 RUNNABLE
"ajp-0.0.0.0-8009-103" daemon prio=5 tid=239 RUNNABLE
"ajp-0.0.0.0-8009-204" daemon prio=5 tid=369 RUNNABLE
"ajp-0.0.0.0-8009-252" daemon prio=5 tid=417 RUNNABLE
"http-0.0.0.0-8080-3" daemon prio=5 tid=100 WAITING
Aucun commentaire:
Enregistrer un commentaire