The application is running on Tomcat 5 and the IBM JVM and peaks at about 220-250 calls per minute.  The hardware is an IBM P6 4-core 4.2 GHz machine with 32GB of RAM.  The hardware is fully dedicated to the application.

Hardware CPU Load Before Tuning

For 18 months the application performance has been slow, and users are complaining of very poor response times.  The system and application administrators notice that most days the CPU load is very high and several times a day the load is up over 90%.

Surprisingly the JVM has not been configured with verbose GC logging enabled and we have no other insight into how the JVM is performing.  Before making any changes it was decided that two things needed to happen first: the JVM needs to be configured with verbose GC logging enabled, and the JVM needs to be configured with the IBM Health Center agent (only available for use with the IBM JVM).

It didn’t take much analysis to notice right away that the 4GB heap was much too small for the application workload.  With a 4GB heap, after a collection the heap occupancy was still over 80%.  And the JVM was spending more than 8% of time in garbage collection.  We also discovered that every hour the application was issuing a System.gc() call.  This was incredibly disruptive to the application performance and the automatic garbage collection.  The developers could never pinpoint the source of these calls so we configured the JVM to ignore the System.gc() calls (-Xdisableexplicitgc).

We resized the heap to 6GB and found that it was also too small.  Eventually we settled on an 8GB heap, although I suspect that it might be too large.  At this point application response time was much improved and we shaved about 4% from the CPU load.

But we did notice that with the 8GB heap the garbage collection time was still about 4%.  And we observed paused times up to 5.5 seconds, the average pause time was about 2.2 seconds.  We thought we could improve on this too.

The JVM was configured with the IBM default garbage collection policy Optthruput.  This policy is a stop-the-world free-list flat heap policy and is not well suited for heap sizes over 4GB.  Our IBM garbage collection policy research lead us to choose the Gencon policy.  The Gencon policy is a generational copying collection policy. This policy combines small pauses with excellent throughput.

Optthruput Policy CPU Load

After making the Gencon garbage collection policy change we saw a radical improvement in the pause times and time spent in garbage collection.  The average pause time now was about 600 ms, the longest observed pause time was less than 2 seconds.  And remarkably the portion of time spent in garbage collection was now .84%, down from 4% (with Optthruput policy), and down from the original 8%!

And the CPU load average was now down under 10%; much better than the 15% (with Optthruput policy).  And even better than the original CPU average of over 20%.  We had another interesting observation too, before making any tuning changes the CPU load high was routinely 100%.  With the changes we made it was now between 18-20%.

Gencon Policy CPU Load

It’s incredible how the wrong heap size and GC settings can be so damaging to application performance.  And what I think is so incredible about this situation was how the application pushed the hardware so hard.

We also visited the Notre-Dame-de-Bon-Secours Chapel which is one of the oldest churches in Montreal. The chapel is over 300 years old. I really enjoyed this tour, maybe even more than the Notre-Dame Basilica. You feel much more connected to the history in this chapel. Climbing the old narrow and creaky stairs to tower of the chapel is a lot of fun, and a little scary. Of course the views of Old Montreal at the top are worth the climb. The tour also includes museum and a crypt with archaeological materials.

Walking along the Quays is also a lot of fun. There a lot family activities here too. We liked to spend our evenings here people watching!

Hiking up Mount Royal and visiting Mount Royal Park is a very enjoyable way to spend part of a day. The hike is easy and the park is very nice. The views of downtown Montreal from the Kondiaronk Belvedere are fantastic. It’s a great way to appreciate the city. We hiked back down and walked through the Plateau Mont-Royal. I believe this area is one of the more popular neighborhoods in Montreal and I can see why! The residential streets are lined with homes that make people think of Montreal. The streets are packed with people, restaurants, shops, and bars.

Of course the food and restaurants in Montreal are first rate. In the mornings we liked to go Espace Cafe and Espresso Bar, it was close to Place d’Armes and the coffee and pastries were like nothing I’ve tasted before. It was never overly crowded and we could take our time and enjoy the coffee. The service was also outstanding, they make you feel very comfortable!

Dinner at Verses was quite good and the atmosphere and pace is very relaxed. The staff was friendly and their wine and food recommendations were excellent. The Tartare de Bison fumé is very good, but I’m not sure I would try it again. The Suprême de volaille bio was excellent, the organic chicken breast was very moist and flavorful. I would recommend Verses.

Olive + Gourmudo is a really fun place for lunch and the sandwiches are very good. I would bet this place is packed for lunch on most days and it was on the day we were there too. We tried the Cubain and the Gustavo hot paninis and they were both good. This place is a little crazy so don’t expect to take a leisurely lunch here.

• Replaced the factory pedals with Look KeO Classic pedals
• Replaced the factory tires with Michelin Pro Optimum tires
• Replaced the factory chain with a Shimano 105 asymmetrical designed chain
• Removed the brake levers on the top bar

Felt Z100 Front End

In June I rode in the Elephant Rock ride in Castle Rock. This was my first organized ride and I thought it was a lot of fun, the course was very enjoyable. I completed the 62 mile course in just under four hours and twenty minutes. I think if I were to do it again I could get the time well under four hours. In the ride I thought the bike was very fast on the descents. I noticed that I was often passing heavier riders with ease. Maybe it’s just my riding style!

Felt Z100 Max Speed (44.7 mph)

The Michelin Pro Optimum tires have been outstanding. I’ve had no flats (yet!), and the wear seems very good. The tires have been over miles of rough Colorado roads with a lot of debris and they’ve done well.

I’ve also been pleased with the Look KeO pedals. I didn’t like the “grip” cleats that shipped with the pedals. The cleats wore very quickly and I thought they were more difficult to get into the pedals, especially compared to the bi-Material Teflon cleats.

The bike has been serviced twice by Full Cycle, at 250 and 1, 000 miles. I’m not sure I was fully satisfied with either job. The shop seems to be in such a rush to turn the bike around that I think they skip the details. The last time the bike was returned with dirt lodged in the rear cassette and the wash job was sloppy. And I don’t think they spend enough time adjusting the derailleurs either. I think I will find another shop, there are plenty of them in Colorado!

I don’t think I’ll change much more with the bike in the future. And I’d have to say that the Z100 was worth the purchase price. I would recommend it to anyone looking for an entry level road bike.

I won’t be creating new posts about LMS operations or technologies or probably much about IT in higher education.  While I’ll miss working in this area it’s on to new challenges.

You can also use a Garbage Collection and Memory Visualizer tool in the Support Assistant. I thought I would use this tool to analyze some of our HotSpot JVM verbose GC logs. I gathered a week worth of logs from two of our WebLogic managed nodes and our JMS node.

The tool worked quickly on the logs and produced some interesting results. The visualization options appear to be a bit limited when working on HotSpot logs but you can visualize: free heap (before and after collection), heap size, used heap (before and after collection), amount freed, and the pause times.

Pause times for managed node JVM

Heap usage for managed node JVM

My impressions after looking at the visualization:

- garbage collection on all three nodes was very efficient
- used heap never exceed more than 1.37GB
- the managed node JVMs performed 3 full collections with pause times around 30 secs
- mean garbage collection pause time was about 60 ms

The tool also offers a Tuning recommendation that I didn’t think was very applicable, maybe it would produce better advice for the IBM JDK.

Of course the first thing I noticed was that membership events represented the largest group of processed events. Group events was the smallest. Looking at the events by type and the date they were processed revealed some interesting trends too.

Group events were very heavy in the beginning of the year and for some unknown reason they were heaviest before the Fall 2010 term was activated in Banner. The other busy periods appear to be in May, August and the end of the year.

Membership events seem to closely mirror the group events except for the activity in March. This makes sense, as instructors request Banner sections (group events) to be taught in Blackboard the instructor and student enrollment (membership events) follow. The busiest periods for membership events was the begging and end of the year.

Person events seemed to be busiest in the beginning of the year. After the end of March person events were steady with a few exceptions. For yet another unknown reason in the middle of September Banner delivered 73,000 person events. My guess is this was the result of some type of data maintenance process but I’m not sure.

After a term becomes active in Banner (open for instructors to request term sections be taught in Bb) it doesn’t appear that instructors move quickly to request sections for the term. I thought maybe there would be a spike in group and membership events after a term was activated. But that’s not the case at all. Instructors seem to make the requests when it’s convient for them.

Unfortunately I can’t embed the Tableau viz in this blog, but it can be viewed here.

2010 has been a rad year for our LMS operations team. It was also a busy year. The single biggest project of the year was moving all of the hardware (two full racks) to a new location in the data center. We also re-aligned hardware in the racks to create a production rack and test rack. The move was a massive team effort and it was a huge success with no problems!

On the front end we expanded the capacity to serve 9000 concurrent users. We hope the additional capacity brings the application to the end of it’s life in 2012. An interesting note on capacity, this is our Fall finals week and yesterday we had a peak of 7600 concurrent users! We had been planning for a peak of 7400.

Several architecture improvements to eliminate single points of failure and improve user session replication were also completed. Next year I would imagine we’ll make virtually no changes to the architecture.

We also successfully completed one major application upgrade and installed 3 minor application patches with almost no issues. Although we did learn a hard lesson about making too many changes at one time.

We had one minor unscheduled outage of 135 minutes (due to a bad fiber switch). A large part of the outage time can be attributed to the slow startup of the WebLogic cluster. We suffered no data loss in the outage. This puts our MTBF at more than 300 days!

And our uptime for the year is 99.97% (our best in 3 years)!

Personal Success:

I’ve made significant progress in my ability to lead and manage team meetings. It was a bit of a painful process but I’m glad I put the effort into it.

I was fortunate enough to author several important documents this year (although sometimes I wonder who reads them)! I learned in doing this that I enjoy writing and I’m working to improve in this area too.

And I’ve continued to improve my organization and communication styles. I think this is going to be critical to the success of next year as a lot of change is coming!

More information about SunGard LIS 2.0.

UPDATE:
John Fontaine at Blackboard has more information about the progress Blackboard is making on supporting LIS. It continues to sound very promising and exciting!

Theo Schlossnagle’s chapter (Web Operations: The Career) serves as a good introduction to web operations as a career. In this chapter Theo looks at the very broad and undefined career path for web operations engineers. Desirable qualities like a strong background in computing, practiced decisiveness, a calm disposition, and experience are all examined. I think anyone who may be considering a career in any IT operations group would benefit from reading this chapter.

Chapter three (Infrastructure and Application Metrics) focuses on the importance of metrics gathering. Important considerations like time resolution and retention are explored. The collection metrics used at Flickr are discussed in some detail. It’s made very clear in this chapter how important gathering metrics is for any operation. It’s nice to read in detail how others are gathering, storing and displaying metrics. If you are new to the idea of gathering metrics and monitoring web infrastructures this chapter would be very helpful.

The sixth chapter (Monitoring) offers a good discussion about how to measure a site’s availability. The goal of increasing availability by decreasing MTTD (Mean Time to Detect) and MTTR (Mean Time to Respond) and increasing MTTF (Mean Time to Failure) is examined using a story as an example. The process of architecture monitoring is discussed in great detail through a series of clear steps. In each step of the process the author shares his own personal experience. This might be one of the better chapters in the book.

Chapter seven (How Complex Systems Fail) is in my opinion the most interesting chapter in the book. The chapter is dedicated to Dr. Richard Cook’s essay “How Complex Systems Fail”. Cook’s essay is very informative and applies almost directly to the complex systems in web operations.

Cook makes excellent points about system defenses and the impossibility of removing all system flaws. He cites economic costs and the lack of being able to identify failure points before they lead to an issue as the biggest contributors to eliminating all system flaws. It’s also noted that system failures almost always change as technology changes and operators make efforts to remove system failures.

I really enjoyed reading the section about post-accident assessments. Knowledge of the outcome makes it difficult to make a fair assessment of human performance. This knowledge leads to a belief that operators should have known that factors or flaws would lead to a failure. I know I’ve experienced this several times in my current position.

How Complex Systems Fail

Postmortems can be dreaded and sometimes far from helpful. I know I’ve been in some bad ones. Thankfully the thirteenth chapter (How to Make Failure Beautiful: The Art and Science of Postmortems) offers some clear strategies to improving postmortems. The author covers how to run a postmortem, the information you need to cover, and what to do after the postmortem is over. It’s a brief chapter but it’s clearly written and well worth reading.

Most of the material in the “Things That Go Bump in the Night (and How to Sleep Through Them)” chapter is good common sense. The author takes a practical look at developing a Business Continuity Plan. The focus of the chapter is really on BCP to keep a site up, not against drastic end of world disasters. It’s far more common that site availabilty will be affected by day to day accidents, so plan around them! I really like the adive on focusing your architecture on minimizing impact duration, rather than incident duration.