I hope everyone had a great Memorial Day Weekend.
In December 2011, I was recovering from my surgery. This was enough of a problem that I had to work mostly from home over the next month. The tools we had allowed me to do so and I was able to do a lot of things in some more relative peace than at other times.
The biggest thing I did with that extra alone time was to try to correlate the database performance issues we were having with something else going on. I looked at traffic by raw bytes. I looked at the number of messages. I looked at Outbound and Inbound traffic. I looked for Large Messages. My conclusion from all of it is that this issue was not directly traffic related. I began to look in other directions and wondered if there was some underlying bugs that were being triggered. I started searching error logs and found a number of small issues, but nothing that seemed to help. I started looking at the number of users and seeing if there was some magic spot that caused issues.
Let me back up one minute and provide a bit more information. This problem occurred regularly on a few systems. The rest seemed to have this happen once or twice over the couple of months that this issue reared its head. We had systems that never displayed any issues at all. It was quite baffling.
During my first full week back in the office, I got an email to meet the CEO for breakfast. This was a rather normal event as he was on the Board of Directors of another company in Petaluma. When he came up for a Board meeting we would have breakfast before hand to catch up. So, I got to the hotel at 7AM on Monday morning in February of 2012 and told that my services were no longer required. We stayed and had breakfast and talked over the transition. I asked a few questions about who would be taking over various portions of my job. I asked that they rework the proposed new organization chart for a few people and that was done. Other than that, I noted to the team that was taking over the roles that they missed a few things that I do. I was immediately offered a contract for 3 months to do those portions of the job and to train someone to take them over.
I spent time with the team to make sure we had a path forward on the problem. In the end it turned out to be two things that proceeded my time by over a year. The smaller (in my estimation) of the two problems was some basic problems in the way the database software was configured. Because we produced small units (remember the clarity issue here), the scale between the biggest and smallest databases meant we really needed two boot up configurations. The larger issue was a problem with the audit log. One of the biggest problems in mail service is that people call you up and ask, “What happened to my mail?” We needed to be able to show records around that. The founders never put a cap on the size of the audit log nor did we clean up old records. Some of the older systems had been in service for several years and this database was growing in an unbounded way. Eventually the scale of the data became a problem. This was especially true for that first ISP which wrote software to delete old SPAM messages daily.
So, what went wrong for me? Well, essentially I did a bad job of managing upward. The product was “late” because of the dates we talked about at that first call set a date. I had not even heard about the product and it had a date. I then slipped that date that I had not created. Then there was the apparent product quality issue that was created by the new software. On top of the ongoing issue there was the upset customer (even though they were upset at something that someone else did). It clearly looked like terrible performance at a glance. Clearly, I should have been explaining myself better and spending more time on it. Lesson to you all. One of my old colleagues (Rich Waldschmidt) would remind us all that our job is to make our boss successful. I definitely failed at that.
From here, I am going to get to some lessons in place from this story arc. After that, I will exit story arc mode for awhile and get back to some basics and more general articles.
Focal Point Business Coaching
Change Your Business – Change Your Life!
Business Coaching, Sales Training, Web Marketing, Behavioral Assessments, Financial Analysis