Math and Terror

My son just celebrated his eighth birthday. While he was eating his birthday breakfast before heading to school, he came out with this gem:

"I'm half of sixteen today!"

Now part of me is happy that he's thinking of mathematical relationships at the breakfast table. But another part of me is thinking how I am so incredibly not yet ready for sixteen. These first eight years have gone by so quickly, I can't begin to imagine how quickly the next will fly.

So the next time you're wondering if the glass is half empty or half full, just be glad that it isn't half of sixteen...

Know Your Data... Werewolves

(That's right. Werewolves.)

Over a year ago, I posted a list of SSIS best practices, and I've presented on this topic a dozen or more times since then at conferences and user groups and seminars. One of these best practices was "Really know your data – really!" and the advice went something like this:

"If there is one lesson I've learned it is that source systems never behave the way you expect them to and behave as documented even less frequently. Question everything, and then test to validate the answers you retrieve. You need to understand not only the static nature of the data - what is stored where - but also the dynamic nature of the data - how it changes when it changes, and what processes initiate those changes, and when, and how, and why. Odds are you will never understand a complex source system well enough, so make sure you are very friendly (may I recommend including a line item for chocolate and/or alcohol in your project budget?) with the business domain experts for the systems from which you will be extracting data. Really."

Well, last night I stumbled across some old meeting notes that reminded me of why this is so very important. The notes were incredibly brief, and consisted largely of a quote from the client project owner. But before I share the quote, please let me share the context and story[1]...

I was helping to design and build a BI application that my client was then re-selling as a service to their customers.[2] We were loading in data from a variety of source systems that provided the same type of data, but which were implemented by different vendors. One of the systems, called FooBar[3] was consistently causing us problems. The data import was running without error, and everything looked good on the surface, but the FooBar-using customers were unhappy[4] because the data they were seeing through the BI portal was incorrect.

Yuck.

The client project owner had, years earlier, worked with the development group that built FooBar, and with his insight we were able to discover that occasionally the data export process in FooBar was failing silently, so that we were getting only partial exports, and there was no way to tell - no error was raised, no checksum existed, and because of some nasty internal details (which I won't go into here) the effort involved in updating FooBar would be very significant. It was not a good time.

So what's the quote? It was this:

"You guys planned for wolves. FooBar is werewolves."

Even after all this time, I still remember the deadpan delivery for this gem. And it was so true. We'd gone in thinking we were prepared for all of the usual problems - and we were. But we weren't prepared for the horrifying reality of the data problems that were lying in wait. We weren't prepared for werewolves.

So what's the moral of the story? Plan for werewolves. Assume the worst. Test early and often, and test failure scenarios, not just happy-day scenarios. Because the time to learn that there really are werewolves is when there is still time to pack a crossbow and some silver bolts, not when the full moon is rising.

Trust me.

 

[1] Names have been changed to protect the innocent. And the other people too...

[2] This has been a recurring theme for me, so it's probably not the company you're thinking of.

[3] I changed the names, remember?

[4] I may also use a literary tool called "understatement" in this story...

What's Best?

I was flying through Cincinnati yesterday and while waiting for my connecting flight I overheard (I wasn't trying to eavesdrop, but the guy was standing right next to me and talking loudly and clearly) a phone conversation about various aspects of IT consulting. During the conversation the person speaking next to me came out with this great quote:

"There's no such thing as best practices - only best practitioners."

Ok, I don't know if I really agree with this 100% (although I did smile and say "that's right!" to the man on the phone) but it is certainly an interesting idea that raises a whole set of interesting questions related to knowing what's "best" for a project.

Obviously, best practices exist, but they are generally just rules of thumb. They're general guidelines to follow in common situations, but they're not a panacea. In order for them to be applied to greatest effect, the person following the best practices needs to understand not just what to do, but also why.

In order to really get the best, you need not only best practices, but you also need the right people to implement them - the best practitioners, as it were. This is because context is practically everything when it comes to determining meaning, and in order to understand the context of a best practice recommendation, the practitioner needs to first understand two primary things:

1) The business context of the project - without understanding the real problems that need to be solved, no one can select an ideal solution.

2) The technology platform on which the solution will be implemented.

These two factors are vital when applying documented best practices because the best practices are generally presented as a set of prescriptive guidance - do this, don't do this, also do this - without a great deal of explanation. There's rarely[1] information about why a given step/task/configuration is considered "best." Because of this, the practitioner needs a solid understanding of how the technology platform works - this will let him interpret the best practice guidance and decide whether a given "best practice" recommendation applies to his current project context.

So what's best?

The answer of course is "it depends." The answer always depends on the context in which the question is asked.

But when you're looking for the best solution to your vexing problems, you need not only best practices, but you also need the right people to implement them. I've seen too many companies over the years who try to save money by only hiring "junior" personnel, but who then expect these inexperienced folks to deliver top-quality software on aggressive schedules. Although sometimes this can work, it's never repeatable and generally only succeeds through great personal sacrifice on the part of the team members.

The next time you're starting a project, think about the skills you're going to need, and realize that sometimes there's no substitute for the experience and deep knowledge of a "best practitioner."[2]

 

[1] Although this is beginning to change as technologies mature - the patterns & practices group at Microsoft does a great job here.

[2] No, I'm not trying to advertise or promote myself or anyone else. I'm booked solid (and then some) for as far as I can see, and don't have any ulterior motives behind the post.

Getting Back What You Put In

Today was Christmas, and one of the gifts I received is a delightful book titled Making Artisan Chocolates by Andrew Garrison Shotts. I've spent the day with family and friends, so I haven't had much time to read it, but the author's dedication at the front of the book contains a quote I just could not wait to share:

"From my father, Don, I learned work ethic and discipline. He taught me that a worthwhile outcome is fully dependent on the time and effort it takes to get there."

It's obvious that Mr. Shotts' father was a wise and thoughtful man. It doesn't matter if you're a chocolatier, a data architect or a mechanic, the effort that you are willing to put into your work determines the quality that you get back out. For me this usually means spending those late nights researching how SSIS works under the hood, or how to eek just a little better performance out of a stored procedure (although now that I have this book I may well be spending some time dusting off my chocolate tempering skills) but no matter who you are and what you do, these are words truly to live by.

And now, those raspberry-wasabi chocolates look irresistible...

Design Conflict

I had a dinner meeting a few weeks back with some members of my development team at work (including the new VP!) and one of the topics of the discussion was that of team dynamics. I've found over the years that the most successful teams are teams of "invested equals." Invested because people only give their best when they think that they can make a difference and when they know that they'll be rewarded, and equals because people (for some strange reason) tend to do what their bosses tell them to, as opposed to pushing back and demanding that their voice and their ideas be heard.

As part of this wonderful dinner conversation, I told a story of how Ted Malone (who still claims to not regret recruiting me) and I were both attempting to come up with a design for a vital new application component. He presented his idea. It was so wrong. I presented my delightfully well thought out idea. He failed to see its beauty. (Please keep in mind that Ted may have somewhat different recollections of this morning.) So we dueled, whiteboard markers at dawn, as it were, and by the time the dust had settled the resulting design was more complete, more elegant and more satisfying than anything either one of us could have come up with alone.

Which brings me to a quote:

"A design that comes out of an argument is always better than a design that comes out of a committee."

Believe it or not, this is actually my own quote. (I'm not usually nearly this pithy, and am forced to quote the brilliant people around me.) It's been sitting in the back of my brain for the last few weeks, waiting for a chance to come out again.

And that chance may well be on the horizon. I've been involved in an email discussion with our CTO (who may well have the biggest brain on the planet - this guy is scary sometimes) on how we may be able to apply Microsoft BI technologies to solve some very interesting (and by "interesting" I mean oppressively difficult) problems in the configuration analytics space. The opportunity I see ahead lies in the fact that I’m the type of person who needs a real, concrete problem to look at and to wrap my brain around. Then I can step back, generalize and come up with an abstract “problem domain” that represents the whole problem to be solved, of which my concrete example was only one instance. But I need that one concrete instance in order to begin.

Dennis, on the other hand, is the type of person who always thinks in abstracts. (Or in any event this is the impression I've gotten; I honestly haven't asked him. Yet.) He always has that overarching “big picture” in mind, and even though he can drill down into the little details at will, that’s not how he looks at the problems natively.

So Dennis and I have some whiteboard time scheduled for next month. My brain is almost literally salivating (yeah, picture that one) at the thought of the mental duel that lies ahead. Bring on the conflict!

Developer's Crossing

Miller's Crossing is one of my favorite movies, and is in my opinion the brightest gem in the Coen brothers' crown, despite the fact that few people know about it when compared to Fargo or Raising Arizona and the rest. If you've never seen it, and you like complex plots, flawed anti-heroes and a bit of odd humor, and don't mind a little violence (ok, a lot of violence) here and there, you owe it to yourself to watch it.


But what does this have to do with SQL Server and BI?


Complexity, that's what.


Consider this quote from Tom Reagan (played by Gabriel Byrne)

"Nobody knows anybody. Not that well."

Sound familiar? When was the last time you felt like you really knew a lot (or knew "everything" for the really smart readers out there) about SQL Server? I remember when I decided to "specialize in SQL Server." Then I decided to "specialize in SQL Server development." And now I'm on the cusp of specializing in SQL Server BI (which actually seems like an increase in scope, but perhaps that's just me) because the breadth of the product has gotten so large that one brain (ok, my brain) cannot encompass it all.

This all struck home today when someone I know (I won't name any names, but he doesn't have a blog) asked me a question that I thought fell very well into his personal specialization. As this is a person whose technical knowledge I respect greatly, it got me thinking, and in the process of working with him to find the solution, this quote filtered up from the depths of my brain.

And that's it. So as the USB hub said to the thumb drive... "Take your flunky and dongle."

Bad Math Day

I've always been fond of the quote "the lottery is a tax on people who are bad at math." It's sort of like "There are 10 types of people in the world - those who understand binary and those who do not," but without the geeky overtones.

But The Register today has a news story that really drives home someone having a bad math day: http://www.theregister.co.uk/2007/11/08/scratchcard_anarchy/

My favorite quote from the story:

"I phoned Camelot and they fobbed me off with some story that -6 is higher, not lower, than -8, but I'm not having it."

Right...

Another Profound Quote

I'm participating in a day-long online meeting via phone, and someone just came out with this gem:

"No one should watch sausage or software being made."

It works on so many levels. It's wrong on so many levels. I love it.

Another Thought on Competition

I posted some thoughts on competition a few months back, and some recent conversations have made me visit this topic again. One was a recurring conversation that I've had with my six year old son. He's very competitive (I wonder where he gets that trait, eh?) but hates to lose. I've been trying to explain to him for years that losing (and by extension, competing against someone better than yourself) is the only way to get better, and that getting better is the only way to win. It's taken a while, but the lesson is starting to stick.

But it doesn't stick everywhere.

I had a conversation a few weeks back with my friend and colleague Ted Malone. We were talking about team dynamics, and Ted came up with this great quote:

Tier one people always try to surround themselves with tier one people.
Tier two people always try to surround themselves with tier three people.

Wow, that's profound. I don't know if Ted gets credit for this quote (I think he said it was not his own, but my memory is poor for that type of thing) but he certainly gets my thanks for sharing it with me. In so many walks of life - and software development and architecture is certainly one of them - the people who shine the brightest[1] are the ones who try to surround themselves with the best and the brightest they can find. Any candle, no matter how dim, will look bright in a dark room; you never know how bright your light shines until you take it out in the sunlight.

So what's the moral of the story?

Good question. I always have trouble with that whole "wrapping up and making a point" thing. But if there is a moral, it is that we should always attempt to raise the bar against which we are being judged, and against which we are judging ourselves. Think of it as a process of continuous improvement for the individual. What better investment could we make?

[1] Please note that I'm not trying to say that I fall into this category! Although I like to think that I'm always learning, any time I start to think of myself as an "expert" in any given topic I meet someone whose knowledge puts mine to shame. So I try to hang out with them. :-)

Happy Halloween!

I can't believe October has come and gone already. Usually this is my favorite month of the year, but this year I've barely noticed it. Things have been so busy I've even forgotten to update my email signature. Usually I will update it to use this great seasonal quote:

"October. When the leaves turn blood and the wind turns bone: a time for doings
dark and strange."

This is attributed to the author Glen Cook, as it's taken from the cover of his classic novel October's Baby[1] but it was probably written by the same guy the publisher got to write the summary for the back cover. Still, I like it anyway, and it generally sums up the way that October usually feels for me.

This year was strange in its own way with trips all over the US, speaking gigs both at home and as far away as Stockholm, and more work than any one person should ever have. Not that this is a bad thing, but hopefully next year October will have a little more time for blogging. And baking...

[1] Which is finally back in print after 20 years as part of the A Cruel Wind compilation.

Wisdom from Microsoft's Bill Buxton

Yet again I've taken an unplanned and extended leave of absence from blogging. My work and travel schedule lately has been overwhelming, and after spending seven weeks on the road (not the entire week each week, but it's still been brutal) I'm finally back in the office and looking up to see that there is still a world around me.

One of the things I've done this morning is walk through my favorites, checking the news and so on. And on The Register there was an interesting article about Bill Buxton of Microsoft Research. And in that article was this great quote:

"The renaissance is over - the problems are far too difficult for any one individual to have sufficient knowledge to advance them. On the other hand, the renaissance team is possible and our only hope is the collective - the cross disciplinary team. Engineering and computer science, interaction design, ethnography, marketing and sales."

Now of course there are still opportunities for a talented lone software developer to make a big difference, but most "interesting" problems today need to be viewed from a broad range of contexts in order to be solved. This is something that I've thought for years (albeit on a much smaller scale, I'm sure) so it makes me happy to see that my views are shared by someone as brilliant as Mr. Buxton.

Keeping Things in Perspective

I learned today that Todd, a good friend of mine, has been hospitalized for a very aggressive B-Cell lymphoma; he was diagnosed yesterday, hospitalized yesterday and is starting chemotherapy today.

That kind of makes Active Directory and Exchange Server problems seem sort of trivial, doesn't it?

I spoke with him on the phone this evening, and he was in great spirits, all things considered. He was full of jokes, and according to his wife he's been driving the doctors and nurses crazy with his bizarre sense of humor.[1] While we were talking he gave me two great quotes:

"I have an alien the size of a softball sitting on my heart."

and

"The one bad thing about all this is that with all the scans they've been doing, I know that I don't actually have a heart of steel."

This second quote was a reference to the Manowar song titled "Heart of Steel" from their "Kings of Metal" album. I have always found this song to be particularly inspirational[2] so I'm going to include its lyrics here.

Build a fire a thousand miles away
To light my long way home
I ride a comet
My trail is long to stay
Silence is a heavy stone
I fight the world and take all they can give
There are times my heart hangs low
Born to walk against the wind
Born to hear my name
No matter where I stand I'm alone

Stand and fight
Live by your heart
Always one more try
I'm not afraid to die
Stand and fight
Say what you feel
Born with a heart of steel

Burn the bridge behind you
Leave no retreat
There's only one way home
Those who laugh and crowd the path
And cut each others throats
Will fall like melting snow
They'll watch us rise
with fire in our eyes
They'll bow their heads
Their hearts will hang low
Then we'll laugh and they will kneel
And know this heart of steel was
Too hard to break
Too hard to hold

Stand and fight
Live by your heart
Always one more try
I'm not afraid to die
Stand and fight
Say what you feel
Born with a heart of steel

There are many people out here thinking of you and praying for you, Todd. Don't forget it.

[1] To put this in context, he's the best QA person I've ever worked with, and I truly believe that a bizarre sense of humor, combined with a strong desire to irritate others, is a prerequisite to excel in this role.
[2] To the point that I have some of its lyrics tattooed on my leg, but that's another story.

I'm Not Ready for Teenagers

I was talking this morning with a friend and colleague who has children older than mine[1] and he had this delightful insight on puberty:

"We understand why there was a Vienna Boys Choir. It had nothing to do with music."

I still have a few years to go before I need to worry about this, but needless to say, I'm not ready yet...


[1] He said I could quote him, but asked that I not use his name.

More MOF Quotes

In the Microsoft Operations Framework class I’m taking this week, we were talking about the MOF Risk Management Discipline[1] and there was another quote that made me smile:

“If everyone owns it, no one owns it”

And there was another related quote that came up yesterday while talking about the MOF Process Model, which fits in very nicely:

“Every process needs an owner: One throat to choke!”

These two quotes really drive home the fact that on any project, on any team and in any process there is a real need for accountability and ownership. Not only does this help foster an environment of responsibility, it also helps define channels of communication between groups and members of groups.

It’s a big thing. When was the last time you were working on a project and asked “who’s responsible for this server[2]?” and no one could get you the answer you needed? I’ll bet it hasn’t been too long.

Ownership is a core component to doing things right, because if no one owns something, no one will take care of it. If it’s not worth assigning an owner, it’s probably not worth doing, so you should question why you’re doing it in the first place.

Take a look at your teams and projects and processes. What do you own? Do other people on your team agree that you own it? Does management? Only you know the answers – or do you?

[1] Risk management discipline is not unique to MOF; it’s also a core part of MSF (the Microsoft Solutions Framework) and any well-run software, database and BI project. If you don’t use it, you really should.
[2] Or software, or resource or process or whatever.

Clear Eye for the KPI Guy

I'm attending a Microsoft Operations Framework class this week[1] and the instructor[2] had a great quote today:

If your KPIs are always green, or if your KPIs are always red, then you're measuring the wrong thing.

Now he didn't actually say KPI - we were talking about the metrics you might use when defining and auditing service level agreements, and not about BI, but the concepts are still the same. In order for a KPI to have the most value there needs to be circumstances when it is not green, and if a KPI is always red, then either you've set the goal unrealistically hard, or else you're not willing or able to use the KPI the way it was intended.


[1] This is one way I'm trying to practice what I preach - MOF and ITIL are essentially the "business side" of the BI project I'm working on, so the more I understand its nuances, the better I'll be able to help bring business value to my users.
[2] Monte Whitbeck from Microsoft - he's doing a killer job. I wish I got to attend more training from people like Monte who really understand the material they present and have worked with it extensively in the real world.

Putting Things in Perspective

A close friend of mine was recently visiting (I think this was late June, if that puts into perspective how insane my schedule has been of late) and I was playing some new music for him. He lives out on the west coast, so it's rare that we get a chance to catch up in person.

Since the last time we've sat down together, he's had surgery for Acoustic Neuroma, which is a potentially life-threatening (although non-cancerous) tumor on the acoustic nerve. His (also potentially life-threatening) surgery was successful; the tumor was removed, but he lost all of his hearing in his right ear.

While we were listening to the music, I asked him what he thought. As part of the conversation that followed, he said:

It's in mono. The whole world is in mono. But it's better than being dead.

Kind of puts things in perspective, doesn't it?

I'll try to keep this in mind while I wrestle with SSIS deployment stuff today. ;-)

Are Donald Farmer and Dr. Ivan Brady Related?

Another great quote:

"You have to understand the nature of your exceptions"

You need to understand the underlying business context of the data and problems with the data before you can solve the problems. And you need to remember that it may be a business solution, not a technical solution.

Sound familiar?

ETL Lifecycle - An Interesting Thought

During this ETL Framework session, Larry Barnes (the person with whom Donald Farmer is co-presenting) raised an interesting point:

ETL processes tend to stick around forever, because people are horrified to
touch them.

He used the image of a Perl script that runs on an Oracle server pulling data from a Sybase database and loads it into Informix (can't you just picture that?) but I have no trouble visualizing the same thing happening with SSIS as well.

This may not be technical or of interest to anyone but me, but it strikes me as being profound in some way. I tend to be very aggressive in my commenting, annotation and documentation, but this really drives home how important this is for SSIS. In addition, this really demonstrates the importance of having a well-documented and understood process for making changes, moving from dev to test to prod just like with traditional application code, because only through this process will people truly be comfortable making changes to such business-critical code.

From the mouths of babes...

"They used to be pigs, but now they're bacon!"

-- Moira, age 4

Happy birthday, Moira!

Dr. Ivan Brady Strikes Again!

If you've ever gotten an email from me, odds are it's had this as the signature:

“Context is practically everything when it comes to determining meaning”
-- Dr. Ivan Brady

Dr. Ivan Brady was my favorite professor when I was in college. He almost convinced me to switch majors from Computer Science to Anthropology. Words fell like pearls of wisdom from his mouth. And this was one of his favorite quotes - he used it a lot.

And still, so many years later, I keep being reminded how incredibly true this is. On so many projects - and BI projects in particular - far too much time is lost on false starts and poor communication. Everyone seems to always want to start working, to start doing something, anything, instead of calming down, establish a common context, and truly understanding what the other parties are talking about.

I know, this isn't really news, but this week has reminded me once again how true, how profound, and how important this is. If more people listened to Dr. Brady, the world would be a better place, and we'd get a lot more done...