Archive for July, 2009
So my mind operates on analogies/similes/metaphors, and I often find it difficult to explain concepts to people without a proper analogy. Today I came across a great one that I’d like to share. While I’m using developers as an example, it also applies to DBAs and most of IT- it’s just more pronounced in the dev world since there’s usually so many of them.
Development is war. It’s a never-ending series of fast-paced, multi-fronted episodes that take place in multiple theaters. Developers are broken into squads to fight different fronts, and there’s a whole command structure in place over top of it. You may win a battle (finish an application), but normally it results in you moving to the next fight. This continues on forever. Foreign Legions (H1B) and Mercenaries (contractors) also fit into the equation. Directors take on the role of a generals, business and marketing people of bureaucrats, and upper management assumes the role of the rulers.
I have a friend who’s a developer at a company with morale problems. Like many companies, the recession has hit them hard. Three rounds of layoffs, 10% paycuts, and outsourcing have done quite a bit of damage, but it’s the continued poor management that has lead to a morale death spiral.
Poor management decisions have angered many of the better developers, often being pulled off an important project to work on the pet project of a low level bureaucrat. Contracts are signed with vendors before developers even know about the project. Questions like “what is the SLA?” and “How much load can you handle?” are never asked. Accountability is a word often extolled, but rarely practiced.
The talented, knowledgeable developers have begun to flee. Upper management has made no efforts to hang onto the people, and have even admitted that they will not do anything about the exodus. These are the same developers who have taken a crumbling system and over the last three years rebuilt it into something far, far better. Their true value is not in what they’ve done, but what they will do. The younger and more inexperienced developers look up to them and learn from them. These talented developers can mentor, answer questions and prevent design mistakes before they get into production.
What makes these talented developers so important? They are usually a combination of three traits:
- Experience- they’ve been developing for 25 years and have had multiple jobs, so they’ve been exposed to many styles and pitfalls. Often wary of fads.
- History- They’ve been with the company for 13 years and know the systems, why cruft exists, and what exists that can be reused. Hesitant to change the status quo.
- Raw Talent- The most infuriating type- brash, cocky, yet brilliant. They’ll come up with an awesome new product that will bring the customers in (let them prototype, but not implement).
These are the three types of people you need on your team. The rest of us are mere peons. We do the grunt work, but rely on these guys to lead us. Without them, bad design and implementation decisions are made.
Now, to come back to the war analogy- these senior developers are the Heroes. They’re the King Arthurs, Conan the Barbarians, and Hercules. They rally the troops and lead the charge. Without them, all is lost- this is something that is often forgotten by those in power.
When times are tough and cuts need to be made, sometimes a hero is sacrificed. When one is killed in a sacrificial move (layoff), there is mourning and anger, but those left press on and find others to follow.
Now, you can lose a lot of peons, or one or two heroes, and morale will still be good, but when you take continual losses, the troops begin to wonder- how long before we’re overwhelmed? Do we have the manpower to still succeed? The rulers (upper management) may hire some mercenaries (contractors) to fill the ranks, but at the end of the day they don’t care what happens- they’re there to get paid then get out. It’s not the same thing- you need leaders.
The troops still press forward, shaken. Meanwhile the rules continue to pick fights (new projects), switch priorities, and stretch the teams further and further. Worse is when they hire a group of mercenaries who talk a big game but don’t know how to fight (code). If they’d let someone who knew how to code do the vetting, it could have been caught before they get on the battlefield and get annihilated.
With all the losses, sacrifices, and overall ineptitude, one of those heroes finally realizes they’re on the wrong side. In disgust, they throw down their weapon and walk away. Some call them traitors, some are saddened to see them go, and yet others will follow them out. The rulers never bother to ask why they left- they’re too narcissistic, or worse, they don’t care. They don’t need that hero to lead the troops- THEY are the leader, not the hero. Besides, they can just hire more mercenaries (you know, the ones with dubious motives).
So, what happens when ALL of the heroes start to walk away? The [smart] peons follow suit. Some will try to hide and hope no one notices them. The end result is an aimless, weakened force of mediocrity with no leadership. You can’t win a war with a couple of peasants too dumb to even bring a pitchfork (let alone a sword) to battle. They just don’t have what it takes to do it by themselves.
This is where my friend’s company is at right now. The surrounding army of 80 IT people has dwindled down to half it’s size. They had been fortunate to have a large collection of heroes- close to 20% of the troops were incredibly talented. Now all that’s left? Three (including him). All three have doubts as to why they’re still there, and are looking to abandon ship.
As for me? I’m just a lowly peon. If I was in their shoes, I’d stick around just for spite. I’d stay, surrender, and watch with a smile as the rulers who destroyed everything good are beheaded, and new rulers put in their place.
But hey, that’s the fortunes of war.
If you ever see me talking to myself, I’m just working through ideas for my book, honest… Here’s a good example of what I’m chomping on right now.
So I’ve been thinking, why does Ziggy’s Aunt Makuran hate Ziggy so much? Why does his cousin resent him?
First off, Ziggy is adopted, so take that into account.
Originally I thought it was because she was upset that Keltrem was made chief instead of her(since she’s female), and resented him for it because she is the eldest child. When Keltrem’s biological son died, she knew her son Gunthorm would be the next chief. When Keltrem adopted Ziggy, she blamed him, and raised her son to hate them both.
…but that didn’t Gel. Why didn’t Kinnon, Marukan’s husband hate Ziggy? There was power to be had if his son became chief… unless it wasn’t his son.
What if Marukan’s first husband was killed in battle by Keltrem’s decision? That is a much better reason not to forgive her brother. But why would Gunthorm share that hate? He was the son of the first husband… and Kinnon adopted him and had another child with Marukan. Kinnon would feel empathy towards Keltrem’s plight with an adopted son, and give ziggy the benefit of the doubt in private, but be distant when around Marukan.
Marukan’s second son (Agaleron) was fathered by Kinnon, and doesn’t care about all that crap, but since his older brother hates ziggy, he hates ziggy as well.
Gunthorm doesn’t empathize with ziggy being adopted because he doesn’t see Kinnon as a father figure, he sees him as a guy who married his mom. There’s tension there.
Lets back up a bit. Keltrem and his wife Zimissa had a son named Botulf. When Botulf was very young, he wandered near some contaminated turnips and ate one, then died of poisoning. That’s what everyone things. … but did Marukan have a role in that? intentional or unintentional?
Zimissa was heartbroken. Keltrem comforts her. Years later, she becomes pregnant again, but miscarries several months short. (did Marukan have a role in that? just how evil is she??)
Zimissa is devastated, tries to commit suicide (did Marukan have a role in that?) She does so by walking past the dark curtain into the plane of chaos. Keltrem follows, but watches her die, unable to save her. Kinnon (the best tracker in the tribe) witnesses this.
Keltrem is despondent, returns to the tribe, only to walk away without saying a word. Marukan takes command of the tribe much to the anger of the elders.
Keltrem wanders, finds Ziggy, thinks it’s a gift from the gods for his loss.
Keltrem returns several days later, tells all what happened. They accept this as truth, and kick Marukan to the curb.
This adds much more depth to the family dynamics.
This is what I was thinking about this morning.
“Never attribute to ignorance what can be attributed to bad grammar.”
…which is a corollary of “never attribute to malice what can be attributed to ignorance/stupidity.”
I caught some interesting behavior on one of our servers today- I’m using subversion to manage a jboss instance (there’ll be a writeup on this later, I’m sure) and caught the following behavior. Here’s the simplified setup.
Two servers, X and Y. both of which are checked out of the subversion repository as a whole and fired up. They are clustered with each other. Each server is hosting a ROOT.war file (binary) that contains the site’s code base. This file is stored in the farm/ directory, and under normal circumstances, when a file is deployed on one to the farm directory, it’s ‘farmed’ over to the other one.
Eventually I plan on disabling the deployment scanner and farming (probably), but it revealed some interesting behavior in the meantime.
A new version of the code base was given for us to deploy. Since we’re half way though implementing these changes, I deployed per our current convention.
- shut down Y
- deploy to X
- restart X
- start Y
For reasons beyond the scope of this discussion, that’s how we do it to get it to farm cleanly. So now I have a [M]odified ROOT.war file on both servers. I commit the file on server X to the repository as revision 537.
At this point, the file on X, Y and in the repository are all the same- same contents, same md5sum, even the same date on the file between X and Y.
The only problem left is Y now has an out of date revision and a modified warfile- It still thinks it’s on rev. 536.
Had this file been a simple text file, running svn up would have noticed the local modifications were the same as the repo changes, and would have mer[G]ed the changes. Unfortunately svn doesn’t know what to do with binary files and treats them like they’re [C]onflicting, resulting in the following three files (md5sums included for illustration):
ROOT.war is the current version that’s there- the one that was farmed over from X. .r536 is the previously mentioned version of the file, and r537 is the latest version from the repository.
Now, under normal conditions, this isn’t a problem… But remember, this is happening in the farm/ directory, so these two new versions may be passed out to the cluster (X). This isn’t optimal.
How could it be more cleanly resolved (no svn pun intended)? subversion could do an md5sum on binary files to see if they’re “the same”, and if so, merge it rather than mark it as [C]onflicted.
I’m pretty sure I’m running an older version of subversion, so this functionality might already be there, but if it isn’t, it’d be a great feature to add.