Duane Gran’s log of thoughts

Its hard to see the big picture when you are working on implementation details and it is no wonder that programmers are often viewed as being out of touch with the organization as a whole. When the developer is deliberating depth-first versus breadth-first for an algorithm they are bound to appear disconnected from the conversation at hand.

Implementation matters. Make no doubt about it. A colleague of mine recently explained his distress at collaborating with a manager who describes software systems in term of food preparation. When the metaphors fail to describe the practical limits of software the manager thinks the programmer is inept. I’ve been there and it is supremely frustrating but I think it is a two way street and both parties need to find common area.

It doesn’t work for the programmer to live in “implementation land” all day long and it doesn’t work for managers to waive off that “implementation stuff” as unimportant jargon. Another missive that hits this point for programmers comes by way of Neil Mcallister’s article. Here are some practical suggestions for programmers to break the mode a bit:

1. Set a reminder in the morning and afternoon to stop and put your head up. By this I mean to make sure that you create a pause in your work to contemplate what may be going on around you. Rather than waiting for someone to shadow your doorstep maybe you notice a colleague has a pressing deadline (not code related) where you can lend expertise. Get into the rhythm of the workplace.
2. In meetings with colleagues and managers begin by making a clear statement of the issue and solution. Don’t start discussions centered on implementation. This is an opportunity to affirm that everyone agrees on what is being done.
3. Ask a trusted colleague for feedback about your “presence” in the organization. Are you viewed as engaged or reactive? Ask for constructive ideas to improve your communication of technical concepts. You may be surprised what comes of it.

Other suggestions and ideas here are most welcome in the comments.

Blaise Pascal famously started a letter stating, “I would have written a shorter letter but I didn’t have time.” Its often easier to make things hard, complex or time intensive than to achieve the short and simple version.

If you would have asked me three months ago what I thought of twitter I would have described it as yet another sign of the collapse of civilization, akin to necktie-less men and poor punctuation. I would have told these kids to get off my lawn.

I’m eating (er, tweeting) my words. If used properly, twitter is an excellent antidote to the verbal diarrhea of weblog (present company included) and article writing. It isn’t a replacement, but the simplicity and 140 character constraint does wonders to focus an idea. Pascal would have liked it.

I recently upgraded to Wordpress version 2.6.5 and it appears that comments are currently broken. I’m investigating it.

Feel free to leave a comment with suggestions. :)

Update: This has been sort of low priority lately, but the comments are working again. The solution was rather simple and involved performing a MySQL table repair operation.

It doesn’t get any more succinct than this:

The very best manager you could ever have manages the people above him, not the people below him.

I’ll only add that I think great managers do both, but insulating developers from upper management is an often overlooked function of good gate control. At least that is how I think of management.

A slashdot article features a question from a nervous database administrator who is worried about giving direct access to a database to a customer. He ends with the plea, “How would you advice me to keep my customer away from my precious tables?”

The right question is, “How can I give my customer better access to reports without compromising security or service availability?” Too often in IT we focus on diplomatic ways of saying “no” instead of seeking creative ways to say “yes”.

If I were in this situation I would draft a proposal for a dedicated server with replicated data. Such a proposal would make it clear that tables may (and likely will) change over time and that they would have read-only access to the secondary server. This would likely result in a happy customer, more revenue and very little impact to operations.

If you are involved in a start up or work with technology in small organizations, be sure to read Keith Casey’s article about technology and startups. That he was the sole technology representative on such a panel is telling.

People familiar with the potential and limits of technology have a lot to contribute to start ups beyond slinging code.

I’ve used Quicken for ages to organize (or at the very least categorize) personal financial transactions. Today I learned that Intuit will rewrite the application for the Mac. The application is a bit long in the tooth, but this announcement gives me little comfort and this may well be the best news possible for Intuit’s competitors. Why, you may ask?

A software rewrite is almost always a bad idea that percolates up from the development team and eventually makes it way into the ear of management. The typical explanation goes that we, the programmers, are spending so much time supporting the idiosyncracies of the old system that a fresh code base is required to get it right. Probably the best explanation for why this is folly comes by way of Joel Spotsky:

The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they’ve been fixed. There’s nothing wrong with it.

I have a hunch that I’m like most people who value reliability and a seamless upgrade in my next version of Quicken. The last question on my mind is whether they wrote the application with the Carbon or Cocoa API. I’ll let other people take that plunge, thank you.

Johanna Rothman looks at several definitions of what constitutes a “project” and offers up a goal-oriented definition. Here is my stab at it:

A project is a temporary endeavor with measurable outcomes.

If that sounds too simplistic, consider for a moment the last project you worked on that never ended. Most people are better at starting projects than finishing them, so I like to emphasize the temporal relationship in addition to meeting the end goal.

As a fitting complement to Waldo’s entry I thought I would offer up a view from the other side, as it were. I’ve been managing software projects for years and suffered my share of management experiences described in Paul Graham’s article.

Software development is creative and structured. You ruminate about the problem and test a variety of solutions until a good match emerges. Few people can do this while fielding interruptions nor do many composers do their best work while balancing life obligations. Software development requires devoted attention.

Management is less creative and far less structured. You operate a general plan and re-route as circumstances dictate. Good managers know when to interrupt software development while bad managers pull the fire alarm at every bump in the road.

I like to think of management as being a traffic officer. Multiple streams of potential decisions bombard me and I have to decide whether they warrant the cost of interrupting developers, finance officers and other business partners. I believe a good manager estimates the opportunity cost of conversations and meetings.

Dave Koelle has produced an elegant solution to a pervasive problem in computer sorting. The default sort order of a list places “z100″ ahead of “z2″ and makes the output of an application less polished. I don’t have an immediate use for this, but I’ve seen the problem enough times to know that I and others will benefit from it. The algorithm is available in Java, C++ and Perl flavors.

Michael Kay wrote the following to the XSL mailing list recently:

THE FIRST LAW OF DEBUGGING: if you can’t find the bug after two minutes, it’s because you’re looking in the wrong place.

CORROLLARY: if you post the problem to a forum, you will snip the very code that contains the bug.

It couldn’t be more true and I’ll readily confess that the kind people on that list have helped me debug several obvious (in retrospect) errors I’ve made with XSL programming.

The old maxim of business that nothing happens until a sale is made holds true, but force of reality isn’t always enough to make truth ubiquitous. An impassioned counter claim from a technologist caught my attention. The point is simple enough and not so controversial. Business people need technologists to implement systems and solutions — that much is clear — but engineering fulfills one part of a total business endeavor.

I like what Guy Kawasaki said, in a list of his top ten lies from engineers:

My theory is that for version 1.0 of a product, the maximum allowable distance between the engineers and marketers is thirty feet.

Marketing and engineering have a torturous relationship. Marketers distill product capability into language suitable for customers and salespeople occasionally promise features or deliverables which frustrate engineers. On the flip side, engineers hedge about building new functionality and infrequently describe systems in useful terms for marketing.

Because sales and marketing have a potentially better grasp on the customer interest it only makes sense for their vision to direct software projects. Unfortunately the attention span of the marketplace is more fickle than most development cycles, so marketers often refine, revise or scrap a product design. This is supremely frustrating for engineers, but building the wrong system or finishing one that won’t sell has no value.

My point is that technologists need business people. They sell stuff and put lipstick on the pig for investors and a host of stake holders. As a career programmer turned IT manager, I’ll readily admit that customer focus is hard to maintain when building the guts of technology. Looking back on my entrepreneur days I see better my blind spots. Ignorance — not knowing or not caring — about the customer is common, but it is almost the rule among technologists.

As an illustration, I’m fond of Ron Garrett’s list of ten geek business myths. Several of the points could be summarized as “my ideas matter” whereas successful ventures are aligned with the goals of the customer.

I recently had lunch with a nineteen year old Computer Science major who requested my advice on many things software, but mostly about how to pursue a fulfilling career in IT. We had a wonderful discussion which concluded about how academic computing differs from what one finds in the industry. To summarize:

• An academic project is nearly always a fresh code base with few interactions, whereas most software development involves improving existing functionality or adjusting systems to comply with current business policy.
• The system requirements of academic projects are nearly always discrete and clear, whereas most software requirements are vague and it is the task of management and developers to translate requirements into a software design.
• An academic project is mostly evaluated by how well it matches specific requirements, whereas most software is judged by its correctness in addition to how well it scales and resists malicious or careless data inputs

In all fairness, when I studied Computer Science I had an instructor for a database design class who masterfully tested software. He would make students demonstrate in front of the class, except that he would interrupt the demo and ask you to enter in awkward data. Nearly everyone’s program crashed or corrupted data as a result. The lesson remains with me today and represents one of my most practical classroom experiences.

A British Computer Society essay identifies an even larger delta emerging over an increasing mode of developing software as a service.

If the gap between public knowledge and academic curriculum isn’t large enough, the gap between academia and industry practice is a gaping hole. While academic departments concentrate on developing new computer systems in an ideal organisational environment, a lot of industry has moved away from in-house development to a focus on delivering a service.

…

Interrupts, loops, algorithms, formal methods are not on the agenda. IT is about deploying resources to meet the information needs of its customers. Implementation, facility management, systems integration, service management, organisational change even environmental audit are the language of IT. These hardly feature on computer science courses.

If all of this hand wringing seems irrelevant, consider that Department of Labor forecasts, as summarized by Business Week, that “computer/math scientist jobs, which include programming, will increase by 40%, from 2.5 million in 2002 to 3.5 million in 2012.” Yet, “Colleges aren’t keeping up with demand. A 2005 survey of freshmen showed that just 1.1% planned to major in computer science, down from 3.7% in 2000.”

The good news may be, as the BCS essay illustrates, that software development is becoming ever more approachable to the masses. Many “good enough” programs are developed by amateurs which don’t justify the rigor of the CS curriculum, but the applications benefit immensely from being written by people fluent in the domain logic. If you have ever witnessed a programmer’s blank stare as you describe finance or metrical poetry you can appreciate why this transition is important and empowering.

The academic curriculum does an excellent job introducing students to computational task and theory, but more projects should be framed by someone outside of the discipline. Every CS graduate knows how to simulate the Towers of Hanoi but few have practical experience collaborating outside of the domain of math and science. Those who do possess a strategic advantage over their peers.

The good news for my young lunch companion is that the field is growing, but the bad news that the field is changing rapidly. Anyone over the past 40 years could say effectively the same thing, but I mean that a software developer must consider supporting and collaborating with stake holders who increasingly expect to build part of the software.

Who would have thought that Jim Cramer, of The Street, would be speaking today in Charlottesville. For a small community we seem to attract an interesting spectrum of speakers.

Cramer will speak at the Darden School of Business at 4pm and stand by tickets for the public will be available on a first come first serve basis at 2:15. He is known for his theatrical antics when talking about finance and investing. His qualifications are impeccable and he is a crowd pleaser for a certain crowd which gets excited about stocks and the like.

I prefer to work with my investment adviser on a more personal (and fiduciary) relationship, but you can’t deny that Cramer has a certain draw about him. He is responsible for enticing many people to make the plunge into investing and his advice is generally good. For example, he advises that you allot five hours per week to research every issue you hold — hence the reason I prefer to work with an adviser.

Others watch closely the results of his recommendations, but most would agree that his performance is a mixture of entertainment and information.

It isn’t that long ago that I sneered at C programmers as I leapt tall buildings and turned water to wine with Java, but consensus is turning rapidly. The legion of slashdot — who always proclaim the death of a technology with every new birth — doesn’t concern me, but it does rattle my cage a little when Apple CEO Steve Jobs says, “Java’s not worth building in. Nobody uses Java anymore. It’s this big heavyweight ball and chain.”

It isn’t that long ago that Apple touted its strong support for Java so this represents a whiplash of sorts, but I respectfully disagree with Jobs. In case I appear entrenched, consider my article on a successful transition from Java to Ruby on Rails. When new technology permits new or better ways of doing things I’m on board.

The heavy weight of Java is more appropriately its breadth of APIs to enable a programmer to integrate software with obscure hardware devices and algorithms. The production of algorithms used to be the bread and butter of software development but for years Java programmers have had a cornucopia of third party APIs, many of them free, to select. When you program in Java you stand on the shoulders of giants, so to speak.

As much as I like ruby and other innovative languages, I’m averse to them fundamentally as a staffing issue. Code has a much longer life than the typical staff appointment for its development.

But the subtext of Jobs’ statement says more about Apple wishes to position the iPhone as a product more than what he thinks of programming languages. Apple has been a bit dodgy about third party application development for its newest gadget. It is based on Mac OS X and could conceivably support Java as well as a host of other application tools but Apple seems intent to dictate the iPhone’s function. As much as I disagree with this choice as well, the pot shot against Java is becoming more common even though it is unfounded.

Update (8 March 2007): See some notes from an Apple engineer who defected from the Java camp. He makes a lot of good points and offers the delicious line that, “[T]he [Java] APIs are too clumsy to code to, and compared to any decent Mac app, the results look like a Soviet tractor built on a Monday.”

Update (1 May 2007): Michael Kay offers up a gem of wisdom (xml-dev mailing list, Sunday, 1 Mar 2007 23:44:48) on the matter of standing on shoulders and algorithms:

[W]hen you write in Java there are teams of people everywhere helping you. That’s why you get faster development, and that’s why you can spend your time thinking about better algorithms rather than debugging the plumbing. That’s what object-oriented software was designed to achieve, and it works.

Clay Shirky summarized my thoughts exactly about the faith people place in virtual reality:

Like video phones, which have been just one technological revolution away from mass adoption since 1964, virtual reality is so appealingly simple that its persistent failure to be a good idea, as measured by user adoption, has done little to dampen enthusiasm for the coming day of Keanu Reeves interfaces and Snow Crash interactions.

I’ve neither had time nor inclination to comprehend the hype of Second Life. I read Snow Crash and still consider much of it to be prophetic, but virtual reality is a niche that has primarily served the plot line of film. It is simply not relevant or useful in any way that I can imagine.

If I’m wrong, you’ll see me wearing a dunce hat on my avatar.

David Pogue recently wrote an excellent critique about the lack of civility in digital communications, but an ancillary thought has begun to percolate with me. The fifteen year-old’s uncivil response — the object of much hand wringing — is noted by Pogue and many readers for his scatter shot use of punctuation and poor grasp of grammar.

Before I throw stones in a glass house I’ll volunteer that I stumble in this area as well. My only defense is that I try to improve my writing.

Consider for the moment the last time you witnessed cursive print. Odds are that it was penned by someone in the elder generation. I don’t know how it is being lost, but you would be hard pressed to find many people under thirty who can write in cursive.

My disturbing hypothesis is that punctuation — those extra characters which slow down the IM and BlackBerry user — may well go to the same dustbin as cursive. For a period of time we elders will wag fingers or tisk away, but gradually the written language will evolve.

i dont know how ill relate to them

Today I will be giving a presentation to The Virginia Local Government Web Alliance on using XML technology. My aim will be to talk more about the “why” of XML rather than the “how”, but I’ll naturally include discussion on the “why not” since there are many reasons to not use XML. My experience lately is peppered with exposure to document-based markup in literary projects, but I hope to be able to give the audience an idea where XML adds value. In particular I aim to show how exposing data in XML enables serendipitous data reuse. Governments should embrace open content and transparency. Sharing data in an open and documented format is an important step in that direction.

For software developers, how many times have you been tasked with solving a bug which involves some mysterious sequence of steps by a user? This frequent task usually goes like this:

1. Ask the user what he did (user can’t remember)
2. Look in log files (log files may lack detail)
3. Make a good faith effort to reproduce the error and fix it

I’m fond of maxim from Martin Fowler, who explains that fixing bugs is easy, but finding them is hard. What can we do to bridge the information gap to explain why software blows up? I think one answer may be to implement log files as a script.

What this means is that user interaction (which could also be machine rather than a flesh and blood user) is recorded to a type of log which could be executed or re-interpreted into the same steps again. Some types of bugs, particular timing-sentive ones, wouldn’t be solved this way, but it would go a long way toward reducing ambiguity in error reporting. I searched around but didn’t see any implementation of the idea, but I’m curious if anyone can punch some holes in it. Would this be useful?

Princeton researchers have made public some troubling facts about the vulnerability of electronic voting machines. The lack of any secondary record, such as a printed ballot, gives license to a hacker to manage the internal state of the device and alter the vote tally.

Technology itself isn’t the problem, but rather the process whereby these devices have been deployed live is faulty. Diebold is a private company that keeps the the design and software closed to outsiders. The Princeton researchers acquired a machine and conducted their experiments in violation of the Digital Millennium Copyright Act, which prohibits reverse engineering and a variety of tinkering with closed systems. Rest assured that malicious persons with talent wouldn’t obey the law, so it is simply a matter of time for before the wrong people would expose the vulnerability. We are lucky today to read about it in a research paper, but tomorrow we may be dealing with untraceable vote fraud.

A better model would be for the government to develop electronic voting internally or to contract it to the private sector, concluding with an open system. Set up a generous prize for researchers and the lay-hacker to find and fix vulnerabilities. When such a system is refined it could be given to burgeoning democracies worldwide.

Instead, the intellectual property is locked away and held by Diebold for profit and the only external oversight comes by way of skirting the law. The integrity of our election process is at stake, and we will have sold it out if our government doesn’t reign this in.