Tuesday, August 11, 2009

The Art and Craft of Software Engineering

With all the exciting new technologies, powerful intelligent tools and frameworks popping out almost every day, have we, perhaps, missed one strange and scary turn the software industry has taken at some point in the past years? Ask a typical hiring manager about what they are looking for in a software engineer, architect, programmer. You will typically be presented with a long list of technologies, methodologies, frameworks, and just plain flavor-of-the-day buzzwords that the manager is dying to see on the applicant's resume. Everybody is talking about Agile development, Scrum, fast-paced environments, etc. There would be nothing wrong with that - if one simple requirement and expectation had not been all but lost behind all these buzzwords. How about the basic talent for software craftsmanship? Since when does the familiarity with specific technologies and processes (often quite superficial, by the way) replace the creative vision and true ability to design and code good quality software? One may argue that the latter is always implied and is such an essential requirement that it would be silly even to mention it! I might believe that if I hadn't seen so many software engineers who couldn't design or write decent code even if their lives depended on that! Oh, and, by the way, all those folks happily and proudly claim having worked with the latest and the greatest technologies and tools. It is only because of what I have seen with my own eyes on so many projects, I present you with the rant that follows... ;)

It should not be a secret to anyone that any tool, technology, or process may be misused by an incompetent person. An incompetent programmer will continue to write bad code even if he or she uses a brilliant framework such as Spring. There is no magic! No matter how many meetings the team has each day, how many status reports they produce, how much their managers talk about agile processes and SCRUM, the project will still be a chaotic mess if there are no good programmers on the team.

Any tool, technology, or process is only as good as the people who use it.

In today's sea of programmers, architects, and all sorts of people who call themselves "software professionals", how many are indeed good, reliable Software Engineers capable of producing good quality software?

There is a common opinion that suggests that the usual “80-20” rule is applicable to the software community as well. Some people, however, are less forgiving. The brilliant Edsger W. Dijkstra - considered by many the Father of Computing Science - once suggested that only 10% of all software engineers are any good at all. The outspoken author and software professional Allen Holub in his brilliant article “The Terror of Code in the Wrong Hands” goes even further and suggests that only about 5% - the so-called elite programmers - can be trusted to get the job done without causing various degrees of harm. So, what is the magic ingredient that distinguishes a good software engineer from everybody else?

Wired for Intellectual Manageability

The most important quality of a true software engineer – in my opinion – is the ability (talent and vision) to effectively represent any complex problem as a set of simple, intellectually manageable parts where each of those parts can be viewed, analyzed, and worked on individually without appearing overwhelmingly complex. This involves the ability to identify, separate, and abstract distinct functional domains, concerns, and concepts within the subject system – however large or small that system itself is. The subject may be a high-level architectural view, or a single low-level operation where distinct groups of micro-steps may be identified and abstracted into smaller and more manageable subroutines. Regardless of the flavor-of-the-day technology or design methodology used, any approach absolutely must focus on de-coupling things that do not have to be - and should not be - hard-wired to one another.

What is even more important is that a good Software Engineer understands - and constantly feels - the necessity to maintain such separation and intellectual manageability of individual parts at any given moment of the development process!

Refactoring is not a buzzword. It is not a separate project scheduled for later either. It is the way of programming, a vital integral part of the development process - every single minute of it. Good programmers refactor subconsciously with almost every line of code they write - not because they want to keep things pretty but because their brains are wired to keep things well organized and intellectually manageable at any given moment. That is the key to crafting software that is always in the stable working condition. That is the only way agile development can really work!

In a reasonable amount of time any new technology or process may be learnt and adopted by anyone with sufficient general intelligence. Unfortunately, not everyone has the vision and ability to model well. Just like the gift of playing a musical instrument or creating works of art, that skill may only be acquired with practice by those who were born with the very specific talent for it. I am absolutely convinced that software engineering requires a unique combination of intelligence, scientific mindset, and artistic vision. Different people are born with different talents and inclinations, and those three qualities are absolutely essential for being a true Software Engineer. None of these three ingredients may be aquired with time: you either have it, or you don't.

A true Software Engineer, without a doubt, is an artist and a craftsman. And with that comes elegance, efficiency, fast results, manageability, and high quality of the software he or she designs.

I believe that creative artistic vision is especially essential in software modeling. The ability of the software engineer to visualize distinct concepts and their relationships, represent them in a clear, elegant, and non-convoluted model – that is what makes the difference between quick results and an endless stressful nightmare, between making profit and loosing money, between the success and failure. When I talk about a “model” or “design” I don't necessarily mean a “design document”, a diagram, or a particular stage of the software development process. I am talking about any type of representation of the subject/problem, including – and foremost – mental visualization, at any given stage of the creative process of designing software. The ability of such visualization and modeling is crucial at any given moment, and at any given level – whether one is working on a high-level architecture or a single low-level method or function. That is the skill that allows the better developers to properly abstract the complexity in components, modules, classes, and subroutines, ensuring intellectual manageability of every individual piece. On the other hand, those who lack that essential ability, end up writing excessively complex, convoluted, hard-to-manage, hard-to-maintain, and very costly software.

Invest in Competence: Why Is It So Difficult?

Building software is not trivial, by any means. Software engineers must deal with many complex issues at a time, overwhelming schedules, confusing and ever-changing business requirements. So why do companies continue to make things even more difficult and stressful by hiring mediocre engineers who are simply not capable of designing and coding well? Why is convoluted spaghetti code not only tolerated but is literally a norm on so many critical projects today?

Building a competent development team requires first of all a very competent technical manager who can identify a good software engineer in the sea of fly-bys and imposters. Such manager must also have the freedom and means to offer good compensation and motivation to the people he or she hires. The manager must also have the freedom to easily let go of those team members who are not performing well, show no improvements or desire to improve. When it comes to offering compensation packages, more often than not, the managers are limited to working with what is given to them by the HR departments or other managers. Unfortunately, even if the company itself can afford paying their employees well, among the people who actually determine the salary/rate caps for software engineers not everyone knows and appreciates the difference between a good software engineer and an "average" one. And, tragically, most non-software folks can't even imagine in their worst nightmare that the fine line between an "average" SE and a really good one in practice translates into a monstrous difference between a complete failure and a huge success. As the result, companies "save" pennies on salaries and waste millions of dollars on disastrous projects that never end.

Although unwillingness of many companies to invest into truly high-quality engineers is a serious problem, it may be - at least, partially, - overcome by a dedication of a competent technical manager who finds other ways to interest and keep highly-skilled top-notch resources. Such managers understand that salary, however important, is not the only motivational factor for a good SE. And this is why it is important to remember that true Software Engineers are artists who take pride in their craft and draw satisfaction from their daily work. For a true software engineer, interesting work on a team with like-minded people who are equally passionate about the common cause is, at the very least, as important as the salary. A good software engineer will think twice before turning down an interesting, creatively challenging assignment. On the other hand, such people are unlikely to stick around long if they find themselves surrounded by lazy unmotivated slouches and drones. Imagine an artist coming back to work each day only to find that some thoughtless fool has painted vulgarities over his masterpiece! Day after day! That is exactly how a good programmer often feels on the team with so called "average software engineers."

A more prosaic and, unfortunately, very common reason why so many teams consist of poor programmers is - sadly - the incompetence and personal insecurities of some hiring managers. This is, of course, true for any industry, and has to do with the human nature. While a good manager tries to hire the best people and let them do what they do best, an incompetent and insecure manager is more concerned with preserving his or her perceived status in the company. Therefore such folks naturally tend to surround themselves with the kind of employees who are not likely to challenge them in any way, who will not expose their weaknesses. It is not uncommon for a good software engineer who's passionate about his/her work to outspoken. Some managers don't like that and may act very annoyed and "concerned" when an "ordinary" engineer speaks up in meetings with new ideas or suggestions to improve things. Go to any popular online software forum, and you are all but guaranteed to come across desperate outbursts and bitter stories about managers or "chief architects" that suppress creativity or enforce old-fashioned dogmas on their teams - just because they themselves don't know any better.

Unfortunately, a truly good development manager is just as hard to come by as a good programmer. Due to the unfortunate lack of appreciation for the value of a good software engineer, most organizations pay the best of their SEs significantly less than they pay development managers. If a SE wants to earn more, their only way - within the given organization - is to grow first into a position of an "Architect", and then into a "Development Manager".

Generally, I believe that the only major difference between an "Architect" and a "Programmer"is that a true architect is expected to have knowledge of a broader spectrum of platforms and technologies, the ability to choose and integrate the right ones, while a "programmer" may get away with simply focusing on a narrower set within a project. Both should be equally capable of understanding the principles of software design and programming. I am convinced that a poor programmer may never become a good software architect. And a truly good software architect is always a good programmer. Architecture and design are not limited to the high-level view. In software, every single artifact, however small, requires thoughtful design and architectural vision. In many organizations, however, the title of an Architect relates to a semi-bureaucratic position somewhere between the enginnering team and the management - usually, closer to the management. "Architects" in such organizations spend most of their time in meetings with the managers and business people. They rarely work closely with programmers, and never write code themselves. However, since such positions usually come with higher salaries, good programmers face the dilemma: to move away from programming into a managerial position, or to look for a way out of that company in favor of something better. Some choose to become free-lance consultants just so that they can continue to do what they do best and what they love, while getting paid fairly well without competing for titles with more political types. It is not a secret that brilliant software engineers not always have great managerial skills. If a good programmer becomes a manager, the organization often loses a good programmer and - quite possibly - acquires a mediocre manager. Some of the good engineers, indeed, make for excellent engineering managers. I am fortunate to have worked with such people. Some, however, become lousy managers - just as before they were sub-par programmers.

IN my opinion, for things to improve, the companies (and I mean, the top management and HR) must realize and admit the following:

  • not everyone who calls himself/herself a Software Engineer actually is one; quite a few of such people do more harm than good by creating a false impression that the project is in full swing, while their incompetent actions are actually leading it away from the successful completion; if there's no one to identify such people on the project, the management may never know that the job that takes over a year could have been completed in a couple of months by just one or two real software engineers.
  • one good software engineer can do more work - faster and better - than a team of (sometimes 10 or 20!) "average" software engineers; therefore, hiring only good software engineers would allow to significantly reduce the sizes of teams and departments and get the jobs done faster and better;
  • good software engineers are the ones who do all the work and are the decisive factor in ensuring the success of theh project;
  • it is okay to pay good SEs at least as much as managers - based on the value they bring and uniqueness of their skills; I have witnessed a situation when the hiring manager's ego stood in the way of hiring an experienced consultant only because the manager could not live with the fact that the consultant's hourly rate - if multiplied by 40 hours and 52 weeks - amounted to a yearly income that apparently was higher than the manager's salary (the manager in question had actually openly voiced that concern); managers often claim that they can't find good people for the job, but what they are often not saying is that they can't find good people for the kind of salary they offer...
  • and finally, it should be made easier for companies to fire those who consistently underperform or demonstrate incompetence and inability to create good software - in order to avoid paying the big bucks to those who don't deserve it.
It seems quite unrealistic to expect all these things happen any time soon. I am only hoping that more and more companies - at least gradually - realize and appreciate the true value of a good software engineer, as well as the magnitude of the potential harm a so called "average" software engineer may bring.

Conclusion

It is not the purpose of a Software Architect to produce thousands of pages of unreadable documentation. It is not to impress the business people and upper management by dropping buzzwords and talking things none of them really understands. I believe that the real duty of any Software Engineer - be it an architect who designs a large system or a programmer who develops a particular module - is to work relentlessly to minimize the complexity and ensure the intellectual manageability of every single artifact. That is what ensures fast, efficient, and successful development. Not everyone is capable of that, and those who are not usually argue that they are not given enough time - under the provided timelines - to write quality software. I am convinced that such arguments are nothing but lame excuses.

Good software engineers often half-jokingly explain their obsession with quality by saying that they are simply... lazy! A good software engineer doesn't want to do the same thing over and over again. Or even twice! A good software engineer hates having to waste hours and days on deciphering cryptic unreadable code, including their own. That is why they write code that reads like a book.

Finally, good sftware engineers hate working endless hours on chasing obscure bugs. They shrug at the thought of being part of some "sustaining team" whose only purpose is fixing bugs and putting patches on a sloppily written code. Good software engineers prefer to get things done well the first time around. That doesn't mean no bugs at all. There may always be something that was not accounted for during the initial iterations of the code. However, in a well designed quality system, any defect in a module is easily traceable and may be corrected quickly without any - or with minimum - impact on the other modules. If your organization has a "sustaining" team, that may be an indication that your "development" team is not very good after all...

No one should be proud of how complex and crafty their systems are. On the contrary, a software engineer should feel ashamed of anything that looks complex, convoluted, unreadable, and difficult to understand.

Of course, it's just my opinion. I could be wrong...

6 comments:

  1. This is the greatest and most accurate description of software engineering I've ever read.

    ReplyDelete
  2. I totally agree with all your words. Brilliant. I'd love to have a "Master" like you.

    ReplyDelete
  3. This is brilliant and are all facts !!!

    There is really a need to educate people on how to hire good software engineers. Many companies ask people to carry interviews who themselves are not competent enough.

    What is your opinion on the kind of questions interviewers ask in technical interviews e.g. questions asking about the syntaxes like regular expression kind, or showing some code and asking whether it will give the compiler error or not, or asking about name of some functions, classes or libraries etc ?

    Don't you think questions should be more conceptual, logical and to check the approch towards giving the solution to any problem ? Is it not better to give some problems and ask to just design a logical structure of solution ?

    Is it practicle to remember every syntax, name of every library for a person who works on wide range of technologies and craft the solution very well ?

    Also the process of hiring software engineers need to be improved. Many times I have seen, software engineers are selected giving more importance to communication skills than logical thinking. Companies conduct rounds like Group Discussions (giving some toppic about polities etc) to shortlist the candidate for software engineers. What is your opinon about it ? Don't you think the person who has good logical thinking and a having good approch to craft a solution can perform well that a person who can speek very well with people ? I understand the neccessity of being a good team player but is it more important ?

    Also I experienced, even though a really passionate good programmer is interested programming or designing a good solutions and wants to improve his technical skills, companies force them to take responsibilities of creating project plans, process documents, presentations just because they are more paid so to make use of them.

    I loved this article and I really wish every hiring managers, HRs, developers must read this article.

    ReplyDelete
  4. @Vishal:
    Sorry for a very delayed response... You bring a great point, and it is a topic for a whole separate discussion... In short, I believe that there is a huge misunderstanding of what the terms "communication skills" and "team work" should mean. It is common for someone to be praised for their "communication skills" and being a "team player" only because the person is friendly, polite, and a smooth talker. What is often overlooked is the ability of that person to contribute meaningful results, getting actual work done vs. being smooth in endless fruitless meetings. Team work should mean - above all - consistently delivering work artifacts that are easily accessible, helpful and self-explanatory to the rest of the team. Writing unreadable and undocumented spaghetti code is NOT good team work! I don't care if you have agreed to work on a weekend. If you are producing stuff that only you can understand, you are not a team player, and I don't care about your social/communication skills if you fail to communicate your thoughts, ideas in a way that the rest of the team do not have to kill themselves deciphering your code, documents, emails. Speaking of emails and IMs... Don't we all hate when we ask someone a question in an email hoping to get a clarification on something very important - and receive an all-lower-case one-liner written without any regard for the English grammar??!! As if the person lets you know that he or she is very busy and your question is not important to them - even if you are asking to provide clarification on something that they had done and now you have to deal with! Anyone who writes such email responses is NOT a team player! Also, if you are the only one in the whole organization who is perceived capable of debugging the code you have written, you are NOT a team player, you are not a hero, you are just a really poor contributor.

    ReplyDelete
  5. I almost totally agree with you. Congrats for your article. One point we could disagree: I think it is possible for motivated people to significantly enhance these capabilities with time and dedication (except perhaps artistic vision, I admit, which as you have said, is an important ingredient). One of the tasks of a good team leader should be to help other team members gain these skills to the extent possible.

    ReplyDelete
    Replies
    1. @Joan: Thank you for your comments. I absolutely agree that supporting and encouraging motivated and hard-working engineers who have a sincere desire to learn and improve is essential.

      Delete