Outsiders generally have no knowledge of the hierarchy that exists among programmers in terms of skills. While everyone can understand that a programmer has had a job in a particular industry or company, and everyone can understand what place they have in the management hierarchy, the place that a programmer has in the skills hierarchy is invisible to most outsiders; moreover, it's not something they're interested in.
I've talked before about the difficulties of managing programmers, particularly when you're not a programmer yourself. I've talked about how strange this is, particularly in comparison to other fields like sports, music and journalism. My book on Software People addresses this subject. It turns out that understanding the skills hierarchy is important not just for understanding the differences among programmers, it has HUGE impact on innovation and the success of entrepreneurial companies.
The usual view
Most outsiders know there are software people, programmers. They know there are professors and that you can get degrees in Computer Science. They know that experience with software in banking is different than software in healthcare, but they're not sure how. They know there are levels of management. Many people even know some of the common buzzwords referring to languages and other things. But then the knowledge, such as it is, dribbles to nothing, other than things that apply to everyone: is s/he a "good" programmer? A team player? Etc.
When new software fashions emerge, many people know the basic buzzwords. Everyone at this point knows about AI, and even Machine Learning. Terms like Big Data are fading in frequency now, while Data Science is on the rise. Self-respecting managers want to be on top of things, and want to assure their organizations are investing in certified AI experts, and leading-edge Data Scientists -- otherwise, we'll lose out!
Sadly, most of these decision-makers don't bother to learn the first thing about what they're spending money on. Or if they do, it's just details that sound impressive, but don't matter. This explains in part the billions of dollars wasted on software fashion, while true innovators working far from the fashion runways make real-life, practical advances. I've illustrated this phenomenon with AI in healthcare.
The hierarchy
The skills hierarchy in software is generally not fashion-driven, though programmers can get caught up in fancy trends like anyone else. So I don't say that programmers are immune to fashion, but that the skills hierarchy itself changes very slowly.
A book could be written on this subject -- it deserves a book, both about the hierarchy itself, its evolution and how it impacts everyone involved. Actually, a chapter in a book has already been written about it, in this book. This is a long way of saying ... this blog post is just a brief introduction to an important subject that is broad and deep and consequential, but sadly largely ignored.
The first thing there is to understand is the incredible depth and breadth of highly specialized knowledge that is required to get anything meaningful done in the world of software. The start of the chapter may give you a flavor that we're talking about WAY more here than "learning a language."
A tremendous amount of specialized knowledge is required to work quickly and well in practical software environments. The knowledge begins with fluency in one of the many extensive, exacting and demanding computer languages. The language itself, however demanding it may be, is just the beginning. The associated “library” (depending on the times, it could be called the run-time library, the foundation class library, or various other things) has to be mastered as well, because it provides the functions that actually get a good deal of your work done. Just by comparing the bulk of documentation, you can easily deduce that the library is as large as the language itself. You can think of the library as being the set of idiomatic expressions, like “raining cats and dogs” that extend any natural language. Depending on the situation, you may need to know two or more complete languages and associated libraries in order to do anything. For example, if you are writing a database-oriented application, you may need to know the application language (e.g., Java), the language’s library (J2EE), the database’s query language (SQL), the database’s stored procedure language (e.g., PL/SQL), and the special library that connects the application language to the database (e.g., JDBC).
Languages and libraries, however elaborate and challenging to master, are just the raw materials for a real program. Knowledge of these is like a carpenter knowing how to cut and nail lumber – he can work with the raw materials and perform the low-level operations on it. Using raw materials to build a wall of a house requires knowledge of the “design pattern” of walls, and the typical way of assembling them, with the special way of framing windows and doors. While the saws, hammers and nails enable you to build a wide variety of wall-like structures, there is just one way to build them that accommodates standard size windows, doors, sheet rock, etc. There’s a special design pattern for when two walls meet to form a corner, and another one when one wall meets another in a “T.” Once you know all that, you know how to build frame walls, and you get to start over learning how to frame a roof, which itself has many details and variations. Design patterns like this exist in software, unfortunately not as standardized as in home construction, but still important and real. You really don’t want someone building a transaction system who is thinking about the issue for the first time. Just as a person building a wall for the first time might forget about windows or not know that walls are typically built on the ground and then raised up, a person building a transaction system for the first time might not think about audit trails or know that paying attention to row-level locking is a typical concern.
Unfortunately, once you know all that, you are still just at the beginning. If you know them well, you are as well-qualified as a college graduate with a major in English would be to lead a corporation’s response to a lawsuit – while it is true that all the legal documents are written in English and all legal proceedings are conducted in English, there are structures and procedures and patterns of speaking, writing and interaction intrinsic to legal English, and unless you conform to them, you and your case will be summarily dismissed. While the law-challenged English major may think in terms of making a date with the opponent and a judge and talking things through, there are formal complaints, responses, interrogatories, motions for discovery, rules of evidence and procedure and on and on and on.
The chapter goes on to cover things like all the tools you have to know to write, build, test and run your program, not to mention find and fix the bugs.
Once you've done all that, you've mastered a single set of myriad tools that are out there to build and test software, in a single application environment. When you move on to the second set of such tools, you begin to appreciate just how different those different environments are -- at least as different as classical Chinese is
from modern American English (this blog post), from medieval Celtic runes.
Having learned a couple of these amazingly different language systems and the wildly different worlds in which they "live," you're still pretty low on the skills hierarchy.
One of the dimensions "up" from "simple" ability to work successfully within one of these language systems is the ability to find patterns in extensive bodies of code and invent ways to leverage the patterns to transform the code in valuable ways. One frequent transformation takes knowledge out of code and puts it into editable data of some kind, with the net result that there's less code and that changes can be made more quickly and safely by changing the data instead of the code.
A closely related dimension to pattern-finding is working with the tools. If there's a problem with one of the tools you use, can you fix it? Can you build a better tool?
This may not sound like a big deal, but there is a HUGE difference between a programmer who is able to write an application that uses a DBMS (database) and a programmer who is able to create a DBMS. Among programmers who work with the internals of a DBMS in any way there is an extensive hierarchy. At the bottom of that hierarchy are people who don't change source code but are nonetheless expert in the use of it, running through various steps up to those who can make strategic changes or even create a new DBMS. This is a dramatic a hierarchy as starting with people who can drive a car, onto car mechanics who can tune up an engine and going up to engineers/scientists who can create a whole new kind of engine.
Within tools, there's even a broad range. For example, someone who can work with a text editor tool is normally worlds away from someone who can build a compiler or a code generator. As different as calculus is from algebra.
This brings us to a whole new dimension. There are people who are so skilled that, not only can they master the available tools to build effective, working software quickly, they can conceive, create and build tool-building engines to automate the process of using the tools to build software. I won't try to describe this here, but it's a kind of level-shift, much like algebra is a level above arithmetic, fully encompassing it while going way beyond it -- while still "boiling down" to numbers in the end.
With tools, we begin to dip our toes into the deep waters of "systems software." Things like the operating system or networking software or storage software or machine/device control software, each of which constitutes a whole world in itself. The people who are proficient in any of these things are truly rare, and near the top of the skills hierarchy.
Just to take one small example, in the world of the internet there are levels of software and skills that are a little exposed to people who are administrators, but like with most of these things, the bulk of the iceberg is hidden from the surface. Something that everyone uses for example is DNS, which is the software that translates the URL that appears in the top of your browser to the low-level IP address that identifies the server which will deliver content to you. DNS operates in a way that is mysterious to the vast, vast majority of software people, even those who are deeply skilled in other areas. You can be skilled in DNS and still know little about the devices and software that get the moving job done, the TCP/IP protocol engines (and extensive supporting software) that moves data through the network.
Hey, you might say, isn't that esoteric stuff only relevant to the nerds who keep things running, like of like the maintenance people in the basement of our office building? Yup, that's typical thinking. There are maintenance people, sure. And there are people who understanding things so well that they can create a new heating and cooling system for your building that is silent, effective, and uses a fraction of the power of the old one.
Conclusion
There's a great more to be said about the multi-dimensional, multi-domain software skills hierarchy than I've said. I've only scratched the surface. But I hope you've gotten as a take-away that the depth of skills in software is much greater than most outsiders, and even most normally-skilled programmers, imagine. Understanding this skills hierarchy is essential to attracting and retaining the very best software people, and unleashing their considerable talents to transform software-using businesses in dramatic ways.
Comments