Over the last couple of weeks, I’ve had some fun combining Agile concepts, humor, and memes. I thought this might be something you could find useful sharing with your team.
May 26, 2014 Leave a comment
(The following is an excerpt from the upcoming “Backlogs 2″ e-book)
Aristotle would have made a terrible programmer because words don’t actually mean anything. Philosophers in general are a big pain in the ass. They’re also responsible for civilization and modern life.
That’s the conclusion you’ve reached after spending all day Saturday at the library. After the pounding your brain has taken, you are looking forward to some down time.
Aristotle lived over two thousand years ago. Back in the day, philosophers like Aristotle were a sort of cross between motivational speaker, law professor and librarian. They could read and write; and they were where you sent your kids when you wanted them to learn how to live a good life, what good and bad were, and how to succeed.
The Sophists were big back then. Sophists believed that justice and meaning were relative and that it was the art of social communication and persuasion that was important. They taught their students, sons of rich and affluent parents, how to argue both sides of an argument, manipulate the emotions of listeners, and handle themselves with an air of dignity. Basically, they taught all the skills you’d need to become and stay powerful in ancient Greece.
Sophists started off very popular, but by Aristotle’s time most people didn’t like them. First, they charged exorbitant amounts of money. Second, they championed the wishy-washy nature of things instead of actually taking a firm stand on anything. “Man is the measure of all things,” said a famous sophist. Folks wondered if sophists really stood for anything at all except for how to look good and become powerful.
Aristotle didn’t like them much either. He created his own school of philosophy, didn’t charge for it, and along the way invented science.
He thought that things had to have meaning. There was some external universal reality that we could learn, categorize, and reason about. It wasn’t all relative. You could use your senses to gain intuitive knowledge of the world around you. Knowledge could allow you to deduce how things work everywhere. There is a real world, with real cause and effect. Learning about universal truths gave us knowledge we could apply everywhere.
Things that we observe have attributes and actions. It was the reasoning about these attributes and actions over a few examples that gave rise to understanding for all the others of the same type. We can share this understanding by creating a body of knowledge that describes the universe. There was a universal idea of, say a cow. Aristotle called this universal idea, the truest version of something possible, a “universal form”. Once we understood things about the universal form of a cow, we could then apply that knowledge to all actual cows we might meet.
Attributes helped us create a master categorization system. You divided the world into categories depending on what attributes things had. This is an animal. This is a plant. Animals have these attributes. Plants have these. This kind of plant has bark. This kind of plant has none. This kind of plant has five leaves. This kind of plant has three. By simply listing the things each plant owns, its attributes, we could start coming up with categories about what kinds of plants they were. Same goes for animals, and everything else, for that matter.
Same goes for actions. This rock falls from a great height. This dandelion seed floats away. Things we observe DO things. They have actions they perform, sometimes on their own, sometimes in response to stimulus. They behave in certain ways. Just like with attributes, by listing the types of actions various things could do; we continue developing our categorization system.
Eventually, once we’ve described all the attributes and actions of something, we determine exactly what it is, its universal form. Then we can use logic and deduction to figure out why it works the way it does. Once we know how the universal form works, we know how all the examples of universal forms we see in life work. If I know how one rock of coal acts when it is set on fire, I know how all rocks of coal will act. This is common sense to us, but Aristotle was the first to come up with it.
Aristotle’s lesson: there’s a master definition of things, a set of universal forms. We discover the meaning of the world by discovering the attributes and actions of things to figure out exactly what they are. If we understand what something is, and we understand how the universal form of that thing behaves, we understand the thing itself. Once we have exact definitions, we can reason about universal forms instead of having to talk about each item separately.
Categorizing by attributes and actions, understanding by deductive logic, having exact definitions, working with abstract universal forms — those ideas grew into everything we call science. Thousands of scientific disciplines came from Aristotle. Quite a legacy for some old Greek guy.
Categorizing the attributes and actions of things was fairly straightforward and could be used no matter what you were talking about. Creating master categorization systems and dictionaries wasn’t so hard. Deductive logic, on the other hand, got more and more complicated the more we picked at it.
Systems of reasoning greatly increased the growth of math. Throughout the centuries, smart people have longed for an answer to the question: if I know certain things, what other things can I use logic to figure out? They created different rigid and formal ways of reasoning about things that depended on universal forms. Each system had pros and cons. There was set theory, formal logic, symbolic logic, predicate calculus, non-euclidean geometry, and a lot more.
Philosophers devised hundreds of systems of reasoning about all sorts of things. They kept creating whole new branches of science as they went along. Some people were interested in how surfaces relate to each other. Some people were fascinated by the relationship between math and physical reality. Some people wanted to know more about what right and wrong were. Some people wanted to find out about diseases. Some people wanted to know about the relationship between truth and beauty. Each of these used the same core Aristotlean principles of categorization of universal forms using exact definitions, then reasoning about those forms.
Highly abstract, formal logic, in particular, invented by another philosopher guy named Bertrand Russell around 1900, led to something called Von Neumann machines. Von Neumann machines led to modern computers.
That’s right, aside from creating science in general, Aristotle’s ideas about deductive logic using universal forms led to an emphasis on logic, then formal logic, then the creation of computers — machines that operate based on rigid universal rules about what is true and false and how to act based on what is true or false.
This science thing was turning out to be a big hit. Meanwhile you never hear much about the sophists. “Sophist” is commonly used to describe somebody who has no sense of right and wrong and uses lots of weasel words.
But everything wasn’t skittles and beer.
First we had a problem with this thing that Aristotle did when he set things up, where he stepped outside of himself and reasoned about things at a universal level. There was a land above science, a meta-science, and philosophers were the folks who operated outside of science asking useful questions. Because Aristotle asked questions about important things, universal truths, we consider him one of the first great philosophers.
The problem was that using reason and logic to work at this higher, universal level caused as much confusion as positive change in the world.
Not to put too fine of a point on things, philosophy through the centuries has been full of really smart people with one or two good ideas that spend their entire lives making those ideas more and more complex and unwieldy. In a simple world, philosopher X comes up with a good idea about why the sky is blue. In reality, philosopher X comes up with a good idea about why the sky is blue, then spends 40 years and writes 200 papers (including 12 books) on the nature of the sky, what kind of blue is important, and how the blue in the sky is actually the key to fish migration in Eastern Tibet and 50 other odd and vaguely understood concepts which he feels you have to know because they are the key to truly understanding his work.
These were not simple-minded or crazy people. They were the smartest of their day. They were simply taking Aristotle’s idea that there are universal truths that we can discover using reason and logic and trying to take it to the next level.
So instead of simple ideas that spawned new sciences, it was more the case that philosophers came up with extremely complex and delicate theories about things that couldn’t be measured or talked about — except by other philosophers. The useful philosopher who spawned a new science was an oddball, and even in that case, most of the time he created as much confusion among later scientists and philosophers as he shined light on anything.
There was confusion over what different terms mean, over which parts of which philosophies apply to which things, if one philosopher actually agrees or disagrees with another one, and even what the philosopher meant when he was writing all that stuff. Frankly this gave most people the impression that all of philosophy was bullshit. That was a shame, because there was some really good stuff in there too. Every now and then it changed the world.
The confusion in terms, meaning, scope, and method of reasoning got philosophers asking questions about philosophy itself and how we actually knew stuff. Then things got really screwy. Philosophers asked if they could really know whether they existed or not, or whether if they went to a swamp and were replaced by an exact duplicate if that new philosopher would be the same person. We had philosophers pushing fat people in front of trolleys and all sorts of other mental experiment shenanigans. There was a lot of smoke but not much fire. Made for some great science fiction, though.
Second, and this was worse, the idea that we could figure out the mechanism of things took a quick hit, and it never recovered. As it turned out, almost all of the time, we could not figure out why things work the way they do. Remember Isaac Newton? He saw an apple fall from a tree and came up with the Law of Universal Gravitation. This was an astounding mathematical accomplishment that allowed us to do all kinds of neat things, like shoot rockets full of men to the moon. There was only one tiny problem: the law didn’t say _why_ gravity worked, it just gave equations for _how_ it would act. For all we knew there were tiny little gerbils running around inside of anything that had mass. Maybe magic dust. Newton didn’t know, and to a large degree, we still don’t know.
Or medicine. Doctors noticed certain patterns of observations, such as chimney sweeps in 19th century England almost always caught testicular cancer. Some doctors speculated that chimneys caused cancer. They stopped people from being chimney sweeps. Testicular cancer dropped. New observations were made, rules guessed at, hypotheses tested. Over time the terms of the debate got finer and finer, finally settling on something approximating “We believe repeated chemical insults to cells by certain agents over time can cause some cells to spontaneously mutate, at times becoming new creatures which can survive and thrive in the host and take their life.” But we don’t know for sure. We don’t know why. There’s a ton of things we don’t know. All we can do is keep making more refined, tentative models that then we test. As these models get more refined, they get more useful for doing practical stuff in the real world, like reducing cancer rates. But we still don’t know the mechanism, the why.
There is a provisional guess, based on the observation of a lot of data. We keep gathering more data, creating possible rules that might explain the data, then creating testable hypotheses to test the rules, then testing the hypotheses, building a model. Then we start over again. The process loop of science contains called abduction, deduction, and induction, and the guy who explained how it all worked is probably the most creative and insightful philosopher-scientist you’ve never heard of.
Charles Sanders Peirce was the smartest thinker in America in the late 1800s, and even fewer people know him than Frederick Winslow Taylor. Peirce was an unknown hero, a rogue, an outsider, discovering things years or decades before others, but rarely getting the credit for it because he never got attention for his work. In the 1880s, he was the first to realize that logical operations could be carried out by electrical circuits — beating the guy who got credit for it by almost 50 years. He was one of the founders of statistics, creating and using techniques that decades later were to be known as Bayesian Statistics. He was the grandfather of symbolic logic, and much debate still exists as to where Bertrand Russell got all the ideas he had when he went about creating formal logic. Many believe Peirce was shortchanged.
But, like many smart people, Peirce was also ill-tempered and prone to tick others off. His only academic job was at Johns Hopkins where he he lectured in logic for a few years. The enemies he made there followed him the rest of his life. When his marriage didn’t work out and he started seeing another woman, they had enough evidence to get him fired. A trustee at Johns Hopkins was informed that Peirce, while a Hopkins employee, had lived and traveled with a woman to whom he was not married. And that, as they say, was the end of that. For one of the greatest thinkers of the late 19th century, it was the end of Peirce’s academic career. He tried the rest of his life to find jobs at other academic institutions without success.
He lived as a poor man, purchasing an old farm in Pennsylvania with inheritance money but never making any money from it. He was always in debt. Family members repeatedly had to bail him out. Although he wrote prolifically, he was unpopular among the scientists of the day, so his work received no recognition. Only as time passed, as other famous scientists and philosophers slowly started giving credit to Peirce many decades later did it finally became known what a great thinker he was. His “rehabilitation” continues to this day, as the study of Peirce has become an academic endeavor of its own. The man who couldn’t get a job at an academic institution and was considered a crank or crackpot now has scholars devoting their careers to studying his work.
Because of his history, Peirce had a unique outsider’s view. Although he was a philosopher, he always thought of himself as just a working scientist. As such, he saw his job as trying to make sense of the foundations of science, just like the philosophers. In his case, though, it was just to get work done.
To start organizing his work, he began by grouping the way we work with knowledge into two parts: speculative, which is the study of the abstract nature of things, and practical, which is the study of how we can gain control over things. Theoretical physics? Speculative. Applied physics? Practical. Philosophy? Speculative. Peirce taught that although the two types of investigations looked almost the same, they were completely different types of experiences. The scientist should never confuse the two.
This led him to creating a new science called semiotics, which was interested in how organisms work with symbols and signs, external and internal, to do things. Every living thing uses signs and symbols to understand and manipulate the world, but nobody had ever studied how they did it.
Thinking about the importance of practical knowledge and manipulating the world around us led to his famous Pragmatic Maxim: “To ascertain the meaning of an intellectual conception one should consider what practical consequences might result from the truth of that conception — and the sum of these consequences constitute the entire meaning of the conception.”
That is, when approaching some system of symbols, reasoning, or meaning, we have to be prepared to abandon everything, philosophy, tradition, reason, pre-existing rules, and what-not, and ask ourselves: what can we use this for? Because at the end of the day, if we can’t use knowledge to manipulate the world around us effectively, it has no value. And, in fact, the only value knowledge has is the ability it provides for us to use it to manipulate the real world. Newton figured out how to model gravity but not the mechanism, and that’s okay. It’s far more important that we have models with practical uses than it is to debate correctness of speculative systems. In fact, we can’t reason at all about speculative systems. The only value a system of symbols can have is being able to change things.
Mental ideas we work with have to _do something_. Pragmatists believe that most philosophical topics, such as the nature of language, knowledge, concepts, belief, meaning, and science — are best looked at in terms of their practical uses, not in terms of how “accurate” they are. Speculative thinking is nonsense in the philosophical sense. That is, we are unable to have any intelligent conversation about it one way or another.
The great pragmatists that followed Peirce took pragmatism everywhere: education, epistemology, metaphysics, the philosophy of science, indeterminism, philosophy of religion, instincts, history, democracy, journalism — the list goes on and on. As always, the sign of a true philosophical breakthrough is one which changes the universe, and Peirce’s pragmatism certainly qualifies.
Peirce’s lesson: in order for us to make sense of the chaos and uncertainty of science and philosophy, we have to hold both to a high standard of only concerning ourselves with the things we can use to change the world around us. This was far more important than arguing about being “right”.
This sounds very familiar to people in the Agile community. The dang thing has to work, consistently, no matter whether you did the process right or not. Right or wrong has nothing to do with it. It’s all about usefulness. A specification means nothing. It has no immediate affect. A test, on the other hand, is useful because it constantly tells us by passing or failing whether or not our output is acceptable or not. It has use.
Nobody ever heard of Peirce, laboring away in his farmhouse producing tons of papers they never read (at least until 100 years later, in some cases) but everybody knew Bertrand Russell and his star pupil Ludwig Wittgenstein, living large at the opposite end of the spectrum. They lived at the same time but in different worlds. Russell was a member Britain’s nobility, well-respected, rich, and widely admired. Everybody considered Wittegnstein to be a genius and he never wanted for anything.
Russell invented formal logic. He is considered the most influential philosopher of the 20th century. Wittgenstein wasn’t much of a slouch, either. Russell took reasoning to the highest levels man has ever reached with formal logic, yet, like always, there were a lot of pieces that didn’t fit together. Reasoning about right or wrong inside speculative systems was a waste of time, no matter how rigorous they were, as Peirce had shown decades earlier, but no one heard him.
Wittgenstein took it on himself to fix it. So he wrote a big book, the Tractatus, that he felt was the final solution to all the problems philosophy faced. He told his friends there wasn’t much more for him to do. Wittgenstein wasn’t much on the humble side, although better than most. (“Annoying genius philosopher guy” seems to a be a recurring theme in these stories.)
After thinking about things some more, Wittgenstein realized that, surprise, he might have made some mistakes in the Tractatus. So he wrote another book, Philosophical Investigations, that wasn’t published until after after his death in 1953. There was a good reason philosophy kept getting tangled up around itself. There was a good reason that we had difficultly separating the speculative and the practical. There was a good reason that philosophers had a good idea or two, then, by trying to tease it out using systems of logic, always ended up out in the weeds somewhere.
Wittgenstein solved the problem from the other end, assuming formal abstract systems had value and looking for fault elsewhere. What he came up with blew your mind. No wonder Collier was stumped by the consultants.
The problem here wasn’t science, or knowledge, or logic, or reason. It was both simpler and more profound. It was language. Human language. Human language was much more slippery than people realized. It wasn’t logic and reason that were broken, it was the idea of a universal categorization system and universal forms based on language. Human language gives us a sense of certainty in meaning that simply does not exist.
It can’t be made to exist, either. Aristotle’s universal forms might still be valid, but they’re not precisely approachable using human language.
Although we do it everyday, we do not truly understand how people communicate with each other. In speculative efforts, where it was all theory, this led to much confusion, as one time a term was used to mean one thing, and another time it meant something slightly different, imperceptibly different. In our everyday lives, people unconsciously looked at whatever the result they were shooting for instead of the language itself, so they didn’t notice. The exact meanings of words didn’t matter to them.
The reason this was important, the reason Collier sent you here, was that there was a special case where lots of theory and speculative talk ran headlong into a wall of whether it was useful or not, and did so on a regular basis: technology development. In technology development, business, marketing, and sales people spoke in abstract, fluffy, speculative terms, much like philosophers did. But at some point, that language had to translate into something practical, just like things in the practical sciences. And so the problems philosophy had been experiencing over and over again over periods of years, decades, and centuries, where there were subtle differences in terms and different ways of looking at things that didn’t agree with each other? Technology development teams experienced those same problems in time periods measured in days, weeks or months.
Technology development is a microcosm of philosophy and science, all rolled into a small period of intense effort. It’s science on steroids.
To illustrate the nature of language, Wittgenstein suggested a thought experiment.
Let’s assume you go to work as a helper for a local carpenter. Neither of you know any language at all, but he knows how to build houses and you want to help. The first day you show up, you begin to play what’s known as a “language game”. He points to a rod with a weight on the end. He says “hammer”. You grab the hammer and hand it to him. This game, of him pointing, naming nouns, and you bringing those things to him, results in your knowledge of what words like “hammer” mean: they mean to pick up a certain type of object and bring it over. Maybe it’s a red object.
The fact is, you don’t know what hammer means, at least not a lot of it. You only know enough to do the job you’ve been given. Another person playing a language game with another carpenter might think of “hammer” as being a rod with a black piece of metal at the end. As Peirce would remind us, that’s good enough. We have results. We have each played the language game to the point where we can gain usefulness from it. Later on, the carpenter might add to the game, showing you different locations and trying two-word sentences “hammer toolbox” or “hammer here.”
Each time you play the game, and each time the game evolves, you learn more and more about what some arbitrary symbol “hammer” means — in that particular language, that particular social construct.
The problem scientists, philosophers, and many others were having was given to us by Aristotle. Turns out science was a gag gift. The assumption was that one word has the same meaning for everybody. That language could represent some unchanging universal form. But the reality is that meaning in language is inherently individualistic, and based on a thousand interactions over the years in various language games that person has played. Sure, for 99% of us, we could identify a hammer based on a picture. That’s because most of us have a high degree of fidelity in our language games around the concept of “hammer.” But even then, some might think “claw hammer”, or “ball peen hammer” while to others there would be no distinction. Would you recognize every possible picture of a hammer as a hammer? Probably not.
Words don’t mean exactly the same between different people. The kicker, the reason this has went on so long without folks noticing is that most of the time, it doesn’t matter. Also, we pick up language games as infants and use them constantly without thinking about it all of our lives. It’s part of our nature.
Where it does matter is when it comes time to convert the philosophical ideas of what might exist into the concrete reality of computer programs, which are built on formal logic — into a system of symbols that assumes that there are universal forms and that things are rigid and relate to each other in rigid and predefined ways.
Let’s say you were asked to write a customer account management program for a large retailer. Given a title “customer account management program”, would you have enough knowledge to write the program? Of course not. You would need more detail.
Being good little Taylorites, over the years we have tried to solve this problem by breaking it down into smaller and smaller processes which can then be optimized. It’s never worked very well. Now you know why.
Just like with Taylor’s Scientific Management and creative processes, it seemed that we could break down behavior and meaning into infinite detail, and there still would be lots of ambiguity remaining. There’s always some critical detail we left out. There’s always some industry context or overloaded jargon that gets omitted.
Suppose you have a spreadsheet. A stranger walks in the room and asks you to create a list with columns to account for customer information, then leaves you alone. Could you complete that task? Of course not. You would need more context. So how much would be enough?
You could look up “customer” in a dictionary, but that wouldn’t help. You could talk to programmers on the internet about what they keep track of for customers. That might help some, but it would be nowhere near being correct. In fact, there is no definition for customer in terms of formal computer logic for your particular application. There is no definition because it hasn’t been created yet. You and the stranger haven’t played any language games to make one. The term “customer” is nonsense to you, meaningless.
Agile had the concept of bringing the end-user (or Product Owner) in with the team to describe things as needed as close to when the work happens as possible. It was to remove waste, or rework; but in reality it was the best solution to a problem that was not fully understood: meanings are subjective and depend on the language games involved in creating them. We get the guy in the room with us because the team needs to play language games right up until the last possible moment. Just like with the carpenter, you play the game until it’s good enough. “Good enough” is vastly different for every team, every Product Owner, and every problem.
Wittgenstein’s lesson: Communities of humans play language games all the time, it’s a state of nature. Language is inherently flexible, vague, and slippery. It gains meaning only to be “good enough” inside that particular community and only for particular uses unique to that community. Nothing means anything until we play language games to make it mean something. We can’t reason about universal forms. Instead, we have to deal with each item separately. Meaning was relative, highly dependent on the person’s experiences, and created by social interactions.
Looks like the sophists weren’t so stupid after all. Aristotle would not be happy. There was no way in hell that Mr. Collier was going to like this.
Aristotle said that there was a universal form for everything, and that by having exact definitions and using formal systems of logic on it we could deduce things about the universal forms that would then apply to everything in the real world. Peirce said that formal abstract systems are by nature speculative, and that speculative systems are nonsense. Unless it can change things in the real world, it is impossible to reason whether things in these systems are correct or incorrect. Computers can change the world using a formal system of logic, so technically, they might be the first devices ever able to translate abstract concepts into real-world effects. Wittgenstein said that didn’t matter: that natural human languages will never, ever match up with the universal forms that all systems of reasoning are built on anyway, so although the computer can work with universal abstract ideas, you’d never get the actual things people spoke about translated directly into the computer.
Science works because at heart it is based on probability, not because it is based on reason and logic.
Language games are terribly non-intuitive for folks brought up to believe that to find the meaning to a word you simply go to the dictionary, or for folks brought up in the scientific method school of thought that says that language can rigorously describe something so that any listener would receive the same meaning. Heck, it would even drive most programmers crazy, with their concept of language being so closely associated with formal systems of logic.
Believing that language can describe reality exactly has sent millions of projects off the rails, and thousands of philosophers to the old folks’ home early. (Wittgenstein grew to loathe philosophy, declaring that it was much more useful as a form of therapy rather than a quest for truth. We should use philosophy the way we would use a conversation at a bar with a particularly smart person, a conversation with a therapist, or a counseling session with a priest: a useful system of beliefs to help us move through life with more understanding and less pain.)
Instead, your imaginary spreadsheet project would proceed like this: you would form a group with people who had initial diverse internal definitions to the thousand or so words surrounding “customer information”. You would play various games, just like the carpenter and helper, until the group came to a common consensus as to what all the words mean and how they relate to one another. This wouldn’t be a formal process — language games seldom are — but it would be a process, a social process, and it would take time.
What you couldn’t do, because it’s impossible, is capture all the terms, definitions, idioms and such required for the project and convey it to another person’s brain. At some far degree of descriptive hell you might get close, maybe close enough, but you’d never achieve the same results as if you just sat down and had everybody naturally create the language on their way to the solution. And even if you managed to somehow describe enough for some initial, tightly-circumscribed work, you’d never cover changes, modifications, re-planning — all the parts of “living the problem” that occurs as part of natural social interaction using language games.
It would be like trying to prepare somebody for a trip to an alien planet and alien civilization using only a book written in English. Could you cover the language, the culture, how to properly communicate during all of the things that might happen? It’s impossible. The best you could hope for would be to get the person into a state where they could begin their own language games and become one with the culture once they got there. Everybody knows this, yet when we talk about specifications and backlogs for technology development, we forget it, and act as if we can join up the abstract and concrete using more and more words as glue. Maybe special pictures. If only we had the right system, we think. If only we included something about fish migration in Tibet. We are all philosophers at heart.
This was why Jones found that throwing away his backlog and re-doing it — restating the backlog — had value. This was why teams that worked problems involving the entire backlog understood and executed on the domain better than teams who were given a few bits at a time. This was why standing teams in an organization had such an advantage. They were playing language games that removed ambiguity and increased performance over time.
You don’t evolve the backlog to be able to work with any team; you evolve a particular team to be able to work with a particular backlog. It’s not the backlog that matures, though lots of things might be added to it. It’s the team that matures through language games.
Solving technology problems always involves creating a new language among the people creating the solution.
Words don’t mean anything, Aristotle would have been a lousy programmer, philosophers were a pain in the ass, Collier was going to kill you, and you were late for your date.
May 14, 2014 Leave a comment
Every couple of weeks, it’s more bickering.
Should teams co-locate? I don’t know, I don’t think there’s a universal answer for all teams, and I want to work from home but I can’t do that and do my job effectively. Does that stop us from arguing? Heck no! One bunch will line up saying co-location is the only thing that works, another bunch will line up and say co-location is 19th-century thinking about should be abolished.
And away we go.
Should TDD be used? Once again, no universal answer, I have my own view, and the way I’d like things to be and the way they are in my current work are different. So let’s all line up and start bickering over whether TDD is dead or not.
How about some of these new program-level Agile constructs, like SAFe? Same answers. Program-level Agility is just now getting some real traction and good anecdotal feedback from the field. Much too early to generalize, and who knows how much generalization would be useful anyway? But, we can go around the mulberry bush a few times on that.
What is it with the bickering? There’s a moderator on a popular LinkedIn Agile forum that decided that anybody who posted a blog link would have the post characterized as “promotions” and sent off to nobody-reads-it-land. I wasn’t crazy about that decision, made my case, then encouraged him to do what he thought was best.
That was over a month ago. He’s still on the same thread arguing about why his policy is the only sane one, and how we should all agree. Good grief!
Seems like a lot of us Agile guys are really good at arguing. Makes you wonder how much fun it would be working alongside them in a team.
Of all the Agile material I’ve consumed over the years, I like the Manifesto the most. The reason I like it was that there was a room full of guys who were all making money with various recipe books for making good software happen — and they managed to agree on what values should be important no matter what processes you were using. This was a moment of sanity.
Then many of them went out and created new branded recipe books and went back to bickering with each other. (I exaggerate, but only by a little. It’s more accurate to say their adherents did this. Many of the original gang have settled down. Not all.)
I know this drives people crazy, but after watching hundreds of teams doing every kind of futzed up process you can imagine, I only care about one thing: how well is the team evolving using the values as a guide?
I don’t care if they use tools, if they all have funny haircuts, if they wear uniforms and salute each other. Are they using a good system of values and changing up how they do things over time? If so, then they’ll be fine. If not, and this is important, even if they were doing the “right” things, I’d have little faith they knew what the hell they were doing.
A bunch of us yahoos coming in and bickering over whether story points should be normalized or not is not helpful — if we do it the wrong way. If we’re having some sort of “future of Agile” discussion, where the end times are upon us and we must turn back now and go back to the old ways? Probably not so useful. If, however, we’re sharing experiences and talking about why some things might work better than others, while acknowledging that many people do things many different ways? Probably much more useful.
We — and I mean me as much as anybody else — tend to make the internet some kind of drama-of-the-week contest. Everything is a disaster. We are all emoting. Well, I’ve got news for you: it’s not. Encourage using the values as a way to establish trust. Insist on teams continuing to try new things and learn. Have some humility about what we actually know as an industry and what we don’t. The rest will work itself out. I promise.
May 7, 2014 Leave a comment
Dear Agile. I’ve been with you a long time. Heck, I was there back in the 80s and 90s, even before you were born. Remember Fast Cycle Time? RAD? Or any of the other systems that emphasized rapid development in close association with the users? Remember all the fun we had before you had a name, when we were doing weekly code drops of tested code? Customers loved us. Developers were happy. It was a good thing. We were all happy then.
And it was even good after you were born. We had a whole bunch of things we could drop into the Agile bucket. There was eXtreme Programming. There was Agile Data modeling. There was even an Agile Unified Process. Seemed like the more we thought about it, the more other things besides development also needed to be Agile. Marketing, Sales, Startups, Program Management. The Agile manifesto was such a good set of values that wherever we applied them, we found improvement.
And that’s the problem.
Agile, it’s time to grow up. When you were born, it was easy to believe in developers as sitting in some small room, customer by their side, banging out great solutions. But that was then, that was your childhood. Now that you’ve gotten older, you have to be aware that most of the time these developers exist as part of some bigger ecosystem: a cross-functional team to develop a new product line, a team of vendors bidding on a government project, or a lean startup team practicing customer development.
It’s not just software any more, if it ever was. Now we’re expecting you to play in the bigger world with a broader view of how developers actually are employed. You have to grow up.
I know what you’re thinking: why can’t everybody just be like Spotify? One big company with a bunch of kick-ass Agile teams in it? Why can’t we just get paid, without having to think about where the money comes from? If the company/customer/client wants us to work in a large group of developers, why can’t we just tell them no? Why should we do estimates? Why do we have to add in all this other stuff anyway? Why can’t it just be the way it used to be, with you and your pair-programming buddy writing killer software for one guy you had to make happy?
I don’t blame you. Growing up is tough. Nobody wants to do it. It’s tough to look out on the world and realize that there are so many other important things besides just the things you’ve been used to. It’s tough realizing so many other people want part of your time, and you have duties and responsibilities in life that might not be so much fun.
I understand this.
But the world needs you. You see, the values you emphasized and the techniques that resulted from them are applicable across a lot more than just software. The world is becoming digital, and the digital world needs Agile to help guide it.
Sure, the world outside the team room has a lot of dysfunctional behavior. There’s management principles that are outdated, there are ideas and models that are harmful to the people who hold them. Many companies are being run in a way that actively destroys morale. Aside from product and intellectual property development, which was our old homestead, there’s also manufacturing and service work. Those activities play by their own rules, and although our value preferences stay the same, the practices and techniques change. That can be very confusing to you, I know. I’ve seen you struggle with trying to apply product development ideas to services and manufacturing. Sometimes it works. Sometimes it hurts.
But I want you to know, I believe in you. I’ve seen you overcome big odds, and the people using Agile principles today are some of the smartest people in the world.
You’re going to do fine. But enough with the whining and insisting on living in a tiny piece of the world. It’s time to grow up.
April 21, 2014 Leave a comment
Agile has a business problem.
I was watching a video of Uncle Bob Martin awhile back, and he said something that struck a nerve.
[Paraphrased] “When we sat down to do the Agile Manifesto, several of us wanted everybody to be sure that the purpose here was re-establishing trust with the business.”
At the time, he was making a case for TDD — how that if we keep writing code in Sprint 1 that breaks in Sprint 3, or delivering buggy software, or not being reliable, you lose the trust with the business. But I think his observation says something about the technology world in general.
Given a choice, we don’t want to be part of the business. Many technology developers work at an arm’s length or more from the actual business. After all, one of the big “benefits” of Scrum is that the Product Owner brings in a list of things for the team to do. The team isn’t expected to know business things, only deliver on a to-do list from the PO.
Most small company development teams have sales, marketing, and management out in the field bringing in the money. They just deliver stuff. Most BigCorp teams are so far away from value creation that I doubt many on the team could describe the relative importance of what they’re delivering or how it fit into the corporate strategy. They just deliver stuff. Non-profit development teams deal with inter-office politics: their goal is to produce something the non-profit likes. They just deliver stuff. Everybody — except for startups — just delivers stuff.
And why not? Let’s face it: business is messy. The amount of work you put into a product is not directly related to the value it might create. There are many factors outside your control. There is no single person to make happy. And other businesses are trying to do the same thing as you are — many of them with more experience and capability than you have.
Wouldn’t you rather just work from a to-do list too?
We tell ourselves what we want to hear, what’s popular. So we focus on our backlog, how well we create things using it, and how to improve our little world. We don’t worry about the larger one.
The problem is: once you draw the line at the Product Owner, once you make the trade-off that, for purposes of “all that business stuff”, the Product Owner is the magic guy, then it becomes a lot easier to start making other arbitrary compromises. Wherever there’s a business consideration, why not just change the rules so to eliminate it?
Is the Product Owner asking you for estimates on when you’ll be done? Stop estimating! Does several new recent corporate acquisitions mean that the new product is going to have 8 teams all working together? Insist on firing everybody until you get one team. After all, the organization should adapt to Agile, not the other way around. Rituals like demos and sprint planning bugging you? Hell, just go to kanban/flow-based work and who cares about cadence, calendar, or commitments?
Sometimes these conversations make me want to laugh. Sometimes I wonder: is there something there? Am I missing something that’s going to provide value for folks down the road? I intuitively like a lot of what I hear: things like flow-based work systems. But I think that’s the problem: we’re confusing the theory in our heads and what we’d like to hear or not with what actually works.
We’re resistant to philosophical change. I know there’s a lot of people in the Agile community who already have everything figured out. As a coach and technologist with 30 years of experience and eyes-on hundreds of teams, back when I started blogging and writing, I thought others would be happy to have me share with them.
I was mistaken. Instead, amazingly, most coaches and practitioners — even those with a year or two of experience and a 2-day class under their belt — already know everything. The old cranky farts are worse, because, well, they’ve been around the block a time or two. Selection bias is a powerful thing, and if you decided you figured something out back in 1995, you’ve had decades of time to convince yourself there isn’t much more to learn. We get the general idea of something in our head, then we insist on the rest of the world conforming to that general idea.
The Agile community is an echo chamber. Instead of viewing this entire enterprise as an evolutionary effort spanning many decades, we view it as a brand-based deal where we’re all competing for market share. People don’t worry which parts of Scrum to adapt here or there, they worry about whether Scrum is still “pure” or not. They worry about whether we have “true” Agile or it has gotten corrupted.
These ideas, instead of being just ideas that we’re supposed to take and learn from, have become some weird kind of Platonic form. There is a “true” Scrum, and if we could only stay in the light of true Scrum, everything will be fine.
We’re not using ideas as stepping stones; we’re coming up with things we like hearing and then branding them. Then we have arguments over branding. This is dumb.
The Agile community continues to have a crisis of conscience: are we part of a system that is trying to learn better ways of providing real value to real people? Or are we trying to create a system of value for the way we work that we will then expect the world to conform to? It’s an important question, and it’s important to determine which side of the fence you come down on.
To me, Agile is a series of ideas about things that mostly work in incremental and iterative development. None of it is perfect, none of it is a magic bullet, and everything I’ve seen so far has places where it works better and places where it doesn’t. I don’t say that to dismiss ideas like Scrum, XP, and so forth. I say it to put them in context. But it’s nothing if it doesn’t do something real for somebody, if it doesn’t deliver.
I expect this body of work to grow, and hopefully I can come up with a thought or two that might last a while. But if it’s going to grow, it’s going to need to keep becoming more and more a part of business and less of its own separate universe.
Note: I didn’t say the entire Agile community. I still have faith that this is the best place to be for people who care about happy and productive teams. My point is that the future is in startups and in adapting what we do in order to create real value (instead of just cranking out technical solutions), not in taking the tiny bit we’ve figured out so far, over-generalizing, and then trying to make the world conform to it.
March 30, 2014 Leave a comment
Lately there has been quite a donnybrook in the Agile community. Is it time to leave Agile behind?
As it turns out, the suits have taken over, taking all that great Agile goodness and turning it into just more command and control, top-down, recipe-book dogmatic nonsense.
So some folks say we should just “return to REAL Agile”. Some folks say the brand Agile is dead: it has been killed by entities intent on adopt, extend, extinguish. Some folks say that Agile was never any good to start with — either it was too wishy-washy and meaningless, unlike Scrum, or that it was too emotional and not logical enough, unlike DAD (or many others).
As for me? I just like collecting shiny objects.
Agile is a brand. It’s a brand without an owner. That means that each of us has to come up with some kind of working definition for what the brand stands for. For me, that brand represents something like “best practices in iterative and incremental development”. If you’ve got one of those, you can publish it under Agile. Works for me.
Of course, people go off the deep end just on that formulation. What do we mean by “best practices”. Isn’t that prescriptive? Are we saying that we know the perfect answer to how teams should act?
I’ve been around this shrubbery many times, and I think it comes down to what you want to do in life.
I’m in this to help developers. I really don’t care what you call it, or how it makes me feel. I don’t even care if it all fits together in some master plan, or whether it has a class or not. If it helps developers, I want to do it.
Some folks are in this to change the world. To them, Agile is a movement. It exists to show what is wrong and what needs to be fixed.
Some folks are in this for a theory of life, the universe, and everything. They want a philosophical grounding that they can use for their development, their team — even the organization.
I believe these last two groups are expecting a bit much. Sadly, though, it’s these powerful emotional constructs that drive adoption of things. So they’re always going to be with us.
I also believe that Nietzsche was right: any sort of structured system of belief is going to be self-contradictory. In other words, no matter what you do, once you make a system out of it, it’s broken. And we technologists love to make systems out of things. Even things that are supposedly non-prescriptive. After watching this industry for decades, I believe this is unavoidable.
All of this has led me to believe that fighting over the term “Agile” is a fool’s game. If you only care about what works, it doesn’t matter what folks call it, and no matter what comes along next, the community is just going to do to it what it did to Agile. If you care so much about terms, then get your head out of your ass and start focusing on what’s important. We have work to do. Give up on allying yourself to a certain brand or term — or proclaiming that you’re opposed to a certain brand or term — and just collect things that make developers’ lives better. Then we can share.
March 11, 2014 Leave a comment
1900. Allentown, Pennsylvania. The dawn of a new century. Manufacturing looked nothing like today.
For one thing, there were no managers. There were owners, There were overseers. Foremen. But not pocket-protectored MBAs wandering the shop floor, clipboard in hand. Back then owners were upper class, landowners, holders of property. They didn’t bother hanging out with riff-raff. Sure, if necessary, you could find the owner or one of his lackeys in the main office. An overseer, a shop foreman or two were on the floor keeping the men sober and busy. But aside from production goals and tradition, the workers were left to their work.
And why not? For skilled work, after completing a lengthy apprenticeship, the job was handled by craftsmen, many times from a guild. They made their own decisions about how their work was performed. Simple work? Workers were hired off the street and paid by how much output was desired. No matter the type of work, each job was a special creation, a snowflake, even if it was just something that fitted into a larger whole.
For unskilled work, workers didn’t do anything. Or rather, they did everything, but weren’t particularly good at any one thing or another. Need more output? Threaten more. Promise more. Workers were miserable.
Craftsmen were the paid same amount no matter how much was produced, and centuries of the apprenticeship system taught workers the wisdom that the more they did, the worse things got. The more they did, the more mistakes they would make. The more they did, the more they would stand out. The more they did, the more they would work themselves out of a job.
Let’s say you got a wild hair up your ass and came in one week and killed it. What do you think would happen next week, kid? I’ll tell you what. Next week the bosses would expect you to work at the same rate, that’s what. And they’d expect everybody else to work at that rate too. Do you have any idea what kind of living hell they’d make for us if we were all constantly working at top speed?
Nope, twice the people working at half efficiency was much better for all concerned than all of us working as hard as we can. It was even better for the company, because then there were extra people around to take up the slack when people got sick. The main job here, if you were smart, was convincing management that you were working as hard as humanly possible — while slacking off as much as you could without getting caught. More people, more money, more leisure time, more security, more everything.
It was, after all, us against them. We have to create our own slack. We can’t trust the business to do it for us.
The early 1900s were the sunset of the Victorian Age, and owners really didn’t much care. They were owners. They were obviously better educated and more hard-charging than the average knuckle-dragger on the shop floor. Foremen provided the raw material, said some encouraging words from time-to-time, and mostly just left workers to sort things out the best they could. Fire the slow workers. Make sure nobody was goofing off. The beatings will continue until morale improves. This is the way it is. This is the way it always was.
It was, after all, us against them. We have to use a stern hand on the workers or nothing gets done.
And so what if things ran 10% better this month than last month? The company made money through political connections, through inside contacts, through an old-boy network. Not by making a better widget.
It sounds odd today, but back then nobody actually organized their work in the abstract. Sure, work was orderly, but nobody sat down and looked at how it all fitted together. There was no master plan or detailed architecture at the task level. There was no optimized design. For management, it was beneath them. How would they know how to do things better than the workers? For workers, using rough guesses and rules of thumb was the way of organizing the work handed down from person to person. Your mentor guessed his material needs a certain way. This was the way you guessed your material needs. Your mentor paced his prep work a certain way. This is the way you paced your prep work — if either of you thought about pacing at all. Or prep.
One mentor taught it one way, another a different way. The standard was that there was no standard.
It was a true collection of artisans, and because everybody was an artisan, nobody was. True skills in such an environment were social ones. Get along with each other. Be respected by your peers. To owners, in terms of measured technical expertise, all the workers looked mostly the same. Other than that, it was all reputation.
Taylor fretted. How much activity was being wasted? How much material over-ordered and never used? How many workers were doing jobs they had no ability in? How many men were living lives of pointlessness, either working hard and not smart, or not working at all while nobody was the wiser?
No. No matter what kind of happy face you wanted to put on it, whether you were owners or workers, this arrangement was killing morale, efficiency, and most importantly, led to distrust, resentment, and a bitter us-versus-them attitude for all involved.
And what of the larger picture? What kind of impact did this work structure have on factories? On the nation as a whole? On the families of the people caught up in it?
Just because we had always done it this way didn’t mean it always had to be done this way.
It was unacceptable. Something had to change.
A hundred years later and it had changed, massively, but somehow we ended up just trading one misery for another.
2001. Snowbird, Utah. A small gaggle of software developers, consultants, and process wonks — nerds, really — met to discuss the state of technology development.
It wasn’t that good. Over the past couple of decades, as more and more people entered the workforce as programmers, dozens of different methods for improving productivity had been tried. Books written. Whitepapers prepared. Doctoral theses defended. People were certified. People set up huge organizational systems to manage technology creation. People honed and tweaked. They lived lives of quiet desperation inside huge corporate and governmental machines dedicated to making technology happen.
For skilled work, nobody did anything, or rather, they were busy, but little got accomplished. Workers were miserable.
Early on, people developing technology started with a simple premise: things happen in a certain order. It’s sequential. People need something. They tell us. We write some code. We debug. We give it to them. They are happy. One thing naturally leads to another. Work naturally happens in a clear sequence.
So it looked like developing technology was just a matter of listing out each step and doing each step correctly, right? Pieces of work come in at a high level, then cascade down to product coming out the other end. It was a big waterfall. You got on at the top, you got off at the bottom. This idea that sequencing was the most important part of the structure of technology development and that we should order our work as some version of a large linear sequence was called “waterfall methodologies” or just “waterfall”.
In theory, waterfall was natural and intuitive. Only problem, in practice it didn’t work so well. In practice, there wasn’t always a single path production could take. There were branches, loops. Once product was released, it might enter the system again as people needed to fix problems or add new features. Instead of a simple waterfall with a few stages from “beginning” to “end”, technology development started looking more and more like a flowchart. Then it started looking like a really complicated flowchart.
The idea that technology development is a flowchart with cycles, not a straight line, over it’s entire lifetime is called a Software Development Life Cycle, or SDLC. The idea is that whatever you’re doing in a modern technology development shop, it’s just work inside a little node on a flowchart somewhere called the SDLC. Your node has inputs and outputs. Rules of operation. Quality criteria. You worked the node. Or the node worked you. Depended on how you looked at it.
Even at this level of complexity things weren’t complete, though. Or rather, no matter how complicated we made our flowchart, there was always something new that needed to be added, and it was important to add it. After all, companies in this world made money by being as precise and efficient as possible, by making a better widget. Chasing the correct SDLC, tweaking it and making it better, looked a lot like something that over time created diagrams of infinite complexity.
So practitioners doubled-back. Instead of looking at things over a lifespan, how about if we changed focus? Looked at things in small increments? You’d do the same things over and over again, releasing a new little bit of stuff when it was ready. Instead of focusing on the tiny details of what you are doing, couldn’t you just tell me what you can do by next Friday? Maybe technology development wasn’t a flowchart, maybe it was just a string of little nodes all the same, each one representing a unit of time.
Focusing on delivering pieces of a whole in small units of time was called incremental and iterative development, and it’s mostly stuck with us. It helped. But not as much as we hoped.
First, we still had the question of process. What did we do inside of each release, or increment? Nobody knew. How long should an increment be? Should there be a standard increment length? Should we just work until we think we’re ready? Obviously if we wanted any chance of improvement, we had to break down, define, and optimize each piece of our work, right?
People added in details to answer all of these questions, and, just like before, a simple idea of doing small increments over and over again in an iterative fashion became many super-huge systems of how to program, how to report your time, how to create one of the standard project management plans, what sorts of status reports were necessary and when, and which of the 83 standard artifacts you needed to complete for the upcoming release meeting next Thursday afternoon. Yes, the nodes were all the same. They were just little boxes of time. Inside each node, though, we crammed a miniature SDLC. And that SDLC could get just as complicated and unwieldy as the old ones did.
It was the dawn of the Age of Dilbert, and workers led days that were full of meetings, mostly management meetings, mostly asking why things weren’t completed yet. Each and every bit of their time was tracked. Everything had a check and a cross-check. There was a standard structure and system for everything they did and everything was meticulously measured and tracked. We define. We measure. We optimize. Productivity was king.
But productivity did not increase. Instead, it plummeted. There was plenty of activity, mind you, at least on paper, but nothing of any value got released in any reasonable time frame. Developers were miserable. Managers were miserable. Customers were miserable.
Trust was lost between the business and the workers. Promises were never kept. Deadlines always slipped. No matter what kind of happy face you wanted to put on it, whether you were owners or workers, this arrangement was killing morale, efficiency, and most importantly, led to distrust, resentment, and a bitter us-versus-them attitude for all involved. Eventually each side recognized the game. It was the same old one. It was, after all, still us against them.
Looking back, some extremely smart people across the industry had worked at this problem of how to organize technology development work over a period of many decades. All sorts of various systems had been erected, diagrams created, but no matter where we started, no matter the philosophy and rules used, technology development always ended up a complex mess of rules, contracts, meetings, flowcharts, late hours, unhappy spouses, and kids with missing-in-action parents.
It was unacceptable. Something had to change.
Back in 1900, Taylor agreed. Something had to change. He looked at the shop floor. The company was spending millions of dollars on labor, and yet nobody was using any kind of system to keep track of how things were done. It beggared belief. Surely we could learn something from technology and science and apply it to the most important thing we do during our day — our work. Surely work had to be something more valuable than simply a force of wills between owners and workers.
What we needed was a scientific method for managing our work.
1. Replace the old rule-of-thumb work methods with methods based on a scientific study of the tasks.
2. Scientifically select, train, and develop each worker rather than passively leaving them to train themselves.
3. Cooperate with the workers to ensure that scientifically developed methods are being followed.
4. Divide work nearly equally between managers and workers, so that the managers apply scientific management principles to planning the work and the workers actually perform the tasks.
In short, we needed to emphasize science, reason, and hands-on management over tradition, old wives’ tales, and class separation. Science was the way forward for all of us. Science was our hope. It was insane not to use science to help us improve the way we created things. In Taylor’s mind, it wasn’t that class distinctions would go away, lord no. You’d never take a lowly worker bee and make much of a thinker out of him. That wasn’t even in the cards. We couldn’t change reality. We couldn’t change attitudes and values, but we could, and should, change our processes and tools; in fact we should break down, define and tweak each and every thing we do during our work day to make sure it makes sense to us and is providing the maximum amount of value.
Looking out over Allentown, Taylor knew our emphasis was correct, we just needed to plan, measure, and change every little bit of our action we took during the day, use the scientific method to meet our goals better.
Which is exactly the opposite conclusion the Agile Consultants reached a century later after living in the world Taylor created.
Looking out over the state of technology industry from Snowbird, no matter where people started, no matter what philosophy and rules they used, technology development always ended up a complex mess of rules, contracts, meetings, flowcharts, and late hours. The business always lost trust in the workers. The workers always lost trust in the business. The customers lost trust in both. Everybody ended up unhappy.
It wasn’t that any of the specific things we did or tools we used were bad, really. In many cases there were really good reasons for folks using them. It just always seemed that somehow things got out of hand, and processes and tools that were supposed to be supporting the work and helping things run smoother instead became prisons that people lived and worked inside of. It wasn’t enjoyable. It didn’t even accomplish the things it was supposed to.
We couldn’t always change our processes and tools. Many times they were the best we had, and we had to use something. But we could, and should, change our attitudes and values.
We needed a change in emphasis. Thus the Agile Manifesto.
The Agile Manifesto is one of those things that started off as a few bullet points in a conference somewhere in the backwoods of industry and somehow gained a life of its own. Sure, there were a lot of other things that came out of the conference — they decided on the buzzword “Agile”, they came up with a set of principles, there were various structured systems of work that spun off. They drank beer. There was a lot of writing and marketing. But nothing with as much staying power as the Manifesto itself.
The reason the Agile Manifesto resonated was because the Manifesto was not just another system of telling folks how to run technology teams. Instead, it was a reaction, a reaction to the hundreds of methods that had already been proposed and tried (and billions wasted) developing technology. It wasn’t a set of instructions. It was a set of values. A polite primordial scream. People who read the manifesto could choose whether to agree with the values or not. If they agreed, they were then free to apply them in any way they desired. They were asked to agree on where our values had gone astray, not about the perfect way to run our next technology development program. There was always plenty of that.
The Agile Manifesto went something like this:
“We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.”
Might want to read that list again, mentally comparing the items on the left to the items on the right.
Pretty simple stuff, and it might have gone without much notice, but the timing was good. There was a website. People saw it and liked what it said. A movement was born.
Then came the hard part: proving that the principles had practical value and weren’t just more happy-talk or managment-speak, sophistry, which is exactly where Frederick Winslow Taylor found himself back in 1900 when he started spreading the gospel of Scientific Management.
At first they hated his ideas, and some of the ones that hated it the most were the ones you’d least suspect: factory owners.
That’s right, because although we associate management today with close oversight and control over the workers, early 20th century owners already had all the control and oversight they wanted. They could hire and fire at will. They were comfortable. They had tradition.
In many cases they even enjoyed very close, paternalistic relationships with folks on the floor. There was rarely outright hatred. A good worker was loved by his boss and he loved his boss too — all while he slacked off as much as possible. All while the boss planned draconian incentives to keep the workers honest. No hard feelings. It’s just the way things have always been. Upper class cracked the whip. Lower class pulled the oxcart.
Owners hated Taylor’s Scientific Management ideas. Here come these guys, these Management Consultants, carrying this radical departure from tradition. Required a lot of oversight and work that looked silly, and at the end of the day, what do you have? Same old lower class, uneducated riff-raff, god bless ‘em, that you had to begin with. You couldn’t make a silk purse out of a sow’s ear. These ideas amounted to nothing much more than an an unnatural mingling. Of ideas. Of classes. Pointless bean counting.
He was the first person to ever use a stopwatch on the factory floor. Who would use a stopwatch for menial tasks? Those were for sporting events. Train schedules. But Taylor would. He would sit and time people as they did their jobs, breaking down each task and cataloging it, watching closely how long it took to accomplish, what the averages looked like, and how much variance there was. Which methods worked better than which others.
Scientific Management: the idea that careful definition, measurement, planning, and architecting could make a huge difference overall.
Taylor watched some men moving pig iron. On average, they moved 12.5 tons per day. If necessary, the company could offer huge sums of money to get them to move more, say 47.5 tons, but experience showed that the only thing that would happen would be that the men would try as hard as they could for a while and then give up, exhausted.
If, on the other hand, a specially trained overseer, a real manager, conducted experiments to determine how much rest was required and at what intervals, then the manager could synchronize the lifting and resting among all the men such that every man could move 47.5 tons per day, every day, all without becoming exhausted. Manager does the thinking. Worker does the working.
Don’t just whip the horse harder and pull on his reins. Take apart everything the horse does, climb inside his brain, and optimize each motion. Think for the horse.
Of course, not every worker would be capable of moving that much pig iron. Perhaps only 1/8 of the general population could. Not all of our skills and weaknesses are the same. Some folks have attributes that make them very good at moving pig iron. Most do not. The company would be doing both the men and society a favor by selecting only those people for which moving pig iron came naturally. This could be accomplished with physical fitness tests. Science helped the managers define the work, plan the work, measure the work, and select the workers for the work.
No detail was too small for Scientific Management to analyze. Taylor studied shovel design. How long does it take to operate a shovel? How does that interval change depending on how much weight the shovel is carrying? What is the optimum weight for an operator to lift and move using a shovel in order to maximize throughput?
Based on time and motion studies, he determined that the optimum weight a worker could lift using a shovel was 21 pounds. Since various materials being shoveled had various densities, obviously shovels should be sized according to the type of material being moved so that they would, at their fullest, weigh 21 pounds. Companies started providing workers with appropriate shovels. Prior to that, workers brought their own shovels to work. Substantial productivity gains were realized — and more importantly, concretely proven — by recording productivity metrics before and after the change. No more running things by anecdote or tradition.
As this caught on, others joined in, including a husband and wife team that used new and highly-experimental motion picture apparatus to study motions of bricklayers. More dramatic improvements were proven.
As it turns out, Scientific Management was something that any educated person with discipline and a scientific mindset could get in on. Just required a new way of looking at things. Which is exactly the way the Agile Manifesto consultants wanted things a hundred years later. A new way of looking at things. It was just common sense: we should share values. Empathize those. No-brainer.
But it wasn’t a no-brainer. People hated the Agile Manifesto in 2000 just like they hated Scientific Management in 1900. And surprisingly, many of those who hated it were also the ones you’d least suspect: developers living in old oppressive systems.
Why should they like it? They had a complex, unwieldy system, sure, but it was their system. It was something they had spent many years putting together and tweaking to optimum performance. Sure, some of the new guys thought the system sucked, was overly complex, and ate people’s souls, but that’s just because they were new guys. Hippies. Slackers. They didn’t understand the reasons why things had to be the way they were because they weren’t looking at all of the things that could go wrong.
And here come these new schmucks, these…Agile Consultants? What do they offer? A bunch of hot air and loose talk to encourage youngsters and all those already unhappy with the way things are going. These Agile Consultant guys would be better off spending their time and energy explaining why things had to be the way they were instead of building fantasy castles in the air. Perhaps doing some real research on processes and procedures we might have missed. Give us more detail. Provide practical advice instead of regurgitated marketing speak.
And yes, things were taking a lot longer than they used to, but that’s not important. What’s important is that we’re finally beginning to operate this place like an engineering shop instead of a mob. Optimization comes later, after standardization and baselining. Everybody knows that. We’re becoming professionals. We have a system. We make complex things, therefore our system should be naturally complex. We are just now getting around to adopting some intricate yet practical and important processes, and these guys are suggesting things that sound like they were invented last week in a beer hall and could be explained on a page or two in the back of a comic book.
But the really crazy thing? The thing that drove us nuts? Most of the time, the practices developed from the Agile Values? They worked.
For example, they had this accomplished, committed, needs-help cycle. Once a day, each person would go through a brief communication cycle that covered — and only covered — their accomplished, committed, and needed-help items. The entire team would stand in a circle and toss around a Nerf ball. When you caught the ball, you communicated to the rest of the team — not a manager! — what you accomplished over the last day. You communicated commitments about what you thought you could accomplish over the coming day. Finally, and this was bizarre, you communicated what you were weak or worried about and asked for help from the others. And the team provided it! Without being told! On their own!
The same thing happened at the next level up, the team. The team repeated this cycle every so often, demonstrating what they had accomplished, making a commitment to do more, admitting their weaknesses, then talking about where they needed help. Then management would be directed, by the team, no less, where they should help. By the team!
It was something you could train people on in five minutes. Heck, just reading about it and you’re already trained. Yet every team that honestly gave it a shot found productivity increasing, happiness increasing, participation increasing. Some even called this communication cycle the “five minute miracle.”
Then some of the other Agile guys had this idea of the team being the unit of work delivery. Instead of assigning work to a person, as has been done since the dawn of time, the idea was to let the team as a unit decide, commit, to what it could accomplish. Then the team, as a unit, was responsible for delivering it, not a specific person. All the problems and variances among different people still existed, of course, but it was up to the team to handle them. From the outside there was the team, and the team was all there was. Work went into the team, the team did work, and work came out of the team. What people did inside the team was of no concern to anybody but those in the team. Insanity.
But genius, too. This freed each team to dynamically optimize its work patterns depending on the project, the team members, the type of work, perhaps even the individual work item or task. The company had a trade to make: time and money to the team for delivered product back to the company. The team had the freedom to make the value creation happen as productively as possible inside its commitment time box, which somebody decided to start calling a sprint.
Teams working in this fashion, with no hangups or baggage carried over from previous systems, customizing their optimizations as they went along, were reporting some astounding successes. So much so that most outsiders found it hard to believe. Two, three, ten times faster? Happier workers? Customers singing praises? Technology development, Management Consulting, had never seen that kind of performance improvement. That kind of excitement. Ever.
Then there was the whole Product Owner and Backlogs thing. Since the team was the unit of production, what was going to be the unit of value for the team to produce? There had to be some way of enumerating and measuring the value of the things the team was working on. There had to be some kind of interface between the team and the outside world. After all, how would the team know what to work on? And so the Product Owner was born. The Product Owner made a list of things that had business value. The Product Owner ordered this list from most valuable to least valuable. The Product Owner gave the list to the team, and the Product Owner was always there to answer questions about the list and the items in it.
This list, the list of things that had business value, was called the Backlog. The Backlog was the Product Owner’s, The Product Owner and his list represented things the business was willing to pay for, things that had real value. Not a lot of explanation was forthcoming at first about what went in the backlog, or what exactly the Product Owner did all day, or how the business knew these things were valuable, or what made a good or bad item on the backlog, but teams had a guy responsible for telling them what to do, and a list of stuff to do, and that seemed good enough for most folks.
A Product Owner and a Backlog eliminated long sessions with users who didn’t know what they wanted. Instead, the Product Owner could handle that. It eliminated huge hunks of time where people dickered about which thing was more important than which other thing. The Product Owner could handle that too. Why should the team care? Give them a list, a person they can talk to, and they can make things happen. We solved the problem by assigning it to somebody.
Big swaths of time that used to be spent coordinating and meeting were returned to the developers to do productive work. Tough problems were made simple. Developers made happy.
Now if it only would take off.
Frederick Taylor had the same struggle with Scientific Management Theory, but he didn’t give up. After working at a few places validating his ideas, he retired at age 50 and spent the rest of his life writing books and proving his ideas correct to anybody that would listen. And because people could see the results, he quickly gained a following. More and more managers saw the natural connection between work, management, and science.
Scientific Management went everywhere. Taylor single-handedly invented the job of Management Consultant. Most all of modern management theory is based in one way or another on Scientific Management Theory. Name a famous management consultant; Deming, Drucker, McKinsey; talk to them or read their work, and pretty soon they’re talking Frederick Winslow Taylor. Name a famous management theory: Six Sigma, TQM, MBO, Balanced Scorecard. It all goes back to Scientific Management Theory.
That’s right. Frederick Winslow Taylor is the guy responsible for just about every management fad you’ve ever had to suffer through.
His ideas of time and motion measurement and optimization, along with his use of the scientific method for process improvement, are used everywhere. In fact, it’s impossible to find anywhere they are not used. The entirety of modern life, with all the great increases in technology and efficiency, is built on Scientific Management Theory.
Even facing stiff initial resistance, the logic of Scientific Management quickly outstripped the emotional appeal of not rocking the boat.
Agile was slow going at first, too, but eventually took off for completely the opposite reason. While Taylor supporters had to argue science and logic, Agile adherents could just jump up on a soapbox and hold forth on the cruelty of the existing system. You couldn’t support Taylorism without supporting precise measurement and studied action, but you could support Agile without actually taking a stand about anything at all. You could just take a stand against stuff. It was enough to know that the system was broken. The system was broken! Tear it all down!
Early Agile adopters faced a ton of friction from the Scientific Management crowd, mostly in the upper echelons, but the emotional appeal of Agile Values quickly outstripped the logical appeal of Taylor’s measurement, planning, and optimization, at least for the people doing the work.
And as much as it sounded like another feel-good management fad, applied Agile had the distinct benefit of actually working. But it worked in a probabilistic, not scientific manner. People could take the values that the manifesto described and create various concrete processes to try in their teams. Then they could compare notes. Over time, some processes were mostly found to work well most of the time. Some only rarely. But practitioners could collect these process tricks, or games, like postage stamps, trading with other practitioners and selecting the appropriate ones depending on what kind of work was going on. This wasn’t about the right or perfectly optimized way to do things. This was about the way to do things that worked better than others most of the time. This difference turned out to be critical. Agile, at least the way the community applied it, wasn’t a standard; it was a heuristic.
Hundreds of books sprang up around Agile. Pick an area of technology development: coding, databases, hardware, the Internet. Successful, proven Agile techniques are there and continue to grow there as people organize new processes and structures of work and then place those processes, games, katas, or tricks, inside the larger controlling Agile value system.
Two opposite philosophies. Each deeply involved in all progress and organized activity on the planet. Each looking at the world in a completely different way. Scientific Management, which makes sense to anybody creating and running the machine of business. Agile, which makes sense to anybody making technology solutions happen.
Scientific Management tells us that life is rational, causal, and works according to logic and reason. If we define what we are doing, break it down into smaller pieces, we can optimize those pieces, thereby optimizing the whole. One set of people plans the work and another set of people implements the work. In fact, that this separation between planning and execution, combined with scrupulous measurement and adaptation, is the only sane way to organize our work.
Agile tells us that no matter how we break down and optimize our work, without our values being in the right place we end up creating a living hell for ourselves and others. That progress involving ideas is many times surreptitious, not directly causal. That you can only break down things so far. That trust is more important than measurements.
It’s as if there’s one world full of business people with suits, stopwatches and blueprints, measuring each little thing and determining how a little tweak here or there will change things by a percentage point or two. Another world is full of technologists with flip-flops, games, and Nerf balls, not talking about measurements or machines yet creating new products that change reality for the rest of us. And doing it on a regular basis.
And backlogs? Right at the boundary, in the hand of the Product Owner.
February 11, 2014 2 Comments
I used to view marketing transactionally and anaytically, I used to think marketing was some kind of thing I did in a separate part of my business, and I used to think it was all easy.
Most all of what I thought about marketing is not true.
Thought I would share some marketing advice for all those sole founders out there selling services online.
- It’s a numbers game. Marketing is introducing yourself to a lot of people over time. That means that whatever your strategy, you have a lot of work to do. You should always be thinking of the number of people-eyeballs you are getting for your effort. In some cases, baseball uniforms for the local little league team might be a tremendously better investment than a TV commercial. But you always have to work the numbers to figure this out.
- It’s a long game. In services, it’s not unusual for there to be a 6-18 month lead time between contact and sales. So not only are you talking to a lot of people, you’re talking to them over a long period of time. Write and publish “evergreen” material online — articles that people will google for and read five years from now.
- It’s all emotional engagement. But it’s not just slinging stuff out into the wild. The trick here is to form emotional engagement. That’s why you do Christmas videos like above and write articles with marketing advice. You should be wanting to participate emotionally in people’s lives. Marketing just isn’t creating full-page newspaper ads any more. It’s digging wells for poor people, participating in startup weekends, helping struggling companies compete. This is good news. It means that the values you have in life, when properly framed, are part of an emotional conversation you have with people who haven’t met you before. That’s marketing today.
- Scale out conversations. So the trick is having these emotionally engaging, long-term conversations around shared values. This is a freaking lot harder than just doing any one thing. Great marketers use all sorts of tech for this: Facebook, email lists, blogs, article publishing, professional groups, and so on. You’re always trying out a dozen or so channels, and slowly optimizing which ones work over time.
- Integrate metrics into the conversation. It’s not so good to live with your head in your computer. Tool companies will sell you all kinds of technology solutions to the simple problem of not owning your image and understanding how to market. Don’t spend your money on this. Instead, get into a lightweight metrics feedback loop where daily you check key numbers but don’t dwell on them. You paint a great painting by being creative and artistic, not by studying the qualities of paintbrushes or obsessing over the perfect canvas stand. Tools are important, but don’t get distracted by them. Message first, tools second.
- Never be afraid to be yourself. Many times as a sole founders we start using “We” in our emails and marketing material. Especially in the professional services field, there’s a perception that the more people that work there, the more value you must bring to the table. Continuing down this road is a mistake. Always use a personal pronoun in your correspondence and encourage people to call or email. It’s fine to be just you.
- Teach. The latest advice is that if you want to engage a market, teach. What this is creating is a world full of crappy advice, as everybody and their brother tries to educate the world. Hopefully this article is not another part of the trend! I don’t have the answers, but I’m willing to share my story of my own struggle. This seems to have value for many.
Take a look at the video above. It’s 40 seconds, and there’s no selling anywhere. The goal is simply to introduce myself to you, make you smile, and tell you what I do. If I get a smile, it’s a win. If I get a smile from a thousand people, it’s a big win. It’s emotions, across a large volume, driving personal engagement, over a long period of time.
It’s a slow Saturday morning. I got up and decided to share my marketing experience. I know a lot of other folks who sell services online, and I think I know more now than I did a year ago.
My wife gets up, starts brewing a cup of tea, and sits beside me.
“What are you doing?”
I think for a moment.
“I’m writing down the things I’ve learned about marketing so that I can help people online not make the mistakes I’ve made and begin a long-term conversation with some of them that might end up in a sale two years from now. But most likely not. I’ll consider myself lucky if I can just help them out in some small way a tiny bit at a time, over the next five to ten years or so.”
That’s sole founder services marketing.
December 21, 2013 2 Comments
But I do believe that if you want to maximize creativity and the ability to learn and pivot, you should work as directly in contact with each other as possible. Like sitting next to each other at a cafeteria table. Perhaps all working on the same keyboard. The closer, the better. (Even though as a developer, I find such arrangements abhorrent in terms of maximizing coding productivity and personal happiness. Such is life.)
This is my opinion. Everybody’s got an opinion. Like most Agile authors, mine is probably wrong to some degree. Form your own opinions and test them in your project with your team. Then we can compare notes
But that’s the key: whatever you want to believe, form your own opinions and test them. I have a theory of operations that I feel holds true most of the time. We can add in some definitions, then we can make my theory of operations testable for a particular situation. We observe whether it works or not. So many times I see Agile coaches and teams working without any theory of operations. Either they think there’s one magic way of doing things, or they don’t know, don’t care. They just want instructions. The lack of structure is seen as a bigger threat than getting results from the practices!
The sales and coaching process itself doesn’t lend itself to this kind of nuanced discussion. Instead of creating a principle that we can then test with practices, what I see are these right-wrong discussions that end in compromise. Everybody is somewhat happy, but nobody really gets anything. We get something that looks like Agile because it resembles what other Agile teams do, but it’s not much of anything.
The discussion goes something like this: should we work colocated? Yep, for you guys I think open plan is best. Well, we can’t do open plan. (Coach thinks for a while). That’s fine, how about you work in your cubes, then use this meeting room here for your team stuff?
Welcome to the Agile Clubhouse, where you can pretend you’re colocated for an hour or two each day.
Burn it down!
Seriously. Co-location addresses communication risk: the idea if we don’t work in close physical proximity to people, we avoid and don’t have difficult, natural, and creative conversations. Walking into a room for a stand-up and some Agile rituals isn’t going to address any of that. It’s just going to look a little bit “more co-located” than doing it some other way. As if the goal were for things to simply appear one way or another.
This is like teams with tools. What are storyboards for? The principle is that physical, tangible pieces of work, in a co-located environment, allow people to use different parts of their brain to work on common problems. Displaying the status and problems of the team in a big, public, visible way, all the time, and having the team talk about and physically move tokens representing those problems around? It allows socialization to occur, new ways of viewing problems to emerge, and the group subconscious to work on understanding and optimizing the nature of the work.
Could you put all the storyboard stuff in a tool? Sure thing! As long as you view project data as simply another form of data, like your bank balance. If that’s all you believe project data is, you can, and should, process it using all kinds of cool online tools. The thing here is that either you accept and understand the principle of tools being visible, public, and social — in which case you can test it — or you substitute it with another one — which you can also then test. Identify a risk. Come up with a principle to address it. Come up with a practice. Test the practice.
Many times, especially in big corporations, all we get is the practice part, and even then we’re not owning the practice, simply executing it for somebody else.
The Agile Clubhouse is a symptom that our head is in the wrong place. We’re trying to compromise recipe books, instead of testing out principles. So we understand that we need to “do some kind of co-location stuff”, but we don’t know why, and we don’t know what to expect from it. To us it’s just something we’re supposed to do — something some of us don’t like, or can’t do. At this point we decide that instead of optimizing our work, we view the problem as finding something we all like doing. So we get a clubhouse.
Compromise is a great thing, and we should always strive for it. But our goal in Agile isn’t to compromise: it’s to use empirical data to optimize and predict performance. It’s better to completely eliminate thinking about what you like or don’t like, as long as you can pick something that’s testable, than it is to adjust whatever you’re doing to make the most number of people happy. Guess what? The best answer for the team might actually be something that nobody likes!. That’s fine. That’s life. Sometimes things work like that. It’s your team and your work life. You deserve the best you can get. At the very least, you deserve to know.
December 11, 2013 Leave a comment
First, our specifications. We want something we can share with a link, that everybody can edit simultaneously. Since we don’t want to get into discussion around what various shapes mean, we need to use UML notation. We’d like to be able to sketch freeform as well, but that’s not critical. Finally, since there actually exists something called “modeling” that is different from playing around with Visio. Visio is fun for doing org charts and whatnot, but in technology development we are actually making a model, where pieces are reused and meaning refined by re-use in other areas, we need a model tree that has components that we can then annotate and reuse.
This product does not exist.
Sure, there are plenty of heavyweight modeling tools that do all sorts of things — for a really expensive price tag. There’s also a hoard of online Visio clones, some of which offer real time collaboration. But nobody has real time collaborative UML modeling.
Here’s the breakdown in features:
|Public link to diagram||Real time collaboration||Freeform sketching||Visio-like sketching||UML Notation||Real structural diagrams||Real behavior diagrams||Actually having a UML model||Deep/rich comment fields||Export Code Gen|
|Google Docs Drawing||YES||YES||YES||YES||NO||NO||NO||NO||NO||NO|
You can assign your own rating to these to figure out what’s best for you. For us, it looks like LucidChart is the winner, but I can’t say I’m happy with that conclusion, because it’s a completely different tool from the one we’re looking for. After all, it’s just a really cool online Visio. The product I really like is GenMyModel, but it lacks enough features to make it a deal-breaker. Such is life. It’s still in early beta, however. Here’s hoping the product evolves well over the next year or so.
All-in-all, I was really surprised that it was 2013 and nobody had mastered online UML modeling. With as much enterprise money as there is on the table, you’d think this would be an attractive area for startups to pursue. Lots of online sketch tools support UML patterns. Perhaps that’s enough for most developers? If so, there are a lot of people missing out on the real value that UML brings.
November 21, 2013 1 Comment