Scrum, Agile, and Modern Tools

Required Reading: https://www.pandastrike.com/posts/20150304-agile

My takeaway quote? "Scrum lags behind the modern toolchain enough that there can be a Potemkin village vibe to the whole thing."

I was clued into this from another takeaway quote someone seconded on Twitter: "Waterfall used too much written communication, but Agile doesn't use enough."

Also read this: http://caines.ca/blog/2014/12/02/i-dont-miss-the-sprint/

Is "sprint" misleading? What about "sprint commitment?"

I'm not sure I object to "sprint" per se.

But I have seen "sprint commitment" turned into an organizational problem, removing what could have been a helpful tool. Folks who start harping on sprint commitments in the sense of "we committed to this, will we meet the deadline?" tend to create a toxic environment. I think the people who hype commitment the most really liked the non-Agile environments: they try bend Agile to meet their Waterfall concepts.

The problem is the word. A "sprint commitment" shouldn't be used like a legally binding "do it or pay penalties" commitment. It should be a metric used to gauge progress. More like a "sprint outcome".

The commitment hype can lead to stories, epics and detailed technical tasks getting muddied up terribly. The story becomes an epic. Little tiny technical tasks get inflated into big important stories. A proper user story gets replaced with nonsense about prepping a database for production rollout, or resolving defects found in QA, or things that -- obviously -- aren't user stories, but are taking up a lot of time.

When it appears that a story is going nowhere, the scrum master breaks it down into things that have status which changes frequently. The sense of end-user meaning behind the actual story gets lost in a haze of technical considerations and tasks that show activity more than accomplishment more than value.

"As an actuary, I want to know that the developers have written syntactically correct DML for my database, so that the product owner don't have to wait as long for the DBA's to build the database."

Really?

Agility and following a "Strictly Agile" approach

I've seen some discussion on Stack Overflow that is best characterized by the question: "What is Strictly Agile?", or "What's the Official Agile Approach?".

Someone shared this with me recently: "Process kills developer passion".

I have also heard some great complaints about organizations that claim "Agile" and actually do nothing of the kind. In some cases it's not a "crunchy agile shell" around a waterfall process; it's a simple lie. Nothing about the process is Agile except a manager insisting that all the status reporting, planning and unprioritized lists of random requirements are Agile.

Finally, I got this weird suggestion: "consider writing a blog about how to test if you are agile or not". It's weird because testing for Agile is like testing for breathing; it's like testing for flammability.

The Agile Test

Testing if your project is Agile can be done two ways.

1. Practical. Make a change to the project. Any change. Requirements, architecture, due dates, staff, anything. Does it derail? If so, it wasn't very Agile, was it?
2. Theoretical. Reread the Agile Manifesto. Make a score card that evaluates the project on each of the eight basic criteria in the Agile manifesto. Convene all the project stakeholders. Conduct careful surveys and have structured walkthroughs to determine the degree of Agility surrounding each person, deliverable, collaborative relationship and issue.

An important point is that Agile is not absolute. Some practices are more Agile than others. There's no "strictly" Agile. There are ways to make a project more Agile; that is, it can effectively cope with change. There are ways to make a project less Agile; that is, change causes problems and can derail the project completely.

The canonical example is a missing, misstated or contradictory requirement that gets uncovered after coding and during user acceptance test. Clearly, that feature has been built and is absolutely wrong. What happens next?

Agile? The product can be released with with the broken feature relegated to the next release. A hack is put in to remove the buttons or menu items or links until they work.

Not Agile? Everyone works around the clock to make that feature work no matter what. Paraphrasing Admiral Farragut: "Technical debt be damned. Development must proceedfull speed ahead." All of this irrespective of the relative value of what's being developed. Schedule comes first; features second.

How Much Process?

The "Process Kills..." blog entry repeats observation that a lot of carefully-defined process isn't really all that helpful. It identifies a cause ("process kills passion") that's can be true, but it's largely irrelevant. Process is—essentially—work that's not focused on delivering anything of real value. Complex processes are "meta" work; it's work focused on IT internals; it's work that creates no value for the users of the software; work that replaces the more valuable elements of the Agile Manifesto.

One can argue that processes, documentation, contracts and plans "assure" success or demonstrate some level of quality. To an extent all the process and meta-work creates trust that—eventually—the resulting software product will solve the original problem.

The mistake is that non-Agile methods use a series of surrogates—processes, documentation, contracts and plans—instead of actual software. The point of Agile methods it to release software early and often and avoid using surrogates.

Key Points of Agile

Here are the key points of the Agile Manifesto.

• Individuals and interactions over processes and tools. A more Agile project will use the best people and encourage them to talk amongst themselves. A less Agile project will write a lot of things (which folks don't have time or reward for reading.) There will be misunderstandings, leading to large, boring meetings where someone reads powerpoint slides to other folks to try and clear up misunderstandings.
• Working software over comprehensive documentation. A more Agile project uses frequent release cycles of incremental software. A less Agile project attempts to gather all requirements, do all design and then try to do all the coding even though the requirements have already been found to be less than crystal clear.
• Customer collaboration over contract negotiation. A more Agile project uses constant contact with customer and product owner to refine and prioritize the requirements. A less Agile project uses a complex change control process to notify everyone of a requirements change, which leads to design and code changes, and has cost and schedule impact that must be carefully planned and documented.
• Responding to change over following a plan. A more Agile project uses incremental releases, conversation and a modicum of discipline to build things of value. Just because someone thought it should be included in the requirements doesn't mean the feature is really required.

The "Process Kills Passion?" Question

There Process Kills Passion blog lists a bunch of things that—it appears—some folks find burdensome:

• Doing full TDD, writing your tests before you wrote any implementing code.
• Requiring some arbitrary percentage of code coverage before check-in.
• Having full code reviews on all check-ins.
• Using tools like Coverity to generate code complexity numbers and requiring developers to refactor code that has too high a complexity rating.
• Generating headlines, stories and tasks.
• Grooming stories before each sprint.
• Sitting through planning sessions.
• Tracking your time to generate burn-down charts for management.

This list has three different collections of practices.

• Good. TDD, code reviews, generating headlines, stories and tasks, grooming stories before each sprint and doing some planning for each sprint are all simply good ideas. They must be done. "Pure Coding" is not a good way to invest time. Planning and then coding is much smarter, no matter how boring planning appears.
• Difficult. Test code coverage can be helpful, but can also devolve to empty numerosity. 20% more coverage doesn't not mean 20% fewer bugs. Nor does it mean 20% less chance of uncovering a bug at run time. Code complexity ratings are also fussy because they don't have a direct correlation with much. They must be done and used to prioritize work that will reduce technical debt. But mindless thresholds are for cowards who don't want to mediate deep technical discussions.
• Silly. Creating burn-down charts for management shouldn't be necessary. Everyone must read and understand the backlog. Everyone should build the summary charts they want from the backlog. The product owner or even the eventual customer should do this on their own. They must be given a profound level of ownership of the features and the process for creating software.

I don't agree that process kills passion. I think there's a fine line between playing with software development and building software of value. I think that valuable software requires some discipline and requires executing a few burdensome tasks (like TDD) that create real value. Assuring 80% or 100% code coverage doesn't always create real value. Spending time keeping the backlog precise and complete is good; spending time making pictures is less good.

Scrum Made Difficult

Here's a great post called "Toward a Catalog of Scrum Smells". This lists some "Management Smells": specifically doing clumsy, ineffective things and calling it "Scrum".

I found this in StackOverflow question, titled "Any stories where trying to apply Scrum went wrong?"

Another great answer was a link to "Scrummerfall: World's Worst Software Development Methodology."

What's interesting to me is that (1) Scrum works -- or you'd have more horror stories -- and (2) people do it wrong all the time.

Most of the Scrum-Done-Badly smells amount to management-as-usual. Rather than empower the developers, managers insist on long, stupid status meetings the purpose of which is to inform management. Rather than trust the developers to get things done, managers insist on detailed plans of little value to the developers.

Breaking into Agile

I had a recent conversation with some folks who were desperate to "processize" everything. They were asking about Scrum Master certification and what standards organizations define the "official" Scrum method.

Interestingly, I also saw a cool column in Better Software magazine, called "Scrumdamentalism" on the same basic question.

In my conversation, I referred them to the Agile Manifesto. My first point was that process often gets in the way of actual progress. Too much process focus lifts up "activity" in place of "accomplishment".

My second point, however, was that the Agile Manifesto and the Scrum method are responses to a larger problem. Looking for a process isn't an appropriate response to the problem.

The One True Scrum Quest

Claiming that there's one true Scrum method and everything else is "not scrum" is an easy mental habit. The question gets asked on Stack Overflow all the time. The questions are usually one of two kinds.

• What's the "official" or "best practice" Scrum method and how do I define a process that rigidly enforces this for my entire team of 20?
• We are doing our design/code/test/integration/release in a way that diverges from the "official" form in the Ken Schwaber and Mike Beedle book. Or it diverges from the Eclipse version. Or it diverges from the Control Chaos overview. Or the Mountain Goat version. Or the C2 Wiki version. Or this version. Is it okay to diverge from the "standard"?

Sigh. The point of Agile is that we should value "Individuals and interactions over processes and tools". The quest for "One True Scrum" specifically elevates the process above the people.

In The Real World

The biggest issue is that the Agile Manifesto is really a response to some fundamental truths about software development.

In management fantasy world, a "project" as a fixed, definite, limited, clearly articulated scope. From this fixed scope, we can then document "all" the requirements (business and technical). This requirements document is (a) testable against the scope, (b) necessary for all further work and (c) sufficient for design, code, test and transition to production. That's not all. And -- in order to make a point later on -- I'll continue to enumerate the fantasies. The fantasy continues that someone can create a "high-level design" or "specification" that is (a) testable against the requirements, (b) necessary for all further work and (c) sufficient to code, test and transition to production. We can then throw this specification over the transom into another room where a "coder" will "cut code" that matches the specification. The code production happens at a fixed, knowable rate with only small random variation based on risk of illness. The testing, similarly, can be meticulously scheduled and will happen precisely as planned. Most "real-world" (management fantasy) projects do not leave any time for rework after testing -- because rework won't happen. If it won't happen, why test? Finally, there will be no technology transfer issues because putting a freshly-written program into production is the same as installing a game from a DVD.

Managers like to preface things with "In The Real World". As in "In The Real World we need to know how long it will take you to write this."

The "in the real world" speech always means "In My Management Fantasy Land." The reason it's always a fantastic speech is because software development involves the unknowable. I'm not talking about some variable in a formula with a value that's currently unknown. I'm talking about predicting the unknowable future.

The Agile Response to Reality

In the Real real world, software development is extraordinarily hard.

Consider this: the computer clock runs in 100-nanosecond increments (1.0E-7). We expect an application to run 24x7x0.999 = 6.04E5 seconds. That's from 100-nano to half-million: about 12 orders of magnitude to keep in one's head.

Consider this: storage in a largish application may span almost a terabyte, (1.0E12). From bytes to terabytes: about 12 orders of magnitude to keep in one's head.

Consider this: a web application written in a powerful framework (Django) requires one to know the following languages and frameworks. Shell script, Apache Config, Python, Django Templates, SQL, HTML, CSS, Javascript, HTTP (the protocol is it's own language), plus the terminology of the problem domain. That's 9 distinct languages. We also have the OS, TCP/IP Apache, mod_wsgi, Django, Python, browser and our application as distinct frameworks. That's 8 distinct framework API's to keep in one's head.

Consider this: the users can't easily articulate their problem. The business analyst is trying to capture enough information to characterize the problem. The users, the analyst, the project manager (and others outside the team) all have recommendations for a solution, polluting the problem description with "solution speak" that's only adds confusion.

In the Management Fantasy "Real World", this is all knowable and simple. In the Real Real World, this is rather hard.

Adapting to Reality

Once we've recognized that software development is hard, we have several responses.

1. Deny. Claim that software developers are either lazy or stupid (or both). Give them pep-talks that begin "in the real world" and hope that they cough up the required estimates because they're motivated by being told that software development "in the real world" isn't all that hard.
2. Processize(tm). Claim that software development is a process that can be specified to a level where even lazy, stupid programmers can step through the process and create consistent results.
3. Adapt. Adapting to the complexity of software development requires asking, "what -- if anything -- expedites software development?"

What Do We Need to Succeed?

There are essentially two domains of knowledge required to create software: the problem domain and the solution domain.

• Problem Domain. This is the "business rules", the "scope", the "requirements", the "purpose", etc. We have the features and functions. What the software does. The value it creates. The "what", "who", "where", "when" and "why".
• Solution Domain. This is the technology that makes it go. The time and space dimensions (all 12 orders of magnitude in each dimension), all the languages and all the frameworks. The "how".

The issue is this:

We don't start out with complete knowledge of problem and solution.

At the start of the project -- when we're asked to predict the future -- we can never know the whole problem, nor can we ever know the whole solution we're about to try and build.

What we need is this:

Put Problem Domain and Solution Domain knowledge into one person's head.

The question then becomes "Who's head?"

We have two choices:

• Non-Programmers. We can try to teach the various non-programmers all the solution domain stuff. We can make the project manager, business analyst, end-users, executive sponsor -- everyone -- into programmers so that they have problem domain and solution domain knowledge.
• Programmers. We can try to impart the problem domain knowledge on the programmers. If we're seriously going to do this, we need to remove the space between programmer and problem.

That's the core of the Agile Response: Close the gap between Problem Domain and Solution Domain by letting programmers understand the problem.

The Bowl of Bananas Solution(tm)

"But wait", managers like to say, "in the real world, we can't just let you play around until you claim you're done. We have to monitor your activity to make sure that you're making 'progress' toward a 'solution'."

In the Real real world, you can't define the "problem", much less test whether anything is -- or is not -- a solution. I could hand most managers a bowl of bananas and they would not be able to point to any test procedure that would determine if the bowl of bananas solves or fails to solve the user's problems.

Most project scope documents, requirements documents, specifications, designs, etc., require extensive tacit problem domain knowledge to interpret them. Given a bowl of bananas, the best that we can do is say "we still have the problem, so this isn't a solution." Our scope statements and requirements and test procedures all make so many assumptions about the problem and the solution that we can't even figure out how evaluate an out-of-the-box response -- like a bowl of bananas.

In the Real real world, management in organization A demands that information be kept in a one database. Management organization B has a separate database for reasons mired in historical animosity and territorial scent-marking. Management in yet another organization wants them "unified" or "reconciled" and demands that someone manually put the data into spreadsheets. This morphs into requirements for a new application "system" to unify this data, making the results look like poorly-design spreadsheets. This morphs into a multi-year project to create a "framework" for data integration that maintains the poorly-designed spreadsheet as part of the "solution".

A quick SQL script to move data from A to B (or B to A) is the bowl-0f-bananas solution. It cannot be evaluated (or even considered) because it isn't a framework, system or application as specified in the scope document for the data integration framework.

This is the problem domain knowledge issue. It's so hard to define the problem, that we can't trust the executive sponsor, the program office, the project managers, the business analysts or anyone to characterize the problem for the developers.

The problem domain knowledge is so important that we need to allow programmers to interact with users so that both the problem and the solution wind up in the programmer's head.

Meetings

I found this note while cleaning up. I think it's a summary of John Cleese's short movie "Meetings Bloody Meetings".

1. Plan. Why are we meeting? What's the purpose? What are the subjects?
2. Inform. Tell the attendees what, why and the expected outcome.
3. Prepare. Create an agenda, with a time line, and the items to be covered.
4. Structure. If the purpose is to make a decision, this involves presentation of evidence, interpretation of the evidence and a decision. If the purpose is to inform, then present the information.
5. Summarize and Record.

I hear complaints about time wasted in meetings. This complaint is absolutely justified. I've wasted a lot of my professional career in dumb meetings.

Out of Touch Management. The out-of-touch manager requires long meetings in which each direct report provides a complete status report. The all-hands meeting appears to be the only time the out-of-touch manager ever talks to anyone. Consequently, it devolves to a sequence of one-on-one meetings which the rest of the team is forced to attend.

If the goal is for many people to inform one person, those conversations don't require a meeting.

Going Through The Motions. The going-through-the-motions manager is aware that a staff meeting is expected, but has nothing really to say or do. These meetings often devolve into awkward silences as open-ended questions are thrown out to try and stimulate some interaction.

If there's no goal (no decision, no information) then there's no purpose for the meeting.

Best Practice.

In the days before agility, I worked with two project managers who gathered information in one-on-one meetings. These are separate projects for separate customers. But the same best practice.

The PM would stop by your cube, figure out what you were doing, what you needed, what you were planning to accomplish. Then, at the ever-so-brief staff meeting, a few key interactions among the staff would be lifted up from all those one-on-one meetings.

Let's review the brilliance of these two PM's.

1. They stopped by your cube. They did not play the power game and make you come to their office. They stopped by. In later years this was called "Management by Walking Around" -- something hailed as brilliant. These PM's just did it.
2. They hung around long enough to actually get what was going on. They didn't waste time asking you to write and email status to them. They asked, listened, understood and summarized your status for you.
3. They recognized the needs for interaction and made them happen. The status meetings were like daily scrum meetings. Since they were weekly, they weren't as brief, but they were just as focused. No long status reports. No long conversations.

Objections.

Some managers don't have time to sit in everyone's cubicle. This overstates their value as managers by understating the huge cost of wasting everyone's time in serial one-on-one's done in a conference room. 12 direct reports means 13 man-hours wasted in a one hour meeting.

Some managers don't have time to write status reports. Instead they forward emails all day.

It's probably better to get off the email treadmill. Get status, write short, to-the-point status reports.

Focus.

Developers (DBA's, Sys Admins, all technical folk) live in a world of technology delivery.

Mangers, however, live in a world of budgets, status reports, and weird exercises in foretelling the future. ("How long will it take? How much will it cost?")

There's no good reason to impose the world of status reporting and fortune-telling on technical people. There are, however, lots of bad reasons for imposing unplanned, unprepared and unstructured meetings on a team that would rather be building product than talking about building product.