Legacy Code Preservation

Rule One: Writing Software is Capturing Knowledge.

Consequence: Converting Software is Preserving Knowledge.

When software is revised for a new framework or operating system or database or when an algorithm is converted to a new language, then we're "converting" (or "migrating") software. We're preserving code, and preserving the knowledge encoded.

For the next few months, I'm going to post some examples of preserving legacy code and how this ties to the knowledge capture issue.

Once we've looked at some examples of business software, we can turn to something a little less concrete: HamCalc.

These examples are presented in historical order. Each example raises questions and outlines elements of a strategy for legacy code preservation.

• What's the Story? Late 1970's. What user story was encoded in the software?
• Are There Quirks? Late 1970's. Is the encoded knowledge really a useful feature? Or is it a bug? What if we can't be sure?
• What's the Cost? Early 1980's. What if the legacy code is complex and expensive? How can we be sure it doesn't encode some valuable knowledge?
• Paving the Cowpaths. Throughout the 80's. When converting from flat-file to database, how can we distinguish between encoded user stories and encoded technical details? Isn't all code equally valuable? There are several examples; I've combined them into one.
• Data Warehouse and Legacy Operations. This is a digression on how data warehouse implementation tends to preserve a great deal of legacy functionality. Some of that legacy functionality exists in stored procedures, a programming nightmare.
• The Bugs are the Features. Can you do software preservation when user doesn't seem to understand their own use cases?
• Why Preserve An Abomination? How do we preserve shabby code? How can we separating the user stories from the quirks and bugs? There are several instances, I've used one as an example.
• How Do We Manage This? The legacy code base was so old that no one could summarize it. It had devolved to a morass of details. With no intellectual handles, how can we talk about the process of converting and what needs to be preserved?
• Why Preserve the DSL? describes a modern instance of "Test-Driven Reverse Engineering" where the unit test cases were created from the user stories and the legacy code use merely as supporting details. An entirely new application was written which preserved very little of the legacy code, but met all the user's requirements.

These nine examples include some duplicates. It's really more like a dozen individual case studies. Some are simple duplicates; the name of the customer is changed, but little else.

This Seems Irrational... But... HamCalc

Step 1.  Look at the original HamCalc.  Even if you aren't interested in Ham radio, it's an epic, evolving achievement in a specialized kind of engineering support.  It's a repository of mountains of mathematical models, some published by the ARRL, others scattered around the internet.

Step 2.  Look closely at HamCalc.  It's all written in GW basic.  Really.  The more-or-less final update is from 2011 -- it's no longer an active project -- but it's a clever idea that suffers from a horrible constraint imposted by the implementation language.

A long time ago, I was captivated by the idea of rewriting HamCalc as Java Applets.  It seemed like a good idea at the time, but that's a lot of work: 449 programs, 85,000 lines of code.

Recently, I wanted to make some additional use of HamCalc's amazing collection of formulas.

However.  The distribution kit is rather hard to read.  The .BAS files are in the tokenized "binary" format.

I found a Python project to interpret the byte codes into a more useful format. See http://www.danvk.org/wp/gw-basic-program-decoder/  However, it wasn't terribly well written, and didn't prove completely useful.

GW Basic Bytes Codes

Look at http://www.chebucto.ns.ca/~af380/GW-BASIC-tokens.html for some basic rules on the file format.

See http://www.antonis.de/qbebooks/gwbasman/index.html for a reasonably clear definition of the language itself. Quirks are, of course, studiously ignored, so there's a lot of ambiguity on edge cases.

For simple bytes-to-text translation, this is pretty simple.  The next step -- interpreting GW Basic -- is a bit more complex.

Future

The irrational thing is that I'm captivated by the idea of preserving this legacy gift from the authors in another, more useful language. Indeed, the idea of a community of "HamCalc Ports to Other Languages" appeals to me. This base of knowledge is best preserved by being made open so that it can be rewritten into other commonly-used languages.

There's a subset version here: http://www.softpedia.com/get/Science-CAD/HamCalc.shtml and here http://www.dxzone.com/dx11432/hamcalc-v1-3.html. This is just a few of the calculations, carefully rebuilt to include nice versions of the ASCII-art graphics that are central to the original presentation.

The hard part of preserving HamCalc is the absolute lack of any test cases of any kind.

I think the project should work like this.

1. Publish the complete plain text source decoded from the tokenized binary format. It will likely be somewhere on http://www.itmaybeahack.com/ or perhaps a Dropbox.
2. Publish the index of programs and features as a cross-reference to the various programs. This should include the various links and references and documentation snippets that populate the code and output. This forms the backbone of the documentation as well as the unit testing.
3. Do a patient (and relatively lame) translation to Python3.2 to break HamCalc into two tiers. The calculation library and a simple UI veneer using stdio features of the print() and input() statements. The idea is to do a minimalist rewrite of the core feature set so that a GUI can be laminated onto a working calculation library.
4. Work out test cases for the initial suite of 449 legacy programs oriented toward the calculation layer, avoiding the UI behavior. The idea isn't 100% code coverage. The idea is to pick the relevant logic paths based on the more obvious use cases.

A sophisticated GUI is clearly something that was part of the original vision. But the limitations of GW Basic and tiny computers of that era assured that the UI and calculation were inextricably intertwingled.

If we can separate the two, we can provide a useful library that others can build on.

Maybe I should organize http://www.hamcalc.org/ as the jumping-off point for this effort?

RESTful Web Services Testing, Q&A

Some background:

I was vaguely pointed at one call in an API, via a 2-page "tutorial" that uses CURL examples. Told "Test this some more." by the guy who'd been doing some amount (none?) of hand "success path" testing via CURL. This has since morphed into "regression testing things, all 12 calls", "we have a build API as well", and "there's this hot new feature for a vendor conference in a couple weeks ..."

There was more, but you get the idea.  There were so more specific "requirements" for the RESTful unit testing environment.

1) Get "smoke test" coverage vs. all the calls 

A sequence of CURL requests to exercise a server can be viewed as "testing".  It's piss-poor at best.  Indeed, it's often misleading because of the complexity of the technology stack.

In addition to the app, you're also testing Apache (or whatever server they're using) plus the framework, plus the firewall, plus caching and any other components of the server's technology stack.

However, it does get you started ASAP.

2) expand / parameterize that 

CURL isn't the best choice.  You wind up writing shell scripts.  It gets ugly before long.

Python is better for this.

Selenium may also work.  Oh wait.  Selenium is written in Python.

3) build out to response correctness & error codes 

Proper design for testability makes this easy.

However.  When you've be tossed a "finished" RESTful web service that you're supposed to be testing, you have to struggle with expected vs. actual.

It's not trivial because the responses may have legitimate variances: date-time stamps, changing security tokens or nonces, sequence numbers that vary.

Essentially, you can't just use the OS DIFF program to compare actual CURL responses with expected CURL responses.

You're going to have to parse the response, pick out appropriate fields to check and write proper unittest assertions around those fields.

4) layer in at least that much testing for the new, new feature breathlessly happening RIGHT NOW.

Without a proper design for testability, this can be painful.

If you're using a good unit test framework, it shouldn't be impossible.  Your framework must be able to start the target RESTful web service for a TestCase, exercise the TestCase, and then shutdown the target RESTful web service when the test has completed.

Now, you're just writing unittest TestCase instances for the new feature breathlessly happening RIGHT NOW.  That should be manageable.

...tool things I've found so far... [list elided] 

All crap, more or less.  They're REST client libraries, not testing tools.

You need a proper unit testing framework with TestCase and TestSuite classes and a TestRunner.  The tools you identified aren't testing frameworks, they're lower level REST client and client library.  CURL, by itself, isn't really very good for robust testing unless you embed CURL in some test framework.

For defining interfaces (2), I have found these... [list elided]

API's in a typical RESTful environment have little or no formal definition, other than Engrish.  WSDL is for Java/XML/SOAP.  It's not used much for (simpler) REST.  For the most part, REST API definitions (i.e., via JSON or whatever) are mostly experimental.  Not standardized.  Not to be trusted.

The issue is one of parallel maintenance.  The idea is that a REST frameworks can operate without too much additional JSON or XML folderol; just the code should be sufficient.

If there's no WSDL (because it's just REST) then there's no formal definition of anything.  Sorry.

I (perhaps foolishly) figured there's be some standard way to consume the standard format definition of an API, to generate test cases and stubbing at least. Maybe even a standard set of verifications, like error codes. So I went a-googling for 1) a standard / conventional way to spec these APIs, 2) a standard / conventional tool or maybe tools @one per stack, and 3) a standard / conventional way to generate tests or test scaffolding in these tools, consuming the standard / conventional API spec. So far, not so much.

"So far, not so much" is the state of the art.  You have correctly understood what's available.

REST -- in spite of it's trivial simplicity and strict adherence to HTTP -- is a rather open world.  It's also pretty simple.  Fancy tools don't help much.

Why not?

Because decent programming languages already do REST; tools don't add significant value.  In the case of Python, there are relatively few tools (Selenium is the big deal, and it's for browser testing) because there's no real marketplace niche for them.  In general, simple Python using httplib (or Python 3 http.client) can test the living shit out of RESTful API better than CURL/DIFF with no ugly shell-script coding.  Only polite, civilized Python coding.

RESTful Web Service Testing

Unit testing RESTful web services is rather complex.  Ideally, the services are tested in isolation before being packaged as a service.  However, sometimes people will want to test the "finished" or "integrated" web services technology stack because (I suppose) they don't trust their lower-level unit tests.

Or they don't have effective lower-level unit tests.

Before we look at testing a complete RESTful web service, we need to expose some underlying principles.

Principle #1.  Unit does not mean "class".  Unit means unit: a discrete unit of code.  Class, package, module, framework, application.  All are legitimate meanings of unit.  We want to use stable, easy-to-live with unit testing tools.  We don't want to invent something based on shell scripts running CURL and DIFF.

Principle #2.  The code under test cannot have any changes made to it for testing.  It has to be the real, unmodified production code.  This seems self-evident.  But.  It gets violated by folks who have badly-designed RESTful services.

This principle means that all the settings required for testability must be part of an external configuration.  No exceptions.  It also means that your service may need to be refactored so that the guts can be run from the command line outside Apache.

When your RESTful Web Service depends on third-party web service(s), there is an additional principle.

Principle #3.  You must have formal proxy classes for all RESTful services your app consumes.  These proxy classes are going to be really simple, since they must trivially map resource requests to proper HTTP processing.  In Python, it is delightfully simple to create a class where each method simply uses httplib (or http.client in Python 3.2) to make a GET, POST, PUT or DELETE request.  In Java you can do this, also, it's just not delightfully simple.

TestCase Overview

Testing a RESTful web service is a matter of starting an instance of the service, running a standard unit testing TestCase, and then shutting that instance down.  Generally this will involve setUpModule and tearDownModule (in Python parlance) or a @BeforeClass and @AfterClass (in Java parlance).

The class-level (or module-level) setup must start the application server being tested.  The server will start in some known initial condition.  This may involve building and populating known database, too.  This can be fairly complex.

When working with SQL, In-memory databases are essential for this.  SQLite (Python) or http://hsqldb.org (Java) can be life-savers because they're fast and flexible.

What's important is that the client access to the RESTful web service is entirely under control of a unit testing framework.

Mocking The Server

A small, special-purpose server must be built that mocks the full application server without the endless overheads of a full web server.

It can be simpler to mock a server rather than to try to reset the state of a running Apache server.  TestCases often execute a sequence of stateful requests assuming a known starting state.   Starting a fresh mock server is sometimes an easy way to set this known starting state.

Here's a Python script that will start a server.   It writes the PID to a file for the shutdown script.

import http.server
import os
from the_application import some_application_feature
class AppWrapper( http.server.BaseHTTPRequestHandler ):
    def do_GET( self ):
        # Parse the URL

        id= url.split("/")[-1]

        # Invoke the real application's method for GET on this URL.
        body= some_application_feature( id )
        # Respond appropriately
        self.send_response( 200, body )
    ... etc ...

# Database setup before starting the service.
# Filesystem setup before starting the service.
# Other web service proxy processes must be started, too.
with open("someservice.pid","w") as pid_file:
    print( os.getpid(), file=pid_file )
httpd = http.server.HTTPServer("localhost:8000", AppWrapper)
try:
    httpd.serve_forever()
finally:
    # Cleanup other web services.

Here's a shutdown script.

import os, signal
with open("someservice.pid") as pid_file:
    pid= int( pid_file.read() )
os.kill( pid, signal.CTRL_C_EVENT )

These two scripts will start and stop a mock server that wraps the underlying application.

When you're working in Java, it isn't so delightfully simple as Python  But it should be respectably simple.  And you have Jython Java integration so that this Python code can invoke a Java application without too much pain.

Plus,  you can always fall further back to a CGI-like unit testing capability where "body= some_application_feature( id )" becomes a subprocess.call(). Yes it's inefficient.  We're just testing.

This CGI-like access only works if the application is very well-behaved and can be configured to process one request at a time from a local file or from the command line.  This, in turn, may require building a test harness that uses the core application logic in a CGI-like context where STDIN is read and STDOUT is written.

Password Encryption -- Short Answer: Don't.

First, read this.    Why passwords have never been weaker—and crackers have never been stronger.

There are numerous important lessons in this article.

One of the small lessons is that changing your password every sixty or ninety days is farcical.  The rainbow table algorithms can crack a badly-done password in minutes.  Every 60 days, the cracker has to spend a few minutes breaking your new password.  Why bother changing it?  It only annoys the haxorz; they'll be using your account within a few minutes.  However.  That practice is now so ingrained that it's difficult to dislodge from the heads of security consultants.

The big lesson, however, is profound.

Work Experience

Recently, I got a request from a developer on how to encrypt a password.  We have a Python back-end and the developer was asking which crypto package to download and how to install it.

"Crypto?" I asked.  "Why do we need crypto?"

"To encrypt passwords," they replied.

I spat coffee on my monitor.  I felt like hitting Caps Lock in the chat window so I could respond like this: "NEVER ENCRYPT A PASSWORD, YOU DOLT."

I didn't, but I felt like it.

Much Confusion

The conversation took hours.  Chat can be slow that way.  Also, I can be slow because I need to understand what's going on before I reply.  I'm a slow thinker.  But the developer also needed to try stuff and provide concrete code examples, which takes time.

At the time, I knew that passwords must be hashed with salt.  I hadn't read the Ars Technica article cited above, so I didn't know why computationally intensive hash algorithms are best for this.

We had to discuss hash algorithms.

We had to discuss algorithms for generating unique salt.

We had to discuss random number generators and how to use an entropy source for a seed.

We had to discuss http://www.ietf.org/rfc/rfc2617.txt in some depth, since the algorithms in section 3.2.2. show some best practices in creating hash summaries of usernames, passwords, and realms.

All of this was, of course, side topics before we got to the heart of the matter.

What's Been Going On

After several hours, my "why" questions started revealing things.  The specific user story, for example, was slow to surface.

Why?

Partly because I didn't demand it early enough.

But also, many technology folks will conceive of a "solution" and pursue that technical concept no matter how difficult or bizarre.  In some cases, the concept doesn't really solve the problem.

I call this the "Rat Holes of Lost Time" phenomena: we chase some concept through numerous little rat-holes before we realize there's a lot of activity but no tangible progress.  There's a perceptual narrowing that occurs when we focus on the technology.  Often, we're not actually solving the problem.

IT people leap past the problem into the solution as naturally as they breathe. It's a hard habit to break.

It turned out that they were creating some additional RESTful web services.  They knew that the RESTful requests needed proper authentication.  But, they were vague on the details of how to secure the new RESTful services.

So they were chasing down their concept: encrypt a password and provide this encrypted password with each request.  They were half right, here.  A secure "token" is required.  But an encrypted password is a terrible token.

Use The Framework, Luke

What's most disturbing about this is the developer's blind spot.

For some reason, the existence of other web services didn't enter into this developer's head.  Why didn't they read the code for the services created on earlier sprints?

We're using Django.  We already have a RESTful web services framework with a complete (and high quality) security implementation.

Nothing more is required.  Use the RESTful authentication already part of Django.

In most cases, HTTPS is used to encrypt at the socket layer.  This means that Basic Authentication is all that's required.  This is a huge simplification, since all the RESTful frameworks already offer this.

The Django Rest Framework has a nice authentication module.

When using Piston, it's easy to work with their Authentication handler.

It's possible to make RESTful requests with Digest Authentication, if SSL is not being used.  For example, Akoha handles this.  It's easy to extend a framework to add Digest in addition to Basic authentication.

For other customers, I created an authentication handler between Piston and ForgeRock OpenAM so that OpenAM tokens were used with each RESTful request.  (This requires some care to create a solution that is testable.)

Bottom Lines

Don't encrypt passwords.  Ever.

Don't write your own hash and salt algorithm.  Use a framework that offers this to you.

Read the Ars Technica article before doing anything password-related.

Performance "Tuning": running in 1/100th the time

For the 757 Python Meetup group, someone proposed looking at some Python code they had which was slow.  The code implemented a variation on the Elo chess rating system.  It applied the ratings to other sports, and used the points scored as well as basic win/lose/tie to work out a ranking for sports teams.  Very clever stuff.

But.

It was described as horribly slow. Since it was only 400 lines of code, it was a great subject for review in a Python meetup.  I would be able to show some tweaks and performance tips.

Hypothetical Step 1:  Profile

My first thought was to run the profiler and see what popped up as a possible root cause of slowness.

Generally, there are three kinds of performance problems.

• Wrong data structure.  Or wrong algorithm.  (They're simply opposite sides of the same coin.)  This generally leads to dramatic, earth-shattering improvements.
• Piss-Poor Programming Practices.  This kind of fine-tuning yields minuscule improvements.  In some cases, no measurable change at all.
• Bugs.  Every time I've been asked to improve "working" code, I find that it has bugs.  Every time.  These can exacerbate performance problems.  Interestingly, most of these are readily apparent during an initial survey:  i.e., while simply reading the code to see how it works.  Trying to create unit tests for the purposes of refactoring often reveals additional bugs.

I didn't know what I was going to see in this 400-line program.

Hypothetically, profiling will help show what kind of problems we have.

Prior to profiling, of course, we need to run the darn thing.  Thankfully, they provided some typical input files.  For example, 1979 High School Football season results.  159 lines of patiently gathered teams and scores.

It Didn't Run

When we tried to run it with the profiler, we found that we had a bug.  Tracking down the bug, revealed the essential performance problem, also.

The core problem was a failure to make use of Python data structures.  

This manifested itself in a number of really bad design errors.  Plus, it presented as the actual, show-stopping, serious bug.

In order to work out team rankings, the program kept a list of teams.

I'll repeat that for folks who are skimming.

In order to work out team rankings, the program kept a list of teams.

Not a dict mapping from team name to team details.  But a simple list.  Locating a team in the list meant iterating through the list, looking for the matching team.  Really.

for t in team_list:

    if t['name'] == target_name:

        process( t )

This kind of thing was repeated in more than one place.

And one of those repeats had the bug in it.

What we have here is "wrong data structure".  Replacing a list with a dict will have earth-shattering impact on performance.

The Bug

The bug, BTW, was a lookup loop which had the additional requirement of adding missing teams.  It tried to use the for-else structure.  This was the intended code (not the actual code).

for t in team_list:

    if t['name'] == target_name:

        return

else:

    t['name']= init_new_team(target_name)

    

This is a first-class bit of Python.  An else clause on a for statement is a real feature of the language.  However, it's obscure enough that it's easy to get wrong.

However, it's also unique to Python, and the kind of thing that can lead to confusion.  I discourage it's use.  

Test-Driven Reverse Engineering

We're doing TDRE on a little piece of recreational programming.  This means that we need to fabricate unit tests for code that is purported to work.  Some folks like to call this "exploratory testing" (a phrase which is contradictory, and consequently meaningless.)  We're "exploring".  We're not "testing".

Once the core bug is fixed, we can run sample data through the application to get "big picture" results.  We can extract bits from the sample data and exercise various functions in isolation to determine what they actually do now, bugs included.

Since this is so simple--and we're going to review it during a 2-hour meet up--and there's nothing billable going on--we can get by with a few really simple unit tests.  We'll run the darn thing once to create some expected output.  We save the output and use that single output log as the expected results from each refactoring step.

More complex applications require more unit tests.  For a billable TDRE effort last year, I had to create 122 unit tests to successfully refactor a program of over 6,000 lines of code.  It took several weeks.

Real Step 1: Fix The Data Structure

Before profiling (but after running to capture some output) we have to fix the essential data structure.  Repeated scanning of a list is never going to perform well.

The whole point of the study of "Data Structures" is to prevent (or optimize) search.

In this case, we can prevent linear search of a list by using a dictionary.  That, clearly, is job one.

It was a pervasive rewrite.   Essentially, everything in this little program included a goofy loop to lookup a team in the list.  The Elo algorithm itself, which is already O(n²), is dragged down by using the linear search for the opposing team four more times, making it O(n³).

Cyclomatic Complexity

One of the big "issues" is the use of if statements throughout the scoring algorithm.  An if statement creates Cyclomatic Complexity and can lead to performance problems.  Generally, if statements should be avoided.

This algorithm applies some normalization factors to reconcile scoring with win/loss numbers in different sports.  Basketball, specifically, involves generally high scores.  Since there are 2-point and 3-point scoring opportunities, a factor is used to normalize the points into "goals".  Football, similarly, has numerous scoring opportunities with values of 1, 2, 3 and 6 points; the scores here are also normalized.

This normalization was done with an if statement that was evaluated deep inside the Elo algorithm.  Repeatedly. Evaluated.

The two functions that handled the normalizations, plus the normalization factors, are ideal candidates for OO design.  There's a clear hierarchy of classes here.  A superclass handles most sports, and two polymorphic subclasses handle football and basketball normalization.

The if statement is now "pushed up" to the very beginning of the program where an instance of the sports normalization object is created.  This object's methods are then used by the Elo algorithm to normalize scores.

Icing on the Cake

Once we've fixed the bug and replaced a list with a dict, everything else is merely icing.
Some other OO changes.

1. The "Team" information should not be a flat, anonymous dictionary.  It should be a proper class definition with proper attributes.  There aren't many methods, so it's easy to create. 
2. The "Game" information is read by csv.DictReader.  However, it should not remain a simple, anonymous dict.  As with a Team, a simple class can be created to handle Game.
3. The overall structure of the application needs to be broken into two sections.  The command-line interface parses options, opens files, and generally gets everything set up.  The actual ranking algorithm should be a function that is given an open file-like object plus the Sport object for normalization.  This allows the ranking algorithm to be reused in other contexts than the command-line (i.e. a web service).

A more subtle OO design point is the question of "mutability".  A Team in this application is little more than a name.   There are also numerous "stateful" values that are part of the Elo algorithm.   A Game, similarly, is an immutable pair of teams and scores.  However, it has some mutable values that are part of the Elo algorithm.  

Really, we have immutable Team and GameHistory objects, plus a few values that are used as part of the Elo calculation.  I'm a big fan of disentangling these mutable and immutable objects from each other.  

I suspect that the Elo algorithm doesn't really need to update the "state" of an object.  I suspect that it actually creates (and discards) a number of immutable candidate ranking objects.  The iteration that leads to convergence might be a matter of object creation rather than update.  I didn't make this change, since it required real work and we were out of time.

Bottom Line

The more times I do TDRE to improve performance, the more I realize that it's all about bugs and data structures.

This recreational application took 45-60 seconds to process one year's record of games for a given league.  It now takes less than 0.2 seconds to do the same volume of work.  Two test cases involving a complete run of 159 records runs in 0.411 seconds.  That's 1/100th the time simply from switching data structures.

The idea of "tweaking" a working program to improve performance is generally misleading.  It might happen, but the impact is minuscule at best.  

Here's the checklist for getting 100:1 improvements.

• Remove searches.  
• Remove deeply-nested if statements.  

Generally, reduce Cyclomatic Complexity.

How Expensive is a "Waterfall" Project Plan?

It's impossible to step into the same river twice; other waters are flowing toward the sea.  It's impossible to do "head-to-head" project comparisons.  You can't have the same people doing the same thing with only one constraint changed.  You can try to resort to having similar people doing the exact same thing.

It's impossible to provide trivial quantification of the costs of a waterfall approach.  However, there's a lot of places where "up front" engineering adds cost and risk.  A waterfall plan adds friction, zero-value administrative costs and (sometimes) extra work.

For software development that doesn't involve "toy" projects, but apply to the magical "real world" projects, it's difficult to justify building the same "real world" thing twice.  It's hard to create a detailed and purely economic analysis of this question.  Therefore, we have to resort to hand-waving and speculation.

[I put "real world" in quotes, because the "real world" is a kind of dog-whistle phrase that only project managers understand.  It appears to mean "projects that fit my preconceived notions" or "projects that can't be done any other way because of political considerations" or "projects constrained by a customer to have a farcical structure".  Another variant on this dog-whistle phrase is "the realities of running a business."  There's something in their real world that they (a) can't define and (b) prevents any change.]

A few months back, I saw a http://prezi.com presentation that was intended to drive architecture, cost and schedule conversations.  The authors were happy.  I was left relatively mystified.

There are three relevant issues that don't fit well into a prezi presentation.

1. Open Technology questions.  If it's new software development, there are unanswered software API, performance, quality, fitness-for-purpose questions.  If it's a clone or a clean-room reverse engineering, then there may be few questions.  Otherwise, there must be unknowns.
2. Open User Story questions.  If it's new software development, then the user stories are not complete, consistent and optimized.  There must be something missing or extra.
3. Open User Experience questions.  I'm not a UX person.  Tinkering with an Arduino has taught me that even in the most prosaic, simplistic project there is a UX element.  

It's easy to trivialize this as "details" or "supporting information" omitted from the presentation.  Actually, it's considerably more interesting that simply elaborating details.  Indeed, the missing stuff is often more important that the elements of the presentation than were provided.

The Standard Gaps

While a presentation can show some elements of UX, it's not interactive.  It cannot provide any useful depth on the UX.  Failure to do UX exploration (with real interaction) is crazy.  Making assumptions about UX is more likely to create cost and risk.

User Stories can be kicked around in a presentation.  However.  Once the highest-priority user stories are built, and lessons learned about the users, the problem and the emerging solution, then the other user stories can (and should) change based on those lessons learned.  Assuming a list of user stories and then committing to a specific schedule enshrines the costs in a bad plan.  And we all know that the plan (and the dates) are sacred once written down.

Software does not exist in a vacuum.

I'll repeat that, since it's a hidden assumption behind the waterfall.

Software does not exist in a vacuum.

You have a Heisenberg problem when you start building software and releasing it incrementally.  The presence of the new software, even in a first-release, beta-test, barely-working mode changes the ecosystem in which the software exists.  Other applications can (and will) be modified to adjust to the new software.  Requirements can (and should) change.

A waterfall project plan exists to prevent and stifle change.  A long list of tasks and dates exists to assure that people perform those tasks on those dates.  This is done irrespective of value created and lessons learned.

Technology changes.  Our understanding of the technology changes.  One wrong assumption about the technology invalidates the deliverables for a sprint.  In a project with multiple people, multiple assumptions and multiple sprints, this effect is cumulative.  Every assumption by every person for every technology is subject to (considerable) error.

Every technology assumption must be looked at as a needless cost that's being built into the project.

Recently, I've been working on a project with folks that don't know Django very well.  Their assumptions are -- sometimes -- alarming.  Part way through a sprint, I got a raft of technical questions on encrypting passwords.  It's hard to state it strongly enough: never encrypt a password.  What's more useful is this: always hash passwords.  The original password must be unrecoverable.  There are lots of clever hashing techniques.  Some of which are already part of Django.  A significant portion of the sprint (appeared) to be based on reinventing a feature already part of Django.

Do the math: a few wrong assumptions made during project planning are canonized forever as requirements in the project.  With a waterfall project, they're not easy to change.  Project managers are punished for changing the project.  You can't increase the project deadline; that's death.  You can't decrease it, either: you get blamed for sand-bagging the estimates.

Arduino Technology Lessons Learned

After buying several 74HC595 shift registers from Sparkfun, I realized that my assumptions about the interfaces were utterly wrong.  I needed to solder a mix of right-angle header and straight-through headers onto the breakout boards.  I tried twice to order the right mix headers.  It seems so simple.  But assumptions about the technology are often wrong.

This is yet more anecdotal evidence that all assumptions must be treated as simple lies.  Some managers like to treat assumptions as  "possibly true" statements; i.e., these are statements that are unlikely to be false.  This is wrong.  Assumptions always have a very high probability of being false, since they're not based on factual knowledge.

Some managers like to treat assumptions as "boundary conditions":  i.e.,  if the assumption is true, then the project will go according to plan.  Since all of the assumptions will be prove to be incorrect, this seems like simple psychosis.  

[Interestingly, the assumptions-are-boundary folks like to play the "real world" card: "In the real world, we need to make assumptions and go forward."  Since all assumptions will be shown to be incorrect, why make them?  Wouldn't it be more rational to say "we need to explore carefully by addressing high-righ unknowns first"?  Wouldn't it be better to both gather facts and build an early release of the software at the same time?]

Arduino User Story Assumptions

After building a prototype that addressed two of the user stories, it slowly became clear that the third user story didn't actually exist.

Sure, the words were there on paper.  Therefore, there's a user story.

But. 

There was nothing to actually do.  

The whole "As a [role], I need [feature] so that I can [benefit]" that was written on day one was based on a failure to understand precisely how much information was trivially available.  The benefit statement was available with a small software change and no separate user story.  And no separate hardware to support that user story.

Exploration and tinkering reduced the scope of the work.  In the real world.

[In the "real world" where waterfall is considered important, exploration is described as unbounded spending of limited resources.  In the real real world, money must be spent; it can either be long hand-wringing meetings or it can be prototype development.]

The user story (written before a prototype existed) was based on a failure to fully understand the UX.  The only way to fully understand the UX is to build it.  Education costs money.

Arduino UX Learnings

Perhaps the most important issue here is UX.

Once upon a time, UX was expensive, difficult and complex.  So difficult that special-purpose prototyping tools were created to make it possible to create a preliminary UX that could be used to confirm UX and user stories.

This UX prototyping effort was real money spent as part of "requirements" or "design"; it created documentation that flowed over the waterfall for subsequent development.

This notion is obsolete.  And has been obsolete for several years.

UX is now so easy to build that it makes much more sense to build two (or three) competing UX's and compare them to see which is actually better.

Indeed, it makes a lot of sense to build one UX and release it; make a real effort at solving the user's problems.  Then build a second UX for A/B testing purposes to see if there's room for improvement.

I'll repeat that for folks who really like the waterfall.

It's now cheaper to actually build two than it is to write detailed requirements for one.

[In the "real world", this is deprecated as "wasting time playing with the UX".  As if written requirements based on nothing more than a whiteboard are more "real" than hands-on experience with the UX.]

You can prove this to yourself by actually observing actual UX developers knocking out pages and screens.  Long "requirements gathering" meetings with prospective users amount to a waste of time and money.  Long "brainstorming" sessions, similarly, are wasteful.  Short, ongoing conversations, a build, a release, and a follow-up review has more value, and costs less.  

Do the math.  Several users and business analysts in several multiple-hour meetings costs how much?

A pair of developers banging out fully-functioning, working UX for a use case costs how much?

A slavish adherence to waterfall development creates "real world" costs.