RESTful Web Services Testing, Q&A

Some background:

I was vaguely pointed at one call in an API, via a 2-page "tutorial" that uses CURL examples. Told "Test this some more." by the guy who'd been doing some amount (none?) of hand "success path" testing via CURL. This has since morphed into "regression testing things, all 12 calls", "we have a build API as well", and "there's this hot new feature for a vendor conference in a couple weeks ..."

There was more, but you get the idea.  There were so more specific "requirements" for the RESTful unit testing environment.

1) Get "smoke test" coverage vs. all the calls 

A sequence of CURL requests to exercise a server can be viewed as "testing".  It's piss-poor at best.  Indeed, it's often misleading because of the complexity of the technology stack.

In addition to the app, you're also testing Apache (or whatever server they're using) plus the framework, plus the firewall, plus caching and any other components of the server's technology stack.

However, it does get you started ASAP.

2) expand / parameterize that 

CURL isn't the best choice.  You wind up writing shell scripts.  It gets ugly before long.

Python is better for this.

Selenium may also work.  Oh wait.  Selenium is written in Python.

3) build out to response correctness & error codes 

Proper design for testability makes this easy.

However.  When you've be tossed a "finished" RESTful web service that you're supposed to be testing, you have to struggle with expected vs. actual.

It's not trivial because the responses may have legitimate variances: date-time stamps, changing security tokens or nonces, sequence numbers that vary.

Essentially, you can't just use the OS DIFF program to compare actual CURL responses with expected CURL responses.

You're going to have to parse the response, pick out appropriate fields to check and write proper unittest assertions around those fields.

4) layer in at least that much testing for the new, new feature breathlessly happening RIGHT NOW.

Without a proper design for testability, this can be painful.

If you're using a good unit test framework, it shouldn't be impossible.  Your framework must be able to start the target RESTful web service for a TestCase, exercise the TestCase, and then shutdown the target RESTful web service when the test has completed.

Now, you're just writing unittest TestCase instances for the new feature breathlessly happening RIGHT NOW.  That should be manageable.

...tool things I've found so far... [list elided] 

All crap, more or less.  They're REST client libraries, not testing tools.

You need a proper unit testing framework with TestCase and TestSuite classes and a TestRunner.  The tools you identified aren't testing frameworks, they're lower level REST client and client library.  CURL, by itself, isn't really very good for robust testing unless you embed CURL in some test framework.

For defining interfaces (2), I have found these... [list elided]

API's in a typical RESTful environment have little or no formal definition, other than Engrish.  WSDL is for Java/XML/SOAP.  It's not used much for (simpler) REST.  For the most part, REST API definitions (i.e., via JSON or whatever) are mostly experimental.  Not standardized.  Not to be trusted.

The issue is one of parallel maintenance.  The idea is that a REST frameworks can operate without too much additional JSON or XML folderol; just the code should be sufficient.

If there's no WSDL (because it's just REST) then there's no formal definition of anything.  Sorry.

I (perhaps foolishly) figured there's be some standard way to consume the standard format definition of an API, to generate test cases and stubbing at least. Maybe even a standard set of verifications, like error codes. So I went a-googling for 1) a standard / conventional way to spec these APIs, 2) a standard / conventional tool or maybe tools @one per stack, and 3) a standard / conventional way to generate tests or test scaffolding in these tools, consuming the standard / conventional API spec. So far, not so much.

"So far, not so much" is the state of the art.  You have correctly understood what's available.

REST -- in spite of it's trivial simplicity and strict adherence to HTTP -- is a rather open world.  It's also pretty simple.  Fancy tools don't help much.

Why not?

Because decent programming languages already do REST; tools don't add significant value.  In the case of Python, there are relatively few tools (Selenium is the big deal, and it's for browser testing) because there's no real marketplace niche for them.  In general, simple Python using httplib (or Python 3 http.client) can test the living shit out of RESTful API better than CURL/DIFF with no ugly shell-script coding.  Only polite, civilized Python coding.

RESTful Web Service Testing

Unit testing RESTful web services is rather complex.  Ideally, the services are tested in isolation before being packaged as a service.  However, sometimes people will want to test the "finished" or "integrated" web services technology stack because (I suppose) they don't trust their lower-level unit tests.

Or they don't have effective lower-level unit tests.

Before we look at testing a complete RESTful web service, we need to expose some underlying principles.

Principle #1.  Unit does not mean "class".  Unit means unit: a discrete unit of code.  Class, package, module, framework, application.  All are legitimate meanings of unit.  We want to use stable, easy-to-live with unit testing tools.  We don't want to invent something based on shell scripts running CURL and DIFF.

Principle #2.  The code under test cannot have any changes made to it for testing.  It has to be the real, unmodified production code.  This seems self-evident.  But.  It gets violated by folks who have badly-designed RESTful services.

This principle means that all the settings required for testability must be part of an external configuration.  No exceptions.  It also means that your service may need to be refactored so that the guts can be run from the command line outside Apache.

When your RESTful Web Service depends on third-party web service(s), there is an additional principle.

Principle #3.  You must have formal proxy classes for all RESTful services your app consumes.  These proxy classes are going to be really simple, since they must trivially map resource requests to proper HTTP processing.  In Python, it is delightfully simple to create a class where each method simply uses httplib (or http.client in Python 3.2) to make a GET, POST, PUT or DELETE request.  In Java you can do this, also, it's just not delightfully simple.

TestCase Overview

Testing a RESTful web service is a matter of starting an instance of the service, running a standard unit testing TestCase, and then shutting that instance down.  Generally this will involve setUpModule and tearDownModule (in Python parlance) or a @BeforeClass and @AfterClass (in Java parlance).

The class-level (or module-level) setup must start the application server being tested.  The server will start in some known initial condition.  This may involve building and populating known database, too.  This can be fairly complex.

When working with SQL, In-memory databases are essential for this.  SQLite (Python) or http://hsqldb.org (Java) can be life-savers because they're fast and flexible.

What's important is that the client access to the RESTful web service is entirely under control of a unit testing framework.

Mocking The Server

A small, special-purpose server must be built that mocks the full application server without the endless overheads of a full web server.

It can be simpler to mock a server rather than to try to reset the state of a running Apache server.  TestCases often execute a sequence of stateful requests assuming a known starting state.   Starting a fresh mock server is sometimes an easy way to set this known starting state.

Here's a Python script that will start a server.   It writes the PID to a file for the shutdown script.

import http.server
import os
from the_application import some_application_feature
class AppWrapper( http.server.BaseHTTPRequestHandler ):
    def do_GET( self ):
        # Parse the URL

        id= url.split("/")[-1]

        # Invoke the real application's method for GET on this URL.
        body= some_application_feature( id )
        # Respond appropriately
        self.send_response( 200, body )
    ... etc ...

# Database setup before starting the service.
# Filesystem setup before starting the service.
# Other web service proxy processes must be started, too.
with open("someservice.pid","w") as pid_file:
    print( os.getpid(), file=pid_file )
httpd = http.server.HTTPServer("localhost:8000", AppWrapper)
try:
    httpd.serve_forever()
finally:
    # Cleanup other web services.

Here's a shutdown script.

import os, signal
with open("someservice.pid") as pid_file:
    pid= int( pid_file.read() )
os.kill( pid, signal.CTRL_C_EVENT )

These two scripts will start and stop a mock server that wraps the underlying application.

When you're working in Java, it isn't so delightfully simple as Python  But it should be respectably simple.  And you have Jython Java integration so that this Python code can invoke a Java application without too much pain.

Plus,  you can always fall further back to a CGI-like unit testing capability where "body= some_application_feature( id )" becomes a subprocess.call(). Yes it's inefficient.  We're just testing.

This CGI-like access only works if the application is very well-behaved and can be configured to process one request at a time from a local file or from the command line.  This, in turn, may require building a test harness that uses the core application logic in a CGI-like context where STDIN is read and STDOUT is written.

Password Encryption -- Short Answer: Don't.

First, read this.    Why passwords have never been weaker—and crackers have never been stronger.

There are numerous important lessons in this article.

One of the small lessons is that changing your password every sixty or ninety days is farcical.  The rainbow table algorithms can crack a badly-done password in minutes.  Every 60 days, the cracker has to spend a few minutes breaking your new password.  Why bother changing it?  It only annoys the haxorz; they'll be using your account within a few minutes.  However.  That practice is now so ingrained that it's difficult to dislodge from the heads of security consultants.

The big lesson, however, is profound.

Work Experience

Recently, I got a request from a developer on how to encrypt a password.  We have a Python back-end and the developer was asking which crypto package to download and how to install it.

"Crypto?" I asked.  "Why do we need crypto?"

"To encrypt passwords," they replied.

I spat coffee on my monitor.  I felt like hitting Caps Lock in the chat window so I could respond like this: "NEVER ENCRYPT A PASSWORD, YOU DOLT."

I didn't, but I felt like it.

Much Confusion

The conversation took hours.  Chat can be slow that way.  Also, I can be slow because I need to understand what's going on before I reply.  I'm a slow thinker.  But the developer also needed to try stuff and provide concrete code examples, which takes time.

At the time, I knew that passwords must be hashed with salt.  I hadn't read the Ars Technica article cited above, so I didn't know why computationally intensive hash algorithms are best for this.

We had to discuss hash algorithms.

We had to discuss algorithms for generating unique salt.

We had to discuss random number generators and how to use an entropy source for a seed.

We had to discuss http://www.ietf.org/rfc/rfc2617.txt in some depth, since the algorithms in section 3.2.2. show some best practices in creating hash summaries of usernames, passwords, and realms.

All of this was, of course, side topics before we got to the heart of the matter.

What's Been Going On

After several hours, my "why" questions started revealing things.  The specific user story, for example, was slow to surface.

Why?

Partly because I didn't demand it early enough.

But also, many technology folks will conceive of a "solution" and pursue that technical concept no matter how difficult or bizarre.  In some cases, the concept doesn't really solve the problem.

I call this the "Rat Holes of Lost Time" phenomena: we chase some concept through numerous little rat-holes before we realize there's a lot of activity but no tangible progress.  There's a perceptual narrowing that occurs when we focus on the technology.  Often, we're not actually solving the problem.

IT people leap past the problem into the solution as naturally as they breathe. It's a hard habit to break.

It turned out that they were creating some additional RESTful web services.  They knew that the RESTful requests needed proper authentication.  But, they were vague on the details of how to secure the new RESTful services.

So they were chasing down their concept: encrypt a password and provide this encrypted password with each request.  They were half right, here.  A secure "token" is required.  But an encrypted password is a terrible token.

Use The Framework, Luke

What's most disturbing about this is the developer's blind spot.

For some reason, the existence of other web services didn't enter into this developer's head.  Why didn't they read the code for the services created on earlier sprints?

We're using Django.  We already have a RESTful web services framework with a complete (and high quality) security implementation.

Nothing more is required.  Use the RESTful authentication already part of Django.

In most cases, HTTPS is used to encrypt at the socket layer.  This means that Basic Authentication is all that's required.  This is a huge simplification, since all the RESTful frameworks already offer this.

The Django Rest Framework has a nice authentication module.

When using Piston, it's easy to work with their Authentication handler.

It's possible to make RESTful requests with Digest Authentication, if SSL is not being used.  For example, Akoha handles this.  It's easy to extend a framework to add Digest in addition to Basic authentication.

For other customers, I created an authentication handler between Piston and ForgeRock OpenAM so that OpenAM tokens were used with each RESTful request.  (This requires some care to create a solution that is testable.)

Bottom Lines

Don't encrypt passwords.  Ever.

Don't write your own hash and salt algorithm.  Use a framework that offers this to you.

Read the Ars Technica article before doing anything password-related.

Performance "Tuning": running in 1/100th the time

For the 757 Python Meetup group, someone proposed looking at some Python code they had which was slow.  The code implemented a variation on the Elo chess rating system.  It applied the ratings to other sports, and used the points scored as well as basic win/lose/tie to work out a ranking for sports teams.  Very clever stuff.

But.

It was described as horribly slow. Since it was only 400 lines of code, it was a great subject for review in a Python meetup.  I would be able to show some tweaks and performance tips.

Hypothetical Step 1:  Profile

My first thought was to run the profiler and see what popped up as a possible root cause of slowness.

Generally, there are three kinds of performance problems.

• Wrong data structure.  Or wrong algorithm.  (They're simply opposite sides of the same coin.)  This generally leads to dramatic, earth-shattering improvements.
• Piss-Poor Programming Practices.  This kind of fine-tuning yields minuscule improvements.  In some cases, no measurable change at all.
• Bugs.  Every time I've been asked to improve "working" code, I find that it has bugs.  Every time.  These can exacerbate performance problems.  Interestingly, most of these are readily apparent during an initial survey:  i.e., while simply reading the code to see how it works.  Trying to create unit tests for the purposes of refactoring often reveals additional bugs.

I didn't know what I was going to see in this 400-line program.

Hypothetically, profiling will help show what kind of problems we have.

Prior to profiling, of course, we need to run the darn thing.  Thankfully, they provided some typical input files.  For example, 1979 High School Football season results.  159 lines of patiently gathered teams and scores.

It Didn't Run

When we tried to run it with the profiler, we found that we had a bug.  Tracking down the bug, revealed the essential performance problem, also.

The core problem was a failure to make use of Python data structures.  

This manifested itself in a number of really bad design errors.  Plus, it presented as the actual, show-stopping, serious bug.

In order to work out team rankings, the program kept a list of teams.

I'll repeat that for folks who are skimming.

In order to work out team rankings, the program kept a list of teams.

Not a dict mapping from team name to team details.  But a simple list.  Locating a team in the list meant iterating through the list, looking for the matching team.  Really.

for t in team_list:

    if t['name'] == target_name:

        process( t )

This kind of thing was repeated in more than one place.

And one of those repeats had the bug in it.

What we have here is "wrong data structure".  Replacing a list with a dict will have earth-shattering impact on performance.

The Bug

The bug, BTW, was a lookup loop which had the additional requirement of adding missing teams.  It tried to use the for-else structure.  This was the intended code (not the actual code).

for t in team_list:

    if t['name'] == target_name:

        return

else:

    t['name']= init_new_team(target_name)

    

This is a first-class bit of Python.  An else clause on a for statement is a real feature of the language.  However, it's obscure enough that it's easy to get wrong.

However, it's also unique to Python, and the kind of thing that can lead to confusion.  I discourage it's use.  

Test-Driven Reverse Engineering

We're doing TDRE on a little piece of recreational programming.  This means that we need to fabricate unit tests for code that is purported to work.  Some folks like to call this "exploratory testing" (a phrase which is contradictory, and consequently meaningless.)  We're "exploring".  We're not "testing".

Once the core bug is fixed, we can run sample data through the application to get "big picture" results.  We can extract bits from the sample data and exercise various functions in isolation to determine what they actually do now, bugs included.

Since this is so simple--and we're going to review it during a 2-hour meet up--and there's nothing billable going on--we can get by with a few really simple unit tests.  We'll run the darn thing once to create some expected output.  We save the output and use that single output log as the expected results from each refactoring step.

More complex applications require more unit tests.  For a billable TDRE effort last year, I had to create 122 unit tests to successfully refactor a program of over 6,000 lines of code.  It took several weeks.

Real Step 1: Fix The Data Structure

Before profiling (but after running to capture some output) we have to fix the essential data structure.  Repeated scanning of a list is never going to perform well.

The whole point of the study of "Data Structures" is to prevent (or optimize) search.

In this case, we can prevent linear search of a list by using a dictionary.  That, clearly, is job one.

It was a pervasive rewrite.   Essentially, everything in this little program included a goofy loop to lookup a team in the list.  The Elo algorithm itself, which is already O(n²), is dragged down by using the linear search for the opposing team four more times, making it O(n³).

Cyclomatic Complexity

One of the big "issues" is the use of if statements throughout the scoring algorithm.  An if statement creates Cyclomatic Complexity and can lead to performance problems.  Generally, if statements should be avoided.

This algorithm applies some normalization factors to reconcile scoring with win/loss numbers in different sports.  Basketball, specifically, involves generally high scores.  Since there are 2-point and 3-point scoring opportunities, a factor is used to normalize the points into "goals".  Football, similarly, has numerous scoring opportunities with values of 1, 2, 3 and 6 points; the scores here are also normalized.

This normalization was done with an if statement that was evaluated deep inside the Elo algorithm.  Repeatedly. Evaluated.

The two functions that handled the normalizations, plus the normalization factors, are ideal candidates for OO design.  There's a clear hierarchy of classes here.  A superclass handles most sports, and two polymorphic subclasses handle football and basketball normalization.

The if statement is now "pushed up" to the very beginning of the program where an instance of the sports normalization object is created.  This object's methods are then used by the Elo algorithm to normalize scores.

Icing on the Cake

Once we've fixed the bug and replaced a list with a dict, everything else is merely icing.
Some other OO changes.

1. The "Team" information should not be a flat, anonymous dictionary.  It should be a proper class definition with proper attributes.  There aren't many methods, so it's easy to create. 
2. The "Game" information is read by csv.DictReader.  However, it should not remain a simple, anonymous dict.  As with a Team, a simple class can be created to handle Game.
3. The overall structure of the application needs to be broken into two sections.  The command-line interface parses options, opens files, and generally gets everything set up.  The actual ranking algorithm should be a function that is given an open file-like object plus the Sport object for normalization.  This allows the ranking algorithm to be reused in other contexts than the command-line (i.e. a web service).

A more subtle OO design point is the question of "mutability".  A Team in this application is little more than a name.   There are also numerous "stateful" values that are part of the Elo algorithm.   A Game, similarly, is an immutable pair of teams and scores.  However, it has some mutable values that are part of the Elo algorithm.  

Really, we have immutable Team and GameHistory objects, plus a few values that are used as part of the Elo calculation.  I'm a big fan of disentangling these mutable and immutable objects from each other.  

I suspect that the Elo algorithm doesn't really need to update the "state" of an object.  I suspect that it actually creates (and discards) a number of immutable candidate ranking objects.  The iteration that leads to convergence might be a matter of object creation rather than update.  I didn't make this change, since it required real work and we were out of time.

Bottom Line

The more times I do TDRE to improve performance, the more I realize that it's all about bugs and data structures.

This recreational application took 45-60 seconds to process one year's record of games for a given league.  It now takes less than 0.2 seconds to do the same volume of work.  Two test cases involving a complete run of 159 records runs in 0.411 seconds.  That's 1/100th the time simply from switching data structures.

The idea of "tweaking" a working program to improve performance is generally misleading.  It might happen, but the impact is minuscule at best.  

Here's the checklist for getting 100:1 improvements.

• Remove searches.  
• Remove deeply-nested if statements.  

Generally, reduce Cyclomatic Complexity.

How Expensive is a "Waterfall" Project Plan?

It's impossible to step into the same river twice; other waters are flowing toward the sea.  It's impossible to do "head-to-head" project comparisons.  You can't have the same people doing the same thing with only one constraint changed.  You can try to resort to having similar people doing the exact same thing.

It's impossible to provide trivial quantification of the costs of a waterfall approach.  However, there's a lot of places where "up front" engineering adds cost and risk.  A waterfall plan adds friction, zero-value administrative costs and (sometimes) extra work.

For software development that doesn't involve "toy" projects, but apply to the magical "real world" projects, it's difficult to justify building the same "real world" thing twice.  It's hard to create a detailed and purely economic analysis of this question.  Therefore, we have to resort to hand-waving and speculation.

[I put "real world" in quotes, because the "real world" is a kind of dog-whistle phrase that only project managers understand.  It appears to mean "projects that fit my preconceived notions" or "projects that can't be done any other way because of political considerations" or "projects constrained by a customer to have a farcical structure".  Another variant on this dog-whistle phrase is "the realities of running a business."  There's something in their real world that they (a) can't define and (b) prevents any change.]

A few months back, I saw a http://prezi.com presentation that was intended to drive architecture, cost and schedule conversations.  The authors were happy.  I was left relatively mystified.

There are three relevant issues that don't fit well into a prezi presentation.

1. Open Technology questions.  If it's new software development, there are unanswered software API, performance, quality, fitness-for-purpose questions.  If it's a clone or a clean-room reverse engineering, then there may be few questions.  Otherwise, there must be unknowns.
2. Open User Story questions.  If it's new software development, then the user stories are not complete, consistent and optimized.  There must be something missing or extra.
3. Open User Experience questions.  I'm not a UX person.  Tinkering with an Arduino has taught me that even in the most prosaic, simplistic project there is a UX element.  

It's easy to trivialize this as "details" or "supporting information" omitted from the presentation.  Actually, it's considerably more interesting that simply elaborating details.  Indeed, the missing stuff is often more important that the elements of the presentation than were provided.

The Standard Gaps

While a presentation can show some elements of UX, it's not interactive.  It cannot provide any useful depth on the UX.  Failure to do UX exploration (with real interaction) is crazy.  Making assumptions about UX is more likely to create cost and risk.

User Stories can be kicked around in a presentation.  However.  Once the highest-priority user stories are built, and lessons learned about the users, the problem and the emerging solution, then the other user stories can (and should) change based on those lessons learned.  Assuming a list of user stories and then committing to a specific schedule enshrines the costs in a bad plan.  And we all know that the plan (and the dates) are sacred once written down.

Software does not exist in a vacuum.

I'll repeat that, since it's a hidden assumption behind the waterfall.

Software does not exist in a vacuum.

You have a Heisenberg problem when you start building software and releasing it incrementally.  The presence of the new software, even in a first-release, beta-test, barely-working mode changes the ecosystem in which the software exists.  Other applications can (and will) be modified to adjust to the new software.  Requirements can (and should) change.

A waterfall project plan exists to prevent and stifle change.  A long list of tasks and dates exists to assure that people perform those tasks on those dates.  This is done irrespective of value created and lessons learned.

Technology changes.  Our understanding of the technology changes.  One wrong assumption about the technology invalidates the deliverables for a sprint.  In a project with multiple people, multiple assumptions and multiple sprints, this effect is cumulative.  Every assumption by every person for every technology is subject to (considerable) error.

Every technology assumption must be looked at as a needless cost that's being built into the project.

Recently, I've been working on a project with folks that don't know Django very well.  Their assumptions are -- sometimes -- alarming.  Part way through a sprint, I got a raft of technical questions on encrypting passwords.  It's hard to state it strongly enough: never encrypt a password.  What's more useful is this: always hash passwords.  The original password must be unrecoverable.  There are lots of clever hashing techniques.  Some of which are already part of Django.  A significant portion of the sprint (appeared) to be based on reinventing a feature already part of Django.

Do the math: a few wrong assumptions made during project planning are canonized forever as requirements in the project.  With a waterfall project, they're not easy to change.  Project managers are punished for changing the project.  You can't increase the project deadline; that's death.  You can't decrease it, either: you get blamed for sand-bagging the estimates.

Arduino Technology Lessons Learned

After buying several 74HC595 shift registers from Sparkfun, I realized that my assumptions about the interfaces were utterly wrong.  I needed to solder a mix of right-angle header and straight-through headers onto the breakout boards.  I tried twice to order the right mix headers.  It seems so simple.  But assumptions about the technology are often wrong.

This is yet more anecdotal evidence that all assumptions must be treated as simple lies.  Some managers like to treat assumptions as  "possibly true" statements; i.e., these are statements that are unlikely to be false.  This is wrong.  Assumptions always have a very high probability of being false, since they're not based on factual knowledge.

Some managers like to treat assumptions as "boundary conditions":  i.e.,  if the assumption is true, then the project will go according to plan.  Since all of the assumptions will be prove to be incorrect, this seems like simple psychosis.  

[Interestingly, the assumptions-are-boundary folks like to play the "real world" card: "In the real world, we need to make assumptions and go forward."  Since all assumptions will be shown to be incorrect, why make them?  Wouldn't it be more rational to say "we need to explore carefully by addressing high-righ unknowns first"?  Wouldn't it be better to both gather facts and build an early release of the software at the same time?]

Arduino User Story Assumptions

After building a prototype that addressed two of the user stories, it slowly became clear that the third user story didn't actually exist.

Sure, the words were there on paper.  Therefore, there's a user story.

But. 

There was nothing to actually do.  

The whole "As a [role], I need [feature] so that I can [benefit]" that was written on day one was based on a failure to understand precisely how much information was trivially available.  The benefit statement was available with a small software change and no separate user story.  And no separate hardware to support that user story.

Exploration and tinkering reduced the scope of the work.  In the real world.

[In the "real world" where waterfall is considered important, exploration is described as unbounded spending of limited resources.  In the real real world, money must be spent; it can either be long hand-wringing meetings or it can be prototype development.]

The user story (written before a prototype existed) was based on a failure to fully understand the UX.  The only way to fully understand the UX is to build it.  Education costs money.

Arduino UX Learnings

Perhaps the most important issue here is UX.

Once upon a time, UX was expensive, difficult and complex.  So difficult that special-purpose prototyping tools were created to make it possible to create a preliminary UX that could be used to confirm UX and user stories.

This UX prototyping effort was real money spent as part of "requirements" or "design"; it created documentation that flowed over the waterfall for subsequent development.

This notion is obsolete.  And has been obsolete for several years.

UX is now so easy to build that it makes much more sense to build two (or three) competing UX's and compare them to see which is actually better.

Indeed, it makes a lot of sense to build one UX and release it; make a real effort at solving the user's problems.  Then build a second UX for A/B testing purposes to see if there's room for improvement.

I'll repeat that for folks who really like the waterfall.

It's now cheaper to actually build two than it is to write detailed requirements for one.

[In the "real world", this is deprecated as "wasting time playing with the UX".  As if written requirements based on nothing more than a whiteboard are more "real" than hands-on experience with the UX.]

You can prove this to yourself by actually observing actual UX developers knocking out pages and screens.  Long "requirements gathering" meetings with prospective users amount to a waste of time and money.  Long "brainstorming" sessions, similarly, are wasteful.  Short, ongoing conversations, a build, a release, and a follow-up review has more value, and costs less.  

Do the math.  Several users and business analysts in several multiple-hour meetings costs how much?

A pair of developers banging out fully-functioning, working UX for a use case costs how much?

A slavish adherence to waterfall development creates "real world" costs.

Innovation, Arduino and "Tinkering"

Many of my customers (mostly super-large IT shops) wouldn't recognize innovative behavior.  Large organizations tend to punish defectors (folks that don't conform), and innovation is non-conforming.

I've just had two object lessons in innovation.  The state of the art has left many in-house IT development processes in the dust.  The cost and complexity of innovation has fallen, but organizations continue to lumber along pretending that innovation is risky or complex.

You can find endless advice on how to foster a culture of innovation.  Often, this advice includes a suggestion that innovative projects should somehow "fail fast".  I'm deeply suspicious of "fail fast" advice.  I think it misleads IT management into thinking there's a super-cheap way to innovate.  It's misleading because "fail fast" leaves too many questions unanswered.

• How soon do you know something's about to be a failure?  
• What's the deadline that applies so that failure can happen quickly?  
• What's the leading indicator of failure?

If you are gifted enough to predict the future -- and can predict failure -- why not apply that gift to predicting success?  Give up on the silliness of unstructured "innovation" and simply implement what will not fail.

At MADExpo, I saw an eye-opening presentation on the Arduino.  I followed that with viewing Massimo Banzi's TED Talk on the subject of Arduino, imagination and open source.

There are two central parts of the Arduino philosophy.

• Tinkering.
• Interaction Design.

Background

Without delving too deeply, I'm trying to build a device that will measure the position of a hydraulic piston.  It's the hydraulic steering on a boat, and a measurement of the piston position provides the rudder position, something that's handy for adjusting sail trim to reduce the strain on an autopilot.

Clearly, such a device needs to be calibrated with the extreme port and starboard range of motion.  Barring unusual circumstances, the amidships position is simply the center between the two limits.

Part 1.  Buy an Arduino, a Sharp GP2Y0A02YK0F IR distance measurer (for 10-80 cm), plus miscellaneous things like breadboard, jumpers, LED's, test leads, etc.  A trip to Radio Shack covers most of the bases.  The rest comes from Sparkfun, Robot Shop, Digi-Key and Mouser.

Part 2.  Learn the language (a subset of C.)  Learn core algorithms (de-bouncing buttons and the IR sensor).

Tinkering

At this point, we've tinkered.  Heavily.

What's important for IT managers is that tinkering doesn't have a project plan.  It doesn't have a simple schedule and clear milestones.  It's a learning process.  It's knowledge acquisition.

The current replacement for tinkering is training.  Rather than learn by attempting (and failing), IT managers hire experts to pass on knowledge.  This is, generally, limiting and specifically stifles innovation.

Years ago, I worked on embedded systems: hardware software hybrids.  We burned ROMs and programmed in assembler.  Back in those days, this kind of tinkering was difficult, and consequently frowned upon.  It was difficult to specify, locate, source, and assemble the components.  There was a lot of reading complex product data sheets to try and determine what to buy and how few were needed.

What had once been a very serious (and very difficult) electrical engineering exercise (IR sensor, button, LED, power supply, etc., etc.) was a few days of tinkering with commodity parts.  The price was low enough and availability ubiquitous enough that frying a few LED's is of no real consequence.  Even frying an Arduino or two isn't much of a concern.

Interaction Design

The next step is to work out the user interface.  For the normal operating mode, the input comes from the hydraulic piston and the output is some LED's to show the displacement left or right of center.  Pretty simple.

However.

There's the issue of calibration.  Clearly, the left and right limits (as well as center position) need to be calibrated into the device.

Just as clearly, this means that the device needs buttons and LED's to switch from normal mode to calibration mode.  And it needs some careful interaction design.  There are several operating modes (uncalibrated, calibrating, normal) with several submodes for calibrating (setting left, setting right, setting center.)

Once upon a time, we wrote long, wordy documents.  We drew complex UML state charts.  We drew all kinds of pictures to try and capture the important features of the interaction.

Enter Arduino

The point of Arduino is not to spend too much time up front over-specifying something that's probably a bad idea.   The point is to experiment quickly with different user interface and interaction experiences to see what works and what doesn't work.

The same is true of many modern development environments.  Web development, for example, can be done by using sophisticated frameworks, writing little backend code and messing with the jQuery, CSS and HTML5 aspects of the interaction.

The scales fell from my eyes when I started to document the various operating modes.

Arduino doesn't have a great unit testing environment.  It's for tinkering, after all.  It's also for building small, focused things.  Not large, rambling, hyper-complex things.  You can achieve complexity through the interaction of small, easy-to-test things.  But don't start out with complexity.

After writing a few paragraphs, I realized that the piston movements could easily be self-calibrating.  Simply track the maximum and minimum distances ever seen.  That's it.  Nothing more.  In the case of a boat, it means swinging the wheel from stop to stop to define the operating range.  That's it.

A button (to clear the accumulated history) is still useful.  But much simpler since it's a one-time-only reset button.  Nothing more.

Moving from idea to working prototype took less time than writing this blog post.

Next steps are to tinker with various display alternatives.  How many LED's?  What colors?  LCD Text Display?  There are numerous choices.

Rather than wasting times on UML, specifications, whiteboard and diagrams, it's a simpler matter to write the user stories and tinker with display hardware.

Cool stuff I saw at MADExpo

A random list of cool things.

1. HTML5.  Start now.  It's supported (more or less) by all browsers if you add appropriate shims.  Start with sites like http://www.html5rocks.com/en/ and continue reading.  It's largely arrived and there's no compelling reason to delay.
2. JavaScript.  I'm not a fan of the language.  However.  It's clearly here to stay. It's part of many important technologies (like CouchDB).  HTML5 shims (or shivs) are necessary.  Flex (and other browser plug-in languages) are effectively dead.  JavaScript is all that's left.
3. Redis.  Wow! Is that cool?  Rather that fart around trying to get outstanding performance out of a clunky old RDBMS, use a simple, high performance data store.
4. MongoDB.  Now that I've seen it, I have a vague notion of places where Mongo is better and places where Couch might be better.  In 80% of the application space, both are fine.  But there's a 20% where we might be able to split a hair and leverage the slightly different feature sets.

I don't build mobile apps.  But if I did, I think I'd start with Appcelerator Titanium before switching to a "native" development environment.  The idea that several standard features are readily accessible with a convenient development environment is pleasant.

I finally laid eyes on an Arduino.  Let me simply say that it looks like dangerous, dangerous fun.  I think I'll be hanging around at Radio Shack a lot.

Book Deal Fell Apart (sigh)

After spending a couple of years (really) working with a publisher, the deal has gone south.

The problem was—likely—all mine.  The book wasn't really what the publisher wanted.  Perhaps it was close and they thought they could edit it into shape.  And perhaps I wasn't responsive enough to criticism.

I like to think that I was responsive, since I made almost all of the suggested changes.

But the deal fell apart, so my subjective impression isn't too relevant.

What To Do?

I have two essential choices here.

1. Finish up.  Simply press on with the heavily-edited version.  The path of least resistance.
2. Roll back the content.  This involves going back to a less heavily edited version and merging in the relevant changes and corrections from the writing process.

Option 2 is on the table because my editor was unhappy with "digressive" material.  The editor had a vision of some kind of narrative arc that exposed Python in one smooth story line.  My attempted presentation was to expose language features and provide supporting details.

Perhaps I'm too deeply invested in the details of computer science.   Or.  Perhaps I'm just a lousy writer.  But I felt that the digressions were of some value because they could fill in some of the gaps I observed while coaching and teaching programmers over the last few decades.

In addition to the editorial challenge, there's technical challenge.  Do I step back from LaTeX?

• Use LaTeX.  This means that I would have to create the non-PDF version with a LaTeX to HTML translator.  Read Converting LaTeX to HTML in the Modern Age.  See Pandoc.  Or, it means that I don't offer an HTML version (a disservice, I think.)  Also, I need to unwind the publisher's LaTeX style libraries and revert to a plain LaTeX.  Since LaTeX is semantically poor, I need to rework a lot of RST markup.
• Revert to RST.  While RST tools can make both LaTeX and HTML from a common source, it does mean that some of the fancier LaTeX formatting has to go.  Specifically, I really got to know the algorithm and algorithmic packages.  I hate to give those up.  But maybe I can work something out with Sphinx 1.1.3's various features.

The problem is that three of the four combinations of paths have advantages.

• Finish up using LaTeX is easiest.  Remove the publisher's document style.  Use HeVeA or Pandoc to make HTML.  Move on.  
• Finish up but revert to RST.  This means conversion from the publisher's LaTeX to RST followed by some editing to fix the missing RST semantic markup and then some debugging to get the LaTeX and HTML to look good.
• Rollback the content using LaTeX.  This would be challenging because I would have to merge manually edited publisher-specific LaTeX from the heavily-edited version with the Sphinx-generated LaTeX from the less-heavily edited source.  Things like the publishers' style tags would complicate the merge. 
• Rollback the content and revert to RST. This means using Pandoc to convert the heavily-edited LaTeX to RST. Then merging the relevant edits into the RST original text.   This actually seems pretty clean, since the heavily-edited RST (converted from LaTeX) would be short.

Perhaps, if I was a better writer, I wouldn't have these problems.

It appears than the solution is MacFarlane's Pandoc.  This can reverse LaTeX to RST, allowing easy side-by-side merging of texts from various sources.  Or.  It can convert LaTeX to HTML, allowing easy work with the heavily edited LaTeX version.

How to Write Crummy Requirements

Here's an object lesson in bad requirements writing.

"Good" is defined as a nice simple and intuitive GUI interface. I would be able to just pick symbol from a pallette and put it somewhere and the software would automatically adjust the spacing.

Some problems.

1. Noise words.  Phrases like "'Good' is defined as" don't provide any meaning.  The word "just" and "automatically" are approximately useless.  Here is the two-step test for noise words.  1.  Remove the word and see if the meaning changed.  2.  Put the opposite and see if the meaning changed.  If you can't find a simple opposite, it's noise of some kind.  Often it's an empty tautology, but sometimes it's platitudinous buzzwords.
2. Untestable requirements.  "nice, simple and intuitive" are unqualified and possible untestable.  If it's untestable, then everything meets the criteria (or nothing does.)  Elegant.  State of the art.  Again, apply the reverse test:  try "horrid, complex and counter-intuitive" and see if you can find that component.  No?  Then it's untestable and has no place.
3. Silliness.  "GUI".  It's 2012.  What non-GUI interfaces are left?  Oh right.  The GNU/Linux command line.  Apply the reverse test: try "non-GUI" and see if you can even locate a product.  Can't find the opposite?  Don't waste time writing it down.

What's left?  

pick symbol from a palette ... the software would ... adjust the spacing.

That's it.  That's the requirement.  35 words that mean "Drag-n-Drop equation editing".

I have other issues with requirements this poorly done.  One of my standard complaints is that no one has actually talked to actual users about their actual use cases.  In this case, I happen to know that the user did provide input.

Which brings up another important suggestion.

• Don't listen to the users.

By that I mean "Don't passively listen to the users and blindly write down all the words they use.  They're often uninformed."  It's important to understand what they're talking about.  The best way to do this is to actually do their job briefly.  It's also important to provide demos, samples, mock-ups, prototypes or concrete examples.  It's 2012.  These things are inexpensive nowadays. 

In the olden days we used to carefully write down all the users words because it would take months to locate a module, negotiate a contract, take delivery, install, customize, integrate, configure and debug.  With that kind of overhead, all we could do was write down the words and hope we had a mutual understanding of the use case.  [That's a big reason for Agile methods, BTW:  writing down all the user's words and hoping just doesn't work.]

In 2012, you should be able to download, install and play with candidate modules in less time than it takes to write down all the user's words.  Often much less time.  In some cases, you can install something that works before you can get the users to schedule a meeting.

And that leads to another important suggestion.

• Don't fantasize.

Some "Drag-n-Drop" requirements are simple fantasies that ignore the underlying (and complex) semantic issues.  In this specific example, equations aren't random piles of mathematical symbols.  They're fairly complex and have an important semantic structure.  Dragging a ∑ or a √ from a palette will be unsatisfying because the symbol's semantics are essential to how it's placed in the final typeset equation.

I've worked recently with some folks that are starting to look at Hypervideo.  This is often unpleasantly difficult to write requirements around because it seems like simple graphic tools would be all that's required.  A lesson learned from Hypertext editors (even good ones like XXE) is that "WYSIWYG" doesn't apply to semantically rich markup.  There are nesting and association relationships that are no fun to attempt to show visually.  At some point, you just want to edit the XML and be done with it.

Math typesetting is has deep semantics.  LaTeX captures that semantic richness.  

It's often best to use something like LaTeXiT rather than waste time struggling with finding a Drag-n-Drop tool that has appropriate visual cues for the semantics.  The textual rules for LaTeX are simple and—most importantly—fit the mathematical meanings nicely.  It was invented by mathematicians for mathematicians.

MADExpo 2012

http://madexpo.us/


June 27 - 29

Hampton, VA

I hope to see you there.

QR Code

I suddenly realized that QR Codes are everywhere.

Except my business cards.

http://pypi.python.org/pypi/qrcode/2.0

That should allow me to solve that problem and move on.

Dereliction of Duty

Recently started looking into Metadata Registries and UDEF and related semantic technology.

The Wikipedia page lists a bunch of relevant Metadata Registry projects and commercial products.  Very nice.  Easy to follow the links and determine features and benefits.

However.

A client has IBM Cognos.  Is there any easy to to see what kind of Metadata features are part of Cognos?

No.  Not really.

I wondered about this marketing gap.  Why doesn't IBM (or Oracle, they're pretty bad at this also) provide a tidy list of features?

• They're so big (and arrogant) that they don't feel the need to do any marketing?
• They're so paranoid that they don't want to have their products reduce to a simple bullet list?
• They're sales people are so good that they don't need a web presence to sell their products?
• They already have such tight wired-in relationships that they don't need to do any marketing?

Or is it the "growth by acquisition" problem?  Since IBM acquired Cognos, they hesitate to commit to a list of features?

Whatever the reason, it's frankly difficult to include IBM products in an easy-to-understand info-graphic comparing alternative products.

IBM RAMAC Device: 5 MB

Check out this picture.  http://www.petapixel.com/2011/12/27/what-5mb-of-storage-looked-like-in-1956/

Random Reminiscing Follows

When I was in college (1974-1978) 64K of RAM was the size of a refrigerator.

By 1982, 64K of RAM was an Apple ][+ fully tricked out with the 16K expansion card.

I vaguely remember working with a "tower" device that was a 5MB disk drive.  Think Mac Pro case to hold a disk drive.

By 1985 or so, 128K or RAM was a Macintosh and a 5MB disk drive a big desktop console box.  Smaller than a tower.  Irritating because it took up so much real-estate when compared with the Mac itself that was designed to take up an 8 x 11 space (not including keyboard or mouse).

Now 5MB is round-off error.

What's Important?

Has computing gotten that much better?

Old folks (like me) reminisce about running large companies on small computers.  Now, we can't even get our coffee in the morning without the staggering capabilities of a 32GB iPhone.

Old folks, however, are sometimes mired in bad waterfall development methods and don't understand the value of test-driven development.  While the hardware is amazing, the development tools and techniques have improved, also.

The Universal Data Element Framework (UDEF)

Okay.  This is seriously cool.

The Universal Data Element Framework (UDEF)  provides a controlled vocabulary that should be used to seed a project's data model.

See http://www.udef.com/

See http://www.opengroup.org//udef/

We're looking at applying UDEF retroactively to an existing schema.  What a pain in the neck!

Step 1.  Parse the table names.  In our case, they're simply CONTIGUOUSSTRINGSOFCHARS, so we have to work out a quick lexicon and use that to break the names into words.  Then we can find the obvious aliases, spelling mistakes and noise words.  'spec', 'quanitity' and 'for' are examples of each.

Step 2.  Look up the various words in the UDEF vocabulary to create candidate matches.   Since each individual word is matched, each table will have multiple candidate matches to seed the analyst's thinking.

Step 3.  Manually pick UDEF standard names or create internal extensions to the standard for the problem domain or enterprise unique features.

Do a similar thing for the column names.  In that case, they're CamelCaseWithSomeACRONYMS.  This is slightly easier to parse, but not much.

Eventually, we have to apply real human business analyst grey matter to locating standard names which might fit with the host of legacy names.

Here's the column name parser.

def property_word_iter( prop_name ):
    """Find words via case changes.
   
    -   Lower to upper ends a word.
    -   Upper to lower ends a word.  However.
        Sometimes the Upper is an acronym that was all caps.
        A lookahead is required to disambiguate.
    """
    cc_iter= iter(prop_name)
    word=[ next(cc_iter) ]
    for c in cc_iter:
        if c.isdigit():
            yield ''.join(word)
            yield c
            word=[ next(cc_iter) ]
        if word[-1].islower() and c.islower():
            word.append(c)
        elif word[-1].isupper() and c.isupper():
            word.append(c)
        elif word[-1].islower() and c.isupper():
            yield ''.join(word)
            c2= next(cc_iter)
            if c2.isupper():
                word= [c, c2]
            else:
                word= [c.lower(), c2]
        elif word[-1].isupper() and c.islower():
            c0 = word[-1]
            yield ''.join(word[:-1])
            word= [c0.lower(), c]
        else:
            raise Exception( "What? {0!r} {1!r}".format( word[-1], c ) )
    if word:
        yield ''.join(word)

Stingray Schema-Based File Reader

Just updated the Stingray Reader.  There was an egregious error (and a missing test case).  I fixed the error, but didn't add a test case to cover the problem.

It's simple laziness.  TDD is quite clear on how to tackle this kind of thing.  Write the missing test case (which will now fail).  Then make the code change.

But the code change was so simple.

COBOL Rework

See this article: "The COBOL Brain Drain" in ComputerWorld.  This article is very, very well written and repeats a number of common fallacies.

The fallacies lead to expensive, labor-intensive in-house software development and maintenance.  Either there's a lot of effort poking at the COBOL code.  Or there's a lot of extra code to "wrap" the COBOL code so that it's never touched.

"Migrating large-scale systems built in Cobol is costly and risky."  A popular position.  But the risks are actually quite small; possibly non-existent.  The risks of not converting are usually higher than the risk of conversion.

The perception of the COBOL code is that it's filled with decades of tricky, nuanced legacy details that are hard to fully understand.  This is only partially true.

A great deal of the tricky code is simply redundant.  COBOL is often written with copy-and-paste programming and blocks of code are simply repeated.  It's also important to note that some of the code is no longer exercised in the first place.

Mythical Risk

The "risk" comes from the perceived opacity of  the tricky, nuanced legacy details.  It doesn't appear to be clear what they mean.  How can a project be started when the requirements aren't fully understood?

What appears to be the case in reality is is that this tricky code isn't very interesting.  Most COBOL programs don't do much.  They can often be summarized in a few sentences and bullet points.

Detailed analysis (641,000 lines of code, 933 programs) reveals that COBOL programs often contain several commingled feature sets.

• The actual business rules.  These are often easy to find in the code and can also be articulated by key users.  The real work is usually quite simple.  
• A bunch of hackarounds.  These are hacks to work around bugs that occur elsewhere in the processing.  Sometimes a hackaround introduces additional problems which require yet more hackarounds.  All of this can be ignored.  
• Solutions to COBOL data representation issues.  Most of these seem to be "subtype" issues: a flag or indicator is introduced to distinguish between subtypes.  Often, these are extensions.  A field that has a defined range of values ("A", "C" or "D") has a few instances of "*" to indicated another subclass that was introduced with a non-standard code for peculiar historical reasons.

Once we separate the real code from the hackarounds and the representation issues, we find that most COBOL programs are trivial.  About 46% of the lines of code (74% of the distinct programs) involves programs that read one to four files to write a report.  In effect, these programs do a simple "relational join" or query.  These programs have single-sentence summaries.  

The hackaround problem is profound.  When looking at COBOL code, there may be endless copy-and-paste IF-statements to handle some issue.  There may be whole suites of programs designed to work around some issue with a third-party package.  There may be suites of programs to deal with odd product offerings or special customer relationships.  

The remaining 26% of the non-trivial programs follow a normal distribution of 1/4 simple, 1/2 moderately complex, and 1/4 regrettably and stupidly complex.  That final 5% of the programs will also be a whopping 20% of the lines of code.  They are the few big programs that really matter.  

Risk Mitigation

The risk mitigation strategy involves several parts.  

1. Data Profiling.  The COBOL data may have considerable value.  The processing is what we're trying to rewrite into a modern language.  A profile of every field of every file will reveal that (a) most of the data is usable and (b) the unusable portion of the data isn't very valuable.   
2. Triage.  We can summarize 80% of the code in simple sentences.  46% of the code is single-sentence summaries.  34% of the code has multiple sentence summaries.  The remaining 20% requires in-depth analysis because the programs are large; they average of 2400 lines of code each.
3. Test-Driven Reverse Engineering.  Since a 5% of the programs do the real updates, it's important to collect the inputs and outputs of these few programs.  This forms a core test suite.
4. Agile Methods.  Find the user stories which really matter.  Find the COBOL programs that implement those user stories.

The most important risk reduction strategy is to take an Agile approach.  We need to prioritize based on value creation.  All COBOL programs are not created equal.  Some fraction can simply be ignored.  Another fraction produces relatively low value, and can be converted last.

The second most important risk mitigation is to build conversions from legacy COBOL files to the new, preferred databases.  The data is preserved.  

There's almost no risk in rewriting the 46% of low-complexity COBOL lines of code.  This code is trivial.  Note that some of this code actually has no business value.  Simply ignoring the no-value code also reduces risk.  Since we're using live files to drive testing, we can easily prove that our new programs work.

It's not risky to focus on the 20% of high-value COBOL lines of code.  This contains most (or all) of the processing that the business needs to have preserved.  They can articulate the user stories; it's easy to confirm that the COBOL does what the business needs.  It's easy to use live data to drive the reverse engineering.

The remaining 34% of the code base may actually involve a small amount of overlooked complexity.  There may be a nuance here that really matters.    

This overlooked "nuance" is something that the users didn't articulate, but it showed up in our unit testing.  We couldn't reproduce an expected output because we didn't correctly locate all the processing steps.  It wasn't in our summary of the 80% of moderate-to-low complexity programs.  It wasn't in our detailed analysis of the 20% subset of hyper-complex, large programs.  

We've reduced the risk by reducing the volume of code that needs to be examined under the microscope. We've reduced the risk by using live files for parallel testing.  We've reduced the risk by collecting user stories.

The remaining risks are ordinary project risks, unrelated to COBOL or legacy data.

The Other Great Lie

Another popular fallacy is this:

"The business wants us to make investments in programming that buys them new revenue. Rewriting an application doesn't buy them any value-add".

The value-add is to create a nimble business.  Purging the COBOL has a lot of advantages.

• It reduces the total number of lines of code.  Reducing costs.  Improving time-to-market for an IT solution to a business problem.
• It reduces the number of technologies competing for mind-share.  Less thinking about the legacy applications is less time wasted solving problems.
• It reduces the architectural complexity.  If the architecture is a spaghetti-bowl of interconnections between Web and Legacy COBOL and Desktop, then following the spaghetti-like connections is simply a kind of intellectual friction.

The COBOL does not need to be purged all at once through a magical "big-bang" replacement.  It needs to be phased out.  

Agile techniques need to be applied.  A simple backlog of high-value COBOL-based user stories is the place to start.  The prioritization of these stories needs to then be clustered around the data files. 

Ideally all of the programs which create or update a given file (or related group of files) can be rewritten in a tidy package.  The old files can be used for Test-Driven Reverse Engineering.  Once the programs have been rewritten, the remaining COBOL legacy applications can continue to operate, using a file created by a non-COBOL application.

Each file (and related cluster of programs) is replaced from high-value down to low-value.  Each step creates a more nimble organization.

The Passive-Aggressive Programmer (again)

I'm not even interested in psychology.  But.  This kind of thing seems to come up once in a great while.

You're asked (or "forced") to work with someone who—essentially—fails to cooperate.  They don't actively disagree or actively suggest something better.  They passively fail to agree.

In fact, they probably disagree.  They may actually have an idea of their own.

But they prefer to passively "fail to agree."

I have little patience to begin with.  And I had noted my personal inability to cope in The Passive-Aggressive Programmer or Why Nothing Gets Done.

Recently, I received this.

"I thought I was going crazy and started doubting myself when dealing with a PAP co-worker. I went out on the internet searching for help and ran into your blog and its helped me realize that I'm not crazy. The example conversations you posted are almost every day occurrences here at my job when dealing with my co-worker. From outside of the department it was oh those two just butt-heads because I never knew how to communicate or point out that really I'm a targeted victim of a PAP rather than a butting heads issue. No matter what approach I took with the PAP I was doomed and still not quite sure where to go from here. Would you happen to offer any advice on how to actually deal with PAP? It's driven me to a point where I'm looking for new employment because my employer won't deal with it." 

I really have no useful advice.  There's no way to "force" them to agree with anything specific.  In some cases, there's no easy to even determine what they might agree with.

Your employer will only "deal with" problems that cause them real pain.  If you're butting heads, but still getting things done, then there's no real pain.  You're successful, even if you're unhappy.

If you want to be both happy and successful, you need to stop doing things that make you unhappy.  If you can't agree with a co-worker, you can butt heads (which makes you unhappy) or you can ignore them (which may make you happy.)

Ignoring them completely may mean that things will stop getting done.  You may appear less successful.  If you stop being successful, then your employer will start to feel some pain.

When you employer feels pain, they will take action to relieve the pain.

You might want to try to provide clear, complete documentation of your colleague's ideas, whatever they are.  If you write down the Passive-Aggressive Programmer's "suggestions", then you might be able to demonstrate what's causing the pain.  Since a properly passive programmer never actually agrees with anything, it's tricky to pin them down to anything specific.

You might be able to make it clear that they're the roadblock that needs to be removed.

Mid-Atlantic Design Expo (MADExpo)

I'm looking forward to this.  I'll be talking about Python.  

• A Python 3.2 tutorial.  I did a dry run with the help of the 757 Python group.  Made a bunch of changes based on their input.
• A more in-depth presentation on a good architecture of Database Schema Migration scripts.  You know, the "open heart surgery" of a database where the structure changes in some way that's not a trivial ALTER statement.

Are engineers valued?

Check out this post: "Why Quit? Because They Have Bigger Monitors".

Brilliant summary of a few small cost things that have a large impact.

How many other signals are being sent?

Jim B. once pointed out that some organizations are entirely run by the accountants.  You can have a good idea.  You can justify the good idea.  You can show ROI for the good idea.  And you can still be shot down in flames with feeble excuses like "it wasn't part of this year's budget, so we won't do it."

Delusional Project Managers

Maybe it's the circle-R in the PMP® signature that put me off.

What's important is that the PMP-certified project manager is absolutely married to fixed-price software development.  The idea of "Agile" was unthinkable.  They were very clear that we had enough information to create a fixed price.

It's hard to disagree with someone who's sure that their presentation is a pinnacle of project specification.

Slides 2 and 3 are an overview that lists some broad, vague buzzwords.  ("Asset", "Resource")

Slides 4 to 7 are blurry screen shots of the legacy application.  There must be dozens of pages.  We were shown four.

Slide 8 is a list of project objectives with no priorities, actors or user stories.  "Modify tool navigation to be more intuitive and efficient."  That's the kind of stuff that's supposed to define the scope with enough detail to create a fixed price.

Slide 9 is  another list of objectives.  "Create a simple project and resource KPI dashboard showing real-time status."  I'm not even sure we can define a fixed-price task for figuring out what this might mean.

Slide 10 is a schedule in which all of the dates are already in the past.

We were told -- firmly and finally -- that we were supposed to fabricate a fixed price proposal from this PPT.  We were encouraged to ask follow-up questions.

My first follow-up question should be "Are you crazy?"

My second follow-up question should be "How much time are you willing to put into this effort?  If it's less than 4 hours per day, we can't really help much."

I'm guessing that we're just "column fillers".  There's an incumbent team that has an existing proposal to extend or rewrite; we're merely providing an outside, "independent" quote that can be used for comparison purposes.  The incumbent team probably knows (in detail) the actors and user stories, and has a carefully prioritized feature set.  We don't even know (precisely) what's in the data model.

I would like to avoid writing a "column filler" proposal that will be a million dollars more than they had in mind.

An Agile approach (5 people × 8 hrs × 15 days = 600 labor hours per sprint) was refused.

If they would reveal their budget, we could have a more meaningful conversation on what they can get finished and delivered for that little dribble of money.

I think that PMP best practices are getting in the way of a successful software development effort.

Flickr, Innovation and Integration

Read this on Gizmodo: "How Yahoo Killed Flickr and Lost the Internet"

Compelling stuff: "Integration Is The Enemy of Innovation".

"[Corporate Development milestones] often completely ignore what made the smaller target valuable in the first place."

Lessons learned: it's hard to apply structured, formal, financial controls to innovation.  As soon as the accountants show up, innovation will be stopped.  Someone has to champion the freedom to innovate in spite of being part of a profit-seeking corporation.

More of Disruptive Technology Change

There's a cool infographic on technology change in FrugalDad.  See The Great Disruption: The Future of Personal Tech.  It's interesting and informative, but the few predictions it makes are not really disruptive.  You wouldn't see anyone lobbying against the suggested future directions.  They're all good ideas that leverage existing technology.

On the other hand, there's a great graphic that shows how disruptive technology is labeled as illegal.  See Infographic: Why the movie industry is so wrong about SOPA.

Consider just one example.  Digital Movies.  The DVD was so frightening to movie producers (or distributors or theaters or the whole supply chain) that discussion of circumvention of DVD encoding had to be made illegal.  That kind of industry legislative action is evidence that a technology is truly disruptive.

Disruptive change will often lead to fearful rejection and legislative action.

"But wait," you say, "no one tried to make the iPod illegal."  Correct.   The iPod is not the core disruptive change.  Digital music is the disruptive change.  The iPod is just a vehicle.  Apple is making their money by providing a platform for digital content.

If you want to know what the Next Big Thing is, look to the US Congress.  Lobbyists are trying to make some things illegal merely because they're disruptive.

Universal Health Care, as one example, is being fought against.  There are lots of specious and farcical reasons being used to argue against simplifying the insurance mess that has emerged over the last few decades.  If Congress is fighting against it, that means the following:

1.  It's disruptive.  Game Changing.  Terrifying.
2.  The old school companies are spending huge lobbying and campaign budgets to prevent change.  They are unable to adapt to a different set of rules.
3.  Some new school companies stand to be wildly profitable if the change ever gets past the Congressional objections.

For another example, read this brilliant article: How Ma Bell Shelved the Future for 60 Years.  This an example of internal censorship of disruptive technology.  "More precisely, in Bell's imagination, the very knowledge that it was possible to record a conversation would "greatly restrict the use of the telephone," with catastrophic consequences for its business. Businessmen, for instance, the theory supposed, might fear the potential use of a recorded conversation to undo a written contract."

You know it's disruptive when it's actively feared.

Bravo -- Iteration and Generator Functions

See this: http://nedbatchelder.com/blog/201204/python_iteration_presentation.html

Nicely done, very thorough presentation on some Python fundamentals.

The nature of iteration and generator functions is easily overlooked.

GE vs. Apple

Read this

http://www.forbes.com/sites/stevedenning/2012/04/25/david-brooks-competitiveness-vs-creativity-ge-vs-apple/

It's a brilliant analysis of innovation and creativity.

MADExpo

See http://madexpo.us/

The Mid-Atlantic Developer Expo.

In my backyard (more-or-less).

There's going to be this: http://madexpo.us/Sessions/384

and this: http://madexpo.us/Sessions/385

I'm looking forward to http://madexpo.us/Sessions/339, which I'm certainly going to attend.  Many other sessions look like fun, too.

Should the CIO Know How to Code?

Read this Computerworld posting: Should the CIO know how to code?

The answer is "Yes."

The examples of "well-functioning non-technical CIOs" are people as rare as hen's teeth.  "These are leaders who know what they don't know. They are good at asking the right questions, probing for further insight, and then re-framing the answers in such a way that the business side will understand".

I'm sure there are people like this.  In the last 35 years, I've met very, very few.  Two actually.

Larry and Chuck are the two examples.

Larry knew what he didn't know.  He took the time to actually sit with actual developers and actually watch them work.  It was weird the first time he sat and watched you type.  But without deep knowledge, he couldn't be sure the projects would get done.  So he allocated an hour or more each day to sit with key developers and learn.

Chuck did essentially the kind of thing.  He sat with each developer individually every single day.  He did not have all-hands meetings that lasted hours.  He did not have an "around the table" where everyone spent 20 minutes boring the entire rest of the team with irrelevant details.

Could they code?

Essentially, yes.  They looked at code over a developer's shoulder.  They participated in a form of "pair programming" where they watched code happen.  By themselves they couldn't code much.  As pair programmers, however, they could work with another programmer and get stuff done.

Patents vs. Innovation

Read "Why Software Patents are Evil" by Simon Phipps in InfoWorld.
It's an excellent summary of the problems caused by patents applied to software.

There's a great TED Talk by Johanna Blakley on "Lessons Learned from Fashion's Free Culture" which reinforces the essential point.

Software patents don't help anyone.  The open source movement is evidence that folks working outside the constraints of patent lawyers are more innovative and produce high-quality software.  The Internet is built on non-proprietary technology (TCP/IP and related protocols), GNU/Linux, Apache and similar software componentry.  How have patents helped?

Detailed Analysis of Disruptive Technology Change

Read this: Why I doubted Facebook could build a billion dollar business, and what I learned from being horribly wrong.

Don't be afraid to read it again.

when it comes to the exceptional cases, all bets are off. So keep your mind open to weird, young [ideas] that you meet that don’t fit the established pattern

Sound advice.  The best ideas are disruptive.  That means that the idea does not fit an established pattern.

The problem with being an architect is that software architecture is a political game.

In order to justify large projects with large funding, you must cater to the folks with money who (generally) feel that disruption == risk.  The idea of incremental effort and proofs of concept may not fly because they've decided that inappropriate incumbent technology is magically quicker than appropriate but novel technology.

There's a profound Software Process Improvement issue here.  Organizations can (and do) stifle innovation in an effort to "improve" their software development process.  The false hope is that an unchanging technology base is somehow helpful at making people more effective.

Even if you give people second-rate tools, you can eventually get to be pretty good at using them.  However.  Using better tools might be better than trying to get really good at using poor tools.

What I find endlessly funny are folks who want "formal research" or "studies" that prove that some new idea is actually better than existing ideas.  You can read Stack Overflow and programmers.stackexhcange.com questions looking for studies that prove the value of unit testing or prove the value of a NoSQL database or prove that software is simpler without triggers or stored procedures.

For the moment, these are disruptive ideas.

We know they're disruptive because people keep asking for proof.

When they stop asking for proof, you know the idea has finally "arrived" and it's time to move on to find the edge of the envelope again.