On "Taste" in Software Design

Read this: http://www.paulgraham.com/taste.html.

I was originally focused on "beauty". Clearly, good design is beautiful. Isn't that obvious? Why so many words to explain the obvious?

The post seemed useless. Why write it in the first place? Why share it? Why share it now, 12 years after it was written?

Because beauty can be elusive to some people. A more complete definition of some attributes of beauty are helpful.

This is not a throw-away concept. These are fourteen essential elements that need to be used as part of every software architectural design review. Indeed, it should be part of every code review. Although code perhaps shouldn't be "daring."

When we adopt an architecture, it should fit these criteria.

This doesn't replace more pragmatic software quality assurance considerations.  See http://www.sei.cmu.edu/reports/95tr021.pdf.

I'm currently delighted with "Good design is redesign."

On Waiting to Write "Serious Code"

Someone told me they weren't yet ready to write "serious code." They needed to spend more time doing something that's not coding.

I'm unclear on what they were doing. It appears they have some barriers that I can't see.

They had sample data. They had a problem statement. They had an existing solution that was not very good. I couldn't see any reason for waiting. Indeed, I can't figure out what "serious" code is. Does that mean there's frivolous code?

Because there was a previous solution, they had a minimum viable product already defined: it has to do what the previous version did, only be better in some way. One could trivially transform the previous product into unit test cases and an acceptance test case. Few things could be more amenable to coding than having test cases.

Since everything necessary seemed to be in place, I had a complete brain cramp when they mentioned they weren't yet ready to write "serious" code. "Serious?" Seriously?

It appears that this developer suffers from a bad case of Fear of Code™. I know some common sources of this fear.

1. Waterfall Project Experience (WPE™.) Old people (like me,) who started in Waterfall World, were told that we had to produce mountains of design before we produced any code. No one knew why in any precise way. Indeed, there's ample evidence that too much design is simply a way to introduce noise into the process. In spite of real questions, some folks think that you can write a design so detailed that a coder can just type in the code from the design. (This level of design is isomorphic to code; to avoid ambiguity it must be written as code.)
2. Relational Database Hegemony (RDH™.) Folks (like me,) who were DBA's, know that databases require a lot of design and a lot of review before they can be created. Writing stored procedures requires even more design and review time. You don't just slap an SP out there. It might be "bad" or "create problems." Also, when you insist on DBA's writing application code, it takes super-detailed, code-level design details. In effect, you must write the code for the DBA to write your code back to you. 
3. One and Done (OAD™.) Some people like to feel that they can write code once and it can be a thing of beauty and a joy forever. The idea of a rewrite is anathema to these people. While this is obviously silly, people still like the conceit that they can produce some prototype code that will be a proper part of every future release forever and always. It's not possible to make all of the decisions the first time regarding adoption and scaling and user preferences. Your prototype code will get replaced eventually: get over it. Write the prototype, get funding, move forward. Don't dither trying to make a bunch of future-oriented decisions based on a future you cannot actually foresee. You can't "future-proof" your code.
4. Learnings are Expensive (LAE™.) You can find people that think that the sequence of (spike, POC, version 0, version 1) is too expensive. They are sure that learning is a project drag, since no "tangible" results are created by learning. This means that they don't value intellectual property or knowledge work, either; an attitude is actually destructive to the organization. Knowledge is everything: software captures knowledge: a spike followed by a POC followed by version zero will arrive on the scene more quickly than any alternative strategy. Don't waste time trying to write version 1 from a position of ignorance.
5. Tools are Expensive (TAE™.) Some people feel that -- since tools are expensive -- they should be used rarely. Back in the olden days, when a compiler took many minutes to produce an error report, you had to be sure the code was good. (I'm old enough that I remember when compiles took hours. Really.) Those days are gone. Most compilers today work at the "speed of light" -- if they were any faster, you couldn't tell, because you can't click any faster. For dynamic languages, like Python, the speed with which code can be emitted makes all tool considerations quaint and silly.
6. Diagram it to Death (DTD™.) Rather than write code, some folks would rather talk about writing code. To them, email, powerpoint, and whiteboard are cheaper than coding. This is a false economy. Nothing is saved by avoiding code. Time is wasted drawing diagrams of things at a level of detail that mirrors the code. Pictures aren't bad in general. Detailed pictures are simply a stalling tactic.

I find it frustrating when people search for excuses to avoid simply creating code. While I see a number of sources, there are many counter-arguments available. 

1. Waterfall is dead. Make something minimal that works for this sprint. Call it a "spike" if that makes you happier. Clean it up in the next sprint. Create value early. Expand on the features later.
2. Databases are free now. SQLite and similar products mean that we can prototype a database without waiting around for DBA's to give us permission to make progress. Build the database now, get something that works. Rework the database as your understanding of the problem matures. Rework the database as the problem itself matures and morphs. Nothing is static; the universe is expanding; do something now.
3. No code lasts forever. Waiting around to create some kind of perfect value one time only is perfect silliness. Create value early and often. Discarding code means you're making progress. If you think it's important, write "draft" on every electronic document which might get changed. (Hint: version numbers are smarter than putting "draft" everywhere.)
4. A spike and code happens more quickly than code. It's a matter of technical risk: unfinished work is an "exposure" -- an unrealized investment. Failing soon is better than researching extensively in an effort prevent a failure that could have been found quickly.
5. Use a dynamic language and avoid all overheads.
6. Keep the diagrams high-level. Code is the only way to meaningfully capture details. Code endures better than some out-of-date Visio file that's in Sharepoint completely disconnected from GitHub.

It's imperative to break down the roadblocks. All "pre-coding" activities are little more than emotional props: knock them down and start coding.

More Thoughts on the friction of DevOps

Read this: How 'DevOps' is Killing the Developer

My pull-out quote:

This is why we see so many developers that can't pass FizzBuzz: they never really had to write any code.

I agree: It appears that DevOps may be more symptom than solution.

I have one tiny objection to any otherwise excellent series of points: I don't like the totem pole analogy.

I prefer a supply-chain:

• Release Engineers respond to user needs.
• Quality Engineers respond to the Release Engineers' needs for assurance that something is fit for use.
• Developers respond to Release Engineers by providing software.
• Similarly, procurement folks may purchase or lease or download and pay royalties for software. 

I think of it like this:

Developer ⇒ QE ⇒ RE ⇒ Users

No top-to-bottom. More a sequence of more-or-less peers.

I still agree with the central tenet: a developer is able to march the software from concept to user. We don't really expect QE or RE to create software. We might expect some skill sharing between QE and RE.

Many years ago, I posted this: IT’s Drive to Self-Destruction, which is random and whiny but related to this point about DevOps. The idea is that key developers create competitive advantage. Release Engineers put it in the hands of users. Both are important. Without creation there's no deployment. Without deployment, creators can be diverted to deployment, so deployment can still go forward, but it will be slower.

The key point is this:

If a developer is spending time with DevOps (and TechOps) trying to get stuff deployed, who's developing the Next Big Thing? 

Proverbs and Bumper-Sticker Computer Science

There's much to learn. The idea of digesting knowledge down to a pithy little proverb is interesting.

https://github.com/AntJanus/programmers-proverbs

http://futurice.com/blog/programmer-proverbs

http://quotes.cat-v.org/programming/bumper-sticker-computer-science

Some of it's new to me. Some of it's timeless.

Also, this http://www.bowdoin.edu/~ltoma/teaching/cs340/spring05/coursestuff/Bentley_BumperSticker.pdf

The Permissions Issue

Why?

Why are Enterprise Computers so hard to use? What is it about computers that terrifies corporate IT?

They're paying lots of money to have me sit around and wait for mysterious approver folks to decide if I can be given permission to install development tools. (Of course, the real work is done by off-shore subcontractors who are (a) overworked and (b) simply reviewing a decision matrix.)

And they ask, "Are you getting everything you need?"

The answer is universally "No, I'm not getting what I need." Universally. But I can't say that.

You want me to develop software. And you simultaneously erect massive, institutional roadblocks to prevent me from developing software.

I have yet to work somewhere without roadblocks that effectively prevent development.

And I know that some vague "security considerations" trump any productive approach to doing software development. I know that there's really no point in trying to explain that I'm not making progress because I can't actually do anything. And you're stopping me from doing anything.

My first two weeks at every client:

The client tried to "expedite" my arrival by requesting the PC early, so it would be available on day 1. It wasn't. A temporary PC is -- of course -- useless. But that's the balance of days 1-5: piddling around with the temporary PC. That was ordered two weeks earlier.

Day 6 begins with the real PC. It's actually too small for serious development due to an oversight in bringing me on as a developer, but not ordering a developer's PC for me. I'll deal. Things will be slow. That's okay. Some day, you'll discover that I'm wasting time waiting for each build and unit test suite. Right now, I'm doing nothing, so I have no basis to complain.

Day 7 reveals that I need to fill in a form to have the PC you assigned me "unlocked." Without this, I cannot install any development tools.

In order to fill in the form, I need to run an in-house app. Which is known by several names, none of which appear on the intranet site. Day 8 is lost to searching, making some confused phone calls, and waiting for someone to get back to me with something.

Oh. And the email you sent on Day 9 had a broken link. That's not the in-house app anymore. It may have been in the past. But it's not.

Day 10 is looking good. The development request has been rejected because I -- as an outsider -- can't make the request to unlock a PC directly. It has to be made by someone who's away visiting customers or off-shore developers or something.

Remember. This is the two weeks I'm on site. The whole order started 10 business days earlier with the request for the wrong PC without appropriate developer permissions.

Third Time's the Charm: the version 3.0 phenomenon

Somewhere, I have a vague recollection of reading advice from someone (Bill Gates?) that it takes three versions to get things right. The context may have been a justification of the wild success of Windows 3.0.

Or, I could be just making it up.

But one thing I have noticed is that there's a definite bias toward looking at software three times.

I worked (briefly) with an agile project management group that suggested that everything will be released three times, called the "Good", "Better", "Best" releases.

• The good release passed the unit tests.
• The better release included any non-functional (performance, auditability, maintainability, etc.) improvements required.
• The best implementation possible.

Not everything required three releases. Simpler components can merge better and best. Some components simply start out in really, really good shape.

Teaching Moment

What I've also noticed is that the explanation of the component -- writing documentation, presenting to peers in a walkthrough -- leads to profound rethinking. 

May things may appear to be better or best in the sense above. Until we have to explain them. Then they're no longer "best" but merely "better" or perhaps even "good." 

A few minutes spent hand-waving through a design often points to things that aren't quite to easy to explain. A walkthrough is very beneficial to the person doing the presentation.

But, not too early.

When I made military software, we had Preliminary Design Reviews that were done before coding begins. The idea was to surround the difficult coding work with yet more process steps and yet more deliverable intermediate results. 

The intent was noble: if a walkthrough reveals so much, then do the walkthroughs early and often.

However. I'm beginning to think that early isn't ideal.

I think that the design walkthrough should be delayed until after minimally working code exists. Once there's code -- with automated unit tests -- then refactoring to meet non-functional quality factors (like performance) is easier and more likely to be successful.

Also, refactoring to make the software clear, simple, and elegant should probably wait until it works and has a complete suite of automated unit tests. 

Required Reading for everyone who said "If it ain't broke, don't fix it."

http://pythonsweetness.tumblr.com/post/64740079543/how-to-lose-172-222-a-second-for-45-minutes

Here an important lesson.

Code Rot is Real.  It Leads to Subtle and Expensive Bugs.

Claiming that code cleanup is just pointless "gold plating" is the kind of thing that can drive a company out of business.

Obstinate Idiocy, Expanded

See Obstinate Idiocy for some background.

Here are three warning signs I was able to deduce.

• No Rational Justification
• Ineffective Tool Choice
• Random Whining

To which I can now add two more.

Symptom 4 of Obstinate Idiocy is that all questions are rhetorical and they often come with pre-argued answers.

Actual email quote:

Me: ">Excel is almost the stupidest choice possible

OI: "What criteria are you using to make that statement?

My criteria was that I needed a way for non-tech people and non-programmers..."

And on the email spins, pre-arguing points and pre-justifying a bad answer. Since their argument is already presented (in mind-numbing detail), there's no effective way to answer the question they asked. Indeed, there's little point in trying to answer, since the pre-argued response is likely to be the final response.

In order to answer, we have to get past the pre-argued response. And this can be difficult because this devolves to "it's political, you don't need the details." So, if it's not technical, why am I involved?

Symptom 5 of Obstinate Idiocy is Learning is Impossible. This may actually be the root cause for Symptom 3, Ineffective Tool Choice. It now seems to me that the tool was chosen to minimize learning. I had suggested using Mathematica. I got this response: " I don't know Python or R or SAS." The answer seems like a non-sequitur because it is. It's justification for a bad decision.

The problem they're trying to solve is gnarly, perhaps it's time to consider learning a better toolset.

Excel has already failed the OI. They asked for an opinion ("Q2: What do you believe are the pros/cons of ... using Excel with "Excel Solver" ...?") that seems to ignore the fact that they already failed trying to use Excel. They already failed, and they followed up by asking for the pros and cons of a tool they already failed with. 

From this limited exchange it appears that they're so unwilling to learn that they can't gather data from their own experience and learn from it.

How to Manage Risk

Also see "On Risk and Estimating and Agile Methods". This post is yet another angle on a common theme.

Orders of Ignorance and Risk Management.

Software risk management has two sides.  First, there's the classical meaning of risk, we'll call that "casino risk" because it's really random events.  This includes fire, flood, famine, conquest, war, pestilence, death, etc.  Actual risks.

The second meaning of risk is a load of malarkey.  It's a code word that includes two things: "bad management" and "ignorance".  Some things called project risks are just plain old bad management—generally driven by a nonexistent process for handling ignorance.  The events aren't random.  

There are five orders of ignorance, and each of them leads to project management problems.  None of these are "random events"; none of this is like casino gambling.  There aren't any odds; most of these things are certainties.

0 Order Ignorance: Things We Know.

There are two sides to the things we think we know about a project.  There are the things we know which are true, and things which are false.  Falsehoods come from at least two places:  we assumed something or we were actually lied to.  (Other choices, like illusions and hallucinations, are too creepy to pursue.)

Our assumptions aren't facts.  This sounds so obviously stupid, but projects get into trouble based on assumptions that were never checked to see if they were true or not.  Managers insist on doing "risk analysis" and then pad their project estimates with time and money instead of simply challenging their assumptions.

Some "assumptions" are explicit placeholders for facts to be found out later.  These formally documented assumptions are a different thing, they're 1st order ignorance, something we know we don't know.

Example.  The customer says they need an app to do [X].  There's 8 people in the department who are likely actors.  We assume the user population is 8.  With no fact checking.  We don't put it in the plan as a formal, documented assumption, we just assume it.

Bottom line.  What are the odds that a plan is based on false knowledge?  This isn't casino gambling.  There aren't any odds that we assumed something, or odds that we were lied to.  This is simple fact-checking, simple bad management. Every unchecked fact is going to be false.

1st Order Ignorance: Things We Don't Know.

There are two ways to deal with things we don't know.  Make a guess and document this, or actually ask a question.

The formally documented guesses (usually called "assumptions") are the hallmark of software project plans.  Documents are often full of lists of assumptions on which the plan and associated estimates rest.

Each one of these "assumptions" is a question that—for some reason—couldn't be asked or couldn't be answered.  Some questions are "politically sensitive" and can't be asked.  Some questions require lots of research to develop an answer.

The answers, of course, change the project.  In most cases they change the project dramatically.  If they didn't have a big impact, we wouldn't spend any time documenting them so carefully, would we?

And all project managers are punished for making any changes.  We can't expand the scope without a lot of accusatory conversations where people keep repeating the original price back to us.  

The canonical line is something like "I thought this was only 1.8 million dollars, how can you change it now?" or "I've already committed to $750K, we can't change the price, something else has to be changed.  We have to work smarter not harder."  Bleah.

This doesn't involve casino-like odds of finding answers that will change the scope of the project.  We know we have questions.  We know we made guesses and documented them as "assumptions".  There were no odds; it was absolutely certain there would be changes.

Bottom line.  What are the odds that a plan is based on things we guessed at?  Typically, this is a fact of life: parts of the estimate are guesses.  

What are the odds that the real answer will be different from the guess?  This, too, is absolutely certain. It's merely a question of magnitude.

For first-order ignorance problems, we should create a contingency budget for each answer that will diverge from the guesses.  This isn't a book-making exercise, it's a list of alternate guesses (or assumptions).

It isn't enough to simply detail the assumptions.  We have to provide alternative answers and the associated costs when the assumptions turn out to be false.

2nd Order Ignorance: Things We Didn't Know to Ask.

If we didn't know to ask, what's that?  Is that a risk?  What are the odds we forgot to ask something?

Here's the canonical quote: "Are there any other unforeseen problems?"

What? If they're unforeseen, then, uhhh, we can't identify them now.

If we can identify them, then, uhhh, they're not unforeseen.

There aren't any "odds" of an unforeseen problem.  It's an absolute certainty that there will be unforeseen problems.

Remember, these are things we didn't know to ask.  Things that didn't make the list of "assumptions".  These are things that completely escaped our attention.

What will the impact be?  We have no way of knowing. No. Way. Of. Knowing.

We can't even put a contingency in place for these things.  We didn't know to ask.  So we don't know what it will cost for rework when we figure out what we should have asked.

All we can do here is use a good, transparent management process.  Each new piece of information -- each thing that's learned that we didn't know to ask -- will change scope, schedule, cost, deliverables, everything.

This isn't "casino risk".  There are no odds associated with this.  This is just change management. It's inevitable. Calling it a project risk is lying about it.

3rd Order Ignorance: No Process for Managing Ignorance.

When we have second-order ignorance (we didn't know to ask) there are two responses: an organized change-management process, or a leap down to 3rd order ignorance.  Third order ignorance slips from simply not knowing into denying that the level of knowledge changes through time.

When we learn something unexpected, we can either deny that it is something new, or we can expose it.  When a business analyst learns that the "simple" calculation involves a magical MS-Access database with no known author, magical numbers and no discernible calculations, this is going to change the scope of the work.  Or make it impossible to make progress until someone explains the MS-Access database.

Denying this kind of unexpected information is common, it's done by playing the management trump card of "schedule is sacred.  Once the schedule is sacred, all learning is either trivially denied or learning turns into ways of shaving scope or quality to make the schedule.  

3rd Order Ignorance means there's no change process and the "schedule is sacred".  If the only thing that matters is schedule, then buggy, useless software will be delivered on time and on budget.

What are the odds of 3rd order ignorance?  Either 1.0 or 0.0.  Either the organization has an effective change management process (in which case, we don't have 3rd order ignorance) or there will be problems in delivering software that works on time. 

Bottom Line.

Here's the summary of ignorance and mitigation.

0th order ignorance: do basic fact checking to validate your assumptions.

1st order ignorance: do contingency planning. Define specific contingencies around each specific unknown fact.  Don't just document an "assumption", plan for alternatives when the assumption is invalidated.

2nd order ignorance: have a change management process.

3rd order ignorance (i.e., no change management): stop using waterfall-style methodologies. Switch to Agile methods so that change and the management of ignorance become essential features of the overall process.

4th order ignorance is the state of not being aware that ignorance is one of the most significant driving forces behind project failure.  A symptom of 4th-order ignorance is conflating "risk analysis" for a project with "casino risks" (or "insurance risks.")  With rare exceptions, all project risk analysis is just ways of coping with bad management.

When there's 4th order ignorance, folks are told that it's helpful or meaningful to try and assign odds to the veracity of the facts, the presence of things which were forgotten, and the change management process itself.

Avoiding 4th order ignorance means recognizing that software project management "risks" are just bad management (with minor exceptions for fire, flood, famine, conquest, war, pestilence, and death.)

Here's how to manage risk:

• Check the facts, 
• plan specific contingencies, 
• use Agile methods because of their built-in ability to manage change.

Database Conversion or Schema Migration

I was told a horror story about a bad database schema migration. Let's call the author of the horror story HPL.

HPL related a tale of woeful conversion problems from a mismanaged schema migration.

While I could feel HPL's pain, the reasons given for their pain were wrong. They didn't quite get the lessons they had learned. Consequently, HPL sounded like someone doomed to repeat the mistake, or—worse—unlearning their lessons.

Here's HPL's most distressing comment.

"we can't migrate over the weekend and be done w/ it."

Apparently, the horror of a weekend migration was somehow desirable to HPL. Who wants a lost weekend? And who wants to put all of the eggs in a single basket?

Anyone who's done more than one "lost weekend migration"—and who's also honest—knows that they don't go well. There are always subsets of data that (a) don't get converted properly and (b) have to get swept under the carpet in order to claim to meet the schedule.

It's a standard situation to have less than 100% of the data successfully converted and still call the effort a success. If 100% was not required, why lose a weekend over it?

Good Plans and Bad Plans

From far wiser people than me, I learned an important lesson in schema migration.

These Wiser Heads ran a "conversion" business. They moved data and applications from platform to platform. They knew a lot about database schema migrations. A lot.

Their standard plan was to build a schema migration script (usually a sequence of apps) that could be run to convert the database (or files or whatever) from old to new schema as often as was necessary.

I'll repeat that.

As often as was necessary.

They debugged the script to get to an acceptable level of conversion. The data conversion (or schema migration) was perfectly repeatable. Of course, they longed for 100% conversion; but pragmatically, the legacy software had bad data. So some fraction would not convert. And once that fraction was found, the schema migration applications could be modified to treat the non-convertable data in some intelligent way.

Their stated goal was to convert data and run parallel testing with that converted data as often as necessary to create confidence that the new data was as perfect a conversion as was possible. At some point, the confidence became certainty and the parallel testing was deemed complete. Since they were parallel testing with live data, the decision amounted to a formalized "commissioning" of the new application. And by then, the new application was already being used.

There are bad ways to do schema migration, of course. HPL listed many.

Horrible Mistakes

The horror story from HPL included this:

"For the migrated tables, create views in the old system and create instead of triggers on those views to ship data to the new system."

It appears that they used views and triggers to create a new system "facade" over the legacy system. Apparently, they wanted both suites of application software to coexist. Not a good approach to schema migration. It appeared that they were trying to share one database with two application schema.

This seems like it's doomed. Unless they're all geniuses.

Wiser Heads would have strongly suggested that the new system use a extract of the old system's data.

HPL goes on to complain,

"Sometimes we can take over a column or 2 and sometimes we can only take over some of the data in the table".  

HPL emphasizes this point with "This is not that far fetched". I'm not sure why the emphasis was needed.

This is not "far fetched". It doesn't need emphasis. It's not really much of a problem, either. It's a standard part of schema migration. Extracting a copy of the data makes this quite easy. Triggers and views to create some kind of active SQL-based Facade is what created the complexity. Not the number of columns involved.

HPL summarizes,

"So you end up w/ [many] tables/views triggers all moving data back and forth and faking stuff out"

Back and forth. A fundamental mistake. A copy can be much easier to manage. One way data movement: Legacy to New.

HPL concludes with a litany of errors of various types: performance, change management, file system issues, error logging and auditing. Blah blah blah. Yes, it was a nightmare. I feel their pain.

What About Coexistence? 

It appears that HPL was involved in a project where the entire old and new applications were supposed to somehow coexist during the conversion.

It appeared that they failed to do any kind of partitioning.

Coexistence is not a trivial exercise. Nor is it a monolith where the entire legacy application suite must coexist with the entire new schema and the entire new application suite.

Pragmatically, coexistence usually means that some portion of the legacy must be kept running while some other portion is modernized. This means the coexistence requires that the application portfolio be partitioned.

Step 1: Some suite of functionality is migrated. That means data from the legacy database/file system is copied to new. That also means some data from new is copied back into the legacy database file/system. Copied.

Step 2: Some other suite of functionality is migrated. As functionality is moved, less and less data is copied back to the legacy.

At some point, this copying back is of no value and can be discontinued.

What About Timing?

This copying clearly requires some coordination. It's not done haphazardly.

Does it require "real time" data movement? i.e. triggers and views?

Rarely is real time movement required. This is the point behind partitioning wisely. Partitioning includes timing considerations as well as data quality and functionality considerations.

It's remotely possible that timing and partitioning are so pathological that data is required in both legacy and new applications concurrently. This is no reason to throw the baby out with the bathwater. This is nothing more than an indication that the data is being copied back to the legacy application close to real time.

This also means performance must be part of the test plan. As well as error handling and diagnostic logging. None of this is particularly difficult. It simply requires care.

Lessons Learned

HPL appeared to make the claim that schema migration is super hard. Or maybe that coexistence is really hard.

Worse, HPL's horror story may be designed to indicate that a horrifying lost weekend is the only way to do schema migration.

Any or all of these are the wrong lessons to learn.

I think there are several more valuable lessons here.

1. Schema migration can and should be done incrementally. It's ideally tackled as an Agile project using Scrum techniques. It's okay to have release cycles that are just days apart as each phase of the conversion is run in parallel and tested to the user's satisfaction.
2. Coexistence requires partitioning to copy any data back to unconverted legacy components. Triggers and views and coexistence of entire suites of software make a difficult problem harder.
3. The conversion script is just another first-class application. The same quality features apply to the conversion as to every other component of the app suite.
4. The conversion must be trivially repeatable. It must be the kind of thing that can be run as often as necessary to move legacy data to the new schema. 

How to Estimate a Project

A recent question

"what we might expect, in terms of 1) Time to completion, 2) Cost to implement according to the mockup, 3) Monthly server/maintenance costs and 4) approximate team size required"

This question was followed by this acknowledgement:

"it is hard to make the above estimations, given the lack of clarity on the architecture which will be employed, and given the nature of software development itself."

This is an understatement. The statement is generally well written, but the word "hard" is weak. It's not hard. It's essentially impossible.

I.  The Conundrum 

Let's say you decide that the budget for "everything" is two point five kajillion dollars. Clearly, you don't want to just fork that money over to a roomful of developers and wait a year for something to happen.

An Agile approach is a sensible alternative. Instead of building everything, you build a first release that does something. Ideally, something that creates the most value for potential users.

What's that first release? Presently, you don't have a formal, testable specification. More work needs to be done to define a first release. Formal and testable are pretty high barriers.

More important than details is this: there's a nasty circularity issue. Until you build something, you don't know where the technical roadblocks are. Once you build something... well... you've built something. And you're going to be building something potentially releasable merely to get to the point of being able to write a budget.

There's no way to know the budget without having started to build something.

Once you start to build something, you did useful work in advance of having a budget.

One (false) claim floating around the software development world is that we can somehow do more research to resolve the unknowns before we start actually building software. This is simply false. As we do "research" we're doing high-level design: we're building software. You may resolve a few unknowns, but there will be more.

The only way to know all the details about platform and the application is to build the application using the platform. We don't know anything until we're done building something that resolves some unknowns.

Interestingly, the definition of "done" cannot possibly exist. We'll return to the farcical nature of "done", below.

Worse, there's no way to know the budget without knowing the people who will be doing the building.

II. The Productivity Issue

Until there's a relatively stable team, it's essentially impossible to know how quickly programmers can build anything. And even then, there can be unexpected, unforeseeable problems with the team.

Let me tediously pound this point home with an even more detailed analysis. The point is to make it very clear that the future is impossible to predict.

Let's say the first release is essentially a clone of an existing open source project.

This is simple. What's the budget to clone the existing open source project?

Choice 1. Find an unpaid intern. (This may no longer be legal, but it's still popular.) Have them clone the repository, rebrand it, and you have something running. How quickly can they do this? You don't know until we meet the intern and watch them work. After they've cloned the existing open source project you then know how long it will take them. Until you've seen them work, you know nothing about the time they'll require.

Choice 2. Find a kid still in school who knows the technology. Pay them sub-minimimum wage to clone the package and rebrand it. How much will you pay? You don't know until you meet the intern and watch them work. After they've cloned the existing open source project, you'll know how much it costs.

Choice 3. Make up a schedule based on what little is known. Put the "clone the existing open source project"  out to bid on http://freelancer.com and hope that others make bids that fit with your expectations. This is fixed price so the budget is—in principle—known. In order to be sure you get something high quality and usable, you'll need to write a lot of test cases and very detailed specifications. Sadly, that pre-work is of imponderable complexity. When you get bids that are too big, you learn that your specifications weren't good enough; and you need to fix your specifications to narrow the scope of work. Now you're doing much of the work (spec writing and test case writing) in order to get a proposal that includes your budget. Note the circularity where you're doing some of the work to figure out the budget for the work you're doing.

Choice 4. Offer someone a share in the company to clone the existing open source project.  Now you don't have a budget at all. You merely have a schedule. When will they be done? You're back to option 1, the unpaid intern, except now with better incentives to be quick. But you don't know how long they'll take until you've seen them do it once.

Choice 5. Offer someone an hourly rate plus a share in the company to clone the existing open source project.  Now you're back to having a budget, and perhaps it has an upper bound. You can pay up to some amount, after that the share in the company is their incentive to get something done.

I beat this point to death because there actually is no answer.  No matter what strategy you choose, you still can't predict developer productivity. It varies by a factor of at least 10 to 1. Some studies show it varying by 100 to 1.

The idea of forecasting development costs is shameful lie created by accountants. Really. The GAAP requires controls and budgets before spending money and we're supposed to compare plan and actual. This is all farcical in software world. Software development is like R&D: it's structured learning and encoding the learning into software.

III. The Done Issue

One of the Great Lies is that software has a defined "done" state. This only true for reductionist classroom exercises. Real software grows, often without bounds.

"Wait," you say, "I have a vision of what I want, that defines a boundary."

Today, that defines a boundary.

In six weeks, after two releases and some trouble support calls and requests for new features, your original vision is out the window, and you're off chasing the things your real users really are asking for.

Only in-house IT managers for large (dumb) companies stick to the original plan in spite of all the lessons learned along the way.

Then you get partnership offers. And you see new platforms and tools, and you get more user requests. The browser landscape changes. Tablets become faster. Other changes that are impossible to imagine will happen.

The vision will not be stable.

It won't even be finite.  A good business model grows and adapts and expands.

IV. Strategy 1: Estimate

What can you do?

Clearly, you want some kind of budget for creating some kind of software.

Clearly, there's no way to provide a good answer.

You can, however, find a farcical answer.

Step 1: find a developer who's willing to make a sincere commitment to a cost and schedule.

Step 2: trust the sincerity of their commitment, even though it's is absolutely going to be wrong. The Great Lie is that we might only wrong by a factor of 2. In reality we can often be wrong by a factor of 10: the $100,000 job turned out to cost over a million. (See above, 10:1 productivity is just one of the unknowns.) The million dollar job was ill-advised and cancelled after the second release, but the users were happy, so it was successful in many ways. But it was cancelled.

A sincere estimate is just a random number. However, many managers find that the sincerity gives them comfort.

Since productivity is unknowable and "done" is unknowable, a detailed estimate and plan means you must now spend a lot of time writing "change orders" and reallocating the budget every time you learn something new.

I'll repeat that.

When you have an estimate, all you do with it is reallocate the estimated budget as you learn more about the customers, the development team and the product. All you do is reallocate; the idea that there's "plan" which is compared with "actual" is farcical because the plan changes constantly. So constantly as to be meaningless.

[Accountants will claim that this is analysis wrong because the future is somehow knowable. I can only stare dumbfounded at them. The future is knowable? Really? They'll say that a plan is a commitment and comparing actual to plan somehow makes sense. They'll give all kinds of weird analogies that don't apply to software. Software development is not a "production" task like brick laying or making pins from wire. If the future was knowable, the project ROI would be a fixed 150% or 300% or, well anything. Oh. Right. Somethings are unknowable. Like the future. Ahem.]

V. Strategy 2: Agile

The very best you can do—indeed, the only rational thing you can do—is to locate talent who are willing to work for an indefinite period of time.

A person or people you trust.

You establish a release cycle. Two or three weeks are typical sprint cycle times. two weeks works well for very new development. Three weeks is better for more established teams.

You identify the first three or so releases by writing those high-priority, high-value user stories as carefully as you can.  Testable, finite user stories. Clear boundaries on acceptable vs. unacceptable behavior. Too few user stories makes it difficult to foresee the future. Too many user stories can be needless preliminary work since they're going to change anyway.

You do Scrum development with a two-week cycle.  http://www.ambysoft.com/essays/agileLifecycle.html

"Useless," you say, "because there's no overall budget!"

Correct. There's no overall budget. You don't (and shouldn't) have a legally-binding definition of "done". Done means "business death." You have a vision for the first release. From that you'll make enough money to get to the second release. Which gets you to the third release. You're not done until you're out of ideas and no one wants your product anymore.

Done should always be defined as "planned release [X] is the last release." After that, it's donate the intellectual property into the public domain and move on to something profitable.

"Then logically," you say, "There can be a budget for the first release."

Except, as noted above, you don't know how productive the team is. So there's no useful budget for even the first release. Ideally, 1 sprint = 1 release. But. Until you know the team, and the user stories, and the platform, and the application, you can't assume that.

Which gets us to this:

The budget is only enough to get you through the next two-week sprint. A three-person team for two weeks is 240 hours. $50/hr. $12,000 per sprint. Perhaps with a larger team, it may be $20,000.

Each sprint must produce something releasable or everyone is fired. It's that crisp. The company is out of business—as currently organized—when the team can't create something releasable. Either the user stories aren't testable or the sprint planning is too ambitious. Or someone lacks the skills they were thought to have during the interview process. Or something is wrong with the team chemistry.

Sometimes, a sprint's work product is not deployed for marketing purposes. It's saved up into the next sprint so that the monthly release is far cooler than the bi-weekly release.

I'm aware that this is an unsatisfying answer. It's nice to hope that software development is a finite, linear process with just minor bumps in the road. Sadly, it's not. It's a completely out-of-control process that hurtles down the wave fronts making progress in a reasonably desirable direction in spite of currents, winds and weather. It's (by definition) a learning process. As knowledge is accumulated, it's encoded in the form of software. Once all the knowledge is available, the software happens to be done, also.

Also: http://slott-softwarearchitect.blogspot.com/2011/11/justification-of-project-staffing.html.
And this: http://slott-softwarearchitect.blogspot.com/2010/03/great-lies-design-vs-construction.html.
This, too: http://slott-softwarearchitect.blogspot.com/2009/11/on-risk-and-estimating-and-agile.html.
Okay, fine: http://slott-softwarearchitect.blogspot.com/2009/10/breaking-into-agile.html.

Automated Code Modernization: Don't Pave the Cowpaths

After talking about some experience with legacy modernization (or migration), I received information from Blue Phoenix about their approach to modernization.

Before talking about modernization, it's important to think about the following issue from two points of view.

Modernization can amount to nothing more than Paving the Cowpaths.

From a user viewpoint, "paving the cowpaths" means that the legacy usability issues have now been modernized without being fixed. The issues remain. A dumb business process is now implemented in a modern programming language. It's still a dumb business process. The modernization was strictly technical with no user-focused "value-add".

From a technical viewpoint, "paving the cowpaths" means that bad legacy design, bad legacy implementation and legacy platform quirks have now been modernized. A poorly-designed application in a legacy language has been modernized into a poorly-designed application in yet another language. Because of language differences, it may go from poorly-designed to really-poorly-designed.

The real underlying issue is how to avoid low-value modernization. How to avoid merely converting bad design and bad UX from one language to another.

Consider that it's possible to actually reduce the value of a legacy application through poorly-planned modernization. Converting quirks and bad design from one language to another will not magically make a legacy application "better". Converting quirky code to Java will merely canonize the quirks, obscuring the essential business value that was also encoded in the quirky legacy code.

Focus on Value

The fundamental modernization question is "Where's the Value?" Or, more specifically, "What part of this legacy is worth preserving?"

In some cases, it's not even completely clear what the legacy software really is. Old COBOL mainframe systems may contain hundreds (or thousands) of application programs, each of which does some very small thing.

While "Focus on Value" is essential, it's not clear how one achieves this. Here's a process I've used.

Step 1. Create a code and data inventory. 

This is essential for determine what parts of the legacy system have value. Blue Phoenix has "Legacy Indexing" for determine the current state of the application portfolio. Bravo. This is important.

I've done this analysis with Python. It's not difficult. Many organizations can provide a ZIP file with all of the legacy source and and all of the legacy JCL (Z/OS shell scripts). A few days of scanning can produce inventory summaries showing programs, files, inputs and outputs.

A suite of tools would probably be simpler than writing a JCL parser in Python

A large commercial operation will have all kinds of source checked into the repository. Some will be inexplicable. Some will have never been used. In some cases, there will be executable code that was not actually built from the source in the master source repository.

A recreational project (like HamCalc) reveals the same patterns of confusion as large multi-million dollar efforts. There are mystery programs which are probably never used; the code is available, but they don't appear in shell scripts or interactive menus. There are programs which have clear bugs and (apparently) never worked. There are programs with quirks; programs that work because of an undocumented "feature" of the language or platform.

Step 2. Capture the Data.

In most cases, the data is central: the legacy files or databases need to be preserved. The application code is often secondary. In most cases, the application code is almost worthless, and only the data matters. The application programs serve only as a definition of how to interpret and decode the data.

Blue Phoenix has Transition Bridge Services. Bravo. You'll be moving data from legacy to new (and the reverse, also.) We'll return to this "Build Bridges" below.

Regarding the data vs. application programming distinction, I need to repeat my observation: Legacy Code Is Largely Worthless. Some folks are married to legacy application code. The legacy code does stuff to the legacy files. It must be important, right?

"That's simple logic, you idiot," they say to me. "It's only logical that we need to preserve all the code to process all the data."

That's actually false. It's not simple logic. It's just wishful thinking.

When you actually read legacy code, you find that a significant fraction (something like 30%) is trivial recapitulation of SQL's "set" operations: SQL DML statements have an implied loop that operates on a set of data. Large amounts of legacy code merely recapitulates the implied loop. This is trivially true of legacy SQL applications with embedded SQL; explicit FETCH loops are very wordy. There's no sense in preserving this overhead if it can be avoided.

Programs which work with flat files always have long stretches of code that models SQL loops or Map-Reduce loops. There's no value in the loop management parts of these programs.

Another significant fraction is "utility" code that is not application-specific in any way. It's an application program that merely does a "CREATE TABLE XYZ(...) AS SELECT ....": a single line of SQL. There's no sense in preserving this through an "automated" tool, since it doesn't really do anything of value.

Also. The legacy code has usability issues. It doesn't precisely fit the business use cases. (Indeed, it probably hasn't fit the business use cases for decades.) Some parts of the legacy code base are more liability than asset and should be discarded in order to simplify, streamline or improve operations.

What's left?

The high value processing.

Step 3. Extract the Business Rules.

Once we've disposed of overheads, utility code, quirks, bad design, and wrong use cases, what's left are a the real brass tacks. A few lines of code here and there will decode a one-character flag or indicator and determine the processing. This code is of value.

Note that this code will be disappointingly small compared to the total inventory. It will often be widely scattered. Bad copy-and-paste programming will lead to exact copies as well as near-miss copies. It may be opaque.

IF FLAG-2 IS "B" THEN MOVE "R" TO FLAG-BC.

Seriously. What does this mean? This may turn out to be the secret behind paying bonus commissions to highly-valued sales associates. If this isn't preserved, the good folks will all quit en masse.

This is the "Business Rules" layer of a modern application design. These are the nuggets of high-value coding that we need to preserve.

These are things that must be redesigned when moving from the old database (or flat files) to the new database. These one character flag fields should not simply be preserved as a single character. They need to be understood.

The business rules should never be subject to automated translation. These bits of business-specific processing must always be reviewed by the users (or business owners) to be absolutely sure that it's (a) relevant and (b) has a complete suite of unit test cases.

The unique processing rules need to have modern, formal documentation. Minimally, the documentation must be in the form of unit test cases; English as a backup can be helpful.

Step 4. Build Bridges.

A modernization project is not a once-and-done operation.

I've been told that the IT department goal is to pick a long weekend, preferably a federal Monday holiday weekend (Labor Day is always popular), and do a massive one-time-only conversion on that weekend.

This is a terrible plan. It is doomed to failure.

A better plan is a phased coexistence. If a vendor (like Blue Phoenix) offers bridge services, then it's smarter and less risky to convert back and forth between legacy and new over and over again.

The policy is to convert early and convert often.

A good plan is the following.

1. Modernize some set of features in the legacy quagmire of code. This should be a simple rewrite from scratch using the legacy code as a specification and the legacy files (or database) as an interface.
2. Run in parallel to be sure the modern version works. Do frequent data conversions from old to new as part of this parallel test.
3. At some point, simply stop converting from old to new and start using the new because it passes all the tests. Often, the new will have additional features or remove old bugs, so the users will be clamoring for it.

For particularly large and gnarly systems, all features cannot be modernized at once. There will be features that have not yet been modernized. This means that some portion of new data will be converted back to the legacy for processing.

The feature sets are prioritized by value. What's most important to the users? As each feature set is modernized, the remaining bits become less and less valuable. As some point, you get to the situation where you have a portfolio of unconverted code but no missing features. Since there are no more desirable legacy features to convert, the remaining code is -- by definition -- worthless.

The unconverted code is a net cost savings.

Automated Translation

Note that there is very little emphasis on automated translation of legacy code. The important work is uncovering the data and the processing rules that make the data usable. The important tools are inventory tools and data bridging tools.

Language survey tools will be helpful. Tools to look for file operations. Tools to look for places where a particular field of a record is used.

Automated translation will tend to pave all the cowpaths: good, bad and indifferent. Once the good features are located, a manual rewrite is just as efficient as automated translation.

Automated translation cannot capture meaning, identify use cases or write unit test cases. Thoughtful manual analysis of meaning, usability and unit tests is how the value of legacy code and data is preserved.

The Small Class Large Class "Question"

People criticize for making tons of small classes instead of fewer larger classes.How is well-organized code more difficult to understand?
— jim christopher (@beefarino) May 31, 2013

This isn't really a question. Writing a few "large" omnibus classes is simply bad design.

There are several variations on the theme of principles of OO programming. None of them include "a few large omnibus classes with nebulous responsibilities."

Here's one set of principles: Class Responsibility Collaboration. Here's one summary of responsibility definition: "Ask yourselves what each class knows and what each class does".  Here's another: "A responsibility is anything that a class knows or does." from Class Responsibility Collaborator (CRC) Models.

This idea of responsibility defined as "Knows or Does" certainly seems to value focus over sprawling vagueness.

Here's another set of principles from Object-Oriented Design; these echo the SOLID Principles without the clever acronym.

Getting down to S: a single reason to change means that the class must be narrowly-focused. When there are a few large classes, then each large class has to be touched for more than one reason. By more than one developer.

Also, getting to O: open to extension, closed to modification requires extremely narrow focus. When this is done well, new features are added via adding subclasses and (possibly) changing an initialization to switch which Factory subclass is used.

But Why?

Why do people reject "lots of small classes"?

Reason 1. It's hard to trivially inspect a complex solution. I've had an argument similar to the one Beefarino alludes to.  In my case, it was a manager who simply didn't schedule the time to review the design in any depth.

Reason 2. Folks unfamiliar with common design patterns often see them as "over-engineered". Indeed, I've had programmers (real live Java programmers, paid to write Java code) who claimed that the java.util data structures (specifically Map, TreeMap and HashMap) were needless, since they could write all of that using only primitive arrays. And they did, painstakingly write shabby code that had endless loops and lookups and indexing garbage instead of simply using a Map.

Reason 3. Some folks with a strong background in simple procedural programming reject class definitions in a vague, general way. Many good programmers work out ways to do encapsulation in languages like C, Fortran or COBOL via naming conventions or other extra-linguistic tricks.

They deeply understand procedural code and try to map their ideas of functions (or subroutines) and their notions of "encapsulation via naming conventions" onto OO design.

At one customer site, I knew there would be friction because the project manager was very interested in "code conventions" and "naming conventions". This was a little upsetting at the time. But I grew to realize that some folks haven't actually seen any open source code. They don't understand that there are established international, recognized conventions for most programming languages, and examples are available on the World Wide Web. Just download a popular package and read the source.

The "naming conventions" was particularly telling. The idea that Java packages (or Python packages and modules) provide distinct namespaces was not something that this manager understood. The idea that a class defines a scope was not really making much sense to them.

Also Suspicious

Another suspicious design feature are "utility" packages. It's rare (not impossible, but rare) for a class to truly be interpackagial in scope and have no proper home. The "java.util" package, for example, is a strange amalgamation of the collection data structures, national and cultural class definitions (calendars, currency, timzones, etc.) handy pattern abstractions, plus a few algorithms (priority queue, random).

Yes, these have "utility" in that they're useful. They apply broadly to many programming problems. But so does java.lang and java.io. The use of a vague and overly inclusive term like "util" is an abdication of design responsibility to focus on what's really being offered.

These things do not belong together in a sprawling unfocused package.

Nor does disparate functionality belong in a sprawling, unfocused class.

Education

The answer is a lot of eduction. It requires time and patience.

One of the best methods for education is code walkthroughs. This permits reviews of design patterns, and how the SOLID principles are followed (or not followed) by code under development.

HamCalc and Quirks

Careful study of the HamCalc shows a number of quirks. Some are funny, some are just examples of the need for unit test frameworks.

The Wikispaces for the modernization project is here: http://hamcalc.wikispaces.com/home

For example, the following line of code, in GW-Basic, will (usually) set Y to zero.

Y = O

Yes. That's the variable "O", not the number 0.

Why does this work? Why can we use "O" instead of 0?

Most programmers avoid using the variable "O", since it's hard to read.  GW-Basic provides default values of 0 for almost all variables. So, "Y=O" works as well as "Y=0" most of the time. The only time is doesn't work is if the program happens to have "O" used as a variable.

This is one of the examples where people start shouting that a compiled language is so obviously superior that the rest of us must be brain-damaged to use a dynamic language like Python.

This isn't a very compelling argument for the overhead of a compiler. It's a more compelling argument for avoiding languages with default values. Python, for example, would throw an exception if the variable "O" had no value.

This isn't common (so far, I've only found one example) but it's amusing.

Another amusing quirk is the occasional tangle of GOTO/GOSUB logic that defies analysis. There are several examples of GOSUB/RETURN logic that is totally circumvented by a GOTO that circumvents the return. This should (eventually) lead to some kind of stack overflow. But GW-Basic doesn't really handle recursion well, so it would probably just be ignored.

One of my favorites is this.


    730 FOR N=A TO T STEP B
    750 IF T/N=INT(T/N)THEN X=X+1:PN(X)=N:T=T/N:GOTO 730
    760 A=3:B=2
    770 NEXT N

What does the GOTO on line 750 mean? Since GW-Basic doesn't use a stack of any kind, it doesn't create recursion or stack overflow. It appears to "restart" the loop with a new value of T. I think.

Obstinate Idiocy [Updated]

Once in a great while, you see someone engaging in Obstinate Idiocy.

Here's my recent example.

They're solving some kind of differential equation. Not sure why. Symptom 1 of Obstinate Idiocy is No Rational Justification. The explanation is often "that's not relevant, what's relevant is this other thing I want to focus on" or something equivalent to "never mind about that."

The equation is this:

\[y + 3 \ln \lvert y \rvert = \frac{1}{3}(x-1)^3-\frac{11}{3}\]

Pretty gnarly.

Apparently, they were so flummoxed by this that they immediately turned to Excel.  Really.  Excel.

Symptom 2 of Obstinate Idiocy is Random Tool Choice. Or perhaps Ineffective Tool Choice. A kind of weird, unthinking choice of tools.

Of course, Excel struggles with this sort of thing, since it appears to gnarly. I was told that there's an Excel Solver, but there was some problem with using it. It didn't scale, or it required some understanding of the shape of the equation or something.

Symptom 3 of Obstinate Idiocy is Seemingly Random Whining. It's random because there's no rational justification for what's going on and the tool was chosen apparently at random.

Ask a question like "why not use another tool?" and you don't get an answer. You get an argument about tool choice or the politics of the situation or "tool choice isn't the point" or some other dismissive non-answer.

Ask a question like "what are you really trying to do?" and you get user stories that make approximately no sense. We had to endure a long discussion on system-assigned surrogate keys as if that was somehow relevant to the graphing the equation shown above. See Symptom #1. There's just no reason for this. It's Very Important, but No One Else Can Understand The Reason Why.

How To Begin?

So, now we're at this weird impasse.

We have the obstinate idiot who won't discuss their tool choice. Somehow I'm supposed to sprinkle around some Faerie Dust and magically make Excel do something better or different than what it normally does. Indeed, I'm having trouble understanding any of the whining about Excel.

Clearly, they've never heard of MatLab or Mathematica or any commercial product that does this nicely. Apparently, they've never even seen the graph tool on Mac OS X which simply draws the graph with no effort of any kind on the part of the user.

Clearly, they've never seen Google and can't make it work.

They asked how a Pythonista would approach a problem this gnarly. I couldn't even properly understand that question, since they hadn't Googled anything and didn't really have a question that could be answered. As a Pythonista, I use Google. I wasn't sure how to approach an answer, since I couldn't really understand what their goal was or what their knowledge gap was.

Since their principal complaint was about Excel, asking a Python-related question didn't make much sense. Were they planning on dropping Excel? If so, why not use MatLab or Mathematica?

See Symptom 2. The tool choice was fixed. Other tools weren't on the table. If so, why ask about Python?

At this point, the place to begin seems to be this link: http://bit.ly/11usbtH And that's not going to be helpful with the Obstinate Idiot. They'll claim they already knew all of that, they just needed some additional or different help.

They specifically said they weren't going to use Python. Which raises the question "Why ask me anything, then?" To which there was no real answer, just sulking about me not being helpful.

Correct. I'm not being helpful. I can't figure out what the problem is. There's a gnarly formula and Excel somehow doesn't work in some optimal way. And database surrogate keys. And departmental politics.

Did You Try This?

The equation simplifies to

\[ x = (3y + 9 \ln \lvert y \rvert + 11)^{\frac{1}{3}} + 1 \]

Which is really easy to graph. \(x=f(y)\) is, of course, not the usual approach of \(y=f(x)\).

Apparently, the Obstinate Idiot had not actually applied algebra to the equation. Nor had they ever conceived of graphing \(x=f(y)\).

Which brings us to Symptom 4 of Obstinate Idiocy: Slow To Ask For Help.

And the variation on Symptom 1 of Obstinate Idiocy: Goal-Free Activity. By this I mean that the thrashing around with Excel and discussing Python was all just a long, drawn-out and utterly irrelevant side-bar from the real purpose, which apparently was to find something out related to a differential equation. It's still unclear what the equation is being used for and why the graph is helpful.

Python Approach

First: Differential Equations are hard. Nothing makes them easy.

Interactive Python, however, can be of some help after you've taken the first steps with pencil and paper.

Here's a console log of something I did to help the Obstinate Idiot.

>>> import math
>>> import pprint
>>> 
>>> def lde_1(y):
...     try:
...         x = (3*y+9*math.log(abs(y))+11)**(1/3)+1
...     except ValueError:
...         x = float("NaN")
...     return x
... 
>>> def eval(y_lo=-15, y_hi=15, y_step=0.5, f_y=lde_1):
...     # Next smaller power of 2: prettier numbers. Less noise.
...     step_2 = 2**math.floor(math.log(y_step, 2))
...     for t in range(int((y_hi-y_lo) // step_2)):
...         y = y_lo + step_2*t
...         x = f_y(y)
...         yield( x, y )
... 
>>> data1= list(eval())
>>> pprint.pprint(data1)

I'll leave out the dump of the data points. However, it's possible to see the asymptote at zero and the ranges where the results switch from real to complex numbers.

We can drill into the region around zero to see some details.

data2 = list(eval(-2, 2, .0625))
pprint.pprint(data2)

These are just numbers.  A picture is worth a thousand numbers.

We have lots of choices for graphic packages in Python. The point here, however, is that evaluating the gnarly equation required two preliminary steps that were far, far more important than choosing a graphic package.

1. Do some simple algebra.
2. Write a simple loop.

If output to Excel is somehow important, there's always this.

>>> import csv
>>> with open("data.csv","w") as target:
...    wtr= csv.writer(target)
...    wtr.writerows(data1)

That will produce a CSV that Excel will tolerate and display as an X-Y scatter plot.

A jupyter notebook with pyplot will knock out a picture directly, allowing visualization.

Password Encryption -- Short Answer: Don't.

First, read this.    Why passwords have never been weaker—and crackers have never been stronger.

There are numerous important lessons in this article.

One of the small lessons is that changing your password every sixty or ninety days is farcical.  The rainbow table algorithms can crack a badly-done password in minutes.  Every 60 days, the cracker has to spend a few minutes breaking your new password.  Why bother changing it?  It only annoys the haxorz; they'll be using your account within a few minutes.  However.  That practice is now so ingrained that it's difficult to dislodge from the heads of security consultants.

The big lesson, however, is profound.

Work Experience

Recently, I got a request from a developer on how to encrypt a password.  We have a Python back-end and the developer was asking which crypto package to download and how to install it.

"Crypto?" I asked.  "Why do we need crypto?"

"To encrypt passwords," they replied.

I spat coffee on my monitor.  I felt like hitting Caps Lock in the chat window so I could respond like this: "NEVER ENCRYPT A PASSWORD, YOU DOLT."

I didn't, but I felt like it.

Much Confusion

The conversation took hours.  Chat can be slow that way.  Also, I can be slow because I need to understand what's going on before I reply.  I'm a slow thinker.  But the developer also needed to try stuff and provide concrete code examples, which takes time.

At the time, I knew that passwords must be hashed with salt.  I hadn't read the Ars Technica article cited above, so I didn't know why computationally intensive hash algorithms are best for this.

We had to discuss hash algorithms.

We had to discuss algorithms for generating unique salt.

We had to discuss random number generators and how to use an entropy source for a seed.

We had to discuss http://www.ietf.org/rfc/rfc2617.txt in some depth, since the algorithms in section 3.2.2. show some best practices in creating hash summaries of usernames, passwords, and realms.

All of this was, of course, side topics before we got to the heart of the matter.

What's Been Going On

After several hours, my "why" questions started revealing things.  The specific user story, for example, was slow to surface.

Why?

Partly because I didn't demand it early enough.

But also, many technology folks will conceive of a "solution" and pursue that technical concept no matter how difficult or bizarre.  In some cases, the concept doesn't really solve the problem.

I call this the "Rat Holes of Lost Time" phenomena: we chase some concept through numerous little rat-holes before we realize there's a lot of activity but no tangible progress.  There's a perceptual narrowing that occurs when we focus on the technology.  Often, we're not actually solving the problem.

IT people leap past the problem into the solution as naturally as they breathe. It's a hard habit to break.

It turned out that they were creating some additional RESTful web services.  They knew that the RESTful requests needed proper authentication.  But, they were vague on the details of how to secure the new RESTful services.

So they were chasing down their concept: encrypt a password and provide this encrypted password with each request.  They were half right, here.  A secure "token" is required.  But an encrypted password is a terrible token.

Use The Framework, Luke

What's most disturbing about this is the developer's blind spot.

For some reason, the existence of other web services didn't enter into this developer's head.  Why didn't they read the code for the services created on earlier sprints?

We're using Django.  We already have a RESTful web services framework with a complete (and high quality) security implementation.

Nothing more is required.  Use the RESTful authentication already part of Django.

In most cases, HTTPS is used to encrypt at the socket layer.  This means that Basic Authentication is all that's required.  This is a huge simplification, since all the RESTful frameworks already offer this.

The Django Rest Framework has a nice authentication module.

When using Piston, it's easy to work with their Authentication handler.

It's possible to make RESTful requests with Digest Authentication, if SSL is not being used.  For example, Akoha handles this.  It's easy to extend a framework to add Digest in addition to Basic authentication.

For other customers, I created an authentication handler between Piston and ForgeRock OpenAM so that OpenAM tokens were used with each RESTful request.  (This requires some care to create a solution that is testable.)

Bottom Lines

Don't encrypt passwords.  Ever.

Don't write your own hash and salt algorithm.  Use a framework that offers this to you.

Read the Ars Technica article before doing anything password-related.

How Expensive is a "Waterfall" Project Plan?

It's impossible to step into the same river twice; other waters are flowing toward the sea.  It's impossible to do "head-to-head" project comparisons.  You can't have the same people doing the same thing with only one constraint changed.  You can try to resort to having similar people doing the exact same thing.

It's impossible to provide trivial quantification of the costs of a waterfall approach.  However, there's a lot of places where "up front" engineering adds cost and risk.  A waterfall plan adds friction, zero-value administrative costs and (sometimes) extra work.

For software development that doesn't involve "toy" projects, but apply to the magical "real world" projects, it's difficult to justify building the same "real world" thing twice.  It's hard to create a detailed and purely economic analysis of this question.  Therefore, we have to resort to hand-waving and speculation.

[I put "real world" in quotes, because the "real world" is a kind of dog-whistle phrase that only project managers understand.  It appears to mean "projects that fit my preconceived notions" or "projects that can't be done any other way because of political considerations" or "projects constrained by a customer to have a farcical structure".  Another variant on this dog-whistle phrase is "the realities of running a business."  There's something in their real world that they (a) can't define and (b) prevents any change.]

A few months back, I saw a http://prezi.com presentation that was intended to drive architecture, cost and schedule conversations.  The authors were happy.  I was left relatively mystified.

There are three relevant issues that don't fit well into a prezi presentation.

1. Open Technology questions.  If it's new software development, there are unanswered software API, performance, quality, fitness-for-purpose questions.  If it's a clone or a clean-room reverse engineering, then there may be few questions.  Otherwise, there must be unknowns.
2. Open User Story questions.  If it's new software development, then the user stories are not complete, consistent and optimized.  There must be something missing or extra.
3. Open User Experience questions.  I'm not a UX person.  Tinkering with an Arduino has taught me that even in the most prosaic, simplistic project there is a UX element.  

It's easy to trivialize this as "details" or "supporting information" omitted from the presentation.  Actually, it's considerably more interesting that simply elaborating details.  Indeed, the missing stuff is often more important that the elements of the presentation than were provided.

The Standard Gaps

While a presentation can show some elements of UX, it's not interactive.  It cannot provide any useful depth on the UX.  Failure to do UX exploration (with real interaction) is crazy.  Making assumptions about UX is more likely to create cost and risk.

User Stories can be kicked around in a presentation.  However.  Once the highest-priority user stories are built, and lessons learned about the users, the problem and the emerging solution, then the other user stories can (and should) change based on those lessons learned.  Assuming a list of user stories and then committing to a specific schedule enshrines the costs in a bad plan.  And we all know that the plan (and the dates) are sacred once written down.

Software does not exist in a vacuum.

I'll repeat that, since it's a hidden assumption behind the waterfall.

Software does not exist in a vacuum.

You have a Heisenberg problem when you start building software and releasing it incrementally.  The presence of the new software, even in a first-release, beta-test, barely-working mode changes the ecosystem in which the software exists.  Other applications can (and will) be modified to adjust to the new software.  Requirements can (and should) change.

A waterfall project plan exists to prevent and stifle change.  A long list of tasks and dates exists to assure that people perform those tasks on those dates.  This is done irrespective of value created and lessons learned.

Technology changes.  Our understanding of the technology changes.  One wrong assumption about the technology invalidates the deliverables for a sprint.  In a project with multiple people, multiple assumptions and multiple sprints, this effect is cumulative.  Every assumption by every person for every technology is subject to (considerable) error.

Every technology assumption must be looked at as a needless cost that's being built into the project.

Recently, I've been working on a project with folks that don't know Django very well.  Their assumptions are -- sometimes -- alarming.  Part way through a sprint, I got a raft of technical questions on encrypting passwords.  It's hard to state it strongly enough: never encrypt a password.  What's more useful is this: always hash passwords.  The original password must be unrecoverable.  There are lots of clever hashing techniques.  Some of which are already part of Django.  A significant portion of the sprint (appeared) to be based on reinventing a feature already part of Django.

Do the math: a few wrong assumptions made during project planning are canonized forever as requirements in the project.  With a waterfall project, they're not easy to change.  Project managers are punished for changing the project.  You can't increase the project deadline; that's death.  You can't decrease it, either: you get blamed for sand-bagging the estimates.

Arduino Technology Lessons Learned

After buying several 74HC595 shift registers from Sparkfun, I realized that my assumptions about the interfaces were utterly wrong.  I needed to solder a mix of right-angle header and straight-through headers onto the breakout boards.  I tried twice to order the right mix headers.  It seems so simple.  But assumptions about the technology are often wrong.

This is yet more anecdotal evidence that all assumptions must be treated as simple lies.  Some managers like to treat assumptions as  "possibly true" statements; i.e., these are statements that are unlikely to be false.  This is wrong.  Assumptions always have a very high probability of being false, since they're not based on factual knowledge.

Some managers like to treat assumptions as "boundary conditions":  i.e.,  if the assumption is true, then the project will go according to plan.  Since all of the assumptions will be prove to be incorrect, this seems like simple psychosis.  

[Interestingly, the assumptions-are-boundary folks like to play the "real world" card: "In the real world, we need to make assumptions and go forward."  Since all assumptions will be shown to be incorrect, why make them?  Wouldn't it be more rational to say "we need to explore carefully by addressing high-righ unknowns first"?  Wouldn't it be better to both gather facts and build an early release of the software at the same time?]

Arduino User Story Assumptions

After building a prototype that addressed two of the user stories, it slowly became clear that the third user story didn't actually exist.

Sure, the words were there on paper.  Therefore, there's a user story.

But. 

There was nothing to actually do.  

The whole "As a [role], I need [feature] so that I can [benefit]" that was written on day one was based on a failure to understand precisely how much information was trivially available.  The benefit statement was available with a small software change and no separate user story.  And no separate hardware to support that user story.

Exploration and tinkering reduced the scope of the work.  In the real world.

[In the "real world" where waterfall is considered important, exploration is described as unbounded spending of limited resources.  In the real real world, money must be spent; it can either be long hand-wringing meetings or it can be prototype development.]

The user story (written before a prototype existed) was based on a failure to fully understand the UX.  The only way to fully understand the UX is to build it.  Education costs money.

Arduino UX Learnings

Perhaps the most important issue here is UX.

Once upon a time, UX was expensive, difficult and complex.  So difficult that special-purpose prototyping tools were created to make it possible to create a preliminary UX that could be used to confirm UX and user stories.

This UX prototyping effort was real money spent as part of "requirements" or "design"; it created documentation that flowed over the waterfall for subsequent development.

This notion is obsolete.  And has been obsolete for several years.

UX is now so easy to build that it makes much more sense to build two (or three) competing UX's and compare them to see which is actually better.

Indeed, it makes a lot of sense to build one UX and release it; make a real effort at solving the user's problems.  Then build a second UX for A/B testing purposes to see if there's room for improvement.

I'll repeat that for folks who really like the waterfall.

It's now cheaper to actually build two than it is to write detailed requirements for one.

[In the "real world", this is deprecated as "wasting time playing with the UX".  As if written requirements based on nothing more than a whiteboard are more "real" than hands-on experience with the UX.]

You can prove this to yourself by actually observing actual UX developers knocking out pages and screens.  Long "requirements gathering" meetings with prospective users amount to a waste of time and money.  Long "brainstorming" sessions, similarly, are wasteful.  Short, ongoing conversations, a build, a release, and a follow-up review has more value, and costs less.  

Do the math.  Several users and business analysts in several multiple-hour meetings costs how much?

A pair of developers banging out fully-functioning, working UX for a use case costs how much?

A slavish adherence to waterfall development creates "real world" costs.

IBM RAMAC Device: 5 MB

Check out this picture.  http://www.petapixel.com/2011/12/27/what-5mb-of-storage-looked-like-in-1956/

Random Reminiscing Follows

When I was in college (1974-1978) 64K of RAM was the size of a refrigerator.

By 1982, 64K of RAM was an Apple ][+ fully tricked out with the 16K expansion card.

I vaguely remember working with a "tower" device that was a 5MB disk drive.  Think Mac Pro case to hold a disk drive.

By 1985 or so, 128K or RAM was a Macintosh and a 5MB disk drive a big desktop console box.  Smaller than a tower.  Irritating because it took up so much real-estate when compared with the Mac itself that was designed to take up an 8 x 11 space (not including keyboard or mouse).

Now 5MB is round-off error.

What's Important?

Has computing gotten that much better?

Old folks (like me) reminisce about running large companies on small computers.  Now, we can't even get our coffee in the morning without the staggering capabilities of a 32GB iPhone.

Old folks, however, are sometimes mired in bad waterfall development methods and don't understand the value of test-driven development.  While the hardware is amazing, the development tools and techniques have improved, also.