The Value of Microsoft's Tools

See Andrew Binstock's "Windows 8: Microsoft's Development Re-Do".

The costs of these migrations has been enormous and continues to accumulate...

I can only rub my hands with glee and engage in shameless "I Told You So" self-congratulations.

Only you can prevent being held hostage by Microsoft.

More than once, I've observed that a strategy of using only proprietary tools would be expensive and complex.  And every time, the folks I was talking to trivialized my concerns as hardly worth considering.

I've seen orphaned software: it only compiles on an old version of Visual Studio.   I've seen software orphaned so badly that it can only be compiled on one creaky old PC.  The cost to convert was so astronomical that the customer preferred to hope for a product to arise somewhere in the marketplace.  When no suitable product appeared over the decades, the problem reached palpable Pants On Fire (POF) levels of panic.  All due to the hidden costs of Microsoft's tools.

I've even been told that VB is a terrible language, but Visual Studio makes it acceptable.

Justification of Project Staffing

I really dislike being asked to plan a project.  It's hard to predict the future accurately.

In spite of the future being -- well -- the future, and utterly unknowable, we still have to have the following kinds of discussions.

Me: "It's probably going to take a team of six."

Customer: "We don't really have the budget for that.  You're going to have to provide a lot of justification for a team that big."

What's wrong with this picture?  Let's enumerate.

1. Customer is paying me for my opinion based on my experience.  If they want to provide me with the answers, I have a way to save them a lot of money.  Write their own project plan with their own answers and leave me out of it.
2. I've already provided all the justification there is.  I'm predicting the future here.  Software projects are not simple Rate-Time-Distance fourth-grade math problems.  They involve an unknown number of unknowns.  I can't provide a "lot" of justification because there isn't any indisputable basis for the prediction.
3. I don't know the people. The customer -- typically -- hasn't hired them yet.  Since I don't know them, I don't know how "productive" they'll be.  They could hire a dozen n00bz who can't find their asses blindfolded even using both hands.  Or.  They could hire two singular geniuses who can knock the thing out in a weekend.  Or.  They could hire a half-dozen arrogant SOB's who refuse to follow my recommendations. 
4. They're going to do whatever they want no matter what I say.  Seriously.  I could say "six".  They could argue that I should rewrite the plan to say "four" without changing the effort and duration.  Why ask me to change the plan?  A customer can only do what they know to be the right thing. 

Doing the Right Thing

Let's return to that last point.  A customer project manager can only do what they absolutely know is the right thing.  I can suggest all kinds of things.  If they're too new, too different, too disturbing, they're going to get ignored.

Indeed, since people have such a huge Confirmation Bias, it's very, very hard to introduce anything new.  A customer doesn't bring in consultants without having already sold the idea that a software development project is in the offing.  They justify spending a few thousand on consulting by establishing some overall, ball-park, big-picture budget and showing that the consulting fees are just a small fraction of the overall.

As consultants, we have to guess this overall, ball-park, big-picture budget accurately, or the project will be shut down.  If we guess too high, then the budget is out of control, or the scope isn't well-enough defined, or some other smell will stop all progress.  If we guess too low, then we have to lard on additional work to get back to the original concept.

Architectures, components and techniques all have to meet expectations. A customer that isn't familiar with test drive development, for example, will have an endless supply of objections.  "It's unproven."  "We don't have the budget for all that testing."  "We're more comfortable with our existing process."

The final trump card is the passive aggressive "I'll have to see the detailed justification."  It means "Don't you dare."  But it sounds just like passive acceptance.

Since project managers can only do what they know is right, they'll find lots of ways of subverting the new and unfamiliar.

If they don't like the architecture, the first glitch or delay or problem will immediately lead to a change in direction to yank out the new and replace it with the familiar.

If they don't like a component, they'll find numerous great reasons to rework that part of the project to remove the offending component.

If they don't like a technique (e.g., Code Walk Throughs) they'll subvert it.  Either not schedule them.  Or cancel them because there are "more important things to do."  Or interrupt them to pull people out of them.

Overcoming the Confirmation Bias

I find the process of overcoming the confirmation bias to be tedious.  Some people like the one-on-one "influencing" role.  It takes patience and time to overcome the confirmation bias so that the customer is open to new ideas.  I just don't have the patience.  It's too much work to listen patiently to all the objections and slowly work through all the alternatives.

I've worked with folks who really relish this kind of thing.  Endless one-on-one meetings.  Lots of pre-meetings and post-meetings and reviews of drafts.  I suppose it's rewarding.  Sigh.

How to Learn

A recent question.

i came up with two options.
 1.  building skills 1 (+ other references)... then algorithms & data
structures.... then your books 2 & 3

or

 2.  your three books 1,2 & 3... then algo & ds

kindly help me decide so i can start soon. 

I have two pieces of advice.

First.  Programming is a language skill.  Just like English.  If you can't get the English right, the odds of getting Python, Java, HTML or SQL right is considerably reduced.  Please, please, please take more care in grammar, syntax and punctuation.  Otherwise, your future as a programmer doesn't look very good.  For example, the personal pronoun is spelled "I".  In the 20th century, we spell out "and"; we stopped writing "&" as a stand-in for the Latin "et" centuries ago.  Also, ellipses ("...") shouldn't be used except when eliding part of a quote.  Clarity and precision actually matter.

Second, and more relevant, your two choices don't really amount to a significant difference.  If you're waiting around for advice, you're wasting your time.  Both sequences are good ideas. It's more important to get started than it is to carefully choose the precise and exact course of study. Just start doing something immediately.

Learning to program is a life-long exercise. There will always be more to learn. Start as soon as you can. The exact choices don't matter.  Why?  Because, eventually, you'll read all of those books plus  many, many others.

Spend less time waiting for advice and more time studying.

More On Inheritance vs. Delegation

Emphasis on the "More On" as in "Moron".  This is a standard design error story.  The issue is that inheritance happens along an "axis" or "dimension" where the subclasses are at different points along that axis.  Multi-dimensional inheritance is an EPIC FAIL.

Context

Data warehouse processing can involve a fair amount of "big batch" programs.  Loading 40,000 rows of econometric data in a single swoop, updating dimensions and loading facts, for example. 

When you get data from customers and vendors, you have endless file-format problems.  To assure that things will work, each of these big batch programs has at least two operating modes.

• Validate.  Go through all the motions.  Except.  Don't commit any changes to the database; don't make any filesystem changes.  (i.e., write the new files, but don't do the final renames to make the files current.)
• Load.  Go through all the motions including a complete commit to the database and any filesystem changes.

Problem

What's the difference between the two modes?  Clearly, one is a subclass of the other.

• Load can be the superclass.  The Validate subclass simply replaces the save methods stubs that do nothing.
• Validate can be the superclass.  The Load subclass simply implements the save method stubs with methods that do something useful.

Simple, right?

Wrong.

What Doesn't Work

This design has a smell.  The smell is that we can't easily extend the overall processing to include an additional feature. 

Why not? 

This design has the persistence feature set as the inheritance axis or dimension.  This is kind of limited.  We really want a different feature set for inheritance.

Consider a Validate for two dimensions (Company and Time) that loads econometric facts.  It has stub "save" methods.

We subclass the Validate to create the proper Load for these two dimensions and one fact.  We replace the stub save methods with proper database commits. 

After the actuaries think for a while, suddenly we have a file which includes an additional dimension (i.e., business location) or an additional fact (i.e., econometric data at a different level of granularity).  What now?  If we subclass Validate to add the dimension or fact, we have a problem.  We have to repeat the Load subclass methods for the new, extended Load.  Oops.

If we subclass Load to add the dimension or fact, we have a problem.  We have to repeat the Validate stubs in the new extended Load to make it into a Validate.  Oops.

Recognizing Delegation

It's difficult to predict inheritance vs. delegation design problems.

The hand-waving advice is to consider the essential features of the object.  This isn't too helpful.  Often, we're so focused on the database design that persistence seems essential.

Experience shows, however, that some things are not essential.  Persistence, for example, is one of those things that should always be delegated.

Another thing that should always be delegated is the more general problem of representation: JSON, XML, etc., should rely on delegation since this is never essential.  There's always another representation for data.  Representation is always independent of the object's essential internal state changes.

Consequence

In my case, I've got about a dozen implementations using a clunky inheritance that had some copy-and-paste programming.  Oops.

I'm trying to reduce that technical debt by rewriting each to be a proper delegation.  With good unit test coverage, there's no real technical risk.  Just tedious fixing the same mistake that I rushed into production twelve separate times. 

Really.  Colossally dumb.

VMware, VIX and PyVIX2

The topic of VMware came up at my local 757 Python Users Group.

A common administrative need is to control VM farms.  While there are a number of pointy-clicky GUI tools, VMware offers the VIX library to permit writing scripts to control VM's.

Here's some information we looked at recently on PyVIX2 and VMware.

The idea behind PyVIX2 is to provide a relatively simple Python binding to VIX.    This, too, is a command-line interface, following on the heels of More Command-Line Goodness and Command-Line Applications.

The Agile "Religion" -- What?

Received "it seems that software development has caught the agile religion. Personally, I have an issue w/ being unimodal."

What?

First.  "agile religion".  As in the deprecating statement: Agile is nothing more than a religion?  As in Agile is nothing more than a vague religious practice with no tangible value to an organization?  Interesting, I guess.

I'm assuming that the author did not read the Manifesto for Agile Software Development.  Or--worse-- they read it and find that the four values (Individuals and interactions, Working software, Customer collaboration and Responding to change) are just of no tangible value.

That's alarming.  Really.  The alternative (processes and tools, comprehensive documentation, contract negotiation, and following a plan without regard to changes) seems like it's a recipe for cost, risk, low-value work and a cancelled project.  Indeed, it seems like non-Agile project management is the best way to get to the fabled "Software Crisis" where lots of money gets spent but little of value gets created.

Further, it seems that all modifications of the classic waterfall method (e.g., spiral method as a prime example) specifically create "iterative, incremental" approaches to software development.  That is, everything that's not a strict (brain-dead) waterfall has some elements of Agile.

This causes me to think that Agile isn't a religion.  It causes me to think that Waterfall methods were a religious practice of no tangible value.  All the methodology experiments over the last 15 years have been ways of introducing flexibility (agility, brains) into a foolishly inflexible methodology definition.

Indeed, it appears that the heavy-weigh waterfallish methods are an attempt to replace thinking with process.  And it didn't work.  So, we have to go back to the thinking part.  Only, we call it Agile now.

Religious Wars.

Second.  "agile religion" (again).  As in methodology discussions are just religious wars?  As in methodology discussions are just quibbling over no-value details?  Some folks may get this impression of making a choice between Agile vs. Non-Agile methods.  I think that those folks haven't actually had the opportunity to work from a prioritized backlog and build the most valuable part first.  I think that someone who things Agile is just a religious war hasn't been allowed to fix a broken project plan based on lessons learned during the first release.

Agility.

Third.   "unimodal".  As in being exclusively Agile is bad?  As in sometimes you need to have a rigid, unyielding process that sticks strictly to the schedule irrespective of changes which may occur?  That doesn't seem rational.

Change happens.  Forcing the inevitable changes to conform to some farcical schedule made up by people who didn't have all the details seems silly.  Making contract negotiation the focal point of response to change seems like a waste of effort.  Trying to document everything so completely that all possible changes are already accounted for seems impossible.  And replacing change with a process that regulates change seems -- perhaps -- unhinged.

There were some links and some charts and graphs attached.  I couldn't get past the two sentences above to see if there was, perhaps, something more to it.  All I could do was respond with a request for clarification that didn't involve the trivialization of Agile methods.  It doesn't seem sensible to try and remove the human element from software development.

I'll provide whatever follow-up clarification surfaces on this topic.  It's interesting to see if the "agile religion" was misplaced, or if there are folks who think that responding to the messiness of real software development is a bad idea.

We tried the waterfall method.  And it didn't work very well.  Agile isn't a "religion".  It's a simple acknowledgement that reality is messy.

More Command-Line Goodness

In Command-Line Applications, we looked at a Python main-import switch which boiled down to this.


for file in args.file: 
    with open( file, "r" ) as source:
        process_file( source, args )

The point was that each distinct file on the command-line was processed in a more-or-less uniform way by a single function that does the "real work" for that input file.

It turns out that we often have flat files which are spreadsheets or spreadsheet-like.   Indeed, for some people (and some organizations) the spreadsheet is their preferred user interface.  As I've said before, 

Spreadsheets are the universal user interface. Everyone likes them, they're almost inescapable. And they work. There's no reason to attempt to replace the spreadsheet with a web page or a form or a desktop application. It's easier to cope with spreadsheet vagaries than to replace them.

They have problems, but they are surprisingly common.  

Enter Stingray Reader.  This is a small Python library to make it easy to have programs which read workbooks--collections of spreadsheets--or spreadsheet-like files with a degree of transparency.  

And.  It allows a clean command-line interface.

With a little care, we can reduce the main-import switch to something like this.

if __name__ == "__main__":

    logging.basicConfig( stream=sys.stderr )

    args= parse_args()

    logging.getLogger().setLevel( args.verbosity )

    builder= make_builder( args )

    try:

        for file in args:

            with workbook.open_workbook( input ) as source:

                process_workbook( source, builder )

        status= 0

    except Exception as e:

        logging.exception( e )

        status= 3

    logging.shutdown()

    sys.exit( status )

The bold lines are specific to workbook ("spreadsheet") processing.  A "builder" creates application-specific Python objects from spreadsheet rows.  The "workbook.open_workbook" is a function that builds a workbook reader based on the file name.  It can handle a number of file types.  

The process_workbook function is the "real work" function that handles a workbook of individual spreadsheets (or a spreadsheet-like file).