VMware, VIX and PyVIX2

The topic of VMware came up at my local 757 Python Users Group.

A common administrative need is to control VM farms.  While there are a number of pointy-clicky GUI tools, VMware offers the VIX library to permit writing scripts to control VM's.

Here's some information we looked at recently on PyVIX2 and VMware.

The idea behind PyVIX2 is to provide a relatively simple Python binding to VIX.    This, too, is a command-line interface, following on the heels of More Command-Line Goodness and Command-Line Applications.

The Agile "Religion" -- What?

Received "it seems that software development has caught the agile religion. Personally, I have an issue w/ being unimodal."

What?

First.  "agile religion".  As in the deprecating statement: Agile is nothing more than a religion?  As in Agile is nothing more than a vague religious practice with no tangible value to an organization?  Interesting, I guess.

I'm assuming that the author did not read the Manifesto for Agile Software Development.  Or--worse-- they read it and find that the four values (Individuals and interactions, Working software, Customer collaboration and Responding to change) are just of no tangible value.

That's alarming.  Really.  The alternative (processes and tools, comprehensive documentation, contract negotiation, and following a plan without regard to changes) seems like it's a recipe for cost, risk, low-value work and a cancelled project.  Indeed, it seems like non-Agile project management is the best way to get to the fabled "Software Crisis" where lots of money gets spent but little of value gets created.

Further, it seems that all modifications of the classic waterfall method (e.g., spiral method as a prime example) specifically create "iterative, incremental" approaches to software development.  That is, everything that's not a strict (brain-dead) waterfall has some elements of Agile.

This causes me to think that Agile isn't a religion.  It causes me to think that Waterfall methods were a religious practice of no tangible value.  All the methodology experiments over the last 15 years have been ways of introducing flexibility (agility, brains) into a foolishly inflexible methodology definition.

Indeed, it appears that the heavy-weigh waterfallish methods are an attempt to replace thinking with process.  And it didn't work.  So, we have to go back to the thinking part.  Only, we call it Agile now.

Religious Wars.

Second.  "agile religion" (again).  As in methodology discussions are just religious wars?  As in methodology discussions are just quibbling over no-value details?  Some folks may get this impression of making a choice between Agile vs. Non-Agile methods.  I think that those folks haven't actually had the opportunity to work from a prioritized backlog and build the most valuable part first.  I think that someone who things Agile is just a religious war hasn't been allowed to fix a broken project plan based on lessons learned during the first release.

Agility.

Third.   "unimodal".  As in being exclusively Agile is bad?  As in sometimes you need to have a rigid, unyielding process that sticks strictly to the schedule irrespective of changes which may occur?  That doesn't seem rational.

Change happens.  Forcing the inevitable changes to conform to some farcical schedule made up by people who didn't have all the details seems silly.  Making contract negotiation the focal point of response to change seems like a waste of effort.  Trying to document everything so completely that all possible changes are already accounted for seems impossible.  And replacing change with a process that regulates change seems -- perhaps -- unhinged.

There were some links and some charts and graphs attached.  I couldn't get past the two sentences above to see if there was, perhaps, something more to it.  All I could do was respond with a request for clarification that didn't involve the trivialization of Agile methods.  It doesn't seem sensible to try and remove the human element from software development.

I'll provide whatever follow-up clarification surfaces on this topic.  It's interesting to see if the "agile religion" was misplaced, or if there are folks who think that responding to the messiness of real software development is a bad idea.

We tried the waterfall method.  And it didn't work very well.  Agile isn't a "religion".  It's a simple acknowledgement that reality is messy.

More Command-Line Goodness

In Command-Line Applications, we looked at a Python main-import switch which boiled down to this.


for file in args.file: 
    with open( file, "r" ) as source:
        process_file( source, args )

The point was that each distinct file on the command-line was processed in a more-or-less uniform way by a single function that does the "real work" for that input file.

It turns out that we often have flat files which are spreadsheets or spreadsheet-like.   Indeed, for some people (and some organizations) the spreadsheet is their preferred user interface.  As I've said before, 

Spreadsheets are the universal user interface. Everyone likes them, they're almost inescapable. And they work. There's no reason to attempt to replace the spreadsheet with a web page or a form or a desktop application. It's easier to cope with spreadsheet vagaries than to replace them.

They have problems, but they are surprisingly common.  

Enter Stingray Reader.  This is a small Python library to make it easy to have programs which read workbooks--collections of spreadsheets--or spreadsheet-like files with a degree of transparency.  

And.  It allows a clean command-line interface.

With a little care, we can reduce the main-import switch to something like this.

if __name__ == "__main__":

    logging.basicConfig( stream=sys.stderr )

    args= parse_args()

    logging.getLogger().setLevel( args.verbosity )

    builder= make_builder( args )

    try:

        for file in args:

            with workbook.open_workbook( input ) as source:

                process_workbook( source, builder )

        status= 0

    except Exception as e:

        logging.exception( e )

        status= 3

    logging.shutdown()

    sys.exit( status )

The bold lines are specific to workbook ("spreadsheet") processing.  A "builder" creates application-specific Python objects from spreadsheet rows.  The "workbook.open_workbook" is a function that builds a workbook reader based on the file name.  It can handle a number of file types.  

The process_workbook function is the "real work" function that handles a workbook of individual spreadsheets (or a spreadsheet-like file).

A smoothly operating, well-oiled engine for failure

It occurs to me that much of "Big IT" creates a well-oiled organization that makes broken software seem acceptable. The breakage is wrapped in layers of finely-tuned process.

Consider a typical Enterprise Application.  There's a help desk, ticket tracking, a user support organization that does "ad-hoc" processing, and a development organization to handle bug fixes and enhancement requests.  All those people doing all that work.

Why?

If people need all that support, then the application is -- from a simplistic view -- broken.

The organization, however, has coped with the broken application by wrapping it in layers of people, process, tools, technology, management and funding.  The end users have a problem, they call the help desk, and the machine kicks in to resolve their problem.

It is a given -- a going-in assumption -- a normal, standard expectation that any enterprise software is so broken that a huge organization will be essential for pressing forward.  It is expected that good software cannot be built.

We're asked to help a client create a sophisticated plan for the New Enterprise App support organization.  Planning this organization feels like planning for various kinds of known, predicted, expected failures. Failure is the expectation.  Broken is the standard operating mode.

Consider a typical non-Enterprise Application.  Let's say, the GNU C compiler.  Or Python.  Or Linux.  An almost entirely volunteer organization, no help desk, no trouble tickets, no elaborate support organization plan.  Yet.  These products actually work flawlessly.  They're not wrapped in a giant organization.

Why is the bar for acceptability so low for "Enterprise" applications?  Why is this tolerated?

Command Line Applications

I'm old -- I admit it -- and I feel that command-line applications are still very, very important. Linux, for example, is packed full of almost innumerable command-line applications. In some cases, the Linux GUI tools are specifically just wrappers around the underlying command-line applications.

For many types of high-volume data processing, command-line applications are essential.

I've seen command-line applications done very badly.

Overusing Main

When writing OO programs, it's absolutely essential that the OS interface (public static void main in Java or the if __name__ == "__main__": block in Python) does as little as possible.

A good command-line program has the underlying tasks or actions defined in some easy-to-work with class hierarchy built on the Command design pattern. The actual main program part does just a few things: gather the relevant environment variables, parse command-line options and arguments, identify the configuration files, and initiate the appropriate commands. Nothing application-specific.

When the main method does application-specific work, that application functionality is buried in a method that's particularly hard to reuse. It's important to keep the application functionality away from the OS interface.

I'm finding that main programs should look something like this:


if __name__ == "__main__":
    logging.basicConfig( stream=sys.stderr )
    args= parse_args()
    logging.getLogger().setLevel( args.verbosity )
    try:
        for file in args.file:
            with open( file, "r" ) as source:
                process_file( source, args )
        status= 0
    except Exception as e:
        logging.exception( e )
        status= 3
    logging.shutdown()
    sys.exit( status )

That's it.  Nothing more in the top-level main program.  The process_file function becomes a reusable "command" and something that can be tested independently.

"Hard Coding" Business Rules

See this: "Stop hard-coding business rules" in SD Times.

Here's what's exasperating: "Memo to developers: Stop hard-coding business rules into applications. Use business rules engines instead."

Business Rules Engines?  You mean Python?

It appears that they don't mean Python.

"Developers can use [a BPM suite or rules engine] and be more productive, so long as they don’t use C# or Java as a default for development".

I'm guessing that by "C# or Java" they mean "a programming language" and I would bet that Python is included in "bad" languages for development.

Python has all the simplicity and expressive power of a Domain-Specific Language (DSL) for business rules.

Don't hard-code business rules in Java.  Code them in an interpreted language like Python.

Also, don't be mislead by any claims that business analysts or (weirdly) users can somehow "code" business rules.  They can't (and mostly, they won't).  That's what SD Times wisely says "Developers".  That's how coding gets done.