Open Source in the News

Whitehouse.gov is publicly using open source tools.

See Boing Boing blog entry. Plus Huffington Post blog entry.

Most importantly, read this from O'Reilly.

Many places are using open source in stealth mode. Some even deny it. Ask your CIO what the policy on open source is, then check to see if you're using Apache. Often, this is an oops -- policy says "no", practice says "yes".

For a weird perspective on open source, read Binstock's Integration Watch piece in SD Times: From Open Source to Commercial Quality. "rigor is the quality often missing from OSS projects". "Often" missing? I guess Binstock travels in wider circles and sees more bad open source software than I do. The stuff I work with is very, very high quality. Python, Django, Sphinx, the LaTeX stack, PIL, Docutils -- all seem to be outstandingly good.

I guess "rigor" isn't an obvious tangible feature of the software. Indeed, I'm not sure what "commercial quality" means if open source lacks this, also.

All of the commercial software I've seen has been in-house developed stuff. Because there's so much of it, it must have these elusive "rigor" and "commercial quality" features that Binstock values so highly. Yet, the software is really bad: it barely works and they can't maintain it.

My experience is different from Binstock's. Also, most of the key points in his article are process points, not software quality issues. My experience is that the open source product exceeds commercial quality. Since there's no money to support a help desk or marketing or product ownership, the open source process doesn't offer all the features of a commercial operation.

Painful Python Import Lessons

Python's packages and modules are -- generally -- quite elegant.

They're relatively easy to manage. The __init__.py file (to make a module into a package) is very elegant. And stuff can be put into the __init__.py file to create a kind of top-level or header module in a larger package of modules.

To a limit.

It took hours, but I found the edge of the envelope. The hard way.

We have a package with about 10 distinct Django apps. Each Django app is -- itself -- a package. Nothing surprising or difficult here.

At first, just one of those apps used a couple of fancy security-related functions to assure that only certain people could see certain things in the view. It turns out that merely being logged in (and a member of the right group) isn't enough. We have some additional context choices that you must make.

The view functions wind up with a structure that looks like this.

@login_required
def someView( request, object_id, context_from_URL ):
   no_good = check_other_context( context_from_URL )
   if no_good is not None: return no_good
   still_no_good = check_session()
   if still_no_good is not None: return still_no_good
   # you get the idea

At first, just one app had this feature.

Then, it grew. Now several apps need to use check_session and check_other_context.

Where to Put The Common Code?

So, now we have the standard architectural problem of refactoring upwards. We need to move these functions somewhere accessible. It's above the original app, and into the package of apps.

The dumb, obvious choice is the package-level __init__.py file.

Why this is dumb isn't obvious -- at first. This file is implicitly imported. Doesn't seem like a bad thing. With one exception.

The settings.

If the settings file is in a package, and the package-level __init__.py file has any Django stuff in it -- any at all -- that stuff will be imported before your settings have finished being imported. Settings are loaded lazily -- as late as possible. However, in the process of loading settings, there are defaults, and Django may have to use those defaults in order to finish the import of your settings.

This leads to the weird situation that Django is clearly ignoring fundamental things like DATABASE_ENGINE and similar settings. You get the dummy database engine, Yet, a basic from django.conf import settings; print settings.DATABASE_ENGINE shows that you should have your expected database.

Moral Of the Story

Nothing with any Django imports can go into the package-level __init__.py files that may get brought in while importing settings.

Process Not Working -- Must Have More Process

After all, programmers are all lazy and stupid.

Got his complaint recently.

"Developers on a fairly routine basis check in code into the wrong branch."

Followed by a common form of the lazy and stupid complaint. "Someone should think about which branch is used for what and when." Clearly "someone" means the programmers and "should think about" means are stupid.

This was followed by the "more process will fix this process problem" litany of candidate solutions.

"Does CVS / Subversion have a knob which provides the functionality to

prevent developers from checking code into a branch?"

"Is there a canonical way to organize branches?" Really, this means something like what are the lazy, stupid programmers doing wrong?

Plus there where rhetorical non-questions to emphasize the lazy, stupid root cause. "Why is code merging so hard?" (Stupid.) "If code is properly done and not coupled, merging should be easy?" (Lazy; a better design would prevent this.) "Perhaps the developers don't understand the code and screw up the merge?" (Stupid.) "If the code is not coupled, understanding should be easy?" (Both Lazy and Stupid.)

Root Cause Analysis

The complaint is about process failure. Tools do not cause (or even contribute) to process failure. There are two possible contributions to process failure: the process and the people.

The process could be flawed. There could be no earthly way the programmers can locate the correct branch because (a) it doesn't exist when they need it or (b) no one told them which branch to use.

The people could be flawed. For whatever reason, they refuse to execute the process. Perhaps they know a better way, perhaps they're just being jerks.

Technical means will not solve either root cause problem. It will -- generally -- exacerbate it. If the process is broken, then attempting to create CVS / Subversion "controls" will end in expensive, elaborate failure. Either they can't be made to work, or (eventually) someone will realize that they don't actually solve the problem. On the other hand, if the people are broken, they'll just subvert the controls in more interesting, silly and convoluted ways.

My response -- at the time -- was not "analyze the root causes". When I first got this, I could only stare at it dumbfounded. My answer was "You're right, your developers are lazy and stupid. Good call. Add more process to overcome their laziness and stupidity."

After all, the questioner clearly knows -- for a fact -- that more process helps fix a broken organization. The questioner must be convinced that actually talking to people will never help.

The question was not "what can I do?" The question was "can I control these people through changes to CVS?" There's a clear presumption of "process not working -- must have more process."

The better response from me should have been. "Ask them what the problem is." I'll bet dollars against bent pins that no one tells them which branch to use in time to start work. I'll bet they're left guessing. Also, there's a small chance that these are off-shore developers and communication delays make it difficult to use the correct branch. There may be no work-orders, just informal email "communication" between on-shore and off-shore points-of-contact (and, perhaps, the points-of-contact aren't decision-makers.)

Bottom Line. If people can't make CVS work, someone needs to talk to them to find out why. Someone does not need to invent more process to control them.

Breaking into Agile

I had a recent conversation with some folks who were desperate to "processize" everything. They were asking about Scrum Master certification and what standards organizations define the "official" Scrum method.

Interestingly, I also saw a cool column in Better Software magazine, called "Scrumdamentalism" on the same basic question.

In my conversation, I referred them to the Agile Manifesto. My first point was that process often gets in the way of actual progress. Too much process focus lifts up "activity" in place of "accomplishment".

My second point, however, was that the Agile Manifesto and the Scrum method are responses to a larger problem. Looking for a process isn't an appropriate response to the problem.

The One True Scrum Quest

Claiming that there's one true Scrum method and everything else is "not scrum" is an easy mental habit. The question gets asked on Stack Overflow all the time. The questions are usually one of two kinds.

• What's the "official" or "best practice" Scrum method and how do I define a process that rigidly enforces this for my entire team of 20?
• We are doing our design/code/test/integration/release in a way that diverges from the "official" form in the Ken Schwaber and Mike Beedle book. Or it diverges from the Eclipse version. Or it diverges from the Control Chaos overview. Or the Mountain Goat version. Or the C2 Wiki version. Or this version. Is it okay to diverge from the "standard"?

Sigh. The point of Agile is that we should value "Individuals and interactions over processes and tools". The quest for "One True Scrum" specifically elevates the process above the people.

In The Real World

The biggest issue is that the Agile Manifesto is really a response to some fundamental truths about software development.

In management fantasy world, a "project" as a fixed, definite, limited, clearly articulated scope. From this fixed scope, we can then document "all" the requirements (business and technical). This requirements document is (a) testable against the scope, (b) necessary for all further work and (c) sufficient for design, code, test and transition to production. That's not all. And -- in order to make a point later on -- I'll continue to enumerate the fantasies. The fantasy continues that someone can create a "high-level design" or "specification" that is (a) testable against the requirements, (b) necessary for all further work and (c) sufficient to code, test and transition to production. We can then throw this specification over the transom into another room where a "coder" will "cut code" that matches the specification. The code production happens at a fixed, knowable rate with only small random variation based on risk of illness. The testing, similarly, can be meticulously scheduled and will happen precisely as planned. Most "real-world" (management fantasy) projects do not leave any time for rework after testing -- because rework won't happen. If it won't happen, why test? Finally, there will be no technology transfer issues because putting a freshly-written program into production is the same as installing a game from a DVD.

Managers like to preface things with "In The Real World". As in "In The Real World we need to know how long it will take you to write this."

The "in the real world" speech always means "In My Management Fantasy Land." The reason it's always a fantastic speech is because software development involves the unknowable. I'm not talking about some variable in a formula with a value that's currently unknown. I'm talking about predicting the unknowable future.

The Agile Response to Reality

In the Real real world, software development is extraordinarily hard.

Consider this: the computer clock runs in 100-nanosecond increments (1.0E-7). We expect an application to run 24x7x0.999 = 6.04E5 seconds. That's from 100-nano to half-million: about 12 orders of magnitude to keep in one's head.

Consider this: storage in a largish application may span almost a terabyte, (1.0E12). From bytes to terabytes: about 12 orders of magnitude to keep in one's head.

Consider this: a web application written in a powerful framework (Django) requires one to know the following languages and frameworks. Shell script, Apache Config, Python, Django Templates, SQL, HTML, CSS, Javascript, HTTP (the protocol is it's own language), plus the terminology of the problem domain. That's 9 distinct languages. We also have the OS, TCP/IP Apache, mod_wsgi, Django, Python, browser and our application as distinct frameworks. That's 8 distinct framework API's to keep in one's head.

Consider this: the users can't easily articulate their problem. The business analyst is trying to capture enough information to characterize the problem. The users, the analyst, the project manager (and others outside the team) all have recommendations for a solution, polluting the problem description with "solution speak" that's only adds confusion.

In the Management Fantasy "Real World", this is all knowable and simple. In the Real Real World, this is rather hard.

Adapting to Reality

Once we've recognized that software development is hard, we have several responses.

1. Deny. Claim that software developers are either lazy or stupid (or both). Give them pep-talks that begin "in the real world" and hope that they cough up the required estimates because they're motivated by being told that software development "in the real world" isn't all that hard.
2. Processize(tm). Claim that software development is a process that can be specified to a level where even lazy, stupid programmers can step through the process and create consistent results.
3. Adapt. Adapting to the complexity of software development requires asking, "what -- if anything -- expedites software development?"

What Do We Need to Succeed?

There are essentially two domains of knowledge required to create software: the problem domain and the solution domain.

• Problem Domain. This is the "business rules", the "scope", the "requirements", the "purpose", etc. We have the features and functions. What the software does. The value it creates. The "what", "who", "where", "when" and "why".
• Solution Domain. This is the technology that makes it go. The time and space dimensions (all 12 orders of magnitude in each dimension), all the languages and all the frameworks. The "how".

The issue is this:

We don't start out with complete knowledge of problem and solution.

At the start of the project -- when we're asked to predict the future -- we can never know the whole problem, nor can we ever know the whole solution we're about to try and build.

What we need is this:

Put Problem Domain and Solution Domain knowledge into one person's head.

The question then becomes "Who's head?"

We have two choices:

• Non-Programmers. We can try to teach the various non-programmers all the solution domain stuff. We can make the project manager, business analyst, end-users, executive sponsor -- everyone -- into programmers so that they have problem domain and solution domain knowledge.
• Programmers. We can try to impart the problem domain knowledge on the programmers. If we're seriously going to do this, we need to remove the space between programmer and problem.

That's the core of the Agile Response: Close the gap between Problem Domain and Solution Domain by letting programmers understand the problem.

The Bowl of Bananas Solution(tm)

"But wait", managers like to say, "in the real world, we can't just let you play around until you claim you're done. We have to monitor your activity to make sure that you're making 'progress' toward a 'solution'."

In the Real real world, you can't define the "problem", much less test whether anything is -- or is not -- a solution. I could hand most managers a bowl of bananas and they would not be able to point to any test procedure that would determine if the bowl of bananas solves or fails to solve the user's problems.

Most project scope documents, requirements documents, specifications, designs, etc., require extensive tacit problem domain knowledge to interpret them. Given a bowl of bananas, the best that we can do is say "we still have the problem, so this isn't a solution." Our scope statements and requirements and test procedures all make so many assumptions about the problem and the solution that we can't even figure out how evaluate an out-of-the-box response -- like a bowl of bananas.

In the Real real world, management in organization A demands that information be kept in a one database. Management organization B has a separate database for reasons mired in historical animosity and territorial scent-marking. Management in yet another organization wants them "unified" or "reconciled" and demands that someone manually put the data into spreadsheets. This morphs into requirements for a new application "system" to unify this data, making the results look like poorly-design spreadsheets. This morphs into a multi-year project to create a "framework" for data integration that maintains the poorly-designed spreadsheet as part of the "solution".

A quick SQL script to move data from A to B (or B to A) is the bowl-0f-bananas solution. It cannot be evaluated (or even considered) because it isn't a framework, system or application as specified in the scope document for the data integration framework.

This is the problem domain knowledge issue. It's so hard to define the problem, that we can't trust the executive sponsor, the program office, the project managers, the business analysts or anyone to characterize the problem for the developers.

The problem domain knowledge is so important that we need to allow programmers to interact with users so that both the problem and the solution wind up in the programmer's head.

Unit Test Naming [Updated]

Just stumbled across several blog postings on unit test naming.

Essentially the TestCase will name the fixture. That's pretty easy to understand.

The cool part is this: each test method is a two-part clause: condition_"should"_result or "when"_condition_"then"_result.

See https://wiki.openmrs.org/display/docs/Unit+Testing+With+at-should+Annotation,

Or possibly "method_state_behavior".

See http://osherove.com/blog/2005/4/3/naming-standards-for-unit-tests.html

What a handy way to organize test cases. Only took me four years to figure out how important this kind of thing is.

[Updated to follow moved links.]

Django Capacity Planning -- Reading the Meta Model

I find that some people spend way too much time doing "meta" programming. I prefer to use someone's framework rather than (a) write my own or (b) extend theirs. I prefer to learn their features (and quirks).

Having disclaimed an interest in meta programming, I do have to participate in capacity planning.

Capacity planning, generally, means canvassing applications to track down disk storage requirements.

Back In The Day

Back in the day, when we wrote SQL by hand, we were expected to carefully plan all our table and index use down to the kilobyte. I used to have really sophisticated spreadsheets for estimating -- to the byte -- Oracle storage requirements.

Since then, the price of storage has fallen so far that I no longer have to spend a lot of time carefully modelling the byte-by-byte storage allocation. The price has fallen so fast that some people still spend way more time on this than it deserves.

Django ORM

The Django ORM obscures the physical database design. This is a good thing.

For capacity planning purposes, however, it would be good to know row sizes so that we can multiply by expected number of rows and cough out a planned size.

Here's some meta-data programming to extract Table and Column information for the purposes of size estimation.

import sys
from django.conf import settings
from django.db.models.base import ModelBase

class Table( object ):
   def __init__( self, name, comment="" ):
       self.name= name
       self.comment= comment
       self.columns= {}
   def add( self, column ):
       self.columns[column.name]= column
   def row_size( self ):
       return sum( self.columns[c].size for c in self.columns ) + 1*len(self.columns)

class Column( object ):
   def __init__( self, name, type, size ):
       self.name= name
       self.type= type
       self.size= size

sizes = {
   'integer': 4,
   'bool': 1,
   'datetime': 32,
   'text': 255,
   'smallint unsigned': 2,
   'date': 24,
   'real': 8,
   'integer unsigned': 4,
   'decimal': 40,
}
def get_size( db_type, max_length ):
   if max_length is not None:
       return max_length
   return sizes[db_type]

def get_schema():
   tables = {}
   for app in settings.INSTALLED_APPS:
       print app
       try:
           __import__( app + ".models" )
           mod= sys.modules[app + ".models"]
           if mod.__doc__ is not None:
               print mod.__doc__.splitlines()[:1]
           for name in mod.__dict__:
               obj = mod.__dict__[name]
               if isinstance( obj, ModelBase ):
                   t = Table( obj._meta.db_table, obj.__doc__ )
                   for fld in obj._meta.fields:
                       c = Column( fld.attname, fld.db_type(), get_size(fld.db_type(), fld.max_length) )
                       t.add( c )
                   tables[t.name]= t
       except AttributeError, e:
           print e
   return tables

if __name__ == "__main__":
   tables = get_schema()
   for t in tables:
       print t, tables[t].row_size()

This shows how we can get table and column information without too much pain. This will report an estimated row size for each DB table that's reasonably close.

You'll have to add storage for indexes, also. Further, many databases leave free space within each physical block, making the actual database much larger than the raw data.

Finally, you'll need extra storage for non-database files, logs and backups.

Unit Testing in C

I haven't written new C code since the turn of the millennium. Since then it's been almost all Java and Python. Along with Java and Python come JUnit and Python's unittest module.

I've grown completely dependent on unit testing.

I'm looking at some C code, and I want a unit testing framework. For pure C, I can find things like CuTest and CUnit. The documentation makes them look kind of shabby. Until I remembered what a simplistic language C is. Considering what they're working with, they're actually very cool.

I found a helpful posting on C++ unit testing tools. It provided some insight into C++. But this application is pure C.

I'm interested in replacing the shell script in CuTest with a Python application that does the same basic job. That's -- perhaps -- a low-value add-on. Perhaps I should look at CUnit and stay away from replacing the CuTest shell script with something a bit easier to maintain.