S.Lott-Software Architect

Apache Log Parsing

2012-01-26T08:00:00.000-05:00

How much do I love Python? Consider this little snippet that parses Apache logs.


import re
from collections import defaultdict, named tuple

format_pat= re.compile( 
    r"(?P<host>[\d\.]+)\s" 
    r"(?P<identity>\S*)\s" 
    r"(?P<user>\S*)\s"
    r"\[(?P<time>.*?)\]\s"
    r'"(?P<request>.*?)"\s'
    r"(?P<status>\d+)\s"
    r"(?P<bytes>\S*)\s"
    r'"(?P<referer>.*?)"\s' # [SIC]
    r'"(?P<user_agent>.*?)"\s*' 
)

Access = namedtuple('Access',
    ['host', 'identity', 'user', 'time', 'request',
    'status', 'bytes', 'referer', 'user_agent'] )

def access_iter( source_iter ):
    for log in source_iter:
        for line in (l.rstrip() for l in log):
            match= format_pat.match(line)
            if match:
                yield Access( **match.groupdict() )

That's about it. The access log rows are now first-class Access-class objects that can be processed pleasantly by high-level Python applications.

Cool things.

The adjacent string concatenation means that the regular expression can be broken up into bits to make it readable.
When the named tuple attributes match the regular expression names, we can trivially turn the match.groupdict() into a named tuple.
By using a generator, the other parts of the application can simply loop through the results without tying up memory to create vast intermediate structures.

A couple of years back, a sysadmin was trying to justify spending money on a log analyzer product. I suggested they (at the very least) get an open source log analyzer.

I also suggested that they learn Python and save themselves the pain of working with a (potentially) complex tool. Given this as a common library module, log analysis applications are remarkably easy to write.

Building Skills in Programming

2012-01-24T08:00:00.001-05:00

I've revised (and streamlined) my Building Skills in Programming book.

The 2.6.2. edition will simply replace the 2.6.1. edition, leading to the possibility of broken bookmarks because of the changes.

Currently, the non-programmer book accounts for under 10% hits on the http://www.itmaybeaback.com/book site. Consequently, I'm not very worried about the breakage. I know someone will hate me for messing with the content just as they were starting to understand it.

I'm indebted to all my readers for the numerous suggestions, corrections and complements that I've received. In order to simplify the correction process, I've put the source onto SourceForge. See the Programming Book-Python 2.6 project.

The next step will be to add a PayPal donations button.

And... If I can get the PDF into really good shape, I may post it on Lulu for folks who really want a hardcopy.

Traffic

For a random 2-day period, the usage looks like this:

246 distinct "users" (really IP addresses).

{'html only': 161,
('both', 'oodesign-java-2.1'): 9,
('both', 'oodesign-python-2.1'): 5,
('both', 'programming-2.6'): 3,
('both', 'python-2.6'): 7,
('pdf only', 'oodesign-java-2.1'): 21,
('pdf only', 'oodesign-python-2.1'): 14,
('pdf only', 'programming-2.6'): 11,
('pdf only', 'python-2.6'): 37}

The "both" is a count of users reading HTML as well as the identified PDF editions.
For the "html only" and "both" users, there's a detailed list of particular books and sections. Too large and boring to repeat here.

One interesting part is this detail:

82.128.23.63 {'programming-2.6': 37, 'oodesign-java-2.1': 28, 'python-2.6': 37, 'oodesign-python-2.1': 24}

Apparently, Nigeria needs a lot of copies of the PDF. I think I might want to block them, because this can't be anything sensible except endless polling by some botnet script.

Since we don't drop off cookies, we can't really identify user sessions. Maybe in the future, I'll wrap the static content download with a simple WSGI application to drop off and collect cookies to track users instead of IP addresses.

Python 2.7 CSV files with Unicode Characters

2012-01-19T08:00:00.000-05:00

The csv module in Python 2.7 is more-or-less hard-wired to work with ASCII and only ASCII.

Sadly, we're often confronted with CSV files that include Unicode characters. There are numerous Stack Overflow questions on this topic. http://stackoverflow.com/search?q=python+csv+unicode

What to do? Since csv is married to seeing ASCII/bytes, we must explicitly decode the column values.

One solution is to wrap csv.DictReader, something like the following. We need to decode each individual column before attempting to do anything with value.

class UnicodeDictReader( object ):
    def __init__( self, *args, **kw ):
        self.encoding= kw.pop('encoding', 'mac_roman')
        self.reader= csv.DictReader( *args, **kw )
    def __iter__( self ):
        decode= codecs.getdecoder( self.encoding )
        for row in self.reader:
            t= dict( (k,decode(row[k])[0]) for k in row )
            yield t

This new object is an iterable which contains a DictReader. We could subclass DictReader, also.

The use case, then, becomes something simple like this.

with open("some.csv","rU") as source:
    rdr= UnicodeDictReader( source )
    for row in rdr:
        # process the row

We can now get Unicode characters from a CSV file.

Python 3.2 CSV Module -- Very, very nice

2012-01-17T08:00:00.000-05:00

A common (and small) task is reformatting a file that's in some variant of CSV. It could be a SQL database extract, or an export from an application that works well with CSV files.

In Python 2.x, a CSV file with Unicode was a bit of a problem. The CSV module isn't happy with Unicode. The documentation is quite clear that many files need to be opened with a mode of 'rb' to correctly handle Windows line-endings.

Because of this, a CSV file with Unicode required using an explicit decoder on the individual columns (not the line as a whole!)

But with Python 3.2, that's all behind us.

Here's something I did recently. The file has six columns that are relevant. One of them (the "NOTE") column has a big block of text with details buried inside using a kind of RST markup. The data might be three lines with a value like this "words words\n:budget: 1500\nwords words".

The file is UTF-8, and the words have non-ASCII unicode characters randomly through it.

 
def details( source ):
    relevant = ( "TASK", "FOLDER", "CONTEXT", "PRIORITY", "STAR", )
    parse= "NOTE"
    data_pat= re.compile( r"^:(\w+):\s*(.*)\s*$" )
    rdr= csv.DictReader( source )
    for row in rdr:
        txt= row[parse]
        lines= ( data_pat.match(l) for l in txt.splitlines() )
        matches= ( m.groups() for m in lines if m )
        result= dict( (k, row[k]) for k in relevant) 
        result.update( dict(matches) )
        yield result

How much do I love Python? Let me count the ways.

The assignment of lines on line 8 was fun. The "NOTE" column, in row[parse], contains the extra fields. They'll be on a separate line with the :word:value format as shown in the data_pat pattern. We create a generator which will split the text field into lines and apply the pattern to each line.
The assignment to matches on line 9 was equally fun. If the matches generator produced a match object, the lines generator will gather the two groups form the line.
The assignment to result creates a dictionary from the relevant columns.
The second assignment to result updates this dictionary with data parsed out of the "NOTE" column.

That makes it quite pleasant (and fast) to process an extract file, reformatting a "big blob of text" into individual columns.

The rest of the app boils down to this.

def rewrite( input, target=sys.stdout ):
    with io.open(input, 'r', encoding='UTF-8') as source:
        data= list( details( source ) )
    headers= set( k for row in data for k in row  )
    wtr= csv.DictWriter( target, sorted(headers) )
    wtr.writeheader( )
    wtr.writerows( data )

This gathers the raw data into a big old sequence in memory, andthen writes that big old sequence back out to a file. If we knew the headers buried in the "NOTE" field, we could do the entire thing in a single pass just using generators.

We have to explicitly provide the encoding because the file was created via a download and the encoding isn't properly set on the client machine. The important thing is that we can do this when it's necessary. And we no longer have to explicitly decode fields.

Since we don't know the headers in the "NOTE" field, we're forced to create the headers set by examining each row dictionary for it's keys.

Multiprocessing and Shared Objects [Revised]

2012-01-12T09:18:00.000-05:00

Read this: Shared Counter with Python Multiprocessing.

Brilliant. Thank you for this.

Too many of the questions on StackOverflow that include multi-threading are better approached as multi-processing. In Linux, there are times when all threads of a single process are stopped while the process (as a whole) waits for system services to complete. It's a consequence of the way select and poll work. An example of the kind of sophisticated design required to avoid this can be found here. Most I/O-intensive applications should be done via multi processing, not multi threading.

And. The kind of shared objects that multi threading allows are often rare and require locks.

So, simplify your life. When you hear about "threads", replace the word with "processes" and move on. The implementation will be much nicer.

The standard gripe is that process creation is so expensive, and thread creation is relatively cheap. All true. That's why folks use process pools: to amortize the creation cost over a long period of operation.

Innovation is the punctuation at the end of a string of failures

2012-01-10T08:00:00.000-05:00

Read this in Forbes: "Innovation's Return on Failure: ROF".

Also, this: "The Necessity of Failure in Innovation (+ more on CDOs)".

This, too: "Why innovation efforts fail".

While we're at it: "Accepting Failure is Key to Good Overall Returns on High-Risk Development Programs".

I can't say enough about the value of "failure". The big issue here is the label.

A project with a grand scope and strategic vision gets changed to make it smaller and more focused. Did it "fail" to deliver on the original requirements? Or did someone learn that the original grand scope was wrong?

A project that changes isn't failure. It's just lessons learned. Canceling, re-scoping, de-scoping, and otherwise modifying a project is what innovation looks like. It should not be counted as a "failure".

A project "Death March" occurs because failure is not an option, and change will be labeled as failure.

Self-Direction, Mastery, Purpose

2012-01-05T08:00:00.000-05:00

Watch this: http://www.youtube.com/watch?v=u6XAPnuFjJc

Brilliant summary of what really motivates people.

The most important advice: provide a sense of purpose and get out of people's way so that they can do the right thing.

Micromanagement, incentives, annual performance reviews and the like aren't as useful as providing the sense of purpose, the opportunity for mastery and the freedom of self-direction.

Epic indictment of Waterfall Methods

2012-01-03T08:00:00.000-05:00

Saw this recently.

Why do most gov websites look like they were created by someone's 10 year old nephew yet cost millions to make?
— Steve Dekorte (@stevedekorte) December 29, 2011

I think this aptly summarizes the results of a waterfall methodology.

You wrote a lot of requirements, not fully understanding the actors or their use cases.
Your vendor implemented those requirements because they were contractual obligations not because they created value for the actors.

The government's not the only offender. They're just more visible and more bound up in a legally-mandated purchasing cycle that makes the waterfall desirable and more Agile methods undesirable.

LANGSEC: Language-theoretic Security

2011-12-29T08:00:00.000-05:00

Wow. Just wow. See "LANGSEC explained in a few slogans".

Short, easy-to-grasp explanation of why complex protocols create new problems.

I'm happy with REST and the stack of stuff under it (HTTP, TCP/IP, etc.)

Once upon a time (2001), I invented by own version of a RESTful protocol outside HTTP. That was cool. Very simple, and very fast. But relatively inflexible. The syntax was more like FTP and SMTP; the semantics where mostly just CRUD rules and RESTful state transfers.

I was way too dumb to leverage HTTP methods and the genius of a URI.

Technology Refresh

2011-12-27T08:00:00.000-05:00

I've been refurbishing an older project -- written in 2008. Probably with Django 1.0.1. Certainly with Python 2.5.

The Django 1.3 release has been around since March. The change underscored the importance of technology refresh.

The best part was to delete code. There were two significant reasons.

The testserver command allowed me to eliminate a bunch of low-value test harness. Without this command, we had to create our own test database, start a server, run integration tests, and then kill the server. With this command, we simply start and kill the server.
The RESTful web services can be securely integrated into the main web application. A simple piece of middleware can authenticate requests based on headers containing ForgeRock OpenAM tokens. It may be that this was always a feature of Django, but over the last few years, we've figured out how to exploit it with simple middleware.

Few things are better than removing old code and replacing it with code written (and tested) by someone else.

In addition to the deletes, we also rearranged some of the dependencies. We had (incorrectly) thought of the Django project as somehow central or essential. It turns out that a bunch of other Python libraries were actually core to the application. The Django web presentation was just one of the sensible use cases. A suite of command-line apps could also be built around the underlying libraries.

In addition to this cleanup, we also replaced the documentation with a new Sphinx project. The project originally used Epydoc markup. This meant that every single docstring had to be rewritten to use RST markup. The upside of this is that we corrected numerous errors.

There Was Pain

This wasn't without some pain.

Was the cost worth the effort? That's the real question here.

I think that many IT managers adopt a silly "If it ain't broke, don't fix it" policy that focuses on short-term cost and short-term value. It ignores long-term accrual from even tiny short-term cost savings.

Here's are two important lessons.

Money saved today is saved forever.
Savings accrue. Forever.

It's important to avoid short-term thinking about cost and benefit.

Color Schemes

2011-12-20T08:00:00.000-05:00

I worked with this a few years ago to tweak up some web pages.

http://colorschemedesigner.com/

I just rediscovered it. It's a cool toy. You get some colors that all "go" together. If you're careful with your .CSS definitions, you give people this page and let them fuss around until their positively silly with color palettes.

Good Summary of Bad Security Assumptions

2011-12-15T08:00:00.000-05:00

This isn't the OWASP Top 10 list, but it's still very handy.

Top 10 Dumb Computer Security Notions.

I'm particularly fond of the "security can't be perfect; since it can't be perfect, why bother?" approach.

One other notion that amuses me is the silliness of changing a password every 90 days. The argument is that "it's harder to hit a moving target". That's obviously false. A good rainbow table and a bad password without salt can be broken in about half an hour. There's no "moving target" here. At 30 minutes to crack a password, the only way the target can appear to move is making every password a 1-time-only password based on some kind of external source (like a token generator.)

The need for ping

2011-12-13T08:00:00.000-05:00

Years ago, when designing an interface to a vendor's web services, I did the following. This isn't a genius move, but it's worth emphasizing how important it is. And what's most important isn't technical.

I built a simple spike solution to access their service.
I morphed this into a "sanity check" to be sure that their service really was working. Mostly, I cleaned up the code so that it was testable and deliverable without embarrassment.
I morphed this into a "diagnostic tool" to bypass the higher-levels of the application and simply access the vendor (and optionally dump the results) to help determine what wasn't work. This involved adding the dump option to the sanity check and renaming the command-line application.
I morphed this into a "credentials check and diagnostic tool". This was -- ahem -- merely taking the hard-wired credentials out of the application. Yes. The first versions had hard-wired credentials.

That brings us to the version in use today. The "vendor ping" application.

The default behavior is a credentials check.

One optional behavior is to dump the interface details.

Another optional behavior is to allow selection among a small number of simple interactions just to be sure things are working.

Unplanned Work

What's important here isn't that I did all this. What's important is that the deliverables, user stories and project plans didn't include this little nugget of high-value goodness.

It gets run fairly frequently in crunch situations. The actor in the story ("As system admin...") is rarely considered as a first-class user of the application. Yet, the admin is a first-class user, and needs to have proper user stories for confirming that the application is working properly.

Statically Typed Language Nonsense

2011-12-09T08:00:00.000-05:00

Read this: "Here Comes Functional Programming" by Larry O'Brien in SD Times.

people who should know better continue to assert that statically typed languages are "safer, because the compiler can catch errors that otherwise wouldn't show up until runtime." While it's true a statically typed language can detect that you've assigned a string to a double without running your code, no type system is so strict that it can substitute for a test suite, and if you have a test suite, type-assignment errors are discovered and precisely diagnosed with little difficulty.

Thank you. A language like Python, which lacks static type declarations for variables, is not evil or an accident waiting to happen.

The article is about functional languages. But the static declaration statement is universally true.

I'm Confused by this Marketing Ploy

2011-12-06T08:00:00.000-05:00

Got this a few weeks back.

My job is to persuade bloggers to link to our site.

I really love my job! We have a friendly team and good management, but unfortunately I have no idea how to convince a blogger to link to us, I'm afraid I might lose my job because of it :(

And that is why, instead of sending letters to thousands of different blogs, I am reading yours.

Couldn't parse it.

It seems to be a calculated Pity Ploy. "I'm afraid I might lose my job...I am reading your [blog]."

The product seemed cool enough. The pitch, however, was too sketchy for me.

Agile "Religion" Issues

2011-12-01T08:00:00.000-05:00

See this Limitations of Agile Software Development and this The Agile "Religion" -- What?. What's important is that the limitations of Agile are not limitations. They're (mostly) intentional roadblocks to Agile.

Looking for "limitations" in the Agile approach misses the point of Agile in several important ways.
The most important problem with this list of "limitations" is that five of the six issues are simply anti-Agile positions that a company can take.

In addition to being anti-Agile, a company can be anti-Test Driven Development. They can be Anti-Continuous Integration. They can be Anti-NoSQL. There are lots of steps a company can take to subvert any given development practice. Taking a step against a practice does not reveal a limitation.

"1. A team of stars... it takes more than the average Joe to achieve agility". This is not a specific step against agility. I chalk this up to a project manager who really likes autocratic rule. It's also possible that this is from a project manager that's deeply misanthropic. Either way, the underlying assumption is that developers are somehow too dumb or disorganized to be trusted.

Agile only requires working toward a common goal. I can't see how a project manager is an essential feature of working toward a common goal. A manager may make things more clear or more efficient, but that's all. Indeed, the "clarity" issue is emphasized in most Agile methods: a "Scrum Master" is part of the team specifically to foster clarity of purpose.

Further, some Agile methods require a Product Owner to clarify the team's direction.

"A team of stars" is emphatically not required. The experience of folks working in Agile environments confirms this. Real, working Agile teams really really are average.

"2. Fit with organizational culture". This has nothing to do with Agile methods. This is just a sweeping (and true) generalization about organizations. An organization that refuses autonomy and refuses flexibility can't use Agile methods. An organization that refuses to create a "Big Design Up Front" can't use a traditional waterfall method and must use Agile methods.

Organizational fit is not a limitation of Agile. It's just a fact about people.

"3. Small team...Assuming that large projects tend to require large teams, this restriction naturally extends to project size."

The assumption simply contradicts Agile principles. It's not a "limitation" at all. Large projects (with large numbers of people) have a number of smaller teams. I've seen projects with over a dozen parallel Agile teams. This means that in addition to a dozen daily scrums, there's also a scrum-of-scrums by the scrum masters.

Throwing out the small team isn't a limitation of Agile. It's a failure to understand Agile. A project with many small teams works quite well. It's not "religion". It's experience.

A single large team has been shown (for the last few decades) to be expensive and risky.

"4. Collocated team...We can easily think of a number of situations where this limitation prevents using agile:" These are not limitations of Agile, but outright refusals to follow Agile principles. Specifically:

"Office space organized by departments" is not a limitation of Agile. That's a symptom of an organization that refuses to be Agile. See #2 above; this indicates a bad fit with the culture. An organization that doesn't have space organized by department might have trouble executing a traditional waterfall method.
"Distributed environment" is not a limitation of Agile. Phones work. Skype works.
"Subcontracting... We have to acknowledge that there is no substitute for face-to-face". Actually, subcontracting is irrelevant. Further, subcontracting is not a synonym for a failure to be collocated. When subcontractors are located remotely, phones still work. Skype works better and is cheaper.

"5. Where’s my methodology?" This is hard to sort out, since it's full of errors. Essentially, this appears to be a claim that a well-defined, documented processes is somehow essential to software development. Experience over the last few decades is quite clear that the written processes and the work actually performed diverge a great deal. Most of the time, what people do is not documented, and the documented process has no bearing on what people actually do. A documented process -- in most cases -- appears irrelevant to the work actually done.

Agile is not chaos. It's a change in the rules to de-emphasize unthinking adherence to a plan and replace this with focus on working software. Well-organized software analysis, design, code and test still exist even without elaborately documented (and irrelevant) process definitions.

"6. Team ownership vs. individual accountability... how can we implement it since an organization’s performance-reward system assesses individual performance and rewards individuals, not teams...?" Again, the assumption ("performance-reward system assesses individual performance") is simply a rejection of Agile principles. It's not a limitation of Agile, it's an intentional step away from an Agile approach.

If an organization insists on individual performance metrics, see #2. The culture is simply antithetical to Agile. Agile still works; the organization, however, is taking active steps to subvert it.

Agile isn't a religion. It doesn't suffer from hidden or ignored "limitations".

"But did we question the assumption that Agile was indeed superior to traditional methodologies?"

The answer is "yes". A thousand times yes. The whole reason for Agile approaches is specifically and entirely because of folks questioning traditional methodologies. Traditional command-and-control methodologies have a long history of not working out well for software development. The Agile Manifesto is a result of examining the failures of traditional methods.

A traditional "waterfall" methodology works when there are few unknowns. Construction projects, for example, rarely have the kinds of unknowns that software development has. Construction usually involves well-known techniques applied to well-documented plans to produce a well-understood result. Software development rarely involves so many well-known details. Software development is 80% design and 20% construction. And the design part involves 80% learning something new and 20% applying experience.

Agile is not Snake Oil. It's not something to be taken on faith.

The Agile community exists for exactly one reason. Agile methods work.

Agile isn't a money-making product or service offering. Agile -- itself -- is free. Some folks try to leverage Agile techniques to sell supporting products or services, but Agile isn't an IBM or Oracle product. There are no "backers". There's no trail of money to see who profits from Agility.

Folks have been questioning "traditional" methodologies for years. Why? Because "traditional" waterfall methodologies are a crap-shoot. Sometimes they work and sometimes they don't work. The essential features of long term success are summarized in the Agile Manifesto. Well-run projects all seem to have certain common features; the features of well-run projects form the basis for the Agile methods.

The Value of Microsoft's Tools

2011-11-29T08:00:00.000-05:00

See Andrew Binstock's "Windows 8: Microsoft's Development Re-Do".

The costs of these migrations has been enormous and continues to accumulate...

I can only rub my hands with glee and engage in shameless "I Told You So" self-congratulations.

Only you can prevent being held hostage by Microsoft.

More than once, I've observed that a strategy of using only proprietary tools would be expensive and complex. And every time, the folks I was talking to trivialized my concerns as hardly worth considering.

I've seen orphaned software: it only compiles on an old version of Visual Studio. I've seen software orphaned so badly that it can only be compiled on one creaky old PC. The cost to convert was so astronomical that the customer preferred to hope for a product to arise somewhere in the marketplace. When no suitable product appeared over the decades, the problem reached palpable Pants On Fire (POF) levels of panic. All due to the hidden costs of Microsoft's tools.

I've even been told that VB is a terrible language, but Visual Studio makes it acceptable.

Justification of Project Staffing

2011-11-24T08:00:00.000-05:00

I really dislike being asked to plan a project. It's hard to predict the future accurately.

In spite of the future being -- well -- the future, and utterly unknowable, we still have to have the following kinds of discussions.

Me: "It's probably going to take a team of six."

Customer: "We don't really have the budget for that. You're going to have to provide a lot of justification for a team that big."

What's wrong with this picture? Let's enumerate.

Customer is paying me for my opinion based on my experience. If they want to provide me with the answers, I have a way to save them a lot of money. Write their own project plan with their own answers and leave me out of it.
I've already provided all the justification there is. I'm predicting the future here. Software projects are not simple Rate-Time-Distance fourth-grade math problems. They involve an unknown number of unknowns. I can't provide a "lot" of justification because there isn't any indisputable basis for the prediction.
I don't know the people. The customer -- typically -- hasn't hired them yet. Since I don't know them, I don't know how "productive" they'll be. They could hire a dozen n00bz who can't find their asses blindfolded even using both hands. Or. They could hire two singular geniuses who can knock the thing out in a weekend. Or. They could hire a half-dozen arrogant SOB's who refuse to follow my recommendations.
They're going to do whatever they want no matter what I say. Seriously. I could say "six". They could argue that I should rewrite the plan to say "four" without changing the effort and duration. Why ask me to change the plan? A customer can only do what they know to be the right thing.

Doing the Right Thing

Let's return to that last point. A customer project manager can only do what they absolutely know is the right thing. I can suggest all kinds of things. If they're too new, too different, too disturbing, they're going to get ignored.

Indeed, since people have such a huge Confirmation Bias, it's very, very hard to introduce anything new. A customer doesn't bring in consultants without having already sold the idea that a software development project is in the offing. They justify spending a few thousand on consulting by establishing some overall, ball-park, big-picture budget and showing that the consulting fees are just a small fraction of the overall.

As consultants, we have to guess this overall, ball-park, big-picture budget accurately, or the project will be shut down. If we guess too high, then the budget is out of control, or the scope isn't well-enough defined, or some other smell will stop all progress. If we guess too low, then we have to lard on additional work to get back to the original concept.

Architectures, components and techniques all have to meet expectations. A customer that isn't familiar with test drive development, for example, will have an endless supply of objections. "It's unproven." "We don't have the budget for all that testing." "We're more comfortable with our existing process."

The final trump card is the passive aggressive "I'll have to see the detailed justification." It means "Don't you dare." But it sounds just like passive acceptance.

Since project managers can only do what they know is right, they'll find lots of ways of subverting the new and unfamiliar.

If they don't like the architecture, the first glitch or delay or problem will immediately lead to a change in direction to yank out the new and replace it with the familiar.

If they don't like a component, they'll find numerous great reasons to rework that part of the project to remove the offending component.

If they don't like a technique (e.g., Code Walk Throughs) they'll subvert it. Either not schedule them. Or cancel them because there are "more important things to do." Or interrupt them to pull people out of them.

Overcoming the Confirmation Bias

I find the process of overcoming the confirmation bias to be tedious. Some people like the one-on-one "influencing" role. It takes patience and time to overcome the confirmation bias so that the customer is open to new ideas. I just don't have the patience. It's too much work to listen patiently to all the objections and slowly work through all the alternatives.

I've worked with folks who really relish this kind of thing. Endless one-on-one meetings. Lots of pre-meetings and post-meetings and reviews of drafts. I suppose it's rewarding. Sigh.

How to Learn

2011-11-22T08:00:00.000-05:00

A recent question.

i came up with two options.
1. building skills 1 (+ other references)... then algorithms & data
structures.... then your books 2 & 3

or

2. your three books 1,2 & 3... then algo & ds

kindly help me decide so i can start soon.

I have two pieces of advice.

First. Programming is a language skill. Just like English. If you can't get the English right, the odds of getting Python, Java, HTML or SQL right is considerably reduced. Please, please, please take more care in grammar, syntax and punctuation. Otherwise, your future as a programmer doesn't look very good. For example, the personal pronoun is spelled "I". In the 20th century, we spell out "and"; we stopped writing "&" as a stand-in for the Latin "et" centuries ago. Also, ellipses ("...") shouldn't be used except when eliding part of a quote. Clarity and precision actually matter.

Second, and more relevant, your two choices don't really amount to a significant difference. If you're waiting around for advice, you're wasting your time. Both sequences are good ideas. It's more important to get started than it is to carefully choose the precise and exact course of study. Just start doing something immediately.

Learning to program is a life-long exercise. There will always be more to learn. Start as soon as you can. The exact choices don't matter. Why? Because, eventually, you'll read all of those books plus many, many others.

Spend less time waiting for advice and more time studying.

More On Inheritance vs. Delegation

2011-11-17T08:00:00.000-05:00

Emphasis on the "More On" as in "Moron". This is a standard design error story. The issue is that inheritance happens along an "axis" or "dimension" where the subclasses are at different points along that axis. Multi-dimensional inheritance is an EPIC FAIL.

Context

Data warehouse processing can involve a fair amount of "big batch" programs. Loading 40,000 rows of econometric data in a single swoop, updating dimensions and loading facts, for example.

When you get data from customers and vendors, you have endless file-format problems. To assure that things will work, each of these big batch programs has at least two operating modes.

Validate. Go through all the motions. Except. Don't commit any changes to the database; don't make any filesystem changes. (i.e., write the new files, but don't do the final renames to make the files current.)
Load. Go through all the motions including a complete commit to the database and any filesystem changes.

Problem

What's the difference between the two modes? Clearly, one is a subclass of the other.

Load can be the superclass. The Validate subclass simply replaces the save methods stubs that do nothing.
Validate can be the superclass. The Load subclass simply implements the save method stubs with methods that do something useful.

Simple, right?

Wrong.

What Doesn't Work

This design has a smell. The smell is that we can't easily extend the overall processing to include an additional feature.

Why not?

This design has the persistence feature set as the inheritance axis or dimension. This is kind of limited. We really want a different feature set for inheritance.

Consider a Validate for two dimensions (Company and Time) that loads econometric facts. It has stub "save" methods.

We subclass the Validate to create the proper Load for these two dimensions and one fact. We replace the stub save methods with proper database commits.

After the actuaries think for a while, suddenly we have a file which includes an additional dimension (i.e., business location) or an additional fact (i.e., econometric data at a different level of granularity). What now? If we subclass Validate to add the dimension or fact, we have a problem. We have to repeat the Load subclass methods for the new, extended Load. Oops.

If we subclass Load to add the dimension or fact, we have a problem. We have to repeat the Validate stubs in the new extended Load to make it into a Validate. Oops.

Recognizing Delegation

It's difficult to predict inheritance vs. delegation design problems.

The hand-waving advice is to consider the essential features of the object. This isn't too helpful. Often, we're so focused on the database design that persistence seems essential.

Experience shows, however, that some things are not essential. Persistence, for example, is one of those things that should always be delegated.

Another thing that should always be delegated is the more general problem of representation: JSON, XML, etc., should rely on delegation since this is never essential. There's always another representation for data. Representation is always independent of the object's essential internal state changes.

Consequence

In my case, I've got about a dozen implementations using a clunky inheritance that had some copy-and-paste programming. Oops.

I'm trying to reduce that technical debt by rewriting each to be a proper delegation. With good unit test coverage, there's no real technical risk. Just tedious fixing the same mistake that I rushed into production twelve separate times.

Really. Colossally dumb.

VMware, VIX and PyVIX2

2011-10-25T08:00:00.000-04:00

The topic of VMware came up at my local 757 Python Users Group.

A common administrative need is to control VM farms. While there are a number of pointy-clicky GUI tools, VMware offers the VIX library to permit writing scripts to control VM's.

Here's some information we looked at recently on PyVIX2 and VMware.

The idea behind PyVIX2 is to provide a relatively simple Python binding to VIX. This, too, is a command-line interface, following on the heels of More Command-Line Goodness and Command-Line Applications.

The Agile "Religion" -- What?

2011-10-20T08:00:00.000-04:00

Received "it seems that software development has caught the agile religion. Personally, I have an issue w/ being unimodal."

What?

First. "agile religion". As in the deprecating statement: Agile is nothing more than a religion? As in Agile is nothing more than a vague religious practice with no tangible value to an organization? Interesting, I guess.

I'm assuming that the author did not read the Manifesto for Agile Software Development. Or--worse-- they read it and find that the four values (Individuals and interactions, Working software, Customer collaboration and Responding to change) are just of no tangible value.

That's alarming. Really. The alternative (processes and tools, comprehensive documentation, contract negotiation, and following a plan without regard to changes) seems like it's a recipe for cost, risk, low-value work and a cancelled project. Indeed, it seems like non-Agile project management is the best way to get to the fabled "Software Crisis" where lots of money gets spent but little of value gets created.

Further, it seems that all modifications of the classic waterfall method (e.g., spiral method as a prime example) specifically create "iterative, incremental" approaches to software development. That is, everything that's not a strict (brain-dead) waterfall has some elements of Agile.

This causes me to think that Agile isn't a religion. It causes me to think that Waterfall methods were a religious practice of no tangible value. All the methodology experiments over the last 15 years have been ways of introducing flexibility (agility, brains) into a foolishly inflexible methodology definition.

Indeed, it appears that the heavy-weigh waterfallish methods are an attempt to replace thinking with process. And it didn't work. So, we have to go back to the thinking part. Only, we call it Agile now.

Religious Wars.

Second. "agile religion" (again). As in methodology discussions are just religious wars? As in methodology discussions are just quibbling over no-value details? Some folks may get this impression of making a choice between Agile vs. Non-Agile methods. I think that those folks haven't actually had the opportunity to work from a prioritized backlog and build the most valuable part first. I think that someone who things Agile is just a religious war hasn't been allowed to fix a broken project plan based on lessons learned during the first release.

Agility.

Third. "unimodal". As in being exclusively Agile is bad? As in sometimes you need to have a rigid, unyielding process that sticks strictly to the schedule irrespective of changes which may occur? That doesn't seem rational.

Change happens. Forcing the inevitable changes to conform to some farcical schedule made up by people who didn't have all the details seems silly. Making contract negotiation the focal point of response to change seems like a waste of effort. Trying to document everything so completely that all possible changes are already accounted for seems impossible. And replacing change with a process that regulates change seems -- perhaps -- unhinged.

There were some links and some charts and graphs attached. I couldn't get past the two sentences above to see if there was, perhaps, something more to it. All I could do was respond with a request for clarification that didn't involve the trivialization of Agile methods. It doesn't seem sensible to try and remove the human element from software development.

I'll provide whatever follow-up clarification surfaces on this topic. It's interesting to see if the "agile religion" was misplaced, or if there are folks who think that responding to the messiness of real software development is a bad idea.

We tried the waterfall method. And it didn't work very well. Agile isn't a "religion". It's a simple acknowledgement that reality is messy.

More Command-Line Goodness

2011-10-13T08:00:00.000-04:00

In Command-Line Applications, we looked at a Python main-import switch which boiled down to this.

for file in args.file:
with open( file, "r" ) as source:
process_file( source, args )

The point was that each distinct file on the command-line was processed in a more-or-less uniform way by a single function that does the "real work" for that input file.

It turns out that we often have flat files which are spreadsheets or spreadsheet-like. Indeed, for some people (and some organizations) the spreadsheet is their preferred user interface. As I've said before,

Spreadsheets are the universal user interface. Everyone likes them, they're almost inescapable. And they work. There's no reason to attempt to replace the spreadsheet with a web page or a form or a desktop application. It's easier to cope with spreadsheet vagaries than to replace them.

They have problems, but they are surprisingly common.

Enter Stingray Reader. This is a small Python library to make it easy to have programs which read workbooks--collections of spreadsheets--or spreadsheet-like files with a degree of transparency.

And. It allows a clean command-line interface.

With a little care, we can reduce the main-import switch to something like this.

if __name__ == "__main__":

logging.basicConfig( stream=sys.stderr )

args= parse_args()

logging.getLogger().setLevel( args.verbosity )

builder= make_builder( args )

try:

for file in args:

with workbook.open_workbook( input ) as source:

process_workbook( source, builder )

status= 0

except Exception as e:

logging.exception( e )

status= 3

logging.shutdown()

sys.exit( status )

The bold lines are specific to workbook ("spreadsheet") processing. A "builder" creates application-specific Python objects from spreadsheet rows. The "workbook.open_workbook" is a function that builds a workbook reader based on the file name. It can handle a number of file types.

The process_workbook function is the "real work" function that handles a workbook of individual spreadsheets (or a spreadsheet-like file).

A smoothly operating, well-oiled engine for failure

2011-10-11T08:00:00.000-04:00

It occurs to me that much of "Big IT" creates a well-oiled organization that makes broken software seem acceptable. The breakage is wrapped in layers of finely-tuned process.

Consider a typical Enterprise Application. There's a help desk, ticket tracking, a user support organization that does "ad-hoc" processing, and a development organization to handle bug fixes and enhancement requests. All those people doing all that work.

Why?

If people need all that support, then the application is -- from a simplistic view -- broken.

The organization, however, has coped with the broken application by wrapping it in layers of people, process, tools, technology, management and funding. The end users have a problem, they call the help desk, and the machine kicks in to resolve their problem.

It is a given -- a going-in assumption -- a normal, standard expectation that any enterprise software is so broken that a huge organization will be essential for pressing forward. It is expected that good software cannot be built.

We're asked to help a client create a sophisticated plan for the New Enterprise App support organization. Planning this organization feels like planning for various kinds of known, predicted, expected failures. Failure is the expectation. Broken is the standard operating mode.

Consider a typical non-Enterprise Application. Let's say, the GNU C compiler. Or Python. Or Linux. An almost entirely volunteer organization, no help desk, no trouble tickets, no elaborate support organization plan. Yet. These products actually work flawlessly. They're not wrapped in a giant organization.

Why is the bar for acceptability so low for "Enterprise" applications? Why is this tolerated?

Command Line Applications

2011-10-06T08:00:00.000-04:00

I'm old -- I admit it -- and I feel that command-line applications are still very, very important. Linux, for example, is packed full of almost innumerable command-line applications. In some cases, the Linux GUI tools are specifically just wrappers around the underlying command-line applications.

For many types of high-volume data processing, command-line applications are essential.

I've seen command-line applications done very badly.

Overusing Main

When writing OO programs, it's absolutely essential that the OS interface (public static void main in Java or the if __name__ == "__main__": block in Python) does as little as possible.

A good command-line program has the underlying tasks or actions defined in some easy-to-work with class hierarchy built on the Command design pattern. The actual main program part does just a few things: gather the relevant environment variables, parse command-line options and arguments, identify the configuration files, and initiate the appropriate commands. Nothing application-specific.

When the main method does application-specific work, that application functionality is buried in a method that's particularly hard to reuse. It's important to keep the application functionality away from the OS interface.

I'm finding that main programs should look something like this:

if __name__ == "__main__":
logging.basicConfig( stream=sys.stderr )
args= parse_args()
logging.getLogger().setLevel( args.verbosity )
try:
for file in args.file:
with open( file, "r" ) as source:
process_file( source, args )
status= 0
except Exception as e:
logging.exception( e )
status= 3
logging.shutdown()
sys.exit( status )

That's it. Nothing more in the top-level main program. The process_file function becomes a reusable "command" and something that can be tested independently.

"Hard Coding" Business Rules

2011-10-04T08:00:00.000-04:00

See this: "Stop hard-coding business rules" in SD Times.

Here's what's exasperating: "Memo to developers: Stop hard-coding business rules into applications. Use business rules engines instead."

Business Rules Engines? You mean Python?

It appears that they don't mean Python.

"Developers can use [a BPM suite or rules engine] and be more productive, so long as they don’t use C# or Java as a default for development".

I'm guessing that by "C# or Java" they mean "a programming language" and I would bet that Python is included in "bad" languages for development.

Python has all the simplicity and expressive power of a Domain-Specific Language (DSL) for business rules.

Don't hard-code business rules in Java. Code them in an interpreted language like Python.

Also, don't be mislead by any claims that business analysts or (weirdly) users can somehow "code" business rules. They can't (and mostly, they won't). That's what SD Times wisely says "Developers". That's how coding gets done.

The Politics of Estimating

2011-09-29T08:00:00.000-04:00

Computerworld, September 12, page 10.

MicroburstIT Disasters
According to a study of 1,471 big IT projects, 15% turn out to be money pits, with cost overruns averaging 200%.

How is this a politically-charged statement? We hear this kind of thing all the time.

As developers (or project leaders) we're failing to execute.

Right?

Hogwash.

An "overrun" is isomorphic to "badly justified" or "badly budgeted" or "oversold to executive sponsors".

An "overrun" can be a failure to use (or even permit) realistic estimates. It may reflect an executive sponsor restating objectives to make the project large enough to justify it. An overrun can mean anything.

Calling it an overrun is a way to label it as "failure to execute".

I prefer to call it a failure of vision (or whatever it is executive sponsors do). It's more likely to be an under-estimate than it is to be an over-run.

After all, how many times have we been told to reduce an estimate? How many times have folks gotten their "attaboys" and "attagirls" for "sharpening their pencils" and reducing the proposal to the smallest amount that the customer would approve?

Threads and I/O

2011-09-27T09:22:00.000-04:00

Threads don't promote concurrent I/O.

Kernel threads may. Most of us write user threads. Here's a great summary under Thread (Computer Science).

However, the use of blocking system calls in user threads (as opposed to kernel threads) or fibers can be problematic. If a user thread or a fiber performs a system call that blocks, the other user threads and fibers in the process are unable to run until the system call returns. A typical example of this problem is when performing I/O: most programs are written to perform I/O synchronously. When an I/O operation is initiated, a system call is made, and does not return until the I/O operation has been completed. In the intervening period, the entire process is "blocked" by the kernel and cannot run, which starves other user threads and fibers in the same process from executing.

The point is this.

If it involves I/O, multi-threading doesn't help. Processes do.

If it involves computation, multi-threading may help.

"Strict" Unit Testing -- Everything In Isolation Is Too Much Work

2011-09-22T08:00:00.000-04:00

Folks like to claim that unit testing absolutely requires each class be tested in isolation using mocks for all dependencies. This is a noble aspiration, but doesn't work out perfectly well in Python.

First, "unit" is intentionally vague. It could be a class, a function, a module or a package. It's "unit" of code. Anything could be considered a "unit".

Second--and more important--the extensive mocking isn't fully appropriate for Python programming. Mocks are very helpful in statically-typed languages where you must be very fussy about assuring that all of the interface definitions are carefully matched up properly.

In Python, duck typing allows a mock to be defined quite trivially. A mock library isn't terribly helpful, since it doesn't reduce the code volume or complexity in any meaningful way.

Dependencies without Injection

The larger issue with trying to unit test in Python with mock objects is the impact of change.

We have some class with an interface.

class AppFeature( object ):

def app_method( self, anotherObject ):

etc.

class AnotherClass( object ):

def another_method( self ):

etc.

We've properly used dependency injection to make AppFeature depend on an instance of AnotherClass. This means that we're supposed to create a mock of AnotherClass to test the AppFeature.

class MockAnotherClass( object ):

def another_method( self ):

etc.

In Python, this mock isn't a best practice. It can be helpful. But adding a mock can also be confusing and misleading.

Refactoring Scenario

Consider the situation where we're refactoring and change the interface to AnotherClass. We modify another_method to take an additional argument, for example.

How many mocks do we have? How many need to be changed? What happens when we miss one of the mocks and have the mysterious Isolated Test Failure?

While we can use a naming convention and grep to locate the mocks, this can (and does) get murky when we've got a mock that replaces a complex cluster of objects with a simple Facade for testing purposes. Now, we've got a mock that doesn't trivially replace the mocked class.

Alternative: Less Strict Mocking

In Python--and other duck typing languages--a less mock-heavy approach seems more productive. The goal of testing every class in isolation surrounded by mocks needs to be relaxed. A more helpful approach is to work up through the layers.

Test the "low-level" classes--those with few or no dependencies--in isolation. This is easy because they're already isolated by design.
The classes which depend on these low-level classes can simply use the low-level classes without shame or embarrassment. The low-level classes work. Higher-level classes can depend on them. It's okay.
In some cases, mocks are required for particularly complex or difficult classes. Nothing is wrong with mocks. But fussy overuse of mocks does create additional work.

The benefit of this is

The layered architecture is tested the way it's actually used. The low-level classes are tested in isolation as well as being tested in conjunction with the classes that depend on them.
It's easier to refactor. The design changes aren't propagated into mocks.
Layer boundaries can be more strictly enforced. Circularities are exposed in a more useful way through the dependencies and layered testing.

We need to still work out proper dependency injection. If we try to mock every dependency, we are forced to confront every dependency in glorious detail. If we don't mock every single dependency, we can slide by without properly isolating our design.

BOSSIE Awards

2011-09-13T08:00:00.000-04:00

web2py wins an award -- cool.

I really like Django. But...

2011-09-08T05:00:00.000-04:00

I was going to be talking about Schema Migration, tacit knowledge, and -- of course -- Python.

The hard part would have been avoiding a LONG rant on how devilishly hard the problem really is.

Apparently, however, DevDays is cancelled. Sigh.

Data Warehousing and SQL -- Tread Carefully

2011-09-01T08:00:00.052-04:00

"Are you implying that a scalable Data Warehouse solution could be implemented using Python and serialised files?"

Not "implying". I'm trying to state it as clearly as I can.

A scalable data warehouse solution involves a lot of flat file processing.

ETL, for example, is mostly a flat-file pipeline. It starts with source application extract (to create a flat file) and proceeds through a number of transformation steps to filter, cleanse, recode, conform dimensions, and eventually relate facts to dimensions. This is generally very, very fast when done with simple flat files and considerably slower when done with a database.

This is the "Data Warehouse Bus" that Kimball describes in chapter 9 of The Data Warehouse Lifecycle Toolkit.

Ultimately, the cleansed, conformed files will lay around in a "staging area" forever. When a datamart is built, then a subset of these files can be (rapidly) loaded into an RDBMS for query processing.

Doing this in Python is no different from doing it in Java, C++ or (for that matter) Syncsort. Yes. You can build a data warehouse using processing steps written around Syncsort and be quite successful.

The important part of this is to recognize the following.

When trying to do data warehouse flat-file processing in C++ (or Java) you have the ongoing schema maintenance issue. The source data changes. You must tweak the schema mapping from source to warehouse. You can encode this schema mapping as property files or some such, or you can simply use an interpreted language like Python and encode the mappings as Python code.

The "Data Warehouse Bus" is a lot of applications that are trivially written as simple, parallel, multi-processing, small, read-match-write programs. Forget threads. Simply use heavy-weight, OS-level processes so that you can maximize the I/O bandwidth. (Remember: when one thread makes an I/O request, the entire process waits; an I/O-bound application isn't helped by multi-threading.)

with open('some_data','rb') as source:

rdr= csv.DictReader( source )

wtr= csv.DictWriter( sys.stdout, some_schema )

for row in rdr:

if exclude( row ): continue

clean = cleanse( row )

wtr.writerow( clean )

This example writes to stdout so that it can be connected in a pipeline with other steps in the processing. Programs running in an OS pipeline run concurrently. They tie up all the cores available without any real programming effort other than decomposing the problem into discrete parallel steps that apply to each row being touched.

Simple file processing is much, much faster than SQL processing. Why? No overheads for locking or buffer pooling or rollback segments, or logging, or after-image journaling or deadlock detection, etc.

Note that a data warehouse database has no need for sophisticated locking. All of the "updates" are bulk loads. 80% of the activity is "insert". With some Slowly Changing Dimension (SCD) operations there is a trivial status-change update, but this can be handled with a single database-wide lock during insert.

The primary reason for using SQL is to handle "SELECT something ... GROUP BY" queries. SQL does this reasonably well most of the time. Python does it pretty well, also.

sum_col1 = defaultdict( float )

count_group = defaultdict( int )

with connection.cursor() as c:

c.execute( "SELECT COL1, GROUP FROM..." )

for row in c.fetchall():

sum_col1[row.group] += col1

count_group[row.group] += 1

print( sum_col1, count_group )

That's clearly wordier than SQL. But not much wordier. The SELECT statement embedded in the Python is simpler because it omits the GROUP BY clause. Since it's simpler, it's more likely to benefit from being reused in the RDBMS.

The Python may actually run faster than a pure SQL query because it avoids the (potentially expensive) RDBMS sort step. The Python defaultdict (or Java HashMap) is how we avoid sorting. If we need to present the keys in some kind of user-friendly order, we have limited the sort to just the distinct key values, not the entire join result.

Because of the huge cost of group by, there are two hack-arounds. One is "materialized views". The idea is that a group-by view is updated when the base tables are updated to avoid the painful cost of sorting at query time. In addition to this, there are reporting tools which are "aggregate aware". They can leverage the materialized view to avoid the sort.

How about we avoid all the conceptual overhead of materialized views and aggregate aware reporting. Instead we can write simple Python procedures that do the processing we want.

Bottom Line

Data Warehouse does not imply SQL. Indeed, it doesn't even suggest SQL except for datamart processing of flexible ad-hoc queries where there's enough horsepower to endure all the sorting.

Brain-Damaged Data

2011-08-04T08:00:00.000-04:00

We process a fair amount of externally-prepared datasets. 40,000 rows of econometric data that we purchased from a third-party. Mostly, the data is in a usable format: .CSV or .XSLX.

Once in a while, we get CSV with | (pipe). A few times, we got fixed-format COBOL-style records.

Recently, we got a CSV-with-pipe that included 2 records with embedded \n sequences in the middle of a CSV row of data. Really.

Painful Elimination

There are two ways to "eliminate" this problem.

Subclass our input processing to handle this special CSV-with-pipe case.
Actually read and parse the source file creating a clean intermediate file that we can simply process with an existing CSV-with-pipe configuration.

I elected to do the first. The second is (to my mind) an auditing nightmare because we touched the file. We have to prove that we didn't disturb any other fields. While not impossible, it becomes a very strange special case for this one-and-only file.

CSV Simplicity

The CSV module's epic simplicity makes it easy to work around this kind of goofy data. Our subclass for this case had the following extra foolishness put in

def make_reader( self ):
        def filter_damage( aFile ):
            file_iter= iter(aFile)
            for row in file_iter:
                if row.rfind('"') >= len(row)-3:
                    logger.error( "Damaged Line: %r", row )
                    rest= next(file_iter)
                    line= row[:row.rfind('"')] + rest[3:]
                    logger.warning( "Repaired Line: %r", line )
                    yield line
                else:
                    yield row
        tweaked_file= filter_damage( self.sourceFile )
        return csv.reader( tweaked_file, delimiter='|', doublequote=False, escapechar='"' )

That's it. Since the Python CSV reader merely wants an iterator over lines, we can (with a simple generator function) provide the necessary "iterator-over-lines".

Delightful.

Apology

The murky-looking row.rfind('"') >= len(row)-3 condition is one of those consequences of trying to find just a few irregular line endings in an otherwise regular file. For CSV processing, files often have to be opened in "rb" mode because they originate (or will be used with) MS-Excel. This makes the damaged line-ending either '"\n' or maybe '"\r\n'. Rather than spend too much time negotiating with Python's universal newline and "rb" mode, it's slightly easier to look for a '"' near the end.

We're hoping this is a one-time-only subclass that we can safely ignore in the future. If hope is dashed, it's a distinct subclass, so it's easily reused and didn't break anything else.

One of Those Things

2011-07-26T08:00:00.005-04:00

Check out this question on Stack Overflow: "Python: replace a string by a float in txt file".

The question is confusing, but it appears to be a longish and confused description of simple formatting or template substitution. It's hard to be sure, but it sounds like one of Those Things™ (TT).

Most of Those Things (TT) are standard problems with standard solutions. Until you've seen a lot TT's, it seems like your problem is unique and special. It's hard to see TT's for what they are.

In this case, the problem appears to be solved by Python's string.Template class with minor modifications. The documentation for customizing string.Template isn't clear, so here's an example.

from string import Template class MyTemplate( Template ): delimiter= '@' pattern= r"@(?P<escaped>@)|@(?P<named>[_a-z][_a-z0-9]*)@|@(?P<braced>[_a-z][_a-z0-9]*)@|@(?P<invalid>)"

That appears to be the standard solution to the standard problem. Define a new delimiter ('@') and some slightly different delimiter parsing rules and away you go.

This can be used as follows to replace any '@x@' variables in any template file. What's important is that very little actual code is needed, since it's one of Those Things that's already been solved.

with open( 'a.txt', 'r' ) as source:
    t = MyTemplate(source.read())
    result= t.substitute( x=15 )
    print result

Spam Email Footers

2011-07-21T08:00:00.001-04:00

I don't want the spamilicious email. I'm trying to actually unsubscribe.

The footer says "If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of any information contained in or attached to this communication is strictly prohibited. If you have received this message in error, please notify the sender immediately and delete the material from any computer."

I don't feel like the intended recipient because it's just irrelevant junk. Perhaps you should not have disseminated, distributed, copied or sent me this. Wouldn't that have been simpler? Keep it to yourself?

I also think I've received the message in error. Since I don't want the damn thing. And that means that I have to delete it? Why can't you stop sending it? Wouldn't that be simpler for both of us?

757 Python User's Group Meetup

2011-07-18T06:06:00.001-04:00

Wednesday night. At 757 Labs. Be there.

Here's the details on meetup.com

Lacking any other agenda, I'll do some more presentation on the supreme coolness of Django.

I almost wet myself

2011-07-12T08:00:00.004-04:00

Someone sent me this: "“Building Skills in Python” – Steven F. Lott".

I had a vague idea that this book would get some traction. This response was surprising. I guess I should get to work on the upgrades. And focus on the "no-nonsense" comment.

Security Vulnerabilities

2011-07-07T08:00:00.003-04:00

Just saw this for the first time today: http://cwe.mitre.org/top25/

I'd always relied on this: https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project

Both are really good lists of security vulnerabilities.

I once had to listen to a DBA tell me that "we don't know what we don't know" as a way of saying that there was no way to be sure that a web app was "secure". That comment lead the project manager to go through the classic "risk exposure" exercise (and hours of discussion) to determine that security mattered. We defined the risks, the costs and the probability of occurrence so that we could document all kinds of potential exposures or something.

Instead of hand-wringing, these kinds of simple lists of the common vulnerabilities provides actionable steps for design, code, test and audit of operations. Further, they guide selection, configuration and operation of web server technology to assure that the vulnerabilities are addressed.

Implementing the Unsubscribe User Story

2011-06-30T08:00:00.001-04:00

I've been unsubscribing from some junk email recently.

The user story is simple: As a not-very-interested person, I want to get off your dumb-ass mailing list so that I don't have to flag your crap as spam any more.

The implementations vary from good to evil. Here's what I've found.

The best sites have an unsubscribe link that simply presents the facts -- you are unsubscribed. I almost feel like re-subscribing to a site that handles this use case so well.

The first level of crap is a site which forces me to click an OK or Unsubscribe button to confirm that I really want to unsubscribe and wasn't clicking the tiny little links at the end of the message randomly.

The deeper level of "marketing" crap is a form that allows me to "configure my subscription settings". This is done by some marketing genius who wanted to "offer additional value" rather than simply do what I asked. This is a hateful (but not yet evil) practice. I don't want to "configure" my settings. I want out.

The third-from-worst is a form in which I must enter my email address. What? I have several email aliases that redirect to a common mailbox. I have to -- what? -- guess which of the aliases was used? This is pernicious because I can make a spelling mistake and they can continue to send me dunning email. This fill-in-the-blanks unsubscribe is simply evil because it gives them plausible deniability when the continue to send me email. It's now my fault that I didn't spell my name correctly.

The next-to-worst is a "mailto:" link that jumps into my emailer. I have to -- what? -- fill in the magic word "Complete" somewhere? You're kidding, right? This is so 1980's-vintage listserv that I'm hoping these companies can be sued because they failed to actually unsubscribe folks. Again, this gives the spammer a legitimate excuse because I failed to do the arcane step properly.

The worst is no link at all. Just instructions explaining that an email must be send with the magic word "Complete" or "Unsubscribe" in the subject or body. Because I use aliases, this will probably not unsubscribe anything useful, but will only unsubscribe my outbound email address. This is the worst kind of evil. In a way, it meets the user story. But only in a very, very oblique way.

Simplicity vs. Depth

2011-06-27T18:05:00.001-04:00

During chapter technical reviews, the question of technical depth has come up time and again. Essentially, in every single chapter.

In the older Building Skills in Python book, there are a number of topics that feel "digressive" to the reviewer and editor. Too much depth.

However, there are a number of Python tutorials, many of which are very shallow. I'd like to find a way to retain the technical depth, without it feeling "digressive".

Choice 1. Split each chapter into different "basic" and "advanced" sections. This would retain a sensible outline of parts (Language Fundamentals, Data Structures, Classes, Modules and a bunch of advanced projects) and chapters within each part. Some chapters would still have to be split because a number of "advanced" concepts (i.e. alternative function argument passing with * and **) really has to be delayed until after an appropriate data structure chapter.

Choice 2. Separate material two kinds of chapters "basic" and "pro". This would lead to a "basics" thread for n00bz (read all the "basics" chapters) and an "pro" thread for professionals where you'd just read all the chapters in order without skipping. This would create some more chapters, but each chapter would be shorter and more focused.

It's

Hackerspace

2011-06-21T08:00:00.003-04:00

Just started learning about "Hackerspace".

Without really knowing what I was doing, I fell into the 757 Labs Hackerspace.

The 757 Python Users' Group, specifically.

What a great idea. Bright people. Interested in the same area of technology.

It's like hanging around with sailors at a marina.

An Object-Lesson in How to Stifle Innovation

2011-06-09T08:00:00.001-04:00

Read this: How Ma Bell Shelved the Future for 60 Years.

AT&T firmly believed that the answering machine, and its magnetic tapes, would lead the public to abandon the telephone.

How many good ideas are set aside by managers who simply don't have a clue what users actually want?

How many great IT projects are rejected because of this kind of delusional paranoia?

Multithreading -- Fear, Uncertainty and Doubt

2011-06-07T14:10:00.005-04:00

Read this: "How to explain why multi-threading is difficult".

We need to talk. This is not that difficult.

Multi-threading is only difficult if you do it badly. There are an almost infinite number of ways to do it badly. Many magazines and bloggers have decided that the multithreading hurdle is the Next Big Thing (NBT™). We need new, fancy, expensive language and library support for this and we need it right now.

Parallel Computing is the secret to following Moore's Law. All those extra cores will go unused if we can't write multithreaded apps. And we can't write multi-threaded apps because—well—there are lots of reasons, split between ignorance and arrogance. All of which can be solved by throwing money after tools. Right?

Arrogance

One thing that makes multi-threaded applications error-prone is simple arrogance. There are lots and lots of race conditions that can arise. And folks aren't trained to think about how simple it is to have a sequence of instructions interrupted at just the wrong spot. Any sequence of "read, work, update" operations will have threads doing reads (in any order), threads doing the work (in any order) and then doing the updates in the worst possible order.

Compound "read, work, update" sequences need locks. And the locations of the locks can be obscure because we rarely think twice about reading a variable. Setting a variable is a little less confusing. Because we don't think much about reads, we fail to see the consequences of moving the read of a variable around as part of an optimization effort.

Ignorance

The best kind of lock is not a mutex or a semaphore. It surely isn't an RDBMS (but God knows, numerous organizations have used an RDBMS as a large, slow, complex and expensive message queue.)

The best kind of lock seems to be a message queue. The various concurrent elements can simply dequeue pieces of data, do their tasks and enqueue the results. It's really elegant. It has many, simple, uncoupled pieces. It can be scaled by increasing the number of threads sharing a queue.

A queue (read with an official "get") means that the reads aren't casually ignored and moved around during optimization. Further, the creation of a complex object can be done by one thread which gets pieces of data from a queue shared by multiple writers. No locking on the complex object.

Using message queues means that there's no weird race condition when getting data to start doing useful work; a get is atomic and guaranteed to have that property. Each thread gets an thread-local, thread-safe object. There's no weird race condition when passing a result on to the next step in a pipeline. It's dropped into the queue, where it's available to another thread.

Dining Philosophers

The Dining Philosophers Code Kata has a queue-based solution that's pretty cool.

A queue of Forks can be shared by the various Philosopher threads. Each Philosopher must get two Fork resources from the queue, eat, philosophize and then enqueue the two Forks again. It's quite short, easy to write and easy to demonstrate that it must work.

Perhaps the hardest thing is designing the Dining Room (also know as the Waiter, Conductor or Footman) that only allows four of the five philosophers to dine concurrently. To do this, a departing Philosopher must enqueue themselves into a "done eating" queue so that the next waiting Philosopher can be seated.

A queue-based solution is delightfully simple. 200 or so lines of code including docstrings comments so that the documentation looked nice, too.

Additional Constraints

The simplest solution uses a single queue of anonymous Forks. A common constraint is to insist that each Philosopher use only the two adjacent forks. Philosopher p can use forks (p+1 mod 5) and (p-1 mod 5).

This is pleasant to implement. The Philosopher simply dequeues a fork, checks the position, and re-enqueues it if it's a wrong fork.

FUD Factor

I think that the publicity around parallel programming and multithreaded applications is designed to create Fear, Uncertainty and Doubt (FUD™).

Too many questions on StackOverflow seem to indicate that a slow program might magically get faster if somehow threads where involved. For programs that involve scanning the entire hard drive or downloading Wikipedia or doing a giant SQL query, the number of threads has little relevance to the real work involved. These programs are I/O bound; since threads must share the I/O resources of the containing process, multi-threading won't help.
Too many questions on StackOverflow seem to have simple message queue solutions. But folks seem to start out using inappropriate technology. Just learn how to use a message queue. Move on.
Too many vendors of tools (or languages) are pandering to (or creating) the FUD factor. If programmers are made suitably fearful, uncertain or doubtful, they'll lobby for spending lots of money for a language or package that "solves" the problem.

Sigh. The answer isn't software tools, it's design. Break the problem down into independent parallel tasks and feed them from message queues. Collect the results in message queues.

Some Code

class Philosopher( threading.Thread ):
    """A Philosopher.  When invited to dine, they will
    cycle through their standard dining loop.
    
    -   Acquire two forks from the fork Queue
    -   Eat for a random interval
    -   Release the two forks
    -   Philosophize for a random interval
    
    When done, they will enqueue themselves with
    the "footman" to indicate that they are leaving.
    """
    def __init__( self, name, cycles=None ):
        """Create this philosopher.
        
        :param name: the number of this philosopher.  
            This is used by a subclass to find the correct fork.
        :param cycles: the number of cycles they will eat.
            If unspecified, it's a random number, u, 4 <= u < 7
        """
        super( Philosopher, self ).__init__()
        self.name= name
        self.cycles= cycles if cycles is not None else random.randrange(4,7)
        self.log= logging.getLogger( "{0}.{1}".format(self.__class__.__name__, name) )
        self.log.info( "cycles={0:d}".format( self.cycles ) )
        self.forks= None
        self.leaving= None
    def enter( self, forks, leaving ):
        """Enter the dining room.  This must be done before the 
        thread can be started.
        
        :param forks: The queue of available forks
        :param leaving: A queue to notify the footman that they are
            done.
        """
        self.forks= forks
        self.leaving= leaving
    def dine( self ):
        """The standard dining cycle: 
        acquire forks, eat, release forks, philosophize.
        """
        for cycle in range(self.cycles):
            f1= self.acquire_fork()
            f2= self.acquire_fork()
            self.eat()
            self.release_fork( f1 )
            self.release_fork( f2 )
            self.philosophize()
        self.leaving.put( self )
    def eat( self ):
        """Eating task."""
        self.log.info( "Eating" )
        time.sleep( random.random() )
    def philosophize( self ):
        """Philosophizing task."""
        self.log.info( "Philosophizing" )
        time.sleep( random.random() )
    def acquire_fork( self ):
        """Acquire a fork.
        
        :returns: The Fork acquired.
        """
        fork= self.forks.get()
        fork.held_by= self.name
        return fork
    def release_fork( self, fork ):
        """Acquire a fork.
        
        :param fork: The Fork to release.
        """
        fork.held_by= None
        self.forks.put( fork )
    def run( self ):
        """Interface to Thread.  After the Philosopher
        has entered the dining room, they may engage
        in the main dining cycle.
        """
        assert self.forks and self.leaving
        self.dine()

The point is to have the dine method be a direct expression of the Philosopher's dining experience. We might want to override the acquire_fork method to permit different fork acquisition strategies.

For example, a picky philosopher may only want to use the forks adjacent to their place at the table, rather than reaching across the table for the next available Fork.

The Fork, by comparison, is boring.

class Fork( object ):
    """A Fork.  A Philosopher requires two of these to eat."""
    def __init__( self, name ):
        """Create the Fork.
        
        :param name: The number of this fork.  This may 
            be used by a Philosopher looking for the correct Fork.
        """
        self.name= name
        self.holder= None
        self.log= logging.getLogger( "{0}.{1}".format(self.__class__.__name__, name) )
    @property
    def held_by( self ):
        """The Philosopher currently holding this Fork."""
        return self.holder
    @held_by.setter
    def held_by( self, philosopher ):
        if philosopher:
            self.log.info( "Acquired by {0}".format( philosopher ) )
        else:
            self.log.info( "Released by {0}".format( self.holder ) )
        self.holder= philosopher

The Table, however, is interesting. It includes the special "leaving" queue that's not a proper part of the problem domain, but is a part of this particular solution.

class Table( object ):
    """The dining Table.  This uses a queue of Philosophers
    waiting to dine and a queue of forks.
    
    This sets Philosophers, allows them to dine and then
    cleans up after each one is finished dining.
    
    To prevent deadlock, there's a limit on the number
    of concurrent Philosophers allowed to dine.
    """
    def __init__( self, philosophers, forks, limit=4 ):
        """Create the Table.
        :param philosophers: The queue of Philosophers waiting to dine.
        :param forks: The queue of available Forks.
        :param limit: A limit on the number of concurrently dining Philosophers.
        """
        self.philosophers= philosophers
        self.forks= forks
        self.limit= limit
        self.leaving= Queue.Queue()
        self.log= logging.getLogger( "table" )
    def dinner( self ):
        """The essential dinner cycle:
        admit philosophers (to the stated limit);
        as philosophers finish dining, remove them and admit more;
        when the dining queue is empty, simply clean up.
        """
        self.at_table= self.limit
        while not self.philosophers.empty():
            while self.at_table != 0:
                p= self.philosophers.get()
                self.seat( p )
            # Must do a Queue.get() to wait for a resource
            p= self.leaving.get()
            self.excuse( p )
        assert self.philosophers.empty()
        while self.at_table != self.limit:
            p= self.leaving.get()
            self.excuse( p )
        assert self.at_table == self.limit
    def seat( self, philosopher ):
        """Seat a philosopher.  This increments the count 
        of currently-eating Philosophers.
        
        :param philosopher: The Philosopher to be seated.
        """
        self.log.info( "Seating {0}".format(philosopher.name) )
        philosopher.enter( self.forks, self.leaving)
        philosopher.start()
        self.at_table -= 1 # Consume a seat
    def excuse( self, philosopher ):
        """Excuse a philosopher.  This decrements the count 
        of currently-eating Philosophers.
        
        :param philosopher: The Philosopher to be excused.
        """
        philosopher.join() # Cleanup the thread
        self.log.info( "Excusing {0}".format(philosopher.name) )
        self.at_table += 1 # Release a seat

The dinner method assures that all Philosophers eat until they are finished. It also assures that four Philosophers sit at the table and when one finishes, another takes their place. Finally, it also assures that all Philosophers are done eating before the dining room is closed.

Changed the Page Template

2011-06-03T08:52:00.000-04:00

The "default" template I chose was too narrow for presenting code samples. Changed it.

Code Kata : "Simple" Database Design

2011-05-26T08:00:00.001-04:00

Here's a pretty simple set of use cases for a code-kata database application.

This is largely transactional, not analytical.

It's a simple inventory of ingredients, recipes and locations.

Context

42' sailboat.
Lots of places to keep stuff. Lots.

Stuff gets lots or misplaced. It's helpful to marry recipes with ingredients to use up the last of something before it goes bad and stinks up the boat.

Actor is essentially the cook.

Use Cases

Perishables to be eaten soon?
Shopping list for specific recipes.
Where did I put that?

Model

Ingredient. A generic description: "lime", "coconut". Not too much more is needed. A "food safety" notation (refrigeration required, etc.) is a helpful attribute. Maybe a "food group" or other nutrition information.
Location. A text description of where things can be stored. This shouldn't have too many attributes, because boats aren't big grids. Phrases like "port saloon upper cabinet", or "galley outer cooler" make sense to folks who live on the boat.
On Hand. This is simply ingredient, location and a measurement of some kind. Example: 3 limes in the starboard galley center cooler. There's a lot of magic around units and unit conversion that can be fun. But that strays outside the database domain.
Recipe. Example: "One of sour, two of sweet, three of strong, and four of weak.", lime, simple syrup, rum, water. Plain text using a lightweight markup is what's required here. Along with a many-to-many relationship with ingredients. This is not carefully defined above because it should be done as a "more advanced" exercise.

I think this has the right amount of complexity and isn't very abstract. Since the use cases are pretty obvious to anyone who's cooked or been to a grocery store, use case details aren't essential.

Meetup Tonight

2011-05-25T08:49:00.004-04:00

Tonight (May 25th). Red Dog. Colley Ave. Ghent. I'll be wearing my Stack Overflow shirt. I'll be there about 7. I know that at least one other person won't be there until 8.

The Meetup link.

I like this meetup idea a lot. Probably because the WFH life-style is a little isolating.

There's the small "Hampton Stack Overflow Community". We have a common interest in Stack Overflow.

Also, there's the 757 Python Users Group. We have a common interest in Python. I've decided to become the "official" organizer for this. I'm going to join the 757 Labs Hackerspace, also.

Agility and following a "Strictly Agile" approach

2011-05-24T08:00:00.001-04:00

I've seen some discussion on Stack Overflow that is best characterized by the question: "What is Strictly Agile?", or "What's the Official Agile Approach?".

Someone shared this with me recently: "Process kills developer passion".

I have also heard some great complaints about organizations that claim "Agile" and actually do nothing of the kind. In some cases it's not a "crunchy agile shell" around a waterfall process; it's a simple lie. Nothing about the process is Agile except a manager insisting that all the status reporting, planning and unprioritized lists of random requirements are Agile.

Finally, I got this weird suggestion: "consider writing a blog about how to test if you are agile or not". It's weird because testing for Agile is like testing for breathing; it's like testing for flammability.

The Agile Test

Testing if your project is Agile can be done two ways.

Practical. Make a change to the project. Any change. Requirements, architecture, due dates, staff, anything. Does it derail? If so, it wasn't very Agile, was it?
Theoretical. Reread the Agile Manifesto. Make a score card that evaluates the project on each of the eight basic criteria in the Agile manifesto. Convene all the project stakeholders. Conduct careful surveys and have structured walkthroughs to determine the degree of Agility surrounding each person, deliverable, collaborative relationship and issue.

An important point is that Agile is not absolute. Some practices are more Agile than others. There's no "strictly" Agile. There are ways to make a project more Agile; that is, it can effectively cope with change. There are ways to make a project less Agile; that is, change causes problems and can derail the project completely.

The canonical example is a missing, misstated or contradictory requirement that gets uncovered after coding and during user acceptance test. Clearly, that feature has been built and is absolutely wrong. What happens next?

Agile? The product can be released with with the broken feature relegated to the next release. A hack is put in to remove the buttons or menu items or links until they work.

Not Agile? Everyone works around the clock to make that feature work no matter what. Paraphrasing Admiral Farragut: "Technical debt be damned. Development must proceedfull speed ahead." All of this irrespective of the relative value of what's being developed. Schedule comes first; features second.

How Much Process?

The "Process Kills..." blog entry repeats observation that a lot of carefully-defined process isn't really all that helpful. It identifies a cause ("process kills passion") that's can be true, but it's largely irrelevant. Process is—essentially—work that's not focused on delivering anything of real value. Complex processes are "meta" work; it's work focused on IT internals; it's work that creates no value for the users of the software; work that replaces the more valuable elements of the Agile Manifesto.

One can argue that processes, documentation, contracts and plans "assure" success or demonstrate some level of quality. To an extent all the process and meta-work creates trust that—eventually—the resulting software product will solve the original problem.

The mistake is that non-Agile methods use a series of surrogates—processes, documentation, contracts and plans—instead of actual software. The point of Agile methods it to release software early and often and avoid using surrogates.

Key Points of Agile

Here are the key points of the Agile Manifesto.

Individuals and interactions over processes and tools. A more Agile project will use the best people and encourage them to talk amongst themselves. A less Agile project will write a lot of things (which folks don't have time or reward for reading.) There will be misunderstandings, leading to large, boring meetings where someone reads powerpoint slides to other folks to try and clear up misunderstandings.
Working software over comprehensive documentation. A more Agile project uses frequent release cycles of incremental software. A less Agile project attempts to gather all requirements, do all design and then try to do all the coding even though the requirements have already been found to be less than crystal clear.
Customer collaboration over contract negotiation. A more Agile project uses constant contact with customer and product owner to refine and prioritize the requirements. A less Agile project uses a complex change control process to notify everyone of a requirements change, which leads to design and code changes, and has cost and schedule impact that must be carefully planned and documented.
Responding to change over following a plan. A more Agile project uses incremental releases, conversation and a modicum of discipline to build things of value. Just because someone thought it should be included in the requirements doesn't mean the feature is really required.

The "Process Kills Passion?" Question

There Process Kills Passion blog lists a bunch of things that—it appears—some folks find burdensome:

Doing full TDD, writing your tests before you wrote any implementing code.
Requiring some arbitrary percentage of code coverage before check-in.
Having full code reviews on all check-ins.
Using tools like Coverity to generate code complexity numbers and requiring developers to refactor code that has too high a complexity rating.
Generating headlines, stories and tasks.
Grooming stories before each sprint.
Sitting through planning sessions.
Tracking your time to generate burn-down charts for management.

This list has three different collections of practices.

Good. TDD, code reviews, generating headlines, stories and tasks, grooming stories before each sprint and doing some planning for each sprint are all simply good ideas. They must be done. "Pure Coding" is not a good way to invest time. Planning and then coding is much smarter, no matter how boring planning appears.
Difficult. Test code coverage can be helpful, but can also devolve to empty numerosity. 20% more coverage doesn't not mean 20% fewer bugs. Nor does it mean 20% less chance of uncovering a bug at run time. Code complexity ratings are also fussy because they don't have a direct correlation with much. They must be done and used to prioritize work that will reduce technical debt. But mindless thresholds are for cowards who don't want to mediate deep technical discussions.
Silly. Creating burn-down charts for management shouldn't be necessary. Everyone must read and understand the backlog. Everyone should build the summary charts they want from the backlog. The product owner or even the eventual customer should do this on their own. They must be given a profound level of ownership of the features and the process for creating software.

I don't agree that process kills passion. I think there's a fine line between playing with software development and building software of value. I think that valuable software requires some discipline and requires executing a few burdensome tasks (like TDD) that create real value. Assuring 80% or 100% code coverage doesn't always create real value. Spending time keeping the backlog precise and complete is good; spending time making pictures is less good.

Creating UML

2011-05-19T08:00:00.000-04:00

I'm a big fan of plain-text tools. Source Code. ReStructuredText. LaTeX.

I'm not a big fan of proprietary file formats and document formats that are difficult or impossible to decode. JSON and XML rock. .XLS files are painful and difficult to work with.

UML Diagrams are a particularly odious problem. To see a diagram it has to be PNG or PDF or some other graphic format that's optimized for storage and display, but not really optimized for editing. SVG has a text vector markup language, but it's painful because it's so generalized.

Recently, I found two text to UML tools that are exciting prospects.

First, there's YUML.me. This draws pretty nice, if simple, diagrams that you can work with with relatively little pain. It's slow and limited. But it works for simple diagrams.

The best part is that the image is rendered from the URL as plain text.

http://yuml.me/diagram/scruffy/usecase/[Author]-(write text), (render image)-[YUML], [Author]-(share link).

YUML supports simple use case diagrams, simple class diagrams and really simple activity diagrams. It covers a few bases with a pleasant level of flexibility.

The other tool is Plant UML. "PlantUML is used to draw UML diagram, using a simple and human readable text description."

The online Plant UML Server allows a flexible no-software-on-the-desktop way to play with their markup language. The text of the image is not in the URL here, since the text is so much more complex.

The best part of this is that the pictures come from plain text.

The plain text is trivial to put under configuration control.
Plain text system descriptions are easy to write with simple markup.
Plain text documentation of existing software can be derived from simple source analysis.
Plain text design documents can generate some elements of the source code

The 757 Python User's Group

2011-05-18T13:32:00.002-04:00

http://www.meetup.com/757-Python-Users-Group/

I'm looking forward to meeting other Python developers in Hampton Roads.

Tonight. 7:00 PM. See you there.

Decisions and Consequences

2011-05-17T08:00:00.000-04:00

A single poorly-made decision can have profound ripple-effects. Once your stuck with it, you make accommodations, hacks and work-arounds. Eventually, things work, but the result is less than ideal.

Changing tack requires sometimes pervasive rework to the application. How can we reduce the risks and improve the value created?

A Recent Example

When dealing with bulk econometric data (Bloomberg, D&B, Moody's, etc.) you get BIG files with lots of fields. Depending on what you're paying for, the file layouts are frequently different even though the content is similar. I'm a big fan of plain-old CSV data. Even the tab-delimited variant of CSV is not bad to work with.

Further, most vendors will slap some heading rows on the file so that the column names are--more or less--identified. Surprisingly, this doesn't work out well in practice because there are often multiple columns with the same name. Sigh.

Using Python's csv library module lets us cope with CSV (and tab-delim) quite gracefully.

What's wrong with that decision? Nothing.

Variant Column Names

The question arises when you've purchased several files of econometric data and the column names are slightly different. This happens with a single vendor and across vendors. It's part of the game that can't easily be avoided. Column names vary.

What to do?

Here's the less-than-ideal decision. Make the column names a parameter.

In Python, this is not terribly difficult. The csv module's DictReader provides us a dictionary for each row. Each column name becomes a key. We can access the fields with some_row['this_field'] and some_row['that_field']. How bad can it be?

The extra punctuation is fairly hideous.

More importantly, however, is the nature of the metadata.

Consequence One -- Dynamic Metadata

Dynamic metadata, in this case, means that any indexing of the data is done based on character string column names.

index[index_name][row[column_name]].append( row )

That's rather more complex than the alternative where the metadata has a fixed definition.

some_index[row.column].append( row )

Consequence Two -- Murky ORM

Once we have dynamic metadata, we're largely frozen out of ordinary SQL database implementations. We don't know the column names, we don't know the indices. We can't do simple CREATE TABLE statements because we don't really have the column names until we open the working files.

We have to grub through all the code to find out where the dynamic mapping is reasoned out. Once we find that, we can then consider how to make the metadata fixed enough to tackle a SQL database.

We could, of course, generate the SQL CREATE INDEX statements on-the-fly. There's nothing wrong with it. But it slows down analysis and decision-making when we're not sure what indexes there are or what leads to a choice of index.

What's important here is that we want to use SQLite because it ships with Python. We want our application to use an ORM (like SQLAlchemy or SQLObject). We don't want our application to become a kind of ORM because of the dynamic SQL and dynamic column names.

Cleanup

The cleanup road is clear.

Map all variant inputs to one common structure. Rather than work with raw dictionaries from csv, map each row to a standard set of names. For now, we can replace the dictionaries with named tuples to prepare for a migration to an ORM when that's possible.
Replace the row['some field'] syntax with row.some_field syntax. Of course, there's a lot of this. This is a pervasive change.
Find all the dynamic index creation and refactor that into a more static "database-like" place for now.

Item 1 is pretty easy to unit test. We're adding a function to map from dynamic names to fixed names. Nothing much to this testing-wise.

Item 2 requires unit tests with really good code coverage or there's no earthy way we can be sure that each mapping-syntax name has been transformed into an attribute-syntax name.

Item 3 barely requires testing. Indexes and other features are performance enhancements that can be removed and added without altering functionality.

A Taxonomy of Use Case Errors

2011-05-12T08:00:00.001-04:00

First, the definition. A use case describes an actor's interaction with a system to create business value. There are three parts: Actor, Interaction and Business Value.

1. Not Interactive.

1.1. The use case is just features and technical attributes with no actor interaction expressed.

1.2. The use case is just algorithms and processing with no connection to an actor or a goal.

2. No Business Value.

2.1. Incomplete

2.1.1. The use case focus on sequential operations with no value or goal.

2.1.3. The use case simply follows existing precedent without supporting actual business goals. It "paves the cow path".

2.2. Non-Specific

2.2.1. The use case is a result of free-running imagination; it conflates "possibly" vs. "required". It contains descriptions of interactions which could happen or would be nice to happen.

2.3. Covers the Technology Only

2.3.1. The solution technology is conflated with the business problem. Words like "database" or "foreign key" or "error log" or other solution technology are central.

2.4. Contradictory

2.4.1. The use case goal contradicts other goals.

2.4.2. The use case sequence is inconsistent with the stated goal.

3. No Actor.

The Ubiquitous Object

2011-05-10T08:00:00.001-04:00

Objects are everywhere.

Weirdly, some people can't see them. I guess they live in a rarified, HP Lovecraftian world of pure action inhabited by amorphous things that can't be properly called "beings" but rather "doings" because they're pure activity with no existence.

Read "Hypnos". "They were sensations, yet within them lay unbelievable elements of time and space—things which at bottom possess no distinct and definite existence."

Got this comment the other day.

... doing procedural code correctly when you don't want to be bothered w/ OO is a separate and big enough topic that warrants its own book or monograph.

I guess that means that objects, and the reality that they model, are a "bother"—a pitfall to be avoided—a cost with no benefit. This is not the first time I've heard this, and—like Lovecraft—it leads me to wonder how such a rich and weird phantasy world gets constructed.

I had a project manager exclaim "You don't need more than seven or eight objects to write any application." I didn't press the person on that point. I assumed that they were talking about classes (not objects) and, further, had conflated class with "elaborate module-like library packed with amazing features". Or maybe they conflated class with package. Or something. It's hard for me to dig into misapprehensions and false assumptions without being rude.

There are a surprising number of misapprehensions. I'm occasionally tempted to turn NTLK loose on all questions tagged "Python" on Stack Overflow. With some patient reading, I think I could develop a taxonomy of OO confusion. However, let's just focus on this comment.

The Bother Factor

Why is OO a "bother"?

I've been told that OO programming is different. Different from what? From procedural programming without objects, I guess.
I'm been told that some problems are a better fit for OO, and some problems aren't a good fit for OO. This is hard to parse because it makes the more profound claim that some problems weirdly don't involve any "objects" just pure actions.
The Object-Relational Impedance Mismatch problem somehow indicts object-oriented programming as unsuitable when there's a relational database involved.

Let's look at some of these in a little depth to see the underlying fallacies.

Procedural Is More Fundamental

This is subtle and pernicious. An OO language contains within it a procedural language. Because of this, we can use Java, C++ or Python to write Fortran-like (or VB-like) crapola code. It's possible to write everything in a single, massive, static class with piles of random global variables, long lists of disorganized methods, and "adaptation via block comment" buffoonery.

Some folks object to characterizing procedural programming as random, disorganized or buffoonery. They tell me that a purely procedural can be neat and well organized with tidy, focused modules that have narrowly-defined responsibilities, no global variables and clever techniques like pointer-to-function to support adaptation.

Wait. The idea of tidy, focused modules with narrowly-defined responsibilities is exactly what a class is.

This is important. All good procedural programming is isomorphic to object-oriented programming minus the class definitions.

Procedural isn't "fundamental". It's just a "fragmentary". Procedural programming is a subset of object-oriented programming. Not a foundation. We can, for example, do functional-style object-oriented programming by using immutable objects.

Some Problems Aren't A Good Fit

Claiming that there are problems which don't fit the object-oriented paradigm is false. Or such a claim hearkens to a more elaborate ontology in which existence somehow doesn't matter.

This question is typical: "What should be OO and what shouldn't?"

When a program "runs" or "executes" there is state change. In a lazy functional world, state change is characterized by the creation and destruction of immutable objects: the new "4" that's created by "2+2".

In order for there to be state, there must be an object that has a state of being. Objects are inherent in doing any computing of any kind.

Some folks like to lift up stored procedures or shell scripts as "important" examples of non-OO programming. Mostly, these just show that a non-OO language can persist for a long time because clever programmers can work around a lot of limitations. (Turing Completeness is a necessary pre-condition; not a desirable feature set.)

[And yes, I've written multiple-thousand line shell scripts so customers can avoid paying a license fee for a proper compiler. Just because it can be done doesn't mean it should be done.]

This is important. All Programming Involves Objects.

There are really just two "paradigm" decisions. Does the problem involve new class definitions or can it be done using built-in classes? Does the problem involve mutable objects or immutable objects?

Software that uses only built-in classes is termed "procedural". Software that uses only immutable objects is termed "functional". Software that uses mutable objects is mistakenly termed "object-oriented".

Object-Relational Mismatch

This isn't really very interesting, no matter how many times people like to flog it. Use an ORM. Move on.

Further, it's important to recognize that normalization, foreign keys, cascading deletes and other malarky are hacks imposed on us by several relational database limitations. These are not essential parts of any problem.

I don't know how many times I've had to answer the "how do I do foreign keys in Java/C++/Python?" question. The answer is always the same: foreign keys are a hack-around because there are no proper object references in a relational database.

What's Left?

In spite of the obvious logic that OO is central, there is always a residual "It's a bother" sense from folks who's first language was not an OO language.

As far as I can tell, the "bother" stems from simple ignorance of what's really going on. Many programmers can't articulate any design principles. Yet, they tend to follow some principles rather closely. Ask them what they're doing. Read their code. Almost everyone who codes has some set of fundamental principles. (The few exceptions are people who seem to write code more-or-less randomly and still manage to arrive at something that appeared to "work"; these people do exist and are very scary.)

Many programmers don't follow all of the SOLID Principles.

Many programmers follow the SOLID principles using different nomenclature. The SOLID initials and acronyms are just one one goofy terminology. There are more principles than these, and the principles can have other names.

What's important is that (except for rare exceptions) all programmers follow some of the SOLID principles. Some follow all of them. Some follow numerous additional principles beyond these. Some give their principles other names.

The folks who claim OO programming is a "bother" just don't happen to recognize that they're already following some of the SOLID principles and actually doing OO programming with built-in classes.

Doing Procedural Programming Correctly

Bottom Line: "doing procedural code correctly" is simply OO programming using only built-in classes.

It's not a "big" topic. It's entirely an exercise in learning how to apply someone else's nomenclature to one's existing principles.

The curse of procedural design

2011-05-03T08:00:00.001-04:00

After reverse engineering procedural code in C, VB or even Python, I'm finding that procedural programming inevitably leads to bad, bad code-rot.

Consider some of the common design patterns.

Strategy. Confronted with alternative strategy choices, a purely procedural code solution is either

If-statements everywhere the strategy is involved.
Block comments. (Pre-processor #if statements are the logical equivalent of block comments plus a tool to move them around just prior to compilation.)

These lack flexibility and seem to devolve into a quagmire of mystery. The if-statements often become tangled and complex. More importantly, some strategy choices — which are unused — may not be maintained at all. Of course, the block comments are never maintained.

Command. Often a command design requires a "code" or "label" and a big-old sequential switch (BOSS™) statement to select among the procedures which implement the various commands. Once "composite" commands are introduced, this devolves into nonsense. Ideally, it's a simple recursion, where a composite command simply invokes the sub-commands. However, folks get nervous about recursion and try to write weird loops.

State. A state design always seems to involve labels or codes for the state names and a slightly different big-old-state-switch (BOSS™, no accident that this is the same acronym) to sort out the variant behaviors in the distinct states. This shouldn't become too confusing. After all, Turing machines and other mathematical abstractions give us a strong hint on how we should proceed.

The problem with stateful procedural programming is that the state changes can be hidden everywhere. In the Really Bad Languages, variables can change values without an assignment statement! In the Not Bad Languages, we can track down the various assignment statements and try to reason out the state changes. Procedural code—without a lot of adult supervision—never seems to encapsulate state change with the the same in-your-face clarity that OO programs do.

I Could Go On

The point is this. While procedural programming could be done well, there appear to be a lot of obstacles inherent in the paradigm.

The best procedural programming I've seen has always been very object-oriented. Each procedure or function had a distinct data structure it worked with; they were all closely related by virtue of naming or file structure; much like a class definition.

I'm starting to wonder if my Building Skills books are taking the right approach. I start with the procedural aspects of Python. I'm beginning to feel that this may be a disservice to the n00bz.

Perhaps it's better to swap the order of the sections and start with the various Pythonic data structures and introduce the various statements sort of "casually" as part of demonstrating how a data structure is supposed to be used.

Language, Tools, Chickens, Eggs, Java and Python

2011-04-30T08:00:00.001-04:00

Too much of programming is intimately tied up with the tools to support the development of the software.

Example 1. I was told -- with absolute and fierce conviction -- that VB may suck as a language, but Visual Studio more than makes up for the obvious problems. For some people, Tools Trump Language. Sadly, I've also had customers with ancient code they could no longer compile or maintain because the tools were out of support.

On Stack Overflow, you can read questions like this: "What IDE to use for Python?". In spite of this question's immense popularity, it gets re-asked all the time. Search for "Python IDE" to see endless duplicates. One of the most common duplicate forms of this question asks (or demands) code completion. As if there are folks who cannot write code without code completion.

Chickens and Eggs

The issue with sophisticated IDE's (like Eclipse, NetBeans, and even Komodo) is that you have to learn the tools before learning the language. Until you know something about the language, the tools, of course, are useless. Worse, Eclipse is for "enterprise" applications and is so fat with bells (and whistles) that it's hard to determine what to use and what it means.

So the tool is a prerequisite for the language. But the language is a prerequisite for the tool.

How to cut the Gordian Knot?

First Principles

Irrespective of the "Visual Studio makes VB not suck" crowd, language comes first -- and last -- and fills all the spaces in between.

Language is everything. Software is merely encoded knowledge. The language of that encoding is how we determine meaning; how we argue about correctness, adaptability, maintainability and security. Tools don't endure -- they come and go -- but the language remains.

The only thing more important than the language is the data itself. But that's another rant.

Proof, of course, is available everyone except in VB circles. For non-proprietary languages (Java, Python, etc., etc.) there are a large number of competing tools. One language many tools. Take the hint. Language is important.

Yes, some tools are so flexible, they cover several languages. But there's no universal tool any more than there's a universal language. And the bias is clearly very, very many tools for a given language and only a few languages for a given tool.

How To Start

Language comes first.

For Python, that's easy. Run Python, type code at the >>> prompt, and you're learning. Python comes with IDLE which is a minimalist IDE. It will get anyone started. Later, they can try other IDE's.

For Java, however, that's not that easy. It isn't however, impossible to get started. It's just challenging.

Option 1 -- Bare Knuckles. It's possible to edit text and run the javac compiler to learn a great deal of Java without an IDE. It's not a bad idea. It will get complex to manage projects with more than a few files.

Eventually that's what Ant, Maven and SCons are for. But that's not a good place to start. Again, the tools don't make sense until you start writing things big enough that the tools actually help.

Option 2 -- Succession of IDE's. It's probably best to start with a very simple IDE for Java. Something like Komodo Edit, TextMate or BBEdit. There are a lot of choices, but the idea is to find something little more than a text editor with a few tools. I've used these and like their relative simplicity.

The JavaWIDE toolset might be helpful. I haven't used it, but some folks suggest that it simplifies the language learning. Later a "regular" desktop IDE can be used.

Later, one can move to NetBeans or Eclipse.

Classrooms and Autodidacts

In the classroom, it's easy to demonstrate NetBeans and answer questions.

For auto-didacts, however, choosing the wrong tool leads to endless confusion. The chicken and egg issue isn't clarified by wasting time trying to install and use a tool that's too sophisticated for a n00b.

N00b autodidacts really need to start with a simple text-editor. They need to use `javac` to compile and `java` to run the resulting class. For the first week or two, this will do. Once past the fundamentals, however, IDE selection can start to make sense. A BBEdit/TextMate/Komodo thing should be next. This is good for -- perhaps a year or more. Then, when doing "real" programming, a heavier-weight tool makes sense.

Test-Driven Reverse Engineering (TDRE)

2011-04-19T08:00:00.000-04:00

Another case study on TDRE.

Provided: 2,938 lines of Python code which process a handful of large files to create a number of outputs. [Details can't be disclosed.]

Objective: Refactor to distinguish between the overall sequence of transformational steps and the details of each individual step.

Observations

The code is almost purely procedural. There are 11 class definitions. 6 of these wrap built-in types with type conversion and null-handling. 1 is a new exception. 1 is a generic "table" that essentially duplicates features of SQLite. The remaining 3 are actually part of the problem domain.

One reason for reverse engineering is that the code has reached an intellectual limit. It's small, but "dense" with highly-optimized processing steps. The cohesion type is almost all "Temporal". Processing is grouped into successive processing loops; each loop contains a cluster of processing steps. Consequently, it's quite hard to tease apart the algorithm to get a "big picture" of what's going on. It's just a dense stand of trees. No forest.

Another reason for reverse engineering is to support the endless adaptation and modification of the code base. The program is a kind of "spreadsheet on steroids". This isn't a simplistic collection of cells and formulæ that permits simple what-if analysis. This is a more complex set of formulæ that would be challenging (but not impossible) to implement as a spreadsheet. The use case, however, is the spreasheet use case: think, tweak, create results, repeat.

TDRE Approach

Start with an Initial Survey of the legacy code base and sample files.

Create an Outline or "sketch" of the domain model and main program. This will be a modules (or a package) with comments and some preliminary class definitions. Little more.

Pick a processing Step in the legacy code. This often requires creating processing summaries of the legacy code. Most legacy code is procedural, so the processing tends to be sequential in nature.

Instrument the Legacy Code with print statements to gather data. This can be simple. The output can be challenging to interpret.

with open("tdre_results_1","w") as tdre:

# some legacy processing

print( "Case:", foo, bar, ", Expect:", baz, file=tdre )

From the output, Build Unit Test Cases. Fill in parts of the processing sequence and domain model. Debug code until the tests pass.

Initial Survey

The Initial Survey locates several things.

The usable, working modules. It appears that all reverse engineering involves a code base with dead or unused code. Even a small project (3,000 lines) will have a remarkable amount of dead code.
Priorities for the implemented functionality. Not every "main" module is relevant.
Example inputs and outputs.

If the software cannot be run (as is the case with organically developed systems that depend on large, complex corporate databases), then the example inputs and outputs may not actually match the software. If the software can be run, it should be run and the actuals compared against the samples to confirm that the code base supplied really produced the sample outputs.

Expect that the provided legacy code is slightly different from the code in production use. In some cases, this cannot be resolved; for example, when the executables are older than the source. In other cases, the code matches and no further work is required to establish the legacy baseline.

The sample outputs point in the direction of an acceptance test case. The sample output cannot be taken literally as the one-and-only acceptance test. While it's desirable for reverse engineering to reproduce the sample output, most reverse engineering will involve enhancements or bug fixes. Expect that errors will be found (or may be known to exist) in the sample output.

Create Outline

The outline is -- initially -- just generic MVP. There must be a domain model, some "presenter" that has the application logic, and some "view" for displaying the outputs.

In our case study, above, the "view" is a collection of (mostly text) output files. The model was undefined in the legacy code, which was all "presenter" application logic.

The goal was to extract the underlying model, break the application "presenter" logic into two layers (forest and trees) and build some views for each of the output files.

Pick a Processing Step

This can be challenging, depending on the legacy code base. There are two paths through a procedural code base.

Back to Front. Start with the final results and unit test the final steps based on previous steps that will be defined later.
Front to Back. Start with the first recognizable intermediate result based on the input files. Unit test the initial steps.

It's more rewarding to work front-to-back because progress can be shown a little more clearly.

A better architecture can be created by working back-to-front since dependencies are easier to understand.

Unit Test Volume, Edges and Corners

There are two unit test design challenges when doing reverse engineering.

Volume. The sample data can be large. 100,000 rows of sample data is too many to test. Finding a "representative" subset is difficult. Generally, arbitrary subsets have to be used to get started. Once the application mostly works, more refined unit tests need to be created.
Edge and Corner Cases. While the code may be riddled with if-statements, it can still be difficult to locate sample inputs that exercise the various conditions in the code. It's risky to create data -- we have to assume that the legacy code does unexpected things. In many cases, print statements have to be put into complex if statements to locate any actual data that exercises that logic path.

Once the unit tests are built, this is just Test-Driven Development (TDD).

Performance Discussions and Software Design

2011-04-05T08:00:00.000-04:00

Read this first: "There is something I find interesting about online discussions around performance issues..." It's about Stack Overflow, specifically. Apparently, someone didn't get their question answered and decided it was better to gripe than to rewrite the question.

Let's look at their response in pieces.

"people try to gang up". Since there's almost no social networking capability, this is a bit much to attribute to people responding to a poorly-worded question. But, if you've worked all day on a bad solution to a poorly-conceived problem, it can feel like being ganged up on. When reality leaks in, it can feel unpleasant.

Hint 1. There are no gangs. It's possible that the question really is poorly written.

"cookie-cutter, patronizing, zero-information responses". I'm guessing these are comments suggesting the approach is bad and asking for clarification. I run afoul of this often because I feel compelled to post comments asking for clarification. Some folks just don't like to clarify. More than once I've been told that their question was very clear. Since I'm asking for clarification, it seems odd to insist the question is perfect. Worse, of course, is asking for help on Stack Overflow, but refusing to clarify the help required.

Hint 2. Clarify. Please. Don't insist that the question is perfect.

"they assume, without any basis, that the person has not (a) benchmarked the code," When the question has no bench mark data, this isn't an assumption. It's a response to the lack of benchmark data.

Hint 3. Provide the facts. Don't complain when folks ask for facts.

"(b) is obviously running an inferior algorithm". Again, this isn't an assumption. It's the response to an incomplete question where the algorithm isn't provided. Also, it's a common response to questions where the algorithm really is inferior.

Hint 4. Consider that -- even after spending days banging your head against the wall -- your question might be poorly-written and require both benchmark data and an algorithm.

"advice about premature optimization... is a well assimilated folklore by now and I dont see how repeating that adds value". Without measurements, profiler results and benchmark data, this is our only possible response. After the profiler results are posted, this advice really is useless. Before profiler results are posted, this advice often turns out to be essential.

Hint 5. Whatever you might know is not well-assimilated folklore on Stack Overflow as a whole. We don't know you, sadly. We don't know how much you know. To avoid useless advice, provide evidence -- in the question -- that the advice has already been followed.

"better served if the discussion shifted to ... Pointed out possible bottlenecks ahead of time," Wouldn't that be nice? What's a "possible bottleneck"? It's a badly-design algorithm. So, the responses to performance questions has to be focused on algorithm choice right away. That means details on the code being used, and profiling information.

Hint 6. There is no hint 6. This would simply repeat hints 3 and 5.

"regardless of the fact whether the code construct is actually a bottleneck in the application or not, it is always good to know what the more efficient alternatives are... there is something called intellectual curiosity."

Reducing a question to a hand-waving hypothetical doesn't improve the question. It doesn't rationalize a poor question. The question still needs to be clarified.

Hint 7. If the question raises a lot of comments and useless advice, please rewrite the question.

StackOverflow Meetup

2011-03-31T08:57:00.004-04:00

See http://www.meetup.com/stackoverflow/Hampton-VA/ for information on next week's Stack Exchange meetup.

For other events near you, see the Meetup page.

I'll be wearing my "official" Stack Overflow shirt. I'll try to grab a seat by the door, also, to be easy to find.

#SOMeetup

Code Deletion

2011-03-30T16:36:00.004-04:00

A joyous milestone today. Removed much of our pre-Piston RESTful web services code.

We started with the Django-REST Interface. While nice, it imposed a number of restrictions that were onerous. In particular, we have a lot of non-model responses. They're model-like data that we serialize to be compatible with Django, but without actually being first-class Django Model objects.

In order to provide a generic, but detailed "status" message, we actually defined a Model that we never instantiated in the database. We'd build (but not save) instances, just to make it easy to serialize them.

What a hack.

To further complicate things, I failed to really understand the way that the alternate user authentication sources worked, and how much of the Django authentication process was better handled through middleware. Failing to fully understand that, I wrote too much code. We tinkered with incoming requests to extract HTTP Authorization headers. We tinkered to handle Amazon-style key/signature values in the GET or POST. And we tinkered to handle OpenAM authentication cookies.

Too much code.

And it gets worse. I tried to use urllib2 for a wide variety of RESTful requests. This means more than GET and POST. That was a mistake. httplib works out a little better for doing RESTful web services requests. If you don't have a lot of complex proxy server handling. And if you don't have a lot of complex authentication.

In our case, the urllib2 was handling the 401 retries, cookies and also had some extra handler code to treat a 201 Created response as a non-error (by default, urrlib2 gagged on 201 Created). Also, urllib2 appears to be lazy and doesn't send everything or close the sockets in the event of a problem. This makes unit testing just a bit more complex than necessary. Also, urllib2 required a couple of monkey patches to let us use PUT and DELETE without problems.

Needless Complexity.

It turns out that handling a 401 retry in httplib isn't really all that difficult. That ended the use case for urllib2.

What's nice is

Being able to unmake some bad decisions.
Rerunning the entire unit test suite to ferret out the remaining concealed dependencies.
Removing hack-arounds, volume and complexity.

We still have a lot of work to make full use of Piston. That will lead to removing yet more code. It will, however, also change the API's slightly because the ".../xml/..." URL's will have a different format and we'll introduce ".../django/..." URL's which will have the current format.

Where is Python Used? (Update)

2011-03-29T08:00:00.003-04:00

This is a fair-to-partly silly question that shows up on places like StackOverflow once in a while.

Python is used widely and pretty heavily.

It's a built-in feature to many operating systems in common use. The exception, of course, is Windows.

I just found out -- the hard way -- that Python 2.6 is an integral part of Apple's iLife suite of products.

Important safety tip for Mac OS X users. System.Libary.Frameworks should not be touched.

Also, it helps to get used to the idea of typing python3 on the command-line. Further, it helps to skip Python 3.1 and go straight to Python 3.2.

Python 3.2 has argparse and the new dictionary-based configuration of logging.

XBox Live -- Can't Unsubscribe

2011-03-15T08:00:00.000-04:00

Here's a lack of a use case for you.

Someone -- fraudulently -- used my email address to subscribe to XBox live. I cannot remedy this. Apparently, neither can Microsoft.

I get spam from XBox. I change my passwords all over the place.

I go to the XBox live web site to cancel this fraudulent account. I can't. There's no place to do that. I cannot cancel the account because it can only be done through the XBox console. Except -- of course -- in the case of fraud, the email user doesn't have a console.

So I call the help desk. "Please remove my email from this account that fraudulently uses it." They can't. Absolutely can't. All I can do is route xbox.com email into the spam folder. That's it.

Nice help desk agent. Doing the best she can. But, she cannot find the email address and disconnect me from spam or XBox or XBox live. Someone at the console needs to do that.

How do we contact the person at the XBox Console? Can't send them email -- it goes to me!

Somehow, someone at Microsoft has to call "beezyNdetroit" on the phone (I guess) and break the bad news to them that they're fraudulently using one of my email addresses for their XBox spam.

To Robert Fulton, Regarding the "Steam Boat"

2011-03-10T08:00:00.000-05:00

"What sir, would you make a ship sail against the wind and currents by lighting a bonfire under her deck? I pray you excuse me. I have no time to listen to such nonsense."

-- Napoleon Bonaparte

There's no authoritative source for this quote. Since Fulton was commissioned to build a submarine and did build a steam-powered boat in France, it's unlikely for this quote to be actually true.

A great list of related quotes: Famous Authoritative Pronouncements.

Improving the "Velocity" of IT

2011-03-03T08:00:00.001-05:00

Check this out: "IT Is Too Darn Slow".

This article is packed with helpful advice on how to improve "velocity" and the pace of innovation.

Good quotes: "Once IT decides to focus on speed, two obstacles get in the way: security and governance." This is important. Manage security without it becoming an impediment.

One of the most important things not mentioned here is the idea that internal security must be a simple, cheap commodity. A identity manager and SSO framework needs to be standard, available, and well-understood. Each project shouldn't involve head-scratching and deep thinking about security. Like an OS and a file system, security infrastructure should be a given.

Another good quote: "IT can't set priorities for 10 projects spread over the next two years because, once projects one, two, and three are done, that will change what would have been four, five, and six."

What isn't provided is the a way to handle the accounting practices that always seem to take IT hostage. The idea of "capital" vs. "expense" can serve as a weird artificial boundary on innovative projects. Lots of innovation gets chopped off when the "capital" budget is spent, and we switch over to "expense" where we can't invent anything new. It's the same people. Yet, the money is "different".

DIY and the Dumb Rules of Prevention

2011-03-01T08:00:00.000-05:00

Check out this little item in eWeek. "Transforming DIY Projects from the Painful to the Productive".

As an outside consultant in a large number of organizations, I've seen a lot of DIY projects—what we used to call "end user computing". Indeed, I've even been hired by the user organizations because IT was ineffective.

But IT gets in it's own way. There are some Dumb Rules of IT that creates an environment in which software development and project management are nightmares over burdensome inefficiency. Why? How?

How is easy. IT has too much process. I sat with angry users at meetings where they begged me to influence the CIO to stop making everything into a large-scale, heavy-weight, closely-monitored "project". All that the users wanted were simple conversations on how best to interface with existing applications and frameworks.

They didn't want a "project".

Root Cause

Why does IT anger the users with too much process?

That's sometimes hard to see when we're an insider. From end-users, I learned that IT's response to the inherent complexity of software development is to treat the act of creating software solutions as if the development work were—itself—just more software. Software executed by people and organizations. They write process they way they write code.

Here's Dumb Rule 1 of success in IT: "When in doubt, define a process."

And that rule is often just wrong. Users hate it.

Process is wrong? For software development, yes, too much process is a bad thing. Why do you think we had to invent Agile methods? The way we create complex software is not helped in any way by having a large-scale, heavy-weight process defined. Adding process steps to the already complex task of creating software merely slows the work down.

All processes aren't bad. Processes to create software are demonstrably unhelpful.

Suggestion One: Lighten UP

"The first step is to bring business developers out of the shadows and into the limelight."

This is important, and difficult. IT is so enamored of large processes that simply embracing non-IT developers means wrapping them in processes they don't need and can't comply with.

While good advice, this isn't the real first step.

The real first step is stop weighing down everything with overly-defined, heavy-weight process definitions. Embrace Agile development where there are sprints and releases and leave it at that.

DIY folks who are not in IT can understand sprint and release, since that's how they work anyway. IT needs to help them get started and help them release. The rest has to be hands-off. Inside corporate IT and outside the corporate IT enclave.

Suggestion Two: Open The Door

"Next, it’s important for IT to take the lead in providing tools and guidelines that make it easy for business developers to start off on the right foot."

While also good advice, it's almost impossible to actually do.

Why? IT folks who have drunk the Proprietary (i.e., Microsoft) Kool Aid laced with contractual terms and conditions won't let end-users have expensive Visual Studio toolkits. And the true believers in Microsoft hegemony can't let users have non-Microsoft tools. All the large IT organizations that are all-singing, all-dancing, all-Microsoft can't empower end users very well at all.

They offer up Sharepoint and MS-Access and then shut down all further access to tools.

Here's Dumb Rule 2 of success in IT: "Measure Success By Licensing Fees".

Visual Studio works because its expensive. Right? Oracle is "enterprise" scale because it's expensive. SQLite is unacceptable because there's no support, right?

To provide tools and guidelines, Corporate IT has to either (1) stop paying so much for tools or (2) budget for user departments to have access to real tools or both.

If we don't allow the DIY folks a decent tool set they'll still create software using crappy tools. They're remarkably capable of building vast shadow systems in MS-Access. Seven interlocked Access databases that all magically work together on a single desktop is the kind of thing they'll do if we don't help them do something better.

Suggestion Three: Stop Driving and Start Guiding

DIY folks want first class access to servers, data and tools.

"IT can take the lead in providing guidelines and even code snippets or basic scripts that work with platforms that are commonly used to create DIY projects in your organization."

"Provide small doses of advice on how to design applications for scale and security that are led by your professional-development staff. "

This can't happen, of course. The reason is that we can't "trust" the end users. If we give them access to the data, they'll make a total mess of it. We know they'll make a mess of it because we have columns in the database where the users have stopped entering the data we carefully defined in the data dictionary and started entering other data.

Why, oh why do end users corrupt the columns in the database? We tear at our hair and rend our clothes asking this. Everyone asks this. It's the great rhetorical argument against allowing users into the hallowed halls of IT.

No one tries to answer it, of course. It's a rhetorical question. The answer is embarrassing.

Users corrupt and misuse the database because a small change to add a column is a 9-month delay preceded by endless useless meetings, endless useless IT process. While waiting for IT to make a simple change, it's easier to simply enter the data into another unused field.

IT creates they're own nightmare by being unable to adapt to business change.

Here's Dumb Rule 3 of success in IT: "Control The Data".

Rather than adapt to business changes, wrap them in process as a way to say "no". Have elaborate processes. Complex budget negotiations. Project prioritization meetings.

Yes, the resources are finite. More process, however, means fewer resources devoted to solving business problems.

Code that does updates needs a lot of code review and testing no matter who wrote it. If end-users have a DIY project, IT should be deeply involved in the QA process. End users don't particularly like the kind of rigorous, expensive, sophisticated testing that IT imposes on itself. That doesn't remove the need to engage the DIY folks in some kind of test plan.

Making Progress

IT needs to offer a menu of services including "secure, reliable and available data and processing resources".

IT needs to have a simple, easy-to-understand quality threshold for software.

Programming, software purchase, integration installation and configuration and the like can be done by IT or by anyone else who meets the simple quality threshold.

A test plan. Evidence that the tests are passed. Compatibility with some standards.

It should be very, very simple so that DIY can be made to work efficiently. Since it's going to happen anyway, it's better to do it well than to fight against it with Dumb Rules.

TDD -- From SME Spreadsheet to TestCase to Code

2011-02-17T08:00:00.000-05:00

In "Unit Test Case, Subject Matter Experts and Requirements" I suggested that it's often pretty easy to get a spreadsheet of full-worked out examples from subject-matter experts. Indeed, if your following TDD, that spreadsheet of examples is solid gold.

Let's consider something relatively simple. Let's say we're working on some fancy calculations. Our users explain until they're blue in the face. We take careful notes. We think we understand. To confirm, we ask for a simple spreadsheet with inputs and outputs.

We get something like the following. The latitudes and longitudes are inputs. The ranges and bearings are outputs. [The math can be seen at "Calculate distance, bearing and more between Latitude/Longitude points".]

Latitude 1	Longitude 1	Latitude 2	Longitude 2	range	bearing
50 21 50N	004 09 25W	42 21 04N	071 02 27W	2805 nm	260 07 38

Only it has a a few more rows with different examples. Equator Crossing. Prime Meridian Crossing. All the usual suspects.

TDD Means Making Test Cases

Step one, then, is to parse the spreadsheet full of examples and create some domain-specific examples. Since it's far, far easier to work with .CSV files, we'll presume that we can save the carefully-crafted spreadsheet as a simple .CSV with the columns shown above.

Step two will be to create working Python code from the domain-specific examples.

The creation of test cases is a matter of building some intermediate representation out of the spreadsheet. This is where plenty of parsing and obscure special-case data handling may be necessary.

from __future__ import division
import csv
from collections import namedtuple
import re

latlon_pat= re.compile("(\d+)\s+(\d+)\s+(\d+)([NSWE])")
def latlon( txt ):
  match= latlon_pat.match( txt )
  d, m, s, h = match.groups()
  return float(d)+float(m)/60+float(s)/3600, h
angle_pat= re.compile("(\d+)\s+(\d+)\s+(\d+)")
def angle( txt ):
  match= angle_pat.match( txt )
  d, m, s = match.groups()
  return float(d)+float(m)/60+float(s)/3600
range_pat= re.compile("(\d+)\s*(\D+)")
def range( txt ):
  match= range_pat.match( txt )
  d, units = match.groups()
  return float(d), units

RangeBearing= namedtuple("RangeBearing","lat1,lon1,lat2,lon2,rng,brg")

def test_iter( filename="sample_data.csv" ):
  with open(filename,"r") as source:
      rdr= csv.DictReader( source )
      for row in rdr:
          print row
          tc= RangeBearing(
              latlon(row['Latitude 1']),  latlon(row['Longitude 1']),
              latlon(row['Latitude 2']),  latlon(row['Longitude 2']),
              range(row['range']),
              angle(row['bearing'])
              )
          yield tc
    
for tc in test_iter():
  print tc

This is long, but, it handles a lot of the formatting vagaries that users are prone to.

From Abstract to TestCase

Once we have a generator to build test cases as abstraction examples, generating code for Java or Python or anything else is just a little template-fu.

   
from string import Template
testcase= Template("""
class Test_${name}( unittest.TestCase ):
   def setUp( self ):
       self.p1= LatLon( lat=GlobeAngle(*$lat1), lon=GlobeAngle(*$lon1) )
       self.p2= LatLon( lat=GlobeAngle(*$lat2), lon=GlobeAngle(*$lon2) )
   def test_should_compute( self ):
       d, brg = range_bearing( p1, p2, R=$units )
       self.assertEquals( $dist, int(d) )
       self.assertEquals( $brg, map(int,map(round,brg.deg)))
""")
for name, tc in enumerate( test_iter() ):
   units= tc.rng[1].upper()
   dist= tc.rng[0]
   code= testcase.substitute( name=name, dist=dist, units=units, **tc._asdict()  )
   print code

This shows a simple template with values filled in. Often, we have to generate a hair more than this. A few imports, a "unittest.main()" is usually sufficient to transform a spreadsheet into unit tests that we can confidently use for test-driven development.

Unit Test Case, Subject Matter Experts and Requirements

2011-02-08T08:00:00.000-05:00

Here's a typical "I don't like TDD" question: the topic is "Does TDD really work for complex projects?"

Part of the question focused on the difficulty of preparing test cases that cover the requirements. In particular, there was some hand-wringing over conflicting and contradictory requirements.

Here's what's worked for me.

Preparation. The users provide the test cases as a spreadsheet showing the business rules. The columns are attributes of some business document or case. The rows are specific test cases. Users can (and often will) do this at the drop of a hat. Often complex, narrative requirements written by business analysts are based on such a spreadsheet.

This is remarkably easy for must users to produce. It's just a spreadsheet (or multiple spreadsheets) with concrete examples. It's often easier for users to make concrete examples than it is for them to write more general business rules.

Automated Test Case Construction

Here's what can easily happen next.

Write a Python script to parse the spreadsheet and extract the cases. There will be some ad-hoc rules, inconsistent test cases, small technical problems. The spreadsheets will be formatted poorly or inconsistently.

Once the cases are parsed, it's easy to then create a Unittest.TestCase template of some kind. Use Jinja2 or even Python's string.Template class to rough out the template for the test case. The specifics get filled into the unit test template.

The outline of test case construction is something like this. Details vary with target language, test case design, and overall test case packaging approach.

t = SomeTemplate()
for case_dict in testCaseParser( "some.xls" ):
  code= t.render( **case_dict )
  with open(testcaseName(**case_dict ),'w') as result:
     result.write( code )

You now have a fully-populated tree of unit test classes, modules and packages built from the end-user source documents.

You have your tests. You can start doing TDD.

Scenarios

One of the earliest problems you'll have is test case spreadsheets that are broken. Wrong column titles, wrong formatting, something wrong. Go meet with the user or expert that built the spreadsheet and get the thing straightened out.

Perhaps there's some business subtlety to this. Or perhaps they're just careless. What's important is that the spreadsheets have to be parsed by simple scripts to create simple unit tests. If you can't arrive at a workable solution, you have Big Issues and it's better to resolve it now than try to press on to implementation with a user or SME that's uncooperative.

Another problem you'll have is that tests will be inconsistent. This will be confusing at first because you've got code that passed one test, and fails another test and you can't tell what the differences between the tests are. You have to go meet with the users or SME's and resolve what the issue is. Why are the tests inconsistent? Often, attributes are missing from the spreadsheet -- attributes they each assumed -- and attributes you didn't have explicitly written down anywhere. Other times there's confusion that needs to be resolved before any programming should begin.

The Big Payoff

When the tests all pass, you're ready for performance and final acceptance testing. Here's where TDD (and having the users own the test cases) pays out well.

Let's say we're running the final acceptance test cases and the users balk at some result. "Can't be right" they say.

What do we do?

Actually, almost nothing. Get the correct answer into a spreadsheet somewhere. The test cases were incomplete. This always happens. Outside TDD, it's called "requirements problem" or "scope creep" or something else. Inside TDD, it's called "test coverage" and some more test cases are required. Either way, test cases are always incomplete.

It may be that they're actually changing an earlier test case. Users get examples wrong, too. Either way (omission or error) we're just fixing the spreadsheets, regenerating the test cases, and starting up the TDD process with the revised suite of test cases.

Bug Fixing

Interestingly, a bug fix after production roll-out is no different from an acceptance test problem. Indeed it's no different from anything that's happened so far.

A user spots a bug. They report it. We ask for the concrete example that exemplifies the correct answer.

We regenerate the test cases from the spreadsheets and start doing development. 80% of the time, the new example is actually a change to an existing example. And since the users built the example spreadsheets with the test data, they can maintain those spreadsheets to clarify the bugs. 20% of the time it's a new requirement. Either way, the test cases are as complete and consistent as the users are capable of producing.

Escaping the Relational Schema Trap

2011-02-02T08:00:00.000-05:00

We're struggling with our Relational Schema. We're not alone, of course, everyone struggles with the relational model. The technology imposes difficult limitations and we work around them.

There's kind of a 4-step process through which the relational schema erodes into irrelevance. The concept of a schema is not irrelevant. It's the rigid relational schema that's a problem.

Many DBA's will say that the relational model is the ultimate in flexibility. They're right, but they're missing the point. The relational database clearly separates the physical storage from the logical model as seen in tables and columns. It's flexible, but the presence of a rigid relational schema limits the pace of business change.

"Clearly," the DBA says, "you don't know how to use ALTER." I beg to differ. I can use ALTER; however, it doesn't permit the broad, sweeping scope of change that the business demands.

In order to attempt to match the pace of business change, we're using an ORM layer. This allows us to fabricate methods and properties left, right and center. We can tackle some pretty big problems with simple code changes. This, however, is no longer helping.

Straws and Camels

When designing a database, we have to be cognizant of the nature and tempo of change. In highly-regulated, very settled business applications (back-office accounting, for example) the data model is well known. Changes are mostly distinctive reporting changes and the tempo is pretty lethargic. It's the back office. Sorry, but innovation rarely happens there.

Each change is just a another hand-full of straw thrown on the camel's back. It happens fairly slowly. And there aren't many surprises. Hacks, workarounds and technical debt accumulates slowly.

In innovative, novel, experimental businesses, however, the nature and tempo are very different. The changes are disruptive, "what are you saying?" kinds of changes. They are "throw out the bathwater, the babies, the cribs and fire the nursemaid" kinds of changes. The tempo is semi-annual reinvent everything. Hacks, workarounds and technical debt get out of control suddenly.

Important Lesson Learned. When the customer misunderstands the offering and asks for something completely senseless, it's good to listen and try to build that -- even if it wasn't what you were offering. In some cases, the original offering was too complex or contrived. In other cases, the offering didn't create enough value. But when you offer [X] and the customer asks how much it will cost for [Y], you have disruptive, sudden, and surprising database changes.

This is bales of hay through onto an unprepared camel. Backs can get broken.

Coping

One common coping strategy is SQL ALTER statements to fiddle with the logical model. This has to be coupled with CREATE TABLE AS SELECT scripts to do open-heart surgery on the logical model. Married with modified ORM definitions. This requires some careful "schema versioning" techniques.

Another coping strategy is lots of "Expansion" columns in the tables. These can be renamed and repurposed without physical storage changes. The rows haven't physically changed, but the column name morphed from "EXPANSION_INT_01" to "Some_Real_Attribute". This doesn't prevent the CREATE TABLE AS SELECT scripts to do open-heart surgery. It still requires some careful "schema versioning" techniques to be sure that the ORM layer matches the logical schema.

A third -- and perhaps most popular -- coping strategy is manpower. Just having dedicated DBA's and maintenance programmers is a common way to handle this. Some folks object, saying that a large staff isn't a way to "cope with change" but is a basic "cost of doing business".

It's false, by the way, to claim that dedicated DBA's are essential. A solo developer can design and implement a database and application software with no help at all. Indeed, in most organizations, developers design and build databases, then turn them over to DBA's for operational support. If the nature of change is minor and tempo of change is slow, a solo developer can deal perfectly well with the database. A dedicated DBA is someone we add when the developer gets swamped by too much change.

(Some DBA's like to claim that the developers never get normalization or indexing correct. I counter with the observation that some DBA's don't get this right, either. DBA's aren't essential. They're a popular way to cope with the nature and tempo of change.)

In the ORM world, there are schema migration toolkits. Projects like Storm, this list for Django, Embarcadero Change Manager for Oracle, and numerous others attempt to support the schema evolution and change management problem. All of this is a clever way to cope with a problem inherent in our choice of technology.

Chaos Theory

Rather than invent clever coping mechanisms, let's take a step back. If we're inventing technology to work around the fixed relational schema, it might be time to rethink the relational schema.

"Oh noes," DBA's cry, "we must have a fixed logical model otherwise chaos ensues."

Really? How come we're always altering that schema? How come we're always adding tables and restructuring the tables?

"Oh that? That's 'controlled change'," the DBA responds.

No, that's slow chaos.

Here's how it plays out. We have a disruptive change. We negotiate with the DBA's to restructure the database. And the test database. And the QA database. We do the development database without any help from the DBA's. We fix the ORM layers. We unit test the changes.

Then we plan and coordinate the production rollout of this change with the DBA's. Note. We already made the change in development. We're not allowed to make the change in production. The DBA's then suggest design alternatives. Normalization isn't "right". Or there are physical changes that need to be declared in the table definitions. We redo the development database. And the ORM layer. And rerun the unit tests.

Because the production database couldn't be touched -- and we had paying customers -- we copied production data into a development database and started doing "production" in development. Now that we're about to make the official production change, we have two databases. The official database content is out-of-date. The development database is a mixture of live production and test data. Sigh.

Rethinking Schema

If the schema is a problem, perhaps we can live without it. Enter NoSQL databases.

Here's how you start down the slippery slope.

Phase I. You need a fairly radical database change. Rather than wait weeks for the DBA's, you ask for a single "BLOB" column. You take the extra data elements for the radical change, JSON encode them, and store the JSON representation in the BLOB field. Now you have a "subschema" buried inside a single BLOB column.

Since this is a simple ALTER, the DBA's will do it without a lot of negotiation or delay. You have a hybrid database with a mixture of schema and noSQL.

Phase II. You need an even more radical change. Rather than wait weeks for the DBA's, you ask for a few tables that have just a primary key and a BLOB column. You've basically invented a document-structured database inside SQL, bypassing the SQL schema entirely.

Phase III. While waiting for the Phase II changes to be implemented, you convert the customer data from their obscure, stupid format into a simple sequential file of JSON documents and write your own simple map-reduce algorithms in Python. Sure, performance is poor, but you're up and running without any database overheads.

Phase IV. Start looking for alternatives.

MongoDB, CouchDB, MySQL Compare Grid

This MongoDB looks really nice. PyMongo offers lots of hints and guidance.

At least one person is looking at mango, a MongoDB database adapter for Django. For us, this isn't the best idea. We use OpenAM for identity management, so our Users and Sessions are simply cloned from OpenAM by an authentication backend that gets the user from OpenAM. SQLite works fine for this.

We think we can use Django's ORM and a relational database for User and Session. For everything else, we need to look closely and MongoDB.

Wins and Losses

The big win is the ability to handle disruptive change a little bit more gracefully.

The big loss in switching away from the Django ORM is we lose the built-in admin pages. We have to build admin Forms and view functions. While this is a bit of a burden, we've already customized every model form heavily. Switching from ModelForm to Form and adding the missing fields isn't much additional work.

The biggest issue with document-oriented data models is assuring that the documents comply with some essential or core schema. Schemas are inescapable. The question is more a matter of how the schema limits change. Having a Django Form to validate JSON documents for the "essential" features is far more flexible than having a Django Model class and a mapping to a relational database.

Schema migration becomes a non-issue until we have to expand the essential schema, which changes the validation rules, and may render old documents retroactively invalid. This is not a new problem -- Relational folks cope with this, also -- but if it's the only problem, then we may have streamlined the process of making disruptive business changes.

FAERIE DUST™

2011-01-27T08:00:00.000-05:00

Here's how to recognize a Faerie Dust request:

We have identified a problem. It can be with almost anything: scalability, reliability, auditability, any Quality Measure.
We're pursuing a specific technology. Typically, something that has the lowest impact on our architecture.
We can't address anything other than this specific technology variation -- we can't change the application software or buy hardware.

Once we're in the Faerie Dust realm, what can we do?

Laughing doesn't help. They have a serious problem, they need a solution. The fact that they won't address the cause isn't completely relevant -- we have to work on the denial, anger, negotiation, depression cycle first. Hopefully skipping past the anger, or assuring the anger is directed elsewhere.

Helping doesn't help. If we join the quest for their Faerie Dust, what will we accomplish? We'll burn billable hours to -- eventually -- reach an equivocal non-solution with a complex write-up and recommendations that won't be implemented.

Not helping doesn't help. If we obstinately refuse to join the quest for the Faerie Dust... well... then we've done nothing. We haven't advanced their understanding of their problem.

What's left? Is there a middle road that allows us to join the Faerie Dust quest, but still point out the side roads, other monsters and other treasures along the way?

Perhaps there is, but it would require a kind of saintly patient persistence. We would have to start with an enumeration of problem causes, prioritize them, and then focus on their selected bit of Faerie Dust. My idea is that enumerating the possible causes allows us to identify the missed opportunities, and the possible magnitude of fixing something essential (algorithm or data structure) instead of throwing up window-dressing to cover problems in something inessential (reducing the time required for a table scan).

Example

Here's a concrete example of Faerie Dust.

Pick a data model that doesn't fit the use cases. i.e., lumped many discrete details into a single text field that has "rich semantic content". Work around this mistake by using wild-card matches.
Complained about performance and dug into nuanced details of LIKE clause and full-text search. Lots of study time spent on LIKE clause processing and how to improvement performance.
Refused to discuss the actual use case or the mismatch between data structures and requirements.

The design didn't match the use cases. Faerie Dust won't help.

Wild-Card (LIKE-clause) searches are slow. What to do?

2011-01-25T08:00:00.001-05:00

Patient: "Doctor, doctor, it hurts when I do this."

Doctor: "Then don't do that."

I got an email with hundreds of words of content. This part made sense: "...doing wild card searches using Oracle's database engine and are wondering why is it so slow and how do they make it go faster."

The rest made very little sense at all. The programmer in question immediately dove into nuances of indexing, Oracle pattern matching, Oracle Text Query and other technical questions. The entire focus was on the technical ins-and-outs.

Not a single word on why wildcards were even being used in the first place. Wildcards appear to solve a business problem; the business problem was never mentioned.

Use Case for Wildcards

After some back-and-forth, the use case emerged. We'll address it below. Essentially, the invoices have names (really) that have "rich semantic content". These invoice names have the form "{customer} {time period} {offering}".

Apparently, the use case is "slice-and-dice" queries. All invoices for a given customer; all invoices in a given time period; all invoices for a given offering; various combinations.

Really. Rather than provide discrete dimensions and use a star schema, they've (a) combined all attributes into a single free-text field and (b) used wild-card searches and now (c) want to complain about it.

We'll return to this use case below.

Basic Rules

Here's are the two rules.

Wild Cards Are The Last Resort For Human-Friendly Search.

Outside Human-Friendly Search, Wild Cards Are Useless.

Let's look at rule 1: Wild Cards Are The Last Resort For Human-Friendly Search.

When a person enters a search string on a web page, we have two choices.

Trust them to enter the exact field as it appears in the database
Presume that people are fallible and cannot be trusted to enter the exact field.

In case #1 (exact match) we might be using an account number, shipping number, an invoice number or some kind of surrogate key. In this case, we do simple equality checks. If the user can't get it right, bummer. In many cases, this is appropriate to prevent snooping.

In case #2 (partial match), we're forced to use a some kind of SQL LIKE clause for the human-friendly search. We have several implementation choices, some in the database, some out of the database. Some in-the-database solutions benefit from clever indexing. Many in-the-database solutions are pretty slow.

Yes, an out-of-the-database solution may actually be faster. Until we benchmark, we can't know. There's no trivial rule that says the database always does search faster. For real speed, we may have to resort to a hybrid solution.

Search Optimization

We might create a small RESTful server for our searchable text fields. This is a cache; the server should handle CRUD rules to assure cache coherence. This search server can uses a Regular Expression engine, or perhaps compute Levenshtein distances or whatever makes sense to optimize user-oriented search.

If we're searching in larger chunks of text, we might want to use a commercial full-text search.

What's essential about this plan is that we're looking at application-specific optimizations. People need flexibility for specific reasons. It's important to look at the actual use cases where a person cannot make an exact match lookup. What problems do they have?

An application may have to deal with customer names. These are often difficult to spell consistently. (Is it "AT&T" or "ATT"?) For this kind of thing Levenshtein Distance might make more sense than wild-card searches.

An application may have to deal with time periods. "2010", "2Q 2010", "July 2010", etc. This is best handled by decomposing time periods into discrete fields and doing appropriate exact match on the specific, relevant fields. The issue is that there are a lot of formulations and some text parsing can be better than a form with a million drop-downs.

An application may have to deal with oddly-named offerings. Marketing calls it one thing. Sales folks call it another. The customer's invoice may call it a third, and the help desk may not use any of those phrases. This may benefit from wild-cards.

Note that we're looking at the business issues. Not the technology issues.

Design Errors

The proper use for LIKE is only to optimize the human-friendly search. Nothing else. Which brings us to rule 2, Outside Human-Friendly Search, Wild Cards are Useless.

Outside human search, every wild-card in a SQL statement indicates a serious database design error. Serious? Error? Yes.

LIKE clauses outside human search indicate a failure to create a design in first normal form (1NF). A field which is used in a LIKE clause has multiple parts, and should have been decomposed into pieces.

Decomposing a multi-part attribute isn't always trivial. There are two cases.

Simple, regular format or punctuation. For example, SSN, US Phones or ZIP codes: 123-45-6789 or (123)555-1234 or 12345-1234.
Complex, irregular format or punctuation. In this case, we have disjoint subtypes in a single table. Most manufacturing part numbers suffer from this.

In case 1, we have two choices: fully decompose or denormalize. In case 2, we can only denormalize because the rules are irregular.

The decomposition solution does not have to lead to a hideous user interface. We can have a web page with a single text field for phone numbers. We can parse that string and decompose the phone number into area code, exchange and number for purposes of database storage. We don't have to thoughtlessly force the users to decompose a field that they don't see as being in three parts.

The denormalization solution means that we have to do some calculation when we accept the input value. We save the full field, plus we extract the various sub-fields based on whatever hellish, complex rules we're faced with.

Implementation Choices

Whenever we have a single text field with "rich semantic content" (i.e., combines multiple disjoint attributes like customer, time period and offering) what we're seeing is a clever way to push database design onto the users. The expectation is that IT will (1) understand the use cases, (2) provide a proper design and (3) optimize performance around that design.

A big text field and wild-card search (and the attendant email traffic) indicates an explicit unwillingness to discuss the real use cases, unwillingness to do design, and a lame hope that somehow wild-card searches can magically be made faster through magical indexing or other super-natural techniques.

The "rich semantic content" field can be decomposed one of two ways.

In the GUI. Add drop-downs so users pick the customer, time period, and product offering information.
In the Application. Parse the big text field into smaller text fields that don't require wild-card search.

There isn't any magic. If wild-card searches are too slow, they have to be replaced.

Benefits?

The benefit of decomposing (or denormalizing) a complex field is that we can eliminate LIKE processing and wild-cards. Instead of "LONG_TEXT_FIELD LIKE '%2Q 2010%'", we can do "DATE.QUARTER=2 AND DATE.YEAR=2010".

All the technical folderol related to indexing and full-text search and database regular expression engines goes right out the window.

The cost is that we have to "wrap" the INSERT and UPDATE processing in a class definition that does the denormalization. That's what a data model layer is for: these kinds of business rules. The insert/update cost, BTW, will be microscopic compared to the number of SELECTs. The extra time spent at INSERT will be handsomely amortized over all the simplified SELECT operations.

Python Jobs

2011-01-24T08:00:00.000-05:00

Looked interesting enough for someone to email it to me. Still can't figure out why they sent it. Posting this feels like advertising, so perhaps I should charge them a promotional fee.

Web Services - Mobile Developer

"A minimum of 2 years of Web Service development, preferably using an established Python-based framework (Django or Pylons).

Demonstrated proficiency with Python is a must."

http://www.maxhire.net/cp/?E85B6B361D43515B7E571E2877501F6A02627C49

Maybe they sent it as confirmation that Python is somehow a "real" technology.

Client-Server Partitioning

2011-01-11T08:00:00.000-05:00

I have slowly grown to love RESTful web services.

I was asked about a nearly empty section in the code repository labeled "Java client".

"Yes," I said, "it's a place-holder for a Java package that includes classes to wrap our RESTful web services."

"Really?" I was asked, "Why? We use FLEX for the client, not Java Applets."

"Today we use FLEX. In the past, we weren't sure. We have a complete Python client library. Indeed, the original concept was to support our customer's building their own web site to use our web services. The Java package would plug into a J2EE web app. No one wanted that, so we built FLEX clients instead."

Then they said that the super-clean separation between all these clients and the RESTful server was taking "flexibility to a whole new level."

I pointed them to this. EWD340: The Humble Programmer. It has this killer quote:

The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.

I find that this really helps keep the focus on simplicity. I suppose that it leads to flexibility, but that's not the real point. The real point was simplicity.

Java PHP Python -- Which is "Faster In General"?

2011-01-06T08:00:00.001-05:00

Sigh. What a difficult question. There are numerous incarnations on StackOverflow. All nearly unanswerable. The worst part is questions where they add the "in general" qualifier. Which is "faster in general" is essentially impossible to answer. And yet, the question persists.

There are three rules for figuring out which is faster.

And there are three significant problems that make these rules inescapable.

Rule One. Languages don't have speeds. Implementations have speeds.

Info on benchmarking. The idea of a benchmark is to have a single, standard suite of source code, which can be used to compare compilers, run-time libraries or hardware.

Having a standard suite of source is essential because it provides a basis for comparison. A single benchmark source is the fixed reference. We don't compare the top of the Empire State Building with the top of the Stratosphere in Las Vegas without specifying whether we care about height above the ground or height above sea level. There has to be some fixed point of comparison, some common attribute, or the measurements devolve into mere numerosity.

Once we have a basis for comparison (one single body of source code), the other attributes are degrees of freedom; the measurements we make will include the other attributes. This will allow a rational statement of what the experimental results where. We can then compare these various free attributes against each other. For details look at something like the Java Micro Benchmark.

Rule Two. Statistics Aren't a Panacea.

The reason there's no "in general" comparison among languages is because there are too many degrees of freedom to make any kind of rational comparison. We can make irrational comparisons, but that's the trap of numerosity -- throwing numbers around. 1250 vs. 1149, 1300 vs. 3177. What do they mean? Height above ground? Height above sea level? What's being measured?

There's a huge problem with claiming that statistics will yield an answer to which language implementation is faster "in general". We need some population that we can sample and measure. Problem 1: What the population are we measuring? It can't be "programs": we can't compare grep against Apache httpd. Those two programs have almost no common features.

What makes the population of programs difficult to define is the language differences. If we're trying to compare PHP, Python and Java, we need to find a program which somehow -- magically -- is common across all three languages.

The Basis For Comparison

Finding common programs degenerates into Problem 2: what programs could be comparable? For example, we have the Tomcat application, written in Java. We wouldn't want to write Tomcat in Python (since Tomcat is a Java Servlet container). We could probably write something Tomcat-like in PHP, but why try? So we can't just grab programs randomly.

At this point, we devolve to subjectivity. We need to find some kind of problem domain in which these languages overlap. This gets icky. Clearly, big servers aren't a good problem domain. Almost as clearly, command-line applications aren't the best idea. PHP does run from the command-line, but it's always contrived-looking because it doesn't exploit PHP's strengths. So we wind up looking at web applications because that's where PHP excels.

Web applications? Subjective? Correct. PHP is a language plus a web application framework bundled together. Java and Python -- by themselves -- are just languages and require a framework. Which Java (and Python) framework is identical to PHP's framework? Spring, Struts, Django, Pylons? None of these reflects a code base that's even remotely similar. Maybe Java JSP is similar enough to PHP. For Python there are several implementations. Sigh.

Crappy Program Problem

We can't easily compare programs because we're really comparing implementations of an algorithm. This leads to Problem 3: we picked a poor algorithm or did a lousy job of implementing it in the target language.

In order to be "comparable", we don't want to exploit highly-optimized or unique features of a language. So we tried to be generic. This is fraught with risks.

For example, Java and PHP don't have list comprehensions. Do we forbid them from our Python behchmark? In Python, everything is a reference, values cannot be copied. If we pick an algorithm implementation which depends on copying objects, Java may appear to excel. If we pick an algorithm implementation which depends on sharing references, Python may appear to excel.

Somehow we have to get past language differences and programmer mistakes. What to do?

Synthetic Benchmarks

Since we can't easily find comparable programs -- as whole programs -- we're left with the need to create some kind of benchmark based on language primitives. Statements or expressions or something. We can try to follow the Whetstone/Dhrystone approach of analyzing a bunch of programs to find the primitive constructs and their relative frequency.

Here's the plan. We'll take 100 PHP programs, 100 Java programs and 100 Python programs and analyze them to find the relative frequency of different kinds of statements. What then?

The goal is to create one source that reflects the statements actually used in the 300 programs we analyzed. In three different languages. Hmmm... Okay. We'll need to create a magical mapping among the statement constructs in the three languages. Well, that's hard. The languages aren't terribly comparable. A Python expression using a List Comprehension is the same thing a multi-statement Java loop. Rats.

The languages aren't very comparable at the statement level at all. And if we force them to be comparable, we're not comparing real programs, but an artificial mapping.

Virtual Machine Benchmarks

Since we can't compare the languages at the program level or the statement level, what's left? Clearly, the underlying interpreter is what we care about.

We're really comparing the Java Virtual Machine, the PHP interpreter and the Python interpreter. That's what we really care about.

And life is simple because we can compare Java, The Project Zero PHP Interpreter based on the JVM and Jython. We can look at "compiled" PHP, Java Class Files and Python .PYC files to find the VM primitives used by each language and then -- what? Compare the run-time of the various VM primitives? No, that's silly, since the run-times are all JVM run-times.

What We're Left With

The very best we can can do is to compare the statistical distribution of the VM instructions created by Java, PHP or Jython compilers. We could note that maybe PHP or Python uses too many "slow" VM instructions, where Java used more "fast" VM instructions. That would be an "in general" comparison. Right?

See? You can measure anything.

In this case, the compiler itself is a degree of freedom. Sadly, we're not comparing languages "in general". We're comparing the bytecodes created by various compilers. We're actually comparing compilers and compiler optimizations of the bytecode. Sigh.

That's not what we were hoping for. We were hoping for some kind of "in general" comparison of the language, not the JVM compiler.

Java has pretty sophisticated optimization. Python, however, eschews optimization. PHP has it's own weird issues. See this paper from Rob Nicholson from the CodeZero project on how to implement PHP in the JVM. PHP doesn't fit the JVM as well as Python does. So there's a weird bias.

Rule Three. Benchmarking Is Hard.

There is no "in general" comparison of programming languages. All that we can do is benchmark something specific.

It works like this.

Stop quibbling about language performance "in general".
Find something specific and concrete we plan to implement.
Actually write the performance-critical piece in Java, PHP, Python, Ruby, whatever. Yes. Build it several times. Really. We don't want to use "language-independent" or "common" features. We want to optimize ruthlessly -- use the language the way it was meant to be used. -- use the various unique-to-the language features correctly and completely.
Actually run the performance-critical piece to get actual timings.
Since run-time libraries and hardware are degrees of freedom, we have to use multiple run-time libraries, multiple compiler optimization settings and multiple hardware configurations to make a proper decision on which language to use for our specific problem.

Now we know something about our specific problem domain and the available languages. That's the best we can do.

We can only compare a specific problem, with a specific algorithm. That's the basis for all benchmark comparisons. Since each implementation was well-done and properly optimized, the degree of freedom is the language -- and the run-time implementation of that language -- and the selected OS and hardware.

Search For Expertise

2011-01-06T07:50:00.002-05:00

I'm looking for a unbiased Python expert to help with a book I'm working on. We need "an unbiased python expert with a keen eye for detail."

The role is technical reviewer. I've never done this before, but it appears that the tech reviewer is a paid position somewhere in a publication pipeline along with copy editing and other production steps. It doesn't sound like a full-time job, since the chapters trickle through the pipeline slowly.

I'm guessing that you'd get to correct the misstatements and run all the code examples.

Integration Testing, unittest and Python 2.7

2011-01-04T08:00:00.002-05:00

Many folks use Python's unittest module for integration testing. It sometimes leads to whining and hand-wringing, but it is very effective.

Ordinary "unit" tests use mocks and focus on a class or a module more-or-less in isolation. The purists say "complete isolation". But that's sometimes unrealistic. A class that's part of a State design pattern or a Strategy design pattern is often so trivial that "pure" unit tests aren't really very enlightening. Using the stateful class along with it's state class hierarchy is usually far more interesting and helpful.

I argue that it's still "unit" testing because parts of the application have been extracted from their processing context.

OS and Module Mocks

Some folks will hand-wring over the use of OS API's or other library API's in a unit test. They feel that the OS API's should be completely mocked in order to call it a "unit" test. In most cases, simplistic mocks can be created, but for most use cases, the mock grows to hellish complexity.

For example, we have a large number of tests which depend on Python's csv module. While we could mock csv to produce row after row of mocked data, it seems much simpler to trust that our infrastructure works, use csv and simply provide files of test data in CSV format. The file is part of the test fixture and is locked up in the source code repository.

Is this "unit" testing when we're integrated with some of the underlying modules?

We don't separately test csv. We simply trust that it library modules have their own tests. If we're willing to trust the already-supplied tests for csv, why not use it in our tests? Why Mock something we've decided to trust?

[For that matter, we also trust Python, the OS and the unittest module itself. Why draw lines between Python, unittest and csv?]

ORM and RDBMS Mocks

A similar kind of worry comes with an ORM layer. Somehow, the "trust" factor starts to break down here. Folks want to mock the ORM layer. Or -- worse -- they want to try and mock the RDBMS layer so that they're testing the application and ORM layer. This is another artificial distinction between stuff we'll trust (the RDBMS layer) and stuff we feel we must write additional tests for (the ORM layer).

It's so much nicer to adopt the Django philosophy of building a test database as part of the test fixture. The ORM and RDBMS are integrated into the test itself. The thing that's "mocked" is the data which gets pre-loaded into the database.

Outside of Django, this can be a bit complex to set up properly. You need to create a temporary database, execute the SQL DDL, load the fixture data. This is an annoying bit of code to write the first time. However, it has handsome rewards because the "unit" testing includes the ORM and RDBMS at a higher level where it's easier to work with.

Integration Testing

We use the unittest module to do integration testing also. In Python 2.6, this involved a fair amount of work. We had to start our RESTful server, run the unit test, and then kill the server. Because of the way the subprocess module works, we can't be completely sure the server is running, so we simply cheat and use a time.sleep(12) to wait for the DB to be built and loaded.

Python 2.7 adds module-level test cases. It checks for setUpModule and tearDownModule functions. I've been gleefully revising all our unit tests to make ready for this. Our previous testing needed pervasive (but minor) rewrites to refactor the database setup and tear down and provide proper names.

Once that's in place, our big test shell script can be simplified down to a little Python unittest module. This module will build a unittest.Suite from all the other test modules (there are dozens) and simply execute the suite as a single, integrated whole.

pyWeb Literate Programming Tool | Download pyWeb Literate Programming Tool software for free at SourceForge.net

2010-12-30T13:29:00.002-05:00

I've (finally) updated the pyWeb Literate Programming Tool.

There were feature requests and bug reports. Much to do. Sadly, I'm really slow at doing it.

pyWeb Literate Programming Tool | Download pyWeb Literate Programming Tool software for free at SourceForge.net

Top Language Skills

2010-12-30T08:00:00.000-05:00

Check out this item on eWeek: Java, C, C++: Top Programming Languages for 2011 - Application Development - News & Reviews - eWeek.com.

The presentation starts with Java, C, C++, C# -- not surprising. These are clearly the most popular programming languages. These seem to be the first choice made by many organizations.

In some cases, it's also the last choice. Many places are simply "All C#" or "All Java" without any further thought. This parallels the "All COBOL" mentality that was so pervasive when I started my career. The "All Singing-All Dancing-All One Language" folks find the most shattering disruptions when their business is eclipsed by competitors with language and platform as a technical edge.

The next tier of languages starts with JavaScript, which is expected. Just about every web site in common use has some JavaScript somewhere. Browsers being what they are, there's really no viable alternative.

Weirdly, Perl is 6th. I say weirdly because the TIOBE Programming Community Index puts Perl much further down the popularity list.

PHP is next. Not surprising.

Visual Basic weighs in above Python. Being above Python is less weird than seeing Perl in 6th place. This position is closer to the TIOBE index. It is distressing to think that VB is still so wildly popular. I'm not sure what VB's strong suit is. C# seems to have every possible advantage over VB. Yet, there it is.

Python and Ruby are the next two. Again, this is more-or-less in the order I expected to see them. This is is the second tier of languages: really popular, but not in the same league as Java or one of the innumerable C variants.

After this, they list Objective-C as number 11. This language is tied to Apple's iOS and MacOS platforms, so it's popularity (like C# and VB) is driven in part by platform popularity.

Third Tier

Once we get past the top 10 Java/C/C++/C#/Objective C and PHP/Python/Perl/Ruby/Javascript tier, we get into a third realm of languages that are less popular, but still garnering a large community of users.

ActionScript. A little bit surprising. But -- really -- it fills the same client-side niche as JavaScript, so this makes sense. Further, almost all ActionScript-powered pages will also have a little bit of JavaScript to help launch things smoothly.

Now we're into interesting -- "perhaps I should learn this next" -- languages: Groovy, Go, Scala, Erlang, Clojure and F#. Notable by their absence are Haskell, Lua and Lisp. These seem like languages to learn in order to grab the good ideas that make them both popular and distinctive from Java or Python.

Amazing Speedup

2010-12-28T08:00:00.000-05:00

A library had unit tests that ran for almost 600 seconds. Two small changes dropped the run time to 26 seconds.

I was amazed.

Step 1. I turned on the cProfile. I added two methods to the slowest unit test module.

def profile():
    import cProfile
    cProfile.run( 'main()', 'the_slow_module.prof' )
    report()

def report():
    import pstats
    p = pstats.Stats( 'the_slow_module.prof' )
    p.sort_stats('time').print_callees(24)

Now I can add profiling or simply review the report. Looking at the "callees" provided some hints as to why a particular method was so slow.

Step 2. I replaced ElementTree with cElementTree (duh.) Everyone should know this. I didn't realize how much this mattered. The trick is to note how much time was spent doing XML parsing. In the case of this unit test suite, it was a LOT of time. In the case of the overall application that uses this library, that won't be true.

Step 3. The slowest method was assembling a list. It did a lot of list.append(), and list.__len__(). It looked approximately like the following.

def something( self ):
result= []
for index, value in some_source:
    while len(result)+1 != index:
        result.append( None )
    result.append( SomeClass( value ) )
return result

This is easily replaced by a generator. The API changes, so every use of this method function may need to be modified to use the generator instead of the list object.

def something_iter( self ):
 counter= 0
 for index, value in some_source:
     while counter+1 != index:
         yield None
         counter += 1
     yield SomeClass( value )
     counter += 1

The generator was significantly faster than list assembly.

Two minor code changes and a significant speed-up.

The Anti-IF Campaign

2010-12-23T08:00:00.001-05:00

Check this out: http://www.antiifcampaign.com/.

I'm totally in favor of reducing complexity. I've seen too many places where a Strategy or some other kind of Delegation design pattern should have been used. Instead a cluster of if-statements was used. Sometimes these if-statements suffer copy-and-paste repetition because someone didn't recognize the design pattern.

What's important is the the if statement -- in general -- isn't the issue. The anti-if folks are simply demanding that folks don't use if as a stand-in for proper polymorphism.

Related Issues

Related to abuse of the if statement is abuse of the else clause.

My pet-peeve is code like this.

if condition1:
   work
elif condition2:
   work
elif condition3:
   work
else:
   what condition applies here?

When the various conditions share common variables it can be very difficult to deduce the condition that applies for the else clause.

My suggestion is to Avoid Else.

Write it like this.

if condition1:
   work
elif condition2:
   work
elif condition3:
   work
elif not (condition1 or condition2 or condition3)
   work
else:
   raise AssertionError( "Oops.  Design Error.  Sorry" )

Then you'll know when you've screwed up.

[Update]

Using an assert coupled with an else clause is a kind of code-golf optimization that doesn't seem to help much. An elif will have the same conditional expression as the assert would have. But the comment did lead to rewriting this to use AssertionError instead of vague, generic Exception.

Code Base Fragmentation -- Again

2010-12-14T08:00:00.000-05:00

Check this out: "Stupid Template Languages".

Love this: "The biggest annoyance I have with smart template languages (Mako, Genshi, Jinja2, PHP, Perl, ColdFusion, etc) is that you have the capability to mix core business logic with your end views, hence violating the rules of Model-View-Controller architecture."

Yes, too much power in the template leads to code base fragmentation: critical information is not in the applications, but is pushed into the presentation. This also happens with stored procedures and triggers.

I love the questions on Stack Overflow (like this one) asking how to do something super-sophisticated in the Django Template language. And the answer is often "Don't. That's what view functions are for."

The Wrapper vs. Library vs. Aspect Problem

2010-12-09T08:00:00.000-05:00

Imagine that we've got a collection of applications used by customers to provide data, a collection of applications we use to collect data from vendors. We've got a third collection of analytical tools.

Currently, they share a common database, but the focus, use cases, and interfaces are different.

Okay so far? Three closely-related groups or families of applications.

We need to introduce a new cross-cutting capability. Let's imagine that it's something central like using celery to manage long-running batch jobs. Clearly, we don't want to just hack celery features into all three families of applications. Do we?

Choices

It appears that we have three choices.

A "wrapper" application that unifies all the application families and provides a new central application. Responsibilities shift to the new application.
A site-specific library that layers some common features so that our various families of applications can be more consistent. This involves less of a responsibility shift.
An "aspect" via Aspect-Oriented programming techniques. Perhaps some additional decorators added to the various applications to make them use the new functionality in a consistent way.

Lessons Learned

Adding a new application to be an overall wrapper turned out to be a bad idea. After implementing it, it was difficult to extend. We had two dimensions of extension.

The workflows in the "wrapper" application needed constant tweaking as the other applications evolved. Every time we wanted to add a step, we had to update the real application and also update the wrapper. Python has a lot of introspection, but these aren't technical changes, these are user-visible workflow changes.
Introducing a new data types and file formats was painful. The responsibility for this is effectively split between the wrapper and the underlying applications. The wrapper merely serves to dilute the responsibilities.

Libraries/Aspects

It appears that new common features are almost always new aspects of existing applications.

What makes this realization painful is the process of retrofitting a supporting library into multiple, existing applications. It seems like a lot of cut-and-paste to add the new import statements, add the new decorators and lines of code. However, it's a pervasive change. The point is to add the common decorator in all the right places.

Trying to "finesse" a pervasive change by introducing a higher-level wrapper isn't a very good idea.

A pervasive change is simply a lot of changes and regression tests. Okay, I'm over it.

Intuition and Experience

2010-12-07T08:00:00.001-05:00

First, read EWD800.

It has harsh things to say about relying on intuition in programming.

Stack Overflow is full of questions where someone takes their experience with one language and applies it incorrectly and inappropriately to another language.

I get email, periodically, also on this subject. I got one recently on the question of "cast", "coercion" and "conversion" which I found incomprehensible for a long time. I had to reread EWD800 to realize that someone was relying on some sort of vague intuition; it appears that they were desperate to map Java (or C++) concepts on Python.

Casting

In my Python 2.6 book, I use the word "cast" exactly twice. In the same paragraph. Here it is.

This also means the "casting" an object to match the declared type
of a variable isn't meaningful in Python. You don't use C++ or Java-style
casting.

I though that would be enough information to close the subject. I guess not. It appears that some folks have some intuition about type casting that they need to see reflected in other languages, no matter how inappropriate the concept is.

The email asked for a "a nice summary with a simple specific example to hit the point home."
It's quite hard to provide an example of something that doesn't exist. But, I guess, intuition provides a strong incentive to see things which aren't there. I'm not sure how to word it more strongly or clearly. I hate to devolve into blow-by-blow comparison between languages because there are concepts that don't map. I'll work on being more forceful on casting.

Coercion

The words coercion (and coerce) occur more often, since they're sensible Python concepts. After all, Python 2 has formal type coercion rules. See "Coercion Rules". I guess my summary ("Section 3.4.8 of the Python Language Reference covers this in more detail; along with the caveat that the Python 2 rules have gotten too complex.") wasn't detailed or explicit enough.

The relevant quote from the Language manual is this: "As the language has evolved, the coercion rules have become hard to document precisely; documenting what one version of one particular implementation does is undesirable. Instead, here are some informal guidelines regarding coercion. In Python 3.0, coercion will not be supported."

I guess I could provide examples of coercion. However, the fact that it is going to be expunged from the language seems to indicate that it isn't deeply relevant. It appears that some readers have an intuition about coercion that requires some kind of additional details. I guess I have to include the entire quote to dissuade people from relying on their intuition regarding coercion.

Further, the request for "a nice summary with a simple specific example to hit the point home" doesn't fit well with something that -- in the long run -- is going to be removed. Maybe I'm wrong, but omitting examples entirely seemed to hit the point home.

Conversion

Conversion gets it's own section, since it's sensible in a Python context. I kind of thought that a whole section on conversion would cement the concepts. Indeed, there are (IMO) too many examples of conversions in the conversion section. But I guess that showing all of the numeric conversions somehow wasn't enough. I have certainly failed at least one reader. However, I can't imagine what more could be helpful, since it is -- essentially -- an exhaustive enumeration of all conversions for all built-in numeric types.

What I'm guessing is that (a) there's some lurking intuition and (b) Python doesn't match that intuition. Hence the question -- in spite of exhaustively enumerating the conversions. I'm not sure what more can be done to make the concept clear.

It appears that all those examples weren't "nice", "simple" or "specific" enough. Okay. I'll work on that.

More Open Source and More Agile News

2010-12-02T08:00:00.000-05:00

ComputerWorld, November 22, 2010, has this: "Open Source Grows Up". The news of the weird is "It's clear that open-source software has moved beyond the zealotry phase." I wasn't aware this phase existed. I hope to see the project plan with "zealotry" in it.

The real news is "More than two-thirds (69%) of the respondents said they expect to increase their investments in open source." That's cool.

Be sure to read the sidebar "Many Enterprises Aren't Giving Back." There's still a lot of concern over intellectual property. I've seen a lot of corporate software -- it's not that good. Most companies that are wringing their hands over losing control of their trade secrets should really be wringing their hands because their in-house software won't measure up to open-source standards.

I like this other quote: 'Five years ago, the South Carolina government was "considering writing a policy to prohibit or at least 'control' open source".' I like the "Must Control Open Source" feeling that IT leadership has. Without this mysterious "control", the organization could be swamped by software it didn't write. How's that different from being swamped by software products that involve contracts and fees? And requires Patch Tuesday?

Agility

SD Times has two articles on Agile methods. Both on the front page of a print publication. That's how you know the technique has "arrived".

First, there's "VersionOne survey finds agile knowledge and use on the rise". My favorite quote: "Interestingly, management support, the ability to change organizational culture and general resistance to change, remained at the forefronts of participants’ minds when indicating barriers to further agile adoption." I like the management barriers. I like it when management tries to exert more 'control' over a process (like software creation) that's so poorly understood.

Here's the companion piece, "For agile success, leaders must let teams loose". This is all good advice. Particularly, this: '"It’s hard to not command and control, but leadership is not about managing work. It’s about creating a capable organization that can manage work," [Rick Simmons] added.'

If you're micro-managing, you're not building an organization. Excellent advice. However, tell that to the financial control crowd.

Budgets and "Control"

Finally, be sure to read this by Frank Hayes in ComputerWorld: "Big Projects, Done Small". Here are the relevant quotes: "The logical conclusion: We should break up all IT projects into sub-million-dollar pieces." "The political reality: Everybody wants multimillion-dollar behemoths." "...huge projects get big political support."

In short, Agile is the right thing to do until you're trying to get approval. Bottom line: use Agile methods. But for purposes of pandering to executives who want to see large numbers with lots of zeroes, it's often necessary to write giant project "plans" that you don't actually use.

Go ahead, write waterfall plans. Don't feel guilty or conflicted. Some folks won't catch up with Agility because they think "Control" is better. Pander to them. It's okay.

Questions, or, How to Ask For Help

2010-11-30T08:00:00.000-05:00

Half the fun on Stack Overflow is the endless use of closed-ended questions. "Can I do this in Python?" being so common and so hilarious.

The answer is "Yes." You can do it.

Perhaps that's not the question they really meant to ask.

See "Open versus Closed Ended Questions" for a great list of examples.

Closed-Ended Questions have short answers, essentially yes or no. Leading Questions and presuming questions are common variations on this theme. A closed-ended question is sometimes called "dichotomous" because there are only two choices. They can also be called "saturated", possibly because all the possible answers are laid out in the question.

Asking Questions

The most important part about asking questions is to go through a few steps of preparation.

Search. Use Google, use the Stack Overflow search. A huge number of people seem to bang questions into Stack Overflow without taking the time to see if it's been asked (and answered) already.
Define Your Goal. Seriously. Write down your objective. In words. Be sure the goal includes an active-voice verb -- something you want to be able to do. If you want to be able to write code, write down the words "I want to write code for [X]". If you want to be able to tell the difference between two nearly identical things, write down the words "I want to distinguish [Y] from [Z]". When in doubt, use active voice verbs to write down the thing you want to do. Focus on actions you want to take.
Frame Your Question. Rewrite your goal into a sentence by changing the fewest words. 90% of the time, you'll switch "I want to" to "How do I". The rest of the time, you'll have to think for a moment because your goal didn't make sense. If your goal is not an active-voice verb phrase (something you want to do) then you'll have trouble with the rewrite.

In some cases, folks will skip one or more steps. Hilarity Ensues.

Leading/Presuming Questions

Another form of closed-ended question is the veiled complaint. "Why doesn't Python do [X] the way Perl/PHP/Haskell/Java/C# does it?"

Essentially, this is "my favorite other language has a feature Python is missing." The question boils down to, "Why is [Y] not like [Z]?" Often it's qualified by some feature, but the question is the same: "Regarding [X], why is Python not like language [Z]?"

The answer is "Because they're different." The two languages are not the same, that's why there's a difference.

This leads to "probing" questions of no real value. "Why did Python designers decide to leave out [X]" and other variants on this theme.

If the answer was "Because they're evil gnomes" what does it matter? If the answer was "because it's inefficient" how does that help? Feature [X] is still missing, and all the "why?" questions won't really help add it back into the language.

It's possible that there's a legitimate question hidden under the invective. It might be "How do I implement [X] in Python? For examples, see Perl/PHP/Haskell/Java/C#." Notice that this question is transformed into an active-voice verb: "implement".

If we look at the three-step question approach above, there's no active-voice verb behind a "why question". What you "know" isn't really all that easy to provide answers for. Knowledge is simply hard to provide. Questions about what you want to do, are much, much easier to answer.

Probing/Confirming Questions

One other category are the "questions" that post a pile of details looking for confirmation. There are three common variations.

tl;dr. The wealth of detail was overwhelming. I'm a big fan of the "detail beat-down". It seems like some folks don't need to summarize. There appear to be people with massive brains that don't need models, abstractions or summaries, but are perfectly capable of coping with endless details. It would be helpful if these folks could "write down" to those of us with small brains who need summaries.
No question at all, or the question is a closed-ended "Do you agree?" An answer of "No." is probably not what they wanted. But what can you do? That's all they asked for.
Sometimes the question is "Any comments?" This often stems from having no clear goal. Generally, if you've done a lot of research and you simply want confirmation, there's no question there. If you've got doubts, that means you need to do something to correct the problems.

Here's what is really important with tl;dr questions: What do you want to do?

80% of the time, it's "Fix my big, complex tl;dr proposal to correct problem [X]." [X] could be "security" or "deadlock" or "patent infringement" or "cost overrun" or "testability".

Here's how to adjust this question from something difficult to answer to something good.

You want to know if your tl;dr proposal have problem [X]. You're really looking for confirmation that your tl;dr proposal is free from problem [X]. This is something you want to know -- but knowledge is not a great goal. It's too hard to guess what you don't know; lots of answers can provide almost the right information.

Reframe your goal: drop knowledge and switch to action. What do you want to do? You want to show that your tl;dr proposal is free from problem [X]. So ask that: "How do I show my tl;dr proposal is free from problem [X]?"

Once you write that down, you now have to focus your tl;dr proposal to get the answer to this question. In many cases, you can pare things down to some relevant parts that can shown to be free from problem [X]. In most cases, you'll uncover the problem on your own. In other cases, you've got a good open-ended question to start a useful conversation that will give you something you can do.

Open-Source, moving from "when" to "how"

2010-11-23T08:00:00.001-05:00

Interesting item in the November 1 eWeek: "Open-Source Software in the Enterprise".

Here's the key quote: "rather than asking if or when, organizations are increasingly focusing on how".

Interestingly, the article then goes on to talk about licensing and intellectual property management. I suppose those count, but they're fringe issues, only relevant to lawyers.

Here's the two real issues:

Configuration Management
Quality Assurance

Many organizations do things so poorly that open source software is unusable.

Configuration Management

Many organizations have non-existent or very primitive CM. They may have some source code control and some change management. But the configuration of the test and production technology stacks are absolutely mystifying. No one can positively say what versions of what products are in production or in test.

The funniest conversations center on the interconnectedness of open source projects. You don't just take a library and plug it in. It's not like neatly-stacked laundry, all washed and folded and ready to be used. Open Source software is more like a dryer full of a tangled collection of stuff that's tied in knots and suffers from major static cling.

"How do we upgrade [X]"? You don't simply replace a component. You create a new tech stack with the upgraded [X] and all of the stuff that's knotted together with [X].

Changing from Python 2.5 to 2.6 changes any binary-compiled libraries like PIL or MySQL_python, mod_wsgi, etc. These, in turn, may require OS library upgrades.

A tech stack must be a hallowed thing. Someone must actively manage change to be sure they're complete and consistent across the enterprise.

Quality Assurance

Many organizations have very weak QA. They have an organization, but it has no authority and developers are permitted to run rough-shod over QA any time they use the magic words "the user's demand it".

The truly funny conversations center on how the organization can be sure that open source software works, or is free of hidden malware. I've been asked how a client can vet an open source package to be sure that it is malware free. As if the client's Windows PC's are pristine works of art and the Apache POI project is just a logic bomb.

The idea that you might do acceptance testing on open source software always seems foreign to everyone involved. You test your in-house software. Why not test the downloaded software? Indeed, why not test commercial software for which you pay fees? Why does QA only seem to apply to in-house software?

Goals vs. Directions

I think one other thing that's endlessly confusing is "Architecture is a Direction not a Goal." I get the feeling that many organizations strive for a crazy level of stability where everything is fixed, unchanging and completely static (except for patches.)

The idea that we have systems on a new tech stack and systems on an old tech stack seems to lead to angry words and stalled projects. However, there's really no sensible alternative.

We have tech stack [X.1], [X.2] and [X.3] running in production. We have [X.4] in final quality assurance testing. We have [X.5] in development. The legacy servers running version 1 won't be upgraded, they'll be retired. The legacy servers running version 2 may be upgraded, depending on the value of the new features vs. the cost of upgrading. The data in the version 3 servers will be migrated to the version 4, and the old servers retired.

It can be complex. The architecture is a direction in which most (but not all) servers are heading. The architecture changes, and some servers catch up to the golden ideal and some servers never catch up. Sometimes the upgrade doesn't create enough value.

These are "how" questions that are more important than studying the various licensing provisions.

Software Patents

2010-11-18T08:00:00.000-05:00

Here's an interesting news item: "Red Hat’s Secret Patent Deal and the Fate of JBoss Developers".

Here's an ancient -- but still relevant -- piece from Tim O'Reilly: "Software and Business Method Patents".

Here's a great article in Slate on the consequences of software patents. "Weapons of Business Destruction: How a tiny little 'patent troll' got BlackBerry in a headlock".

The biggest issue with software patents is always the "non-obvious" issue. Generally, this can be debated, so a prior art review is far more valuable.

See "Peer Review Starts for Software Patent Applications: IEEE Spectrum talks to the founder to Peer-to-Patent Beth Noveck". This is where the rubber hits the road.

To participate, see Peer To Patent. Locate prior art and make patent trolls get real jobs.

Also, see this: "A patent improvement: Intellectual property: A new scheme will solicit comments via the internet to improve the vetting of patent applications".

Hadoop and SQL/Relational Hegemony

2010-11-11T08:00:00.000-05:00

Here's a nice article on why Facebook, Yahoo and eBay use Hadoop: "Asking Any Question Of All Your Data".

The article has one tiny element of pandering to the SQL hegemonists.

Yes, it sounds like a conspiracy theory, but it seems like there really are folks who will tell you that the relational database is effectively perfect for all data processing and should not be questioned. To bolster their point, they often have to conflate all data processing into one amorphous void. Relational transactions aren't central to all processing, just certain elements of data processing. There, I said it.

Here's the pandering quote: "But this only works if the underlying data storage and compute engine is powerful enough to operate on a large dataset in a time-efficient manner".

What?

Is he saying that relational databases do not impose the same constraint?

Clearly, the RDBMS has the same "catch". The relational database only works if "...the underlying data storage and compute engine is powerful enough to operate on a large dataset in a time-efficient manner."

Pandering? Really?

Here's why it seems like the article is pandering. Because it worked. It totally appealed to the target audience. I saw this piece because a DBA -- a card-carrying member of the SQL Hegemony cabal -- sent me the link, and highlighted two things. The DBA highlighted the "powerful enough" quote.

As if to say, "See, it won't happen any time soon, Hadoop is too resource intensive to displace the RDBMS."

Which appears to assume that the RDBMS isn't resource intensive.

Further, the DBA had to add the following. "The other catch which is not stated is the skill level required of the people doing the work."

As if to say, "It won't happen any time soon, ordinary programmers can't understand it."

Which appears to assume that ordinary programmers totally understand SQL and the relational model. If they did understand SQL and the relational model perfectly, why would we have DBA's? Why would we have performance tuning? Why would we have DBA's adjusting normalization to correct application design problems?

Weaknesses

So the weaknesses of Hadoop are that it (a) demands resources and (b) requires specialized skills. Okay. But isn't that the exact same weakness as the relational database?

Which causes me to ask why an article like this has to pander to the SQL cabal by suggesting that Hadoop requires a big compute engine? Or is this just my own conspiracy theory?

Data Mapping and Conversion Tools -- Sigh

2010-11-09T08:00:00.000-05:00

Yes, ETL is interesting and important.

But creating a home-brewed data mapping and conversion tool isn't interesting or important. Indeed, it's just an attractive nuisance. Sure, it's fun, but it isn't valuable work. The world doesn't need another ETL tool.

The core problem is talking management (and other developers) into a change of course. How do we stop development of Yet Another ETL Tool (YAETLT)?

First, there's products like Talend, CloverETL and Pentaho open source data integration. Open Source. ETL. Done.

Then, there's this list of Open Source ETL products on the Manageability blog. This list all Java, but there's nothing wrong with Java. There are a lot of jumping-off points in this list. Most importantly, the world doesn't need another ETL tool.

Here's a piece on Open Source BI, just to drive the point home.

Business Rules

The ETL tools must have rules. Either simple field alignment or more complex transformations. The rules can either be interpreted ("engine-based" ETL) or used to build a stand-alone program ("code-generating" ETL).

The engine-based ETL, when written in Java, is creepy. We have a JVM running a Java app. The Java app is an interpreter for a bunch of ETL rules. Two levels of interpreter. Why?

Code-generating ETL, OTOH, is a huge pain in the neck because you have to produce reasonably portable code. In Java, that's hard. Your rules are used to build Java code; the resulting Java code can be compiled and run. And it's often very efficient. [Commercial products often produce portable C (or COBOL) so that they can be very efficient. That's really hard to do well.]

Code-generating, BTW, has an additional complication. Bad Behavior. Folks often tweak the resulting code. Either because the tool wasn't able to generate all the proper nuances, or because the tool-generated code was inefficient in a way that's so grotesque that it couldn't be fixed by an optimizing compiler. It happens that we can have rules that run afoul of the boilerplate loops.

Old-School Architecture

First, we need to focus on the "TL" part of ETL. Our applications receive files from our customers. We don't do the extract -- they do. This means that each file we receive has a unique and distinctive "feature". We have a clear SoW and examples. That doesn't help. Each file is an experiment in novel data formatting and Semantic Heterogeneity.

A common old-school design pattern for this could be called "The ETL Two-Step". This design breaks the processing into "T" and "L" operations. There are lots of unique, simple, "T" options, one per distinctive file format. The output from "T" is a standardized file. A simple, standardized "L" loads the database from the standardized file.

Indeed, if you follow the ETL Two Step carefully, you don't need to actually write the "L" pass at all. You prepare files which your RDBMS utilities can simply load. So the ETL boils down to "simple" transformation from input file to output file.

Folks working on YAETLT have to focus on just the "T" step. Indeed, they should be writing Yet Another Transformation Tool (YATT) instead of YAETLT.

Enter the Python

If all we're doing is moving data around, what's involved?

import csv
result = {
  'column1': None,
  'colmnn2': None,
  # etc.
}
with open("source","rb") as source:
  rdr= csv.DictReader( source )
  with open( "target","wb") as target:
      wtr= csv.DictWriter( target, result.keys() )
      for row in rdr:
          result['column1']= row['some_column']
          result['column2']= some_func( row['some_column'] )
          # etc.
          wtr.writerow( result )

That's really about it. There appear to be 6 or 7 lines of overhead. The rest is working code.

But let's not be too dismissive of the overhead. An ETL depends on the file format, summarized in the import statement. With a little care we can produce libraries similar to Python's csv that work with XLS directly, as well as XLSX and other formats. Dealing with COBOL-style fixed layout files can also be boiled down to an importable module. The import isn't overhead; it's a central part of the rules.

The file open functions could be seen as overhead. Do we really need a full line of code when we could -- more easily -- read from stdin and write to stdout? If we're willing to endure the inefficiency of processing one input file multiple times to create several standardized outputs, then we could eliminate the two with statements. If, however, we have to merge several input files to create a standardized output file, the one-in-one-out model breaks down and we need the with statements and the open functions.

The for statement could be seen as needless overhead. It goes without saying that we're processing the entire input file. Unless, of course, we're merging several files. Then, perhaps, it's not a simple loop that can be somehow implied.

It's Just Code

The point of Python-based ETL is that the problem "solved" by YATT isn't that interesting. Python is an excellent transformation engine ETL. Rather than write a fancy rule interpreter, just write Python. Done.

We don't need a higher-level data transformation engine written in Java. Emit simple Python code and use the Python engine. (We could try to emit Java code, but it's not as simple and requires a rather complex supporting library. Python's Duck Typing simplifies the supporting library.)

If we don't write a new transformation engine, but use Python, that leaves a tiny space left over for the YATT: producing the ETL rules in Python notation. Rather than waste time writing another engine, the YATT developers could create a GUI that drags and drops column names to write the assignment statements in the body of the loop.

That's right, the easiest part of the Python loop is what we can automate. Indeed, that's about all we can automate. Everything else requires complex coding that can't be built as "drag-and-drop" functionality.

Transformations

There are several standard transformations.

Column order or name changes. Trivial assignment statements handle this.
Mapping functions. Some simple (no hysteresis, idempotent) function is applied to one or more columns to produce one or more columns. This can be as simple as a data type conversion, or a complex calculation.
Filter. Some simple function is used to include or exclude rows.
Reduction. Some summary (sum, count, min, max, etc.) is applied to a collection of input rows to create output rows. This is an ideal spot for Python generator functions. But there's rarely a simple drag-n-drop for these kinds of transformations.
Split. One file comes in, two go out. This breaks the stdin-to-stdout assumption.
Merge. Two go in, one comes out. This breaks the stdin-to-stdout assumption, also. Further, the matching can be of several forms. There's the multi-file merge when several similarly large files are involved. There's the lookup merge when a large file is merged with smaller files. Merging also applies to doing key lookups required to match natural keys to locate database FK's.
Normalization (or Distinct Processing). This is a more subtle form of filter because the function isn't idempotent; it depends on the state of a database or output file. We include the first of many identical items; we exclude the subsequent copies. This is also an ideal place for Python generator functions.

Of these, only the first three are candidates for drag-and-drop. And for mapping and filtering, we either need to write code or have a huge library of pre-built mapping and filtering functions.

Problems and Solutions

The YATT problem has two parts. Creating the rules and executing the rules.

Writing another engine to execute the rules is a bad idea. Just generate Python code. It's a delightfully simple language for describing data transformation. It already works.

Writing a tool to create rules is a bad idea. Just write the Python code and call it the rule set. Easy to maintain. Easy to test. Clear, complete, precise.

Pythonic vs. "Clean"

2010-11-04T08:00:00.000-04:00

This provokes thought: "Pythonic".

Why does Python have a "Pythonic" style? Why not "clean"?

Is it these lines from Tim Peters' "The Zen of Python" (a/k/a import this)

There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.

Perhaps having a PEP 8, a BDFL (and FLUFL) means that there's a certain "pressure" to conform?

Or do we have higher standards than other languages? Or less intellectual diversity?

I think that "pythonic" is just a catchy phrase that rolls off the tongue. I think a similar concept exists in all languages, but there isn't a good phrase for it in most other languages. Although Ned Batchelder has some really good suggestions. (Except for C++, which should be "C-Posh-Posh" for really good coding style.)

History

When I was a COBOL programmer, there were two buzz-phrases used. "Clean" and "Structured". Clean was poorly-defined and really just a kind of cultural norm. In those days, each shop had a different opinion of "clean" and the lack of widespread connectivity meant that each shop had a more-or-less unique style. Indeed, as a traveling consultant, I helped refine and adjust those standards because of the wide variety of code I saw in my travels.

"Structured" is pretty much an absolute. Each GOTO-like thing had to be reworked as properly nested IFs or PERFORMs. No real issue there. Except from folks who argued that "Structured" was slower than non-Structured. A load of malarkey, but one I heard remarkably often.

When I was a Fortran (and Ada) programmer, I worked for the military in which there were simply absolute standards for every feature of the source code. Boring. And no catchy buzz-word. Just "Compliant" or "Wrong".

Since it was the early '90's (and we were sequestered) we didn't have much Internet access. Once in a while we'd have internal discussions on style where the details weren't covered by any standard. Not surprisingly, they amounted to "Code Golf" questions. Ada has to be perfectly clear, which can be verbose, and some folks don't like clarity.

When I become a C programmer, I found a Thomas Plum's Reliable Data Structures in C. That provided a really good set of standards. The buzzword I used was "Reliable".

The problem with C programming is that "Clean" and "Code Golf" get conflated all the time. Folks write the craziest crap, claim it's "clean" and ignore the resulting obscurity. Sigh. I wish folks with stick with "Reliable" or "Maintainable" rather than "Clean".

While doing Perl programming I noticed that some folks didn't seem to realize the golden rule.

No One Wins At Code Golf

I don't know why. Other than to think that some folks felt that Perl programs weren't "real" programs. They were just "scripts" and could be treated with a casual contempt.

When I learned Java, I noted that an objective was to have a syntax that was familiar. It was a stated goal to have the Java style guidelines completely overlap with C and C++ style guidelines. Fair enough. Doesn't solve the "Code Golf" vs. "Clean" problem. But it doesn't confound it with another syntax, either.

Python

From this history, I think that "Pythonic" exists because we have a BDFL with high standards.

"Might Be Misleading" is misleading

2010-11-02T08:00:00.002-04:00

My books (Building Skills in Programming, Building Skills in Python and Building Skills in OO Design) develop a steady stream of email. [Also, as a side note, I need to move them to the me.com server, Apple is decommissioning the homepage.mac.com domain.]

The mail falls into several buckets.

Thanks. Always a delight. Keep 'em coming.

Suggestions. These are suggestions for new topics. Recently, I've had a few requests for Python 3 coverage. I'm working with a publisher on this, and hope -- before too long -- to have real news.

Corrections. I get a lot of these. A lot. Keep 'em coming. I didn't pay a copy editor; I tried to do it myself. It's hard and I did a poor job. More valuable than spelling corrections are technical corrections. (I'm happy to report that I don't get as many of these.) Technical corrections are the most serious kind of correction and I try to fix those as quickly as possible.

Source Code Requests. No. I don't supply any source. If I send you the source, what skill did you build? Asking for source? Not a skill that has much value, IMHO. If you want to learn to program, you have to create the source yourself. That is the job. Sorry for making you work, but you have to actually do the work. There's no royal road to programming.

The "Other" Bucket

I get some emails that I file under "other" because they're so funny. They have the following boilerplate.

"Code fragment [X] might is misleading because [Y]."

First, it's a complaint, not a question. That's not very helpful. That's just complaining. Without a suggested improvement, it's the worst kind of bare negativity.

The best part is that — without exception — the person sending the email was not mislead. They correctly understood the code examples.

Clearly, the issue isn't that the code is "misleading" in the usual sense of "lying" or "mendacious". If it was actually misleading, then (a) they wouldn't have understood it and (b) there'd be a proper question instead of a complaint.

Since they correctly understood it, what's misleading?

User Interface Reviews

In software development, we used to go through the "misleading" crap-ola in user interface reviews. In non-Agile ("waterfall") development, we have to get every nuance, every aspect, every feature of the UI completely specified before we can move on. Everyone has to hand-wring over every word, every font choice, field order, button placement, blah, blah and blah.

It seems like 80% of the comments are "label [X] might be misleading". The least useful comment, of course, is this sort of comment with no suggestion. The least useful reviewer is the person who (1) provides a negative comment and, when asked for an improvement, (2) calls a meeting of random people to come up with replacement text.

[Hint: If you eventually understood the misleading label, please state your understanding in your own words. Often, hilarity ensues when their stated understanding cycles back to the original label.]

The "label [X] might be misleading" comment is — perhaps — the most expensive nuisance comment ever. Projects wind up spinning through warrens of rat-holes chasing down some verbiage that is acceptable. After all, you can't go over the waterfall until the entire UI is specified, right?

Worse, of course, the best sales people do not interpose themselves into the sales process. They connect prospective customers with products (or services). Really excellent sales people can have trouble making suggestions. Their transparency is what makes them good. It's not sensible demanding suggestions from them.

Underneath a "Might Be Misleading" comment, the person complaining completely understood the label. They were not actually mislead at all. If it was misleading, then (a) they wouldn't have understood it and (b) there'd be a proper question instead of a complaint.

Thank goodness for Agile product owners who can discard the bad kind of negativity. The right thing to do is put a UI in front of more than one user and bypass the negativity with a consensus that the UI actually is usable and isn't really misleading.

Might Be Misleading

The "Might be Misleading" comments are often code-speak for "I don't like it because..." And the reason why is often "because I had to think." I know that thinking is bad.

I understand that Krug's famous Don't Make me Think is the benchmark in usability. And I totally agree that some thinking is bad.

There are two levels of thinking.

Thinking about the problem.
Thinking about the UI and how the UI models the problem.

Krug's advice is clear. Don't make users think about the UI and how the UI models the problem. Users still have to think about the problem itself.

In the case of UI labels which "Might Be Misleading", we have to figure out if it's the problem or the UI that folks are complaining about. In many cases, parts of the problem are actually hard and no amount of UI fixup can ever make the problem easier.

Not Completely Accurate

One of the most common UI label complaints is that the label isn't "completely" accurate. They seem to object to fact that a UI label can only contain a few words and they have to actually understand the few words. I assume that folks who complain about UI labels also complain about light switches having just "on" and "off" as labels. Those labels aren't "completely" accurate. It should say "power on". Indeed it should say "110V AC power connected". Indeed it should say "110V AC power connected through load". Indeed it should say "110V AC 15 A power connected via circuit labeled #4 through load with ground".

Apparently this is news. Labels are Summaries.

No label can be "completely" accurate. You heard it here first. Now that you've been notified, you can stop complaining about labels which "might be misleading because they're not completely accurate." They can't be "completely" accurate unless the label recapitulates the entire problem domain description and all source code leading to the value.

Apologies

In too many cases of "Might Be Misleading," people are really complaining that they don't like the UI label (or the code example) because the problem itself is hard. I'm sympathetic that the problem domain is hard and requires thinking.

Please, however, don't complain about what "Might Be Misleading". Please try to focus on "Actually Is Misleading."

Before complaining, please clarify your understanding.

Here's the rule. If you eventually understood it, it may be that the problem itself is hard. If the problem is hard, fixing the label isn't going to help, is it?

If the problem is hard, you have to think. Some days are like that. The UI designer and I apologize for making you think. Can we move on now?

If the label (or example) really is wrong, and you can correct it, that's a good thing. Figure out what is actually misleading. Supply the correction. Try to escalate "Might Be Misleading" to "Actually Mislead Someone". Specifics matter.

Also, please remember that labels are summaries. At some point, details must be elided. If you have trouble with the concept of "summary", you can do this. (1) Write down all the details that you understand. Omit nothing. (2) Rank the details in order of importance. (3) Delete words to pare the description down to an appropriate length to fit in the UI. When you're done, you have a suggestion.

Python and the "Syntactic Whitespace Problem"

2010-10-26T08:00:00.003-04:00

Check out this list of questions on Stack Overflow:

http://stackoverflow.com/search?q=%5Bpython%5D+whitespace+syntax

About 10% of these are really just complaints about Python's syntax. Almost every Stack Overflow question on Python's use of syntactic whitespace is really just a complaint.

Here's today's example: "Python without whitespace requirements".

Here's the money quote: "I could potentially be interested in learning Python but the whitespace restrictions are an absolute no-go for me."

Here's the reality.

Everyone Indents Correctly All The Time In All Languages.

Everyone. All the time. Always.

It's amazing how well, and how carefully people indent code. Not Python code.

All Code. XML. HTML. CSS. Java. C++. SQL. All Code.

Everyone indents. And they always indent correctly. It's truly amazing how well people indent. In particular, when the syntax doesn't require any indentation, they still indent beautifully.

Consider this snippet of C code.

if( a == 0 )
   printf( "a is zero" );
   r = 1;
else
   printf( "a is non-zero" );
   r = a % 2;

Over the last few decades, I've probably spent a complete man-year reading code like that and trying to figure out why it doesn't work. It's not easy to debug.

The indentation completely and accurately reflects the programmer's intention. Everyone gets the indentation right. All the time. In every language.

And people still complain about Python, even when they indent beautifully in other languages.

Code Base Fragmentation

2010-10-21T08:00:00.001-04:00

Here's what I love -- an argument that can only add cost and complexity to a project.

It sounds like this to me: "We need to fragment the code base into several different languages. Some of the application programming simply must be written in a language that's poorly-understood, with tools that are not widely available, and supported by a select few individuals that have exclusive access to this code. We haven't benchmarked the technical benefit."

Further, we'll create complex organizational roadblocks in every single project around this obscure, specialized, hard-to-support language.

Perhaps I'm wrong, but database triggers always seem to create more problems than they solve.

They Totally Solve a Problem

The most common argument boils down to application-specific cross-cutting concerns. The claim is that these concerns (logging, validation, data model integrity, whatever) can only be solved with triggers. For some reason, though, these cross-cutting concerns can't be solved through ordinary software design. I'm not sure why triggers are the only solution when simple OO design would be far simpler.

Some folks like to adopt the "multiple application programming languages" argument. That is, that ordinary OO design won't work because the code would have to be repeated in each language. This is largely bunk. It's mostly folks scent-marking territory and refusing to cooperate.

Step 1. Write a library and share it. It's hard to find a language that can't be used to write a sharable library. It's easy to find an organization where the Visual C# programmers are not on speaking terms with the Java programmers and the isolated Python folks are pariahs. This isn't technology. Any one of the languages can create the necessary shared library. A modicum of cooperation would be simpler than creating triggers.

Step 2. Get over it. "Duplicated" business logic is rampant in most organizations. Now that you know about, you can manage it. You don't need to add Yet Another Language to the problem. Just cooperate to propagate the changes.

They're Totally Essential To The Database

The silly argument is that some business rules are "closer to" or "essential to" the database. The reason I can call this silly is because when the data is converted to another database (or extracted to the data warehouse) the triggers aren't relevant or even needed. If the triggers aren't part of "interpreting" or "using" the data, they aren't essential. They're just convenient.

The data really is separate from the processing. And the data is far, far more valuable than the processing. The processing really is mostly application-specific. Any processing that isn't specific to the application really is a cross-cutting concern (see above). There is no "essential" processing that's magically part of the data.

What If...

Life is simpler if all application programming is done in application programming languages. And all triggers are just methods in classes. And everyone just uses the class library they're supposed to use.

"But what if someone doesn't use the proper library? A trigger would magically prevent problems."

If someone refuses to use the application libraries, they need career coaching. As in "find another job where breaking the rules is tolerated."

Technical Debt

2010-10-19T08:00:00.000-04:00

Love this from Gartner. "Gartner Estimates Global 'IT Debt' to Be $500 Billion This Year, with Potential to Grow to $1 Trillion by 2015".

NetworkWorld ran a quicky version of the story. Gartner: Global 'IT debt' hits $500 billion, on the way to $1 trillion.

ComputerWorld -- to be proper journalists -- have to get a balancing quote. Their version of the story is this: Gartner warns of app maintenance 'debt'. The balancing quote is the following:

"There are many good reasons to NOT upgrade/modernize many applications, and I believe Gartner is out of line using words like 'debt' which have guilt associated with them,"

"Guilt"? That's a problem? Why are we pandering to an organization's (i.e., CIO's) emotional response?

I'm not sure that using a word like "debt" is a problem. Indeed, I think they should ramp up the threat level on this and add words like "short-sighted" and "daft" and perhaps even "idiotic".

Anyone who doesn't believe (or doesn't understand) technical debt needs only to review the Y2K plans and budgets. A bad technology decision lead to a mountain of rework. Yes, it was all successful, but it made IT budgeting difficult for years afterwords.

The rest of the organization was grumpy about having their projects were stalled until after Y2K. IT created it's own problems by letting the technology debt accumulate to a level where it was "fix or face an unacceptable risk of not being able to stay in business."

How many other latent Y2K-like problems are companies ignoring?

Real Security Models

2010-10-13T08:00:00.000-04:00

Lots of folks like to wring their hands over the Big Vague Concept (BVC™) labeled "security".

There's a lot of quibbling. Let's move beyond BVC to the interesting stuff.

I've wasted hours listening to people identify risks and costs of something that's not very complex. I've been plagued by folks throwing up the "We don't know what we don't know" objection to a web services interface. This objection amounts to "We don't know every possible vulnerability; therefore we don't know how to secure it; therefore all architectures are bad and we should stop development right now!" The OWASP top-ten list, for some reason, doesn't sway them into thinking that security is actually manageable.

What's more interesting than quibbling over BVC, is determining the authorization rules.

Basics

Two of the pillars of security are Authentication (who are you?) and Authorization (what are you allowed to do?)

Authentication is not something to be invented. It's something to be used. In our case, with an Apache/Django application, the Django authentication system works nicely for identity management. It supports a simple model of users, passwords and profiles.

We're moving to Open SSO. This takes identity management out of Django.

The point is that authentication is -- largely -- a solved problem. Don't invent. It's solved and it's easy to get wrong. Download or License an established product for identity management

and use it for all authentication.

Authorization

The Authorization problem is always more nuanced, and more interesting, than Authentication. Once we know who the user is, we still have to determine what they're really allowed to do. This varies a lot. A small change to the organization, or a business process, can have a ripple effect through the authorization rules.

In the case of Django, there is a "low-level" set of authorization tests that can be attached to each view function. Each model has an implicit set of three permissions (can_add, can_delete and can_change). Each view function can test to see if the current user has the required permission. This is done through a simple permission_required decorator on each view function.

However, that's rarely enough information for practical — and nuanced — problems.

The auth profile module can be used to provide additional authorization information. In our case, we just figured out that we have some "big picture" authorizations. For sales and marketing purposes, some clusters of features are identified as "products" (or "features" or "options" or something). They aren't smallish things like Django models. They aren't largish things like whole sites. They're intermediate things based on what customers like to pay for (and not pay for).

Some of these "features" map to Django applications. That's easy. The application view functions can all simply refuse to work if the user's contract doesn't include the option.

Sadly, however, some "features" are part of an application. Drat. We have two choices here.

Assure that there's a "default" option and configure the feature or the default at run time. For a simple class (or even a simple module) this isn't too hard. Picking a class to instantiate at run time is pretty standard OO programming.
Rewrite the application to refactor it into two applications: the standard version and the optional version. This can be hard when the feature shows up as one column in a displayed list of objects or one field in a form showing object details. However, it's very Django to have applications configured dynamically in the settings file.

Our current structure is simple: all customers get all applications. We have to move away from that to mix-and-match applications on a per-customer basis. And Django supports this elegantly.

Security In Depth

This leads us to the "Defense in Depth" buzzword bingo. We have SSL. We have SSO. We have high-level "product" authorizations. We have fine-grained Django model authorizations.

So far, all of this is done via Django group memberships, allowing us to tweak permissions through the auth module. Very handy. Very nice. And we didn't invent anything new.

All we invented was our high-level "product" authorization. This is a simple many-to-many relationship between the Django Profile model and a table of license terms and conditions with expiration dates.

Django rocks. The nuanced part is fine-tuning the available bits and pieces to match the marketing and sales pitch and the the legal terms and conditions in the contracts and statements of work.