Decisions and Consequences

A single poorly-made decision can have profound ripple-effects. Once your stuck with it, you make accommodations, hacks and work-arounds. Eventually, things work, but the result is less than ideal.

Changing tack requires sometimes pervasive rework to the application. How can we reduce the risks and improve the value created?

A Recent Example

When dealing with bulk econometric data (Bloomberg, D&B, Moody's, etc.) you get BIG files with lots of fields. Depending on what you're paying for, the file layouts are frequently different even though the content is similar. I'm a big fan of plain-old CSV data. Even the tab-delimited variant of CSV is not bad to work with.

Further, most vendors will slap some heading rows on the file so that the column names are--more or less--identified. Surprisingly, this doesn't work out well in practice because there are often multiple columns with the same name. Sigh.

Using Python's csv library module lets us cope with CSV (and tab-delim) quite gracefully.

What's wrong with that decision? Nothing.

Variant Column Names

The question arises when you've purchased several files of econometric data and the column names are slightly different. This happens with a single vendor and across vendors. It's part of the game that can't easily be avoided. Column names vary.

What to do?

Here's the less-than-ideal decision. Make the column names a parameter.

In Python, this is not terribly difficult. The csv module's DictReader provides us a dictionary for each row. Each column name becomes a key. We can access the fields with some_row['this_field'] and some_row['that_field']. How bad can it be?

The extra punctuation is fairly hideous.

More importantly, however, is the nature of the metadata.

Consequence One -- Dynamic Metadata

Dynamic metadata, in this case, means that any indexing of the data is done based on character string column names.

index[index_name][row[column_name]].append( row )

That's rather more complex than the alternative where the metadata has a fixed definition.

some_index[row.column].append( row )

Consequence Two -- Murky ORM

Once we have dynamic metadata, we're largely frozen out of ordinary SQL database implementations. We don't know the column names, we don't know the indices. We can't do simple CREATE TABLE statements because we don't really have the column names until we open the working files.

We have to grub through all the code to find out where the dynamic mapping is reasoned out. Once we find that, we can then consider how to make the metadata fixed enough to tackle a SQL database.

We could, of course, generate the SQL CREATE INDEX statements on-the-fly. There's nothing wrong with it. But it slows down analysis and decision-making when we're not sure what indexes there are or what leads to a choice of index.

What's important here is that we want to use SQLite because it ships with Python. We want our application to use an ORM (like SQLAlchemy or SQLObject). We don't want our application to become a kind of ORM because of the dynamic SQL and dynamic column names.

Cleanup

The cleanup road is clear.

1. Map all variant inputs to one common structure. Rather than work with raw dictionaries from csv, map each row to a standard set of names. For now, we can replace the dictionaries with named tuples to prepare for a migration to an ORM when that's possible.
2. Replace the row['some field'] syntax with row.some_field syntax. Of course, there's a lot of this. This is a pervasive change.
3. Find all the dynamic index creation and refactor that into a more static "database-like" place for now.

Item 1 is pretty easy to unit test. We're adding a function to map from dynamic names to fixed names. Nothing much to this testing-wise.

Item 2 requires unit tests with really good code coverage or there's no earthy way we can be sure that each mapping-syntax name has been transformed into an attribute-syntax name.

Item 3 barely requires testing. Indexes and other features are performance enhancements that can be removed and added without altering functionality.

A Taxonomy of Use Case Errors

First, the definition. A use case describes an actor's interaction with a system to create business value. There are three parts: Actor, Interaction and Business Value.

1. Not Interactive.

1.1. The use case is just features and technical attributes with no actor interaction expressed.

1.2. The use case is just algorithms and processing with no connection to an actor or a goal.

2. No Business Value.

2.1. Incomplete

2.1.1. The use case focus on sequential operations with no value or goal.

2.1.3. The use case simply follows existing precedent without supporting actual business goals. It "paves the cow path".

2.2. Non-Specific

2.2.1. The use case is a result of free-running imagination; it conflates "possibly" vs. "required". It contains descriptions of interactions which could happen or would be nice to happen.

2.3. Covers the Technology Only

2.3.1. The solution technology is conflated with the business problem. Words like "database" or "foreign key" or "error log" or other solution technology are central.

2.4. Contradictory

2.4.1. The use case goal contradicts other goals.

2.4.2. The use case sequence is inconsistent with the stated goal.

3. No Actor.

The Ubiquitous Object

Objects are everywhere.

Weirdly, some people can't see them. I guess they live in a rarified, HP Lovecraftian world of pure action inhabited by amorphous things that can't be properly called "beings" but rather "doings" because they're pure activity with no existence.

Read "Hypnos". "They were sensations, yet within them lay unbelievable elements of time and space—things which at bottom possess no distinct and definite existence."

Got this comment the other day.

... doing procedural code correctly when you don't want to be bothered w/ OO is a separate and big enough topic that warrants its own book or monograph.

I guess that means that objects, and the reality that they model, are a "bother"—a pitfall to be avoided—a cost with no benefit. This is not the first time I've heard this, and—like Lovecraft—it leads me to wonder how such a rich and weird phantasy world gets constructed.

I had a project manager exclaim "You don't need more than seven or eight objects to write any application." I didn't press the person on that point. I assumed that they were talking about classes (not objects) and, further, had conflated class with "elaborate module-like library packed with amazing features". Or maybe they conflated class with package. Or something. It's hard for me to dig into misapprehensions and false assumptions without being rude.

There are a surprising number of misapprehensions. I'm occasionally tempted to turn NTLK loose on all questions tagged "Python" on Stack Overflow. With some patient reading, I think I could develop a taxonomy of OO confusion. However, let's just focus on this comment.

The Bother Factor

Why is OO a "bother"?

1. I've been told that OO programming is different. Different from what? From procedural programming without objects, I guess.
2. I'm been told that some problems are a better fit for OO, and some problems aren't a good fit for OO. This is hard to parse because it makes the more profound claim that some problems weirdly don't involve any "objects" just pure actions.
3. The Object-Relational Impedance Mismatch problem somehow indicts object-oriented programming as unsuitable when there's a relational database involved.

Let's look at some of these in a little depth to see the underlying fallacies.

Procedural Is More Fundamental

This is subtle and pernicious. An OO language contains within it a procedural language. Because of this, we can use Java, C++ or Python to write Fortran-like (or VB-like) crapola code. It's possible to write everything in a single, massive, static class with piles of random global variables, long lists of disorganized methods, and "adaptation via block comment" buffoonery.

Some folks object to characterizing procedural programming as random, disorganized or buffoonery. They tell me that a purely procedural can be neat and well organized with tidy, focused modules that have narrowly-defined responsibilities, no global variables and clever techniques like pointer-to-function to support adaptation.

Wait. The idea of tidy, focused modules with narrowly-defined responsibilities is exactly what a class is.

This is important. All good procedural programming is isomorphic to object-oriented programming minus the class definitions.

Procedural isn't "fundamental". It's just a "fragmentary". Procedural programming is a subset of object-oriented programming. Not a foundation. We can, for example, do functional-style object-oriented programming by using immutable objects.

Some Problems Aren't A Good Fit

Claiming that there are problems which don't fit the object-oriented paradigm is false. Or such a claim hearkens to a more elaborate ontology in which existence somehow doesn't matter.

This question is typical: "What should be OO and what shouldn't?"

When a program "runs" or "executes" there is state change. In a lazy functional world, state change is characterized by the creation and destruction of immutable objects: the new "4" that's created by "2+2".

In order for there to be state, there must be an object that has a state of being. Objects are inherent in doing any computing of any kind.

Some folks like to lift up stored procedures or shell scripts as "important" examples of non-OO programming. Mostly, these just show that a non-OO language can persist for a long time because clever programmers can work around a lot of limitations. (Turing Completeness is a necessary pre-condition; not a desirable feature set.)

[And yes, I've written multiple-thousand line shell scripts so customers can avoid paying a license fee for a proper compiler. Just because it can be done doesn't mean it should be done.]

This is important. All Programming Involves Objects.

There are really just two "paradigm" decisions. Does the problem involve new class definitions or can it be done using built-in classes? Does the problem involve mutable objects or immutable objects?

Software that uses only built-in classes is termed "procedural". Software that uses only immutable objects is termed "functional". Software that uses mutable objects is mistakenly termed "object-oriented".

Object-Relational Mismatch

This isn't really very interesting, no matter how many times people like to flog it. Use an ORM. Move on.

Further, it's important to recognize that normalization, foreign keys, cascading deletes and other malarky are hacks imposed on us by several relational database limitations. These are not essential parts of any problem.

I don't know how many times I've had to answer the "how do I do foreign keys in Java/C++/Python?" question. The answer is always the same: foreign keys are a hack-around because there are no proper object references in a relational database.

What's Left?

In spite of the obvious logic that OO is central, there is always a residual "It's a bother" sense from folks who's first language was not an OO language.

As far as I can tell, the "bother" stems from simple ignorance of what's really going on. Many programmers can't articulate any design principles. Yet, they tend to follow some principles rather closely. Ask them what they're doing. Read their code. Almost everyone who codes has some set of fundamental principles. (The few exceptions are people who seem to write code more-or-less randomly and still manage to arrive at something that appeared to "work"; these people do exist and are very scary.)

Many programmers don't follow all of the SOLID Principles.

Many programmers follow the SOLID principles using different nomenclature. The SOLID initials and acronyms are just one one goofy terminology. There are more principles than these, and the principles can have other names.

What's important is that (except for rare exceptions) all programmers follow some of the SOLID principles. Some follow all of them. Some follow numerous additional principles beyond these. Some give their principles other names.

The folks who claim OO programming is a "bother" just don't happen to recognize that they're already following some of the SOLID principles and actually doing OO programming with built-in classes.

Doing Procedural Programming Correctly

Bottom Line: "doing procedural code correctly" is simply OO programming using only built-in classes.

It's not a "big" topic. It's entirely an exercise in learning how to apply someone else's nomenclature to one's existing principles.

The curse of procedural design

After reverse engineering procedural code in C, VB or even Python, I'm finding that procedural programming inevitably leads to bad, bad code-rot.

Consider some of the common design patterns.

Strategy. Confronted with alternative strategy choices, a purely procedural code solution is either

• If-statements everywhere the strategy is involved.
• Block comments. (Pre-processor #if statements are the logical equivalent of block comments plus a tool to move them around just prior to compilation.)

These lack flexibility and seem to devolve into a quagmire of mystery. The if-statements often become tangled and complex. More importantly, some strategy choices — which are unused — may not be maintained at all. Of course, the block comments are never maintained.

Command. Often a command design requires a "code" or "label" and a big-old sequential switch (BOSS™) statement to select among the procedures which implement the various commands. Once "composite" commands are introduced, this devolves into nonsense. Ideally, it's a simple recursion, where a composite command simply invokes the sub-commands. However, folks get nervous about recursion and try to write weird loops.

State. A state design always seems to involve labels or codes for the state names and a slightly different big-old-state-switch (BOSS™, no accident that this is the same acronym) to sort out the variant behaviors in the distinct states. This shouldn't become too confusing. After all, Turing machines and other mathematical abstractions give us a strong hint on how we should proceed.

The problem with stateful procedural programming is that the state changes can be hidden everywhere. In the Really Bad Languages, variables can change values without an assignment statement! In the Not Bad Languages, we can track down the various assignment statements and try to reason out the state changes. Procedural code—without a lot of adult supervision—never seems to encapsulate state change with the the same in-your-face clarity that OO programs do.

I Could Go On

The point is this. While procedural programming could be done well, there appear to be a lot of obstacles inherent in the paradigm.

The best procedural programming I've seen has always been very object-oriented. Each procedure or function had a distinct data structure it worked with; they were all closely related by virtue of naming or file structure; much like a class definition.

I'm starting to wonder if my Building Skills books are taking the right approach. I start with the procedural aspects of Python. I'm beginning to feel that this may be a disservice to the n00bz.

Perhaps it's better to swap the order of the sections and start with the various Pythonic data structures and introduce the various statements sort of "casually" as part of demonstrating how a data structure is supposed to be used.

Language, Tools, Chickens, Eggs, Java and Python

Too much of programming is intimately tied up with the tools to support the development of the software.

Example 1. I was told -- with absolute and fierce conviction -- that VB may suck as a language, but Visual Studio more than makes up for the obvious problems. For some people, Tools Trump Language. Sadly, I've also had customers with ancient code they could no longer compile or maintain because the tools were out of support.

On Stack Overflow, you can read questions like this: "What IDE to use for Python?". In spite of this question's immense popularity, it gets re-asked all the time. Search for "Python IDE" to see endless duplicates. One of the most common duplicate forms of this question asks (or demands) code completion. As if there are folks who cannot write code without code completion.

Chickens and Eggs

The issue with sophisticated IDE's (like Eclipse, NetBeans, and even Komodo) is that you have to learn the tools before learning the language. Until you know something about the language, the tools, of course, are useless. Worse, Eclipse is for "enterprise" applications and is so fat with bells (and whistles) that it's hard to determine what to use and what it means.

So the tool is a prerequisite for the language. But the language is a prerequisite for the tool.

How to cut the Gordian Knot?

First Principles

Irrespective of the "Visual Studio makes VB not suck" crowd, language comes first -- and last -- and fills all the spaces in between.

Language is everything. Software is merely encoded knowledge. The language of that encoding is how we determine meaning; how we argue about correctness, adaptability, maintainability and security. Tools don't endure -- they come and go -- but the language remains.

The only thing more important than the language is the data itself. But that's another rant.

Proof, of course, is available everyone except in VB circles. For non-proprietary languages (Java, Python, etc., etc.) there are a large number of competing tools. One language many tools. Take the hint. Language is important.

Yes, some tools are so flexible, they cover several languages. But there's no universal tool any more than there's a universal language. And the bias is clearly very, very many tools for a given language and only a few languages for a given tool.

How To Start

Language comes first.

For Python, that's easy. Run Python, type code at the >>> prompt, and you're learning. Python comes with IDLE which is a minimalist IDE. It will get anyone started. Later, they can try other IDE's.

For Java, however, that's not that easy. It isn't however, impossible to get started. It's just challenging.

Option 1 -- Bare Knuckles. It's possible to edit text and run the javac compiler to learn a great deal of Java without an IDE. It's not a bad idea. It will get complex to manage projects with more than a few files.

Eventually that's what Ant, Maven and SCons are for. But that's not a good place to start. Again, the tools don't make sense until you start writing things big enough that the tools actually help.

Option 2 -- Succession of IDE's. It's probably best to start with a very simple IDE for Java. Something like Komodo Edit, TextMate or BBEdit. There are a lot of choices, but the idea is to find something little more than a text editor with a few tools. I've used these and like their relative simplicity.

The JavaWIDE toolset might be helpful. I haven't used it, but some folks suggest that it simplifies the language learning. Later a "regular" desktop IDE can be used.

Later, one can move to NetBeans or Eclipse.

Classrooms and Autodidacts

In the classroom, it's easy to demonstrate NetBeans and answer questions.

For auto-didacts, however, choosing the wrong tool leads to endless confusion. The chicken and egg issue isn't clarified by wasting time trying to install and use a tool that's too sophisticated for a n00b.

N00b autodidacts really need to start with a simple text-editor. They need to use `javac` to compile and `java` to run the resulting class. For the first week or two, this will do. Once past the fundamentals, however, IDE selection can start to make sense. A BBEdit/TextMate/Komodo thing should be next. This is good for -- perhaps a year or more. Then, when doing "real" programming, a heavier-weight tool makes sense.

Test-Driven Reverse Engineering (TDRE)

Another case study on TDRE.

Provided: 2,938 lines of Python code which process a handful of large files to create a number of outputs. [Details can't be disclosed.]

Objective: Refactor to distinguish between the overall sequence of transformational steps and the details of each individual step.

Observations

The code is almost purely procedural. There are 11 class definitions. 6 of these wrap built-in types with type conversion and null-handling. 1 is a new exception. 1 is a generic "table" that essentially duplicates features of SQLite. The remaining 3 are actually part of the problem domain.

One reason for reverse engineering is that the code has reached an intellectual limit. It's small, but "dense" with highly-optimized processing steps. The cohesion type is almost all "Temporal". Processing is grouped into successive processing loops; each loop contains a cluster of processing steps. Consequently, it's quite hard to tease apart the algorithm to get a "big picture" of what's going on. It's just a dense stand of trees. No forest.

Another reason for reverse engineering is to support the endless adaptation and modification of the code base. The program is a kind of "spreadsheet on steroids". This isn't a simplistic collection of cells and formulæ that permits simple what-if analysis. This is a more complex set of formulæ that would be challenging (but not impossible) to implement as a spreadsheet. The use case, however, is the spreasheet use case: think, tweak, create results, repeat.

TDRE Approach

Start with an Initial Survey of the legacy code base and sample files.

Create an Outline or "sketch" of the domain model and main program. This will be a modules (or a package) with comments and some preliminary class definitions. Little more.

Pick a processing Step in the legacy code. This often requires creating processing summaries of the legacy code. Most legacy code is procedural, so the processing tends to be sequential in nature.

Instrument the Legacy Code with print statements to gather data. This can be simple. The output can be challenging to interpret.

with open("tdre_results_1","w") as tdre:

# some legacy processing

print( "Case:", foo, bar, ", Expect:", baz, file=tdre )

From the output, Build Unit Test Cases. Fill in parts of the processing sequence and domain model. Debug code until the tests pass.

Initial Survey

The Initial Survey locates several things.

1. The usable, working modules. It appears that all reverse engineering involves a code base with dead or unused code. Even a small project (3,000 lines) will have a remarkable amount of dead code.
2. Priorities for the implemented functionality. Not every "main" module is relevant.
3. Example inputs and outputs.

If the software cannot be run (as is the case with organically developed systems that depend on large, complex corporate databases), then the example inputs and outputs may not actually match the software. If the software can be run, it should be run and the actuals compared against the samples to confirm that the code base supplied really produced the sample outputs.

Expect that the provided legacy code is slightly different from the code in production use. In some cases, this cannot be resolved; for example, when the executables are older than the source. In other cases, the code matches and no further work is required to establish the legacy baseline.

The sample outputs point in the direction of an acceptance test case. The sample output cannot be taken literally as the one-and-only acceptance test. While it's desirable for reverse engineering to reproduce the sample output, most reverse engineering will involve enhancements or bug fixes. Expect that errors will be found (or may be known to exist) in the sample output.

Create Outline

The outline is -- initially -- just generic MVP. There must be a domain model, some "presenter" that has the application logic, and some "view" for displaying the outputs.

In our case study, above, the "view" is a collection of (mostly text) output files. The model was undefined in the legacy code, which was all "presenter" application logic.

The goal was to extract the underlying model, break the application "presenter" logic into two layers (forest and trees) and build some views for each of the output files.

Pick a Processing Step

This can be challenging, depending on the legacy code base. There are two paths through a procedural code base.

• Back to Front. Start with the final results and unit test the final steps based on previous steps that will be defined later.
• Front to Back. Start with the first recognizable intermediate result based on the input files. Unit test the initial steps.

It's more rewarding to work front-to-back because progress can be shown a little more clearly.

A better architecture can be created by working back-to-front since dependencies are easier to understand.

Unit Test Volume, Edges and Corners

There are two unit test design challenges when doing reverse engineering.

• Volume. The sample data can be large. 100,000 rows of sample data is too many to test. Finding a "representative" subset is difficult. Generally, arbitrary subsets have to be used to get started. Once the application mostly works, more refined unit tests need to be created.
• Edge and Corner Cases. While the code may be riddled with if-statements, it can still be difficult to locate sample inputs that exercise the various conditions in the code. It's risky to create data -- we have to assume that the legacy code does unexpected things. In many cases, print statements have to be put into complex if statements to locate any actual data that exercises that logic path.

Once the unit tests are built, this is just Test-Driven Development (TDD).

Performance Discussions and Software Design

Read this first: "There is something I find interesting about online discussions around performance issues..." It's about Stack Overflow, specifically. Apparently, someone didn't get their question answered and decided it was better to gripe than to rewrite the question.

Let's look at their response in pieces.

"people try to gang up". Since there's almost no social networking capability, this is a bit much to attribute to people responding to a poorly-worded question. But, if you've worked all day on a bad solution to a poorly-conceived problem, it can feel like being ganged up on. When reality leaks in, it can feel unpleasant.

Hint 1. There are no gangs. It's possible that the question really is poorly written.

"cookie-cutter, patronizing, zero-information responses". I'm guessing these are comments suggesting the approach is bad and asking for clarification. I run afoul of this often because I feel compelled to post comments asking for clarification. Some folks just don't like to clarify. More than once I've been told that their question was very clear. Since I'm asking for clarification, it seems odd to insist the question is perfect. Worse, of course, is asking for help on Stack Overflow, but refusing to clarify the help required.

Hint 2. Clarify. Please. Don't insist that the question is perfect.

"they assume, without any basis, that the person has not (a) benchmarked the code," When the question has no bench mark data, this isn't an assumption. It's a response to the lack of benchmark data.

Hint 3. Provide the facts. Don't complain when folks ask for facts.

"(b) is obviously running an inferior algorithm". Again, this isn't an assumption. It's the response to an incomplete question where the algorithm isn't provided. Also, it's a common response to questions where the algorithm really is inferior.

Hint 4. Consider that -- even after spending days banging your head against the wall -- your question might be poorly-written and require both benchmark data and an algorithm.

"advice about premature optimization... is a well assimilated folklore by now and I dont see how repeating that adds value". Without measurements, profiler results and benchmark data, this is our only possible response. After the profiler results are posted, this advice really is useless. Before profiler results are posted, this advice often turns out to be essential.

Hint 5. Whatever you might know is not well-assimilated folklore on Stack Overflow as a whole. We don't know you, sadly. We don't know how much you know. To avoid useless advice, provide evidence -- in the question -- that the advice has already been followed.

"better served if the discussion shifted to ... Pointed out possible bottlenecks ahead of time," Wouldn't that be nice? What's a "possible bottleneck"? It's a badly-design algorithm. So, the responses to performance questions has to be focused on algorithm choice right away. That means details on the code being used, and profiling information.

Hint 6. There is no hint 6. This would simply repeat hints 3 and 5.

"regardless of the fact whether the code construct is actually a bottleneck in the application or not, it is always good to know what the more efficient alternatives are... there is something called intellectual curiosity."

Reducing a question to a hand-waving hypothetical doesn't improve the question. It doesn't rationalize a poor question. The question still needs to be clarified.

Hint 7. If the question raises a lot of comments and useless advice, please rewrite the question.