Letter to Mom -- What Is This "Computer Programming" Thing?

Happy birthday, mom. Glad to see you're still doing well, avoiding the complications of COVID-19.

You asked what it was I did for a living. Emphasis on the past tense, now that we're both fully retired old people. 

I have to confess that it's not easy describing high-tech work. There's a lot of jargon. Your varied range of careers included many things, one of which was being a school librarian. The world has had libraries and librarians for millennia. The job title is pretty well understood. The world hasn't had electronic computers for very long, making the job of programming them a relative novelty.

Aids to computation include slide rules and other mechanical devices. The idea of a mechanical computer dates from the 1830's. You can read about the first computer programmer, Ada Lovelace, here: https://www.computerhistory.org/babbage/adalovelace/. Proper electronic digital computes didn't arise until the 40's, when you were a child. ENIAC, for example dates from 1945.

While a lot has changed since the ENIAC, there are a few universal truths. I'm going to beat one of those truths like a dead horse because it's both essential and obscured by layers of technology.

This first and most fundamental truth is that a computer -- even something as sophisticated as a laptop with a dozen open browser tabs, zoom, and two different solitaire games -- is really a small device that is patiently waiting for you to type or click; the software works out some response and this is displayed on the screen or burbles out the speakers (or both.)

We can say that a general-purpose computer is "applied" to a specific problem. We shorthand this into creating "Application Software;" the software that applies the computer's hardware to a problem. And we shorthand this into "Apps" or "Applications" that do useful things on a general-purpose device.

The distinction between software (things you download and change) and hardware (the box on your desk) has become pretty common-place. The details of the software are what we need to put under the magnifying glass to look at closely.

To make your computer more useful, clever engineers have worked out a way to interleave activities from a variety of applications, all of which are using your computer concurrently. There's a set of rules to determine which application is in the "foreground"; this is the application software that has access to keyboard, mouse, display, and speakers. When you click on another window, you bring another application to the foreground. Access to the hardware switches and the display updates. It's very slick. They provide a number of visual cues to show you which application's "window" is in the foreground; all the others have different cues to show you they're in the background.

What's important about this foreground/background concept is that each application is -- from one point of view -- free to behave as if it is in total control of the entire computer. In reality, an application emphatically does not have unfettered control over the computing resources; there are a large number of gates and fences forcing applications into an orderly, and disciplined sharing and cooperation.

You taught at a nursery school. You know how important an orderly set of rules is. Applications are no different than unruly three- and four-year olds: they try to grab snacks out of order. They forget how pants work when they try to use the toilet. They need lessons in how to put their coats on to go outside in the winter.

These rules -- the set of policies and procedures that constrain applications -- is collectively called the "Operating System." (Don't ask why, the computer folks borrow terms from other disciplines and imbue them with new meanings. There's rarely a sensible etymology, just conventional usage.) The idea of a "system" of components is essential. There are a lot of layers of engineering in the OS.

The presence of an operating system lets multiple apps cooperate. But, it doesn't change the fundamental truth that originated with Babbage and Lovelace and continued on through Turing and Von Neumann and others and was handed down to me.

The general-purpose computer is applied ("programmed") to a problem; it's set up to respond to inputs by displaying outputs.

So that was boring. What did you do?

Good point. That was boring. But necessary, I think, to bracket the nuanced difference between "computer" as a collection and an individual application. The computer-as-collection includes a lot of software: an OS plus applications. This is distinct from each individual application that's part of the collection. It's all software, but the context shifts from everything the computer is doing to one specific solitaire game.

Above, I mentioned that the OS has layers. In a way it's like a quilt, there's a backing, batting in the middle, and a complex quilt top made from pieces. Most important is the quilting that holds the layers together. 

In a way, it's also like a library. There's the foundational problem of storing and loaning books. But there's a secondary problem of finding the damn things; leading to Dewey or LC codes for topics so we can keep related books together. And there's a third layer problem of having an accurate index or catalog of all the books. Using small cards (the card catalog) gives the library flexibility to make sure the catalog matches the stacks. And there are related problems of loaning them out with some reasonable promise to return them. 

I might even be able to work out an analogy with the Apple Orchard or the Arboretum or the Summer Camp. But, I think you get my drift here, that there are foundational elements that we can't really change, and we build on those foundations to make the whole slightly easier for people to use.

I get it, you built application software. What did you do?

What's important about concept of layers is how pervasive the layering idea is in all of computing.

Because of the potential complexity of a solution to a problem, we take the "layering" idea one step further than simply decreeing there should be layers.

What we found, starting in the 70's, was that the operating system tended to conceal many details of the underlying hardware. A modern programming language also divorced us from details of the hardware. Admiral Grace Murray Hopper's idea was to have an application that would transform a program written in some neutral language into the language of whatever hardware we had on hand. She pioneered the COBOL programming language; the language was utterly unlike any specific piece of hardware, and required a "compiler" application to translate COBOL statements into a form that the OS could run as an application.

We liked this idea: the underlying hardware became a kind of hazy abstraction. We knew it was there, but between our languages, libraries of pre-written software, our compilers, and the OS, we didn't really see the underlying hardware. This lets us decompose a complex problem into a number of smaller problems; giving us a lot of leverage.

The core idea of "abstraction" leads to the idea of layers of abstraction. Within our application software we can can also use this idea of layers to decompose our solution to a problem. An application layer can be quilted to a library layer that we bought or downloaded. The library is -- independently -- quilted to an OS layer. And the whole stack of layers is carefully stitched down to the underlying silicon chip. Maybe it was a Motorola chip, or an Intel chip, or an AMD chip. We didn't much know or care.

Well. We cared a little. Some of the AMD chips were faster than some of the Intel chips. So we would prefer to have our OS and our language focused on those chips because things were faster. Until Intel jumped ahead of AMD. The concept was to remain divorced from gritty details of how the little fleck of silicon with its millions of transistors actually worked. 

Recap

Application software configures the general-purpose computer to a specific task. Applications coexist via an operating system and reusable libraries.

Software (application, operating system, libraries) is created in layers and provide abstractions to hide the details of underlying layers.

My job?

Design the layers. Get other programmers to understand the design for the layers. Help them to create statements ("code") using the language of choice. (I'm a big fan of Python, but I've used many, many other languages.)

Note that I didn't (generally) design the visible quilt top in any detail. My job was to help the visual designers and the user experience (UX) designers create a top that delighted people using the software. I made sure that the top and the layers underneath it all fit together reasonably well for a sensible budget. Cutting and stitching all the blocks was a specialized skill that I tried to avoid.

I did more than design, however. When I say design of the structure, you can imagine an architect or civil engineer looking over drawings of girders and beams and making sure the floor would hold the weight of all those books in the new wing of the library.

While many software designers and architects do pour over drawings, I -- personally -- didn't like to leave it at the drawing stage. This was probably a career-limiting choice, but I liked to get my hands dirty actually digging holes and standing up cinderblocks in the foundation. The idea of swinging a hammer to build components told me -- directly -- how good (or bad) my design was.

There's a fork in the career path for programmers. Some software architects work best with Keynote presentations to developers and executives. They build understanding and consensus. They're trusted with larger projects and larger budgets. If things didn't work out, they could deflect blame to the folks writing the software. This distinction between design and realization can be used to avoid culpability. It worried me.

Other architects (me, specifically) work best with code. I still needed to build understanding and consensus. But I also built software so I could be *sure* things worked. I liked to provide concrete, tangible, "do it like this" code.

To higher-level executives -- people with budget authority -- I was only a low-level programmer. 

For decades, this meant a project would wind down after completion, and I would leave the customer's location, and move on to a new project. That's why I traveled a LOT.

A few clients would come to realize that I did offer significant value by being able to design the layers and abstractions while also helping folks actually build the software. This recognition was a rarity, which is why I call it a career-limiting choice. It happened a few times. There's a particularly memorable offer from a client in the 90's that -- in retrospect -- I should have taken. But, generally, I moved from work site to work site, designing, and building the application software for very, very large computers.

So, you went to meetings a lot?

Precisely.

At first, I needed to talk about the problem. What they want software to do. Why do they think new, custom-built, unique software will solve the problem they have? This means meeting with people to understand the problem in the first place. "What can't you do?" "Why can't you do it?" There's a lot of "Why?" questions that need to be asked to locate the obstacle that's easiest to remove. (Or the lowest-hanging fruit we can pick.)

Then, we need to talk about the solution. How will we solve the problem with computers and software? In some cases, they have departments that aren't talking. Or they have legal obstacles. Or they have a half-wit vice president in charge of being the owner's brother. Eventually, we wind up at "aha. They have software that acts as a kind of 'custodian' for their cloud-based resources, but the language of the rules for that custodian are opaque." 

(Seriously. A real problem. Very, very removed from reality: governance of rented "cloud" resources. Enterprise policies for use of cloud resources. Concrete rules for cleaning up the computers rented from a cloud vendor. Mathematical foundations for those rules. Very. Abstract. https://github.com/cloud-custodian/cel-python)

Once we've got the preferred solution, we need to decompose it into things we can download, and things we have to build. Ideally, we can download most (or all) of it and move on. Realistically, the problem domain is unique or something about the overall context and organization is unique, and means leads to customized software to reflect the unique situation.

Before too long, we have meetings to review some pictures: some contexts, some containers for application software, some components (or I've called them "layers" above). This will lead to people writing some code. (The 4 C's: Context, Container, Component, Code.)

(Side-bar. The "container" is a generalization of the idea of a computer. The OS lets multiple applications cooperate; what if we have multiple OS's cooperating? This idea of layers of abstraction is so compelling, we can apply it in a variety of places. This lets us to talk about abstract containers instead of concrete computers.)

We'll have daily meetings while we're building the code that populates the components that gets installed into the containers that fills out the context. These last 10 minutes. What we've done. What we're doing.

We'll have meetings every two weeks to look at components and containers and be sure they work. People will demo what they've done. It will be fun. We'll have donuts.

We'll have impromptu meetings to talk about how to write tests and do quality assurance on our code and components. The testing and quality checking became my obsession during the last five years of my career. Answering the question "Did you test everything?" 

We'll have meetings to talk about managing the containers to be sure they're working. And how to integrate and deploy the components into the containers. 

In and among the meetings, I wrote code. For the last ten years, it was always in Python. Before that it was in other languages.

So, that's what I did for a living. I went to meetings. I wrote code.

HATEOAS is useless? Or not used enough?

See Why HATEOAS is useless and what that means for REST.

The article provides a background leading up to these observations:

• There are very few good tools to create a REST API using this style
• There are no clients widely used to consume these types of APIs

The "useless" in the title is more like "not used enough."

There's a multi-part conclusion that may be more helpful if it's fleshed out further. For now, however, it appears that the big problems center around:

• You still need to write Open Api Specifications (OAS, f/k/a Swagger). I don't think this is bad. The blog post makes it sound like a problem. I think it's essential.
• You need to put versioning somewhere. The path is less than idea. I'm big on the Accept header containing application-specific MIME types. For example, application/vnd.com.your-name-here.app.json+v1. This doesn't strike me as a problem, either.
• The whole approach is "closer to RPC than some REST lovers like to admit." I think this point revolves around the way JSON-RPC or SOAP involves some overheads above basic HTTP that are unhelpful. I don't think the "closer to RPC" follows logically from the lack of tooling for HATEOS, but it certainly could be true that a badly-done API might involve too many of the wrong kinds of overheads.

I think there's a hidden strawman here. The "automatic discovery" idea. I don't think this idea makes a lick of sense. Some people think it's implied (or required) by REST, and any failure to provide for fully-automated semantically rich discovery of an API is some kind of failure.

I don't think full semantic discovery is possible or even desirable. 

• It's not possible because of the problem of assigning names and meanings to resources and verbs in an end-point. The necessary details can only be exposed with a semantically complete ontology and complex SPARQL queries into the ontology to find resources and end-points. 
• It's not desirable because we replace a human-focused OAS with a complete ontology that has to be rigorously defined, and tested to be sure that all kinds of automated discovery algorithms can understand the provided details. And none of this addresses the actual application, it's all rich, detailed meta-description of the application.

I don't see why we're trying to replace people. API discovery is actually kind of hard. The resources, their relationships, and the verbs for getting or updating those resources involves an essentially difficult knowledge capture and dissemination problem. 

DC API User Group - December Edition

https://www.meetup.com/DC-Web-API-User-Group

https://www.meetup.com/DC-Web-API-User-Group/events/245014240/

Nice bunch of people. Nice facility. Good discussions.

The GUI Problem

I write Microservices. And not-so-micro Services. API's.

I got this email recently.

"Goal: get you to consider adding Gooey to your Python tool set"

"Gooey
What it's for: Turn a console-based Python program into one that sports a platform-native GUI.
Why it's great: Presenting people, especially rank-and-file users, with a command-line application is among the fastest ways to reduce its use. Few beyond the hardcore like figuring out what options to pass, or in what order. Gooey takes arguments expected by the argparse library and presents them users as a GUI form, with all options labeled and presented with appropriate controls (such as a drop-down for a multi-option argument, and so on). Very little additional coding -- a single include and a single decorator -- is needed to make it work, assuming you're already using argparse."

The examples and the GitHub documentation make it look delightful.

However.  It's utterly useless for me.  Interesting but useless.

From my perspective, API's and microservices are vastly more important than desktop GUI's.

I'll repeat that in order to start a food-fight:

API's and microservices are more important than desktop GUI's

I almost forgot the important qualifiers: When working with Big Data. Or When working with DevOps Automation.

I realize that some people like to cling to the desktop GUI as a Very Important Thing™. Which could be why they send me emails touting the advantages of some kind of GUI tool or framework. The Desktop GUI is important, but, from my perspective, it's a niche.

Actually two niches.

Niche 1. The word processor and spreadsheet and a few other generic tools for putting text into a computer. While desktop versions are better than server-side emacs and vi, they fill a similar purpose. An IDE is (from this perspective) is little more than a glorified text editor. In places that use Jenkins and Hudson and uDeploy and all of those server-based tools, the desktop IDE is a place to stage code for Jenkins jobs to do the "real" build.

Niche 2. All the other tools that turn a small-ish computer into a dedicated workstation for specific kinds of media production. Video. Audio. Image. Typesetting. These are not "generic" applications like word processors or spreadsheets; they're very specific and narrowly-focused applications. They rely on effectively transforming the general-purpose computer into a very special-purpose computer.

Super-fancy desktop-based tools for analytics or Big Data processing are not actually too useful. Anyone trying to use a desktop as an enterprise systems of record is asking for trouble.  I work with folks trying to process terabyte datasets on their laptops and wondering why it takes so long. My company has servers. We pay for MongoDB and Hadoop. We have API's to access big databases with big piles of data. I'm automating the toolsets as fast as I can so they can work with giant datasets.

Gooey looks like fun. But not for me.

Today's Milestone: Refactoring and Django Migrations

Once upon a time, when today's old folks were young, we'd debate the two project strategies: Hard Part Do Later (HPDL) vs. Hard Part First (HPF).

The HPDL folks argued that you could pick away at the hard part until -- eventually -- it wasn't hard any more. This doesn't often work out well in practice, but a lot of people like it. Sometimes the attempt to avoid the hard part makes it harder.

The HPF folks, on the other hand, recognized that solving the hard problem correctly, may make the easy problems even easier. It may not, but either way, the hard part was done.

The debate would shift to what -- exactly -- constituted the hard part. Generally, what one person finds hard, another person has already done several times before. It's the part that no one has done before that eventually surfaces as being truly hard.

Young kids today (get off my lawn!) often try to make the case that an Agile approach finesses the "hard part" problem. We define a Minimally Viable Product (MVP) and we (magically) don't have to worry about doing the hard part first or last.

They're wrong.

If the MVP happens to include the hard part, we're back a HPF. If the MVP tries to avoid the hard part, we're looking at HPDL.

The Novelty Factor

Agile methods don't change things. We still have to Confront the Novelty (CTN™). Either it's new technology or it's a new problem domain or a new solution to an existing problem domain. Something must be novel, or we wouldn't be writing software, we'd be downloading it.

I'm a HPF person. If you set the hard part aside to do later, all the things you do instead become constraints, limiting your choices for solving the hard part that comes later. In some rare cases, you can decompose the hard part and solve it in pieces. The decomposition is simply Hard Part First through Decomposition (HPFtD™) followed by Prioritize the Pieces (PtP™) and another round of Hard Part First.

Today, we're at a big milestone in the HPF journey.

The application's data model is simple. However.

The application has a complex pipeline of processing to get from source data to the useful data model.

A strict (and dumb) MVP approach would skip building the complex pipeline and assume that it was magically implemented somehow.

A slightly smarter MVP approach uses some kind of technical spike solution to handle the complex pipeline. We do that manually until we get past MVP and decide to implement the pipeline in something more final and complete.

My HPF strategy tackles the complex pipeline because we have to build it anyway and it's hard. We don't have to build all of it. Just enough to lay out the happy path.

The milestone?

It's time to totally refactor because -- even doing the hard part first -- we have the wrong things in the wrong places. Django application boundaries generally follow the "resources". It's a lot like designing a RESTful API. Define the resources, cluster them together in some kind of ontology that provides a meaningful hierarchy.

Until -- of course -- you get past the problem domain novelty and realize that some portion of the hierarchy is going to become really lopsided. It needs to be restructured so we have a flat group of applications.

Wait. What?

Flatten?

Yes.

When we have a Django application model that's got eleventy-kabillion classes, it's too big. Think the magic number 7±2: there's a limit to our ability to grasp a complex model.

Originally, we thought we'd have apps "A", "B", and "C". However. "A" turned out to be more complex than it seemed when we initially partitioned the apps. Based on the way the classes are named and clustered in the model file, it's clear that we have an internal structure is struggling to emerge. There are too many comments and high-level organizational hints in the docstrings.

It looks like this might be the model that's emerging:

• Former A
• A1
• Conceptual A2
• A2a
• A2b
• A3
• B
• C

This means that there will be classes in A3 that depend on separate apps A2a and A2b. Further, A2 is really just a concept that unifies the design; it doesn't need to be implemented as a proper app. Both A2a and A2b depend on A1. A3 depends on A2a, A2b, and A1.  

Ugh. Refactoring. And the associated migrations. 

Django allows us to have nested apps. But. Do we really want to go there? Is a nested collection of packages really all that helpful? 

Or.

Would it be better to flatten the whole thing, and simply annotate the dependencies among apps?

The Zen Of Python suggests that Flat is Better than Nested.

The hidden benefit of Flat is that the Liskov Substitution Principle is actually a bit easier to exploit. Yes, we have a tangled web of dependencies, but we're slightly less constrained when all of the Django apps are peers. Yes, many things will depend on the A1 app, but that will be less of a problem than the current pile of classes is.

The important part here is to start again. This means I need to discard the spike database and discard the history of migrations to date. I always hate disrupting my development databases, since it has test cases I know and remember.

That's the disruptive milestone for me: discarding the old database and starting again.

TDD, API Design and Refactoring

See this short discussion on a Stingray Reader feature:
https://sourceforge.net/p/stingrayreader/discussion/COBOL/thread/d2132851/?limit=25#2a3a

This turned into an exercise in pure TDD.

<rant>
I'm not a fan of applying TDD in a strict, death-march fashion.

I see the comments on Stack Overflow that indicate that some folks feel strongly that strict TDD is somehow helpful. While "test before code" is laudable and often helpful, there's no royal road to good software.

Design involves a great deal of back and forth between code and test. A great deal.

It's logically impossible to write a test without having thought about the code. In order to write the test first, there must be a notional API against which the test is written. Anyone who requires that the test file must be written before the notional class or module is just playing at petty tyranny.

The notional design -- the rough outline of the class or module -- can be written into a file before any tests. It's okay. It is still test-driven because the considerations of testability drove the design process.

In particular, when starting "from scratch" -- with nothing -- writing tests first is senseless. Some module or package structure must exist for the test modules to import.

</rant>
Having ranted, it still arises that the tests do come before any code under some circumstances.

In this case, the requested functionality was quite difficult to visualize. However, it was possible to cobble together a test case that simplified the problem down to something like this this:

01 Some-Record.
     05 Header PIC XXX.
     05 Body PIC X(17).

01 ABC-Segment.
     05 Field-ABC PIC X(17).

01 DEF-Segment.
     05 Field-DEF PIC X(17).

In COBOL, the program would use logic like IF Header EQUALS "ABC" THEN MOVE Body TO ABC-Segment. We need a way to handle something like this in Python so that we can parse the EBCDIC COBOL data.

This summarized example allowed construction of a test case that made use of a API that might have existed. I was pretty sure I had a test case that showed an approach.

What Actually Happened

Since the application already had 178 unit tests, there was plenty of structure that worked.

The single new unit test relied on a notional API that wasn't really in place. The new test bombed grotesquely.

There are two solutions:

• Modify the test.
• Fix the notional API so that it works properly.

I started out chasing the second option. I tweaked some things. More tests failed. I tweaked some more things. The new test finally passed, but another test was failing.

Some careful study of the failing test revealed that my approach was wrong. Way wrong.

The notional API was a bad idea.

The tweaks to make it work were a worse idea.

Back to the Lab Bench

At this point, I had made enough changes that the only thing to do was copy the new test and use the Git Revoke on the local changes to unwind the awful mistakes.

Staring again, I had a slightly better grip on the relevant code. I had a failing test. I tried a different approach that wasn't quite so inventive. This meant modifying the test.

I actually went through a few iterations of the test, using the test method as a kind of lab bench.

A more Pythonic approach to the lab bench is to work from the >>> prompt. I think that all of the exemplary projects use the >>> prompt examples in their documentation. This is a way to narrow and clarify the API. As projects get big, they can sprawl. New features can wind up with many imports to pick and choose elements from existing modules.

When it becomes difficult to use the >>> prompt as the lab bench, that's a sign that the API is too complex. Refactoring must happen.

Using the unit test framework as the lab bench was a hint that something had drifted out of tolerance.

However. I did get a test which passed. Yay. Sort of.

The test code was hideous.

TDD and API Design

The point of TDD, however, is that we have a working suite of tests. Refactoring won't break anything.

The point was that the hideous API could be rewritten into something that both

• Passed all the tests, and
• Was usable at the >>> prompt.

It's difficult to express how valuable the Python >>> prompt is to help clarify API design issues.

The rule is this:

If the API doesn't make sense at the >>> prompt, it's incomprehensible

Sadly, Java doesn't have this kind of boundary. Java programming can spin into quite complex API's, limited only by the laziness of the programmer who avoids refactoring.

Or the malice of the programmer's manager in not allowing time to refactor.

Decisions and Consequences

A single poorly-made decision can have profound ripple-effects. Once your stuck with it, you make accommodations, hacks and work-arounds. Eventually, things work, but the result is less than ideal.

Changing tack requires sometimes pervasive rework to the application. How can we reduce the risks and improve the value created?

A Recent Example

When dealing with bulk econometric data (Bloomberg, D&B, Moody's, etc.) you get BIG files with lots of fields. Depending on what you're paying for, the file layouts are frequently different even though the content is similar. I'm a big fan of plain-old CSV data. Even the tab-delimited variant of CSV is not bad to work with.

Further, most vendors will slap some heading rows on the file so that the column names are--more or less--identified. Surprisingly, this doesn't work out well in practice because there are often multiple columns with the same name. Sigh.

Using Python's csv library module lets us cope with CSV (and tab-delim) quite gracefully.

What's wrong with that decision? Nothing.

Variant Column Names

The question arises when you've purchased several files of econometric data and the column names are slightly different. This happens with a single vendor and across vendors. It's part of the game that can't easily be avoided. Column names vary.

What to do?

Here's the less-than-ideal decision. Make the column names a parameter.

In Python, this is not terribly difficult. The csv module's DictReader provides us a dictionary for each row. Each column name becomes a key. We can access the fields with some_row['this_field'] and some_row['that_field']. How bad can it be?

The extra punctuation is fairly hideous.

More importantly, however, is the nature of the metadata.

Consequence One -- Dynamic Metadata

Dynamic metadata, in this case, means that any indexing of the data is done based on character string column names.

index[index_name][row[column_name]].append( row )

That's rather more complex than the alternative where the metadata has a fixed definition.

some_index[row.column].append( row )

Consequence Two -- Murky ORM

Once we have dynamic metadata, we're largely frozen out of ordinary SQL database implementations. We don't know the column names, we don't know the indices. We can't do simple CREATE TABLE statements because we don't really have the column names until we open the working files.

We have to grub through all the code to find out where the dynamic mapping is reasoned out. Once we find that, we can then consider how to make the metadata fixed enough to tackle a SQL database.

We could, of course, generate the SQL CREATE INDEX statements on-the-fly. There's nothing wrong with it. But it slows down analysis and decision-making when we're not sure what indexes there are or what leads to a choice of index.

What's important here is that we want to use SQLite because it ships with Python. We want our application to use an ORM (like SQLAlchemy or SQLObject). We don't want our application to become a kind of ORM because of the dynamic SQL and dynamic column names.

Cleanup

The cleanup road is clear.

1. Map all variant inputs to one common structure. Rather than work with raw dictionaries from csv, map each row to a standard set of names. For now, we can replace the dictionaries with named tuples to prepare for a migration to an ORM when that's possible.
2. Replace the row['some field'] syntax with row.some_field syntax. Of course, there's a lot of this. This is a pervasive change.
3. Find all the dynamic index creation and refactor that into a more static "database-like" place for now.

Item 1 is pretty easy to unit test. We're adding a function to map from dynamic names to fixed names. Nothing much to this testing-wise.

Item 2 requires unit tests with really good code coverage or there's no earthy way we can be sure that each mapping-syntax name has been transformed into an attribute-syntax name.

Item 3 barely requires testing. Indexes and other features are performance enhancements that can be removed and added without altering functionality.

Performance Discussions and Software Design

Read this first: "There is something I find interesting about online discussions around performance issues..." It's about Stack Overflow, specifically. Apparently, someone didn't get their question answered and decided it was better to gripe than to rewrite the question.

Let's look at their response in pieces.

"people try to gang up". Since there's almost no social networking capability, this is a bit much to attribute to people responding to a poorly-worded question. But, if you've worked all day on a bad solution to a poorly-conceived problem, it can feel like being ganged up on. When reality leaks in, it can feel unpleasant.

Hint 1. There are no gangs. It's possible that the question really is poorly written.

"cookie-cutter, patronizing, zero-information responses". I'm guessing these are comments suggesting the approach is bad and asking for clarification. I run afoul of this often because I feel compelled to post comments asking for clarification. Some folks just don't like to clarify. More than once I've been told that their question was very clear. Since I'm asking for clarification, it seems odd to insist the question is perfect. Worse, of course, is asking for help on Stack Overflow, but refusing to clarify the help required.

Hint 2. Clarify. Please. Don't insist that the question is perfect.

"they assume, without any basis, that the person has not (a) benchmarked the code," When the question has no bench mark data, this isn't an assumption. It's a response to the lack of benchmark data.

Hint 3. Provide the facts. Don't complain when folks ask for facts.

"(b) is obviously running an inferior algorithm". Again, this isn't an assumption. It's the response to an incomplete question where the algorithm isn't provided. Also, it's a common response to questions where the algorithm really is inferior.

Hint 4. Consider that -- even after spending days banging your head against the wall -- your question might be poorly-written and require both benchmark data and an algorithm.

"advice about premature optimization... is a well assimilated folklore by now and I dont see how repeating that adds value". Without measurements, profiler results and benchmark data, this is our only possible response. After the profiler results are posted, this advice really is useless. Before profiler results are posted, this advice often turns out to be essential.

Hint 5. Whatever you might know is not well-assimilated folklore on Stack Overflow as a whole. We don't know you, sadly. We don't know how much you know. To avoid useless advice, provide evidence -- in the question -- that the advice has already been followed.

"better served if the discussion shifted to ... Pointed out possible bottlenecks ahead of time," Wouldn't that be nice? What's a "possible bottleneck"? It's a badly-design algorithm. So, the responses to performance questions has to be focused on algorithm choice right away. That means details on the code being used, and profiling information.

Hint 6. There is no hint 6. This would simply repeat hints 3 and 5.

"regardless of the fact whether the code construct is actually a bottleneck in the application or not, it is always good to know what the more efficient alternatives are... there is something called intellectual curiosity."

Reducing a question to a hand-waving hypothetical doesn't improve the question. It doesn't rationalize a poor question. The question still needs to be clarified.

Hint 7. If the question raises a lot of comments and useless advice, please rewrite the question.

The Wrapper vs. Library vs. Aspect Problem

Imagine that we've got a collection of applications used by customers to provide data, a collection of applications we use to collect data from vendors. We've got a third collection of analytical tools.

Currently, they share a common database, but the focus, use cases, and interfaces are different.

Okay so far? Three closely-related groups or families of applications.

We need to introduce a new cross-cutting capability. Let's imagine that it's something central like using celery to manage long-running batch jobs. Clearly, we don't want to just hack celery features into all three families of applications. Do we?

Choices

It appears that we have three choices.

1. A "wrapper" application that unifies all the application families and provides a new central application. Responsibilities shift to the new application.
2. A site-specific library that layers some common features so that our various families of applications can be more consistent. This involves less of a responsibility shift.
3. An "aspect" via Aspect-Oriented programming techniques. Perhaps some additional decorators added to the various applications to make them use the new functionality in a consistent way.

Lessons Learned

Adding a new application to be an overall wrapper turned out to be a bad idea. After implementing it, it was difficult to extend. We had two dimensions of extension.

1. The workflows in the "wrapper" application needed constant tweaking as the other applications evolved. Every time we wanted to add a step, we had to update the real application and also update the wrapper. Python has a lot of introspection, but these aren't technical changes, these are user-visible workflow changes.
2. Introducing a new data types and file formats was painful. The responsibility for this is effectively split between the wrapper and the underlying applications. The wrapper merely serves to dilute the responsibilities.

Libraries/Aspects

It appears that new common features are almost always new aspects of existing applications.

What makes this realization painful is the process of retrofitting a supporting library into multiple, existing applications. It seems like a lot of cut-and-paste to add the new import statements, add the new decorators and lines of code. However, it's a pervasive change. The point is to add the common decorator in all the right places.

Trying to "finesse" a pervasive change by introducing a higher-level wrapper isn't a very good idea.

A pervasive change is simply a lot of changes and regression tests. Okay, I'm over it.

If only architectures were this simple

http://xkcd.com/730/

I almost wet myself. This is so typical.

Buzz in the general public regarding software bugs

I got this the other day: "there seems to be a lot of buzz out in the general public regarding software bugs".

Attached to this was an article from The Economist in 2003 plus one from 2010. To me, this doesn't seem to be a "lot" of buzz. But what do I know?

Further, it did not come from someone outside the software/IT industry. It came from a DBA. I guess the presence of this email in my inbox must mean some DBA's are surprised that there are bugs. I guess they were surprised to see "bug" in a general-interest magazine.

They also forwarded a link to http://www.glitchthebook.com/. This looks more interesting than a writer for The Economist (http://www.economist.com/) providing information to a general audience that every professional should already know.

I guess it could be interesting when someone notices "bug" in a general-interest magazine.

Hidden Cost Hogwash

I object, however to this "hidden cost" hogwash. Bugs have an explicit, obvious, direct cost. There may be "hidden costs" but they are largely irrelevant and pale in comparison to direct costs.

What we need are articles not on the "hidden cost", but on actual bugs. In particular, there are two kinds of actual costs that we need to look at: "hidden bugs" and "compound bugs".

• Hidden Bugs. These are things simply below the user interface level. They're present and they're often worked-around by UI hacks. Hidden Bugs are more costly than visible bugs. Complex multi-layered and multi-component architectures are packed with hidden bugs.
• Compound Bugs. These are hidden bugs where the workaround also has a bug. The interface file has an intermittent glitch, so the web services are cluttered with try: statements. The try: statements, themselves, harbor bugs, so we have to then add assert statements and declare it "defensive programming". The net effect is to simply log something that was provided to the interface incorrectly. Sigh.

We shouldn't waste time talking about "hidden costs" of glitches when we aren't even sure what the actual up-front costs are. If we knew the costs, we'd spend a bit more on the software to prevent the bugs in the first place.

We also shouldn't be surprised to see "bug" in a general-interest magazine.

API Quality Check

A recent request for an API quality check sent me into a paroxysm.

The request seemed simple enough. They had two varieties of API design: varietal M had a lot of methods, each with relatively few parameters. Varietal P had a few methods, but each had a boat-load of parameters.

There had been some "reading" on API design and questions were raised. They wanted me to weigh in, telling them that style M was "better" than style P. [It is, but that's not the point.]

I was shocked speechless.

I find it incredible that anyone could even need coaching in API design. Much less find Tulach's book and still be unable to apply the principles.

Here's what bugged me.

If they were coding in any sensible language, they should have mountains of API examples all around them. Java has a huge standard library. C# has the entire .Net framework. Python has a vast library. All of these are tremendous, well-designed, carefully crafted examples of API's.

API's Everywhere

As far as I can see, the world is fat with API examples. Everywhere you look, every vendor, every product, everything has an API.

It just shouldn't be rocket science to compare your API against the established standard for the language in which you're working.

Somehow it was possible for several programmers to be completely unable to find any examples of API design.

I can only assume that they are living in a time-warp; none of them have ever connected to the Internet or seen any code but their own. Perhaps the only API they could think of was JDBC. Or perhaps they were all Visual Basic or PL/SQL programmers and didn't see much open-source code. Or perhaps they had some really obscure language where no one posts any open source API's.

What To Do?

My direct advice was to read Tulach and create a big spreadsheet ranking their code against each principle that Tulach provides.

After thinking about it, I realize I should have asked what API's they were currently using and how their proposed new code stacked up against the existing language and framework they already had.