SOLID Coding in Python

SOLID Coding in Python by Mattia Cinelli.

Download Medium on the App Store or Play Store

This was fun to read. It has some nice examples. 

I submit that the order of presentation (S, O, L, I, D) is misleading. The acronym is fun, but awkward.

My LinkedIn Learning course covers these in (what I think is) a more useful order.

1. Interface Segregation. I think this is the place to start: make your interfaces as small as possible.
2. Liskov Substitution. Where necessary, leverage inheritance.
3. Open/Closed. This is a good quality check to be sure you've followed the first two principles well.
4. Dependency Injection. This is often about test design and future expansion. In Python, where everything really happens at run time, we often fail to parameterize a type properly. We often figure that out a test time, and need to revisit the Open/Closed principle to get things right.
5. Single Responsibility is more of a summary of the previous principles than a distinct, new principle. I think it comes last and should be treated as a collection of good ideas, not a single idea.

I think time spent on the first three -- Interface Segregation, Liskov Substitution, and the Open/Closed principle -- pay off tremendously. The ILODS acronym, though, isn't as cool as SOLID.

The "Single Responsibility" suffers from an ambiguous context. At one level of abstraction, all classes have a single responsibility. As we dive into details, we uncover multiple responsibilities. The further we descend into implementation details the more responsibilities we uncover. I prefer to consider this a poetic summary, not the first step in reviewing a design.

Architectural Boundaries: Which Package/Module/Class Owns That Responsibility?

 The SOLID design principles beat the design boundary issue to death. Here are the principles in my preferred order. (See https://www.linkedin.com/learning/learning-s-o-l-i-d-programming-principles) 

1. Interface Segregation -- minimize the boundaries. Do this first.
2. Liskov Substitution -- keep the boundaries consistent. Do this for hierarchies.
3. Open/Closed -- keep the boundaries stable and allow subclasses. 
4. Dependency [Inversion] Injection -- keep the implementation separate from the design.
5. Single Responsibility -- This is essentially a summary of the above four principles.

The point here is that these principles are pleasantly poetic, but there are those edgy cases where an interface can go either way.

Specifically, here's an Edgy Case that can go either way.

We're reading GPX (GPS Exchange) data. See https://www.topografix.com/GPX/1/1/. 

Associated with this is what's known as the Lowrance USR file format. A lot of devices include the same (or similar) underlying software, and can exchange waypoint and route information in USR format.

We have this as part of the underlying model.

• The underlying Angle as an abstraction. This has two subclasses:
• Latitude. An angle with "N" and "S" for its sign, conventionally shown as a two-digit number of degrees: 25°42.925′N
• Longitude. An angle with "E" and "W" for its sign, conventionally shown as a three-digit number of degrees: 080°13.617′W
• A Point (or LatLon) is a two-tuple, tuple[Lat, Lon].

A waypoint includes name, description, a time-of-last-update (TOLU), and display symbol to be used. It may also include a GUID to track name changes and assure uniqueness in spite of repeated names.

So far, so good. Nothing too edgy there. "Where's the problem?" you ask.

The problem is representation.

In GPX files, latitude and longitude are float values in degrees. You'll see this: <wpt lon="-80.22695124" lat="25.7154147">...</wpt>.

To do any useful computation, they need to be radians. Or a geocode that supports proximity comparisons, like OLC.

And. If you work with a CSV export from a tool like OpenCPN, then you get strings. This can be any combination of degrees and minutes or degrees, minutes, and seconds. And, depending on the software, there may be either ° or º for the degrees. Can't tell the apart? One is U+00B0, the DEGREE SIGN. The other is U+00BA, the MASCULINE ORDINAL INDICATOR. Plus, of course, everyone uses apostrophe (') and quote (") where they should have used prime (′) and double prime (″). These are easy regular expression problems to solve.

This leads to a class like the following:

class Angle(float):
    @classmethod
    def fromdegrees(cls, deg: float, hemisphere: Optional[str] = None) -> "Angle":
        ...
    @classmethod
    def fromstring(cls, value: str) -> "Angle":
        ...

This Angle class converts numbers or strings into useful values; in radians internally. Formatted in degrees externally.  (And yes, this gets a warning from Python 3.9 that we can't usefully extend float like this.)

The problem is USR files. 

In USR files, they use millimeter mercator numbers for latitude and longitude. These are distances from the equator or the prime meridian. Because they're in millimeters, an integer will do nicely. A little computation is done to extract degrees (or radians) from these values.

SEMIMINOR_B = 6_356_752.3142

lon = round(math.degrees(mm_lon / SEMIMINOR_B), 8)
lat = round(
        math.degrees(2 * math.atan(math.exp(mm_lat / SEMIMINOR_B)) - math.pi / 2), 8
    )

These aren't too bad. But.

Here's the question.

Where does this belong? Is it part of the underlying Angle class? It is separate?

Where does Millimeter Mercator representation belong?

This raises a secondary question: Where does ANY representation belong?

Do we separate the essential object (an angle in radians, a float) from all representation questions? If so, how do we properly bind value and representation at run time? 

Is our app full of complex mixins to bind the float with representation choices?  class Latitude(float, DMS, MM, etc.): pass. This seems potentially annoyingly complex: we have to make sure names don't collide, when defining all these aspects separately.

I think the representation for latitudes and longitudes *is* the essential problem here. The math (i.e. computing the loxodromic distance between points) is trivially separated from all of these representation concerns. 

If we buy into the centrality of representation issues, then, we're down to the following argument.

Resolution: millimeter mercator belongs in the Angle class.

Affirmative: it's yet another representation of an angle's value. 

Negative: it's not used outside USR files and belongs in the USR file parser module.

Affirmative Rebuttal: None of the other representations in Angle are tied specifically to a file format.

Negative Rebuttal: Because the other formats (float, string) are intermixed in CSV files and text displays, making them "widely used." While float is used consistently in GPX, this encoding is a pleasant exception that relies on widely-used encodings.

Okay. We seem to have conflicting goals here. Some representation is a generic thing that crosses file formats and some representation is localized to a specific file format and not reused.

The SOLID design principles don't help chose between these designs. Instead, they provide post-hoc justification for the design we chose.

We can exploit the SOLID principles in a variety of ways. Some Examples.

• We could claim that LatitudeMM is a subclass of Latitude with the MM conversions mixed in. Open/Closed. Liskov Substitution. 
• We could claim that Latitude has several load/dump strategies available, including Load from MM. Open/Closed. Dependency is Injected at run-time.

Sigh.

Prior Art

Methods like __str__() and __repr__() are generally considered part of the essential class. That means the most common string representations need to be provided. The parsing of a string, similarly, is the constructor for  an instance of the float class.

So. Some representations are part of the class. Clearly, however, not all representations are part of the class. Representation codecs like pickle, struct, or ctype are kept separate.

I'm going to make the case that there's a very, very fine line between unique and non-unique-but-not-widely-used aspects of a class of objects. And, in this specific case, the millimeter mercator should be kept separate.

I'm going to rely on other representations like PlusCode (also called OLC) as yet another obscure representation and insist these aren't essential to the class. Indeed, I'm going to suggest that proximity-friendly geocoding is clearly separate because it's a hack to replace complex distance computations with substring comparisons. 

Over-Solving or Solving Problems You Don't Have

Sometimes we call them "Belt and Braces" solutions. As a former suspenders person who switched to belts, the idea of wearing both is a little like over-engineering. In the unlikely event of catastrophic failure of one system, your pants can still remain properly hoist. There's a weird, but defensible reason for that. Most over-engineering lacks a coherent reason. 

Sometimes we call them "Bells and Whistles." The solution has both bells and whistles for signaling. This is usually used in a derogatory sense of useless noisemakers, there for show only. Again, there's a really low-value and dumb, but defensible reason for this. 

While colorful, none of this is helpful for describing over-engineered software. Over-engineered software is often over-engineered for incoherent and indefensible reasons.

Over-engineering generally means trying to solve a problem that no user actually has. This leads to throwing around irrelevant features.

Concrete Example

I lived on a boat. I spent a fair amount of time fretting over navigation. 

There are two big questions: 

1. How far apart are two points, really. 
2. What's the real bearing from one point to another.

These are -- in some cases -- easy to answer.

If you have a printed, paper chart at the right scale, you can use dividers to compute a distance. It's actually a very easy task. Similarly, you can read the bearing off the chart directly. There's a trick to comparing a course to a nearby compass rose, but it's easy to learn and very accurate.

Of course, we don't want to painstakingly copy our notes from a paper chart to a spreadsheet to add them up to get total distance. And then fold in speed to get time and fuel consumption. These summary computations are a pain.

What you want is to do all of this with a computer.

1. Plot the points using a piece of software like OpenCPN (https://opencpn.org).
2. Extract the GPX file.
3. Compute distances, bearings, and durations to create a route.

"So?" you ask.

So. When I did this, I researched the math and got a grip on the haversine formula for doing the spherical geometry computation of distances between points on a sphere.

It's not too bad. The formula are big-ish. But manageable. See http://www.edwilliams.org/avform.htm#Dist for the great circle distance formula.

For airplanes and powered freighters crossing oceans, this is perfect.

For a small sailboat going from Annapolis, Maryland, to the Bahamas, this level of complexity is craziness. While accurate, it doesn't really solve the problem I have. 

I don't actually need that much accuracy. 

I need this much accuracy.

And no more. This is the essential hypotenuse distance using an R-factor to convert the difference between latitudes and the distance between longitudes into pretty-close distances. For nautical miles, R is 60×180÷π. 

This is simpler and it solves the problem I actually have. Up to about 232 miles, the answer is within 1 mile of correct. The error grows quickly. Double the distance and the error seems to jump to 8 miles. A 464 mile sailing journey (at 6 knots) takes 3 days. Wind, weather, tides and currents will introduce more error than the simplifying assumptions.

What's important is this can be put into a spreadsheet without pain. I don't need to write sophisticated Python apps to apply haversine to sequences of way-points. I can do a simpler hypotenuse computation on waypoints converted to radians.

Is there a lesson learned?

I think there is.

There's the haversine a super-general solution. It handles great-circle routes elegantly. 

But it doesn't solve my actual problem. And that makes it over-engineering.

My problem is what we call rhumb-line sailing. Over short-enough distances the world may as well be flat. Over slightly longer distances, errors in the ship's compass and speedometer make a hyper-accurate great circle route moot. 

(Even with several fancy GPS-based navigation computers, a prudent mariner has paper backups. The list of waypoints, estimated times and directions are essential when the boat's GPS reciever fails.)

I don't really need the sophistication (and the potential for bugs) with haversine. It doesn't solve a problem I actually have.

Stingray Reader Pervasively Bad Decision

I made some bad decisions when I wrote this a few years ago: https://github.com/slott56/Stingray-Reader. Really bad. And. Recently, I've burdened myself with conflicting goals. Ugh.

I need to upgrade to Python 3.8, and add type hints. This exposed somes badness.

See https://slott-softwarearchitect.blogspot.com/2020/01/stingray-reader-rewrite.html for some status.

The very first version(s) of this were expeditious solutions to some separate-but-related problems. Spreadsheet processing was an important thing for me f. Fixed-format file versions of spreadsheets showed up once in a while mixed with XLS and CSV files. Separately, COBOL code analysis was a thing I'd been involved in going back to the turn of the century.

The two overlap. A lot.

The first working versions of apps to process COBOL data in Python relied on a somewhat-stateful representation of the COBOL DDE (Data Definition Element.) The structure had to be visited more than once to figure out size, offset, and dimensionality. We'll talk about this some more.

A slightly more clever algorithm would leverage the essential parsing as a kind of tree walk, pushing details down into children and summarizing up into the parent when the level number changed. It didn't seem necessary at the time.

Today

I've been working for almost three weeks on trying to disentangle the original DDE's from the newer schema. I've been trying to invert the relationships so a DDE exists independently of a schema attribute. This means some copy-and-paste of data between the DDE source and the more desirable and general schema definition.

It turns out that some design decisions can be pervasively bad. Really bad-foundation-wrecks-the-whole-house kind of bad.

At this point, I think I've teased apart the root cause problem. (Of course, you never know until you have things fixed.)

For the most part, this is a hierarchical schema. It's modeled nicely by JSONSchema or XSD. However. There are two additional, huge problems to solve.

REDEFINES. The first huge problem is a COBOL definition can redefine another field. I'm not sure about the directionality of the reference. I know many languages require things be presented in dependency order: a base definition is provided  lexically first and all redefinitions are subsequent to it. Rather than depend on order of presentation, it seems a little easier to make a "reference resolution" pass. This plugs in useful references from items to the things they redefine, irrespective of any lexical ordering of the definitions.

This means we data can only be processed strictly lazily. A given block of bytes may have multiple, conflicting interpretations. It is, in a way, a free union of types. In some cases, it's a discriminated union, but the discriminating value is not a formal part of the specification. It's part of the legacy COBOL code.

OCCURS DEPENDING ON. The second huge problem is the number of elements in an array can depend on another field in the current record. In the common happy-path cases, occurrences are fixed. Having fixed occurrences means sizes and offsets can be computed as soon as the REDEFINES are sorted out.

Having occurrences depending on data means sizes and offsets cannot be computed until some data is present. The most general case, then, means settings sizes and offsets uniquely for each row of data.

Current Release

The current release (4.5) handles the ODO, size, and offset computation via a stateful DDE object.

Yes. You read that right. There are stateful values in the DDE. The values are adjusted on a row-by-row basis.

Tomorrow

There's got to be a better way.

Part of the problem has been conflicting goals.

• Minimal tweaks required to introduce type hints.
• Minimal tweaks to break the way a generic schema depended on the DDE implementation. This had to be inverted to make the DDE and generic schema independent.

The minimal tweaks idea is really bad. Really bad. 

The intent was to absolutely prevent breaking the demo programs. I may still be able to achieve this, but... There needs to be a clean line between the exposed work-book like functionality, and some behind the scenes COBOL DDE processing.

I now think it's essential to gut two things:

1. Building a schema from the DDE. This is a (relatively) simple transformation from the COBOL-friendly source model to a generic, internal model that's compatible with JSONSchema or XSD. The simple attributes useful for workbooks require some additional details for dimensionality introduced by COBOL.
2. Navigating to the input file bytes and creating Workbook Cell objects in a way that fits with the rest of the Workbook abstraction.

The happy path for Cell processing is more-or-less by attribute name: row.get('attribute').  This changes in the presence of COBOL OCCURS clause items. We have to add an index. row.get('ARRAY-ITEM', index=2) is the Python version of COBOL's ARRAY-ITEM(3).

The COBOL variable names *could* be mapped to Python names, and we *could* overload __getitem__() so that row.array_item[3] could be valid Python to fetch a value.

But nope. COBOL has 1-based indexing, and I'm not going to hide that. COBOL has a global current instance of the row, and I'm not going to work with globals. 

So. Where do I stand?

I'm about to start gutting. Some of the DDE size-and-offset (for a static occurrences)

Building Skills in OO Design

See https://www.patreon.com/posts/30995708

I've (finally) gotten the book content upgraded to Python 3.7.

I've also deleted all the previous versions of the book. I had been keeping them on my web server because -- well -- because I don't know why. They go back to at least 2011, some of the content may be even older than that.

I've also deleted some previous self-published content.

(I started writing about Python almost 20 years ago. Some of the content could have been that old. It deserves to be deleted.)

Python Enhancement Proposal -- Floating an Idea

Consider the following code

def max(m: int, n: int) -> int:
    if m >= n:
        return m
    elif n >= m:
        return n
    else:
        raise Exception(f"Design Error: {vars()}")

There's a question about else: clause and the exception raised there.

• It's impossible. In this specific case, a little algebra can provide that it's impossible. In more complex cases, the algebra can be challenging. In some cases, external dependencies may make the algebra impossible.
• It's needless in general. An else: would have been better than the elif n >= m:.  The problem with else: is that a poor design, or poor coordination with the external dependencies, can lead to undetectable errors.

Let's look at something a little more complex.

def ackermann(m: int, n: int) -> int:
    if m < 0 or n < 0:
        raise ValueError(f"{m} and {n} must be non-negative")
    if m == 0:
        return n + 1
    elif m > 0 and n == 0:
        return ackermann(m - 1, 1)
    elif m > 0 and n > 0:
        return ackermann(m - 1, ackermann(m, n - 1))
    else:
        raise Exception(f"Design Error: {vars()}")

It's somewhat less clear in this case that the else: is impossible. A little more algebra is required to create a necessary proof.

The core argument here is Edge Cases Are Inevitable. While we can try very assiduously to prevent them, they seem to be an emergent feature of complex software. There are two arguments that seem to indicate the inevitability of edge and corner cases:

• Scale. For simple cases, with not too many branches and not too many variables, the algebra is manageable. As the branches and variables grow, the analysis becomes more difficult and more subject to error. 
• Dependencies. For some cases, this kind of branching can be refactored into a polymorphic class hierarchy, and the decision-making superficially simplified. In other cases, there are multiple, disjoint states and multiple conditions related to those states, and the reasoning becomes more prone to errors.

The noble path is to use abstraction techniques to eliminate them. This is aspirational in some cases. While it's always the right thing to do, we need to check our work. And testing isn't always sufficient.

The noble path is subject to simple errors. While we can be very, very, very, very careful in our design, there will still be obscure cases which are very, very, very, very, very subtle. We can omit a condition from our analysis, and our unit tests, and all of our colleagues and everyone reviewing the pull request can be equally snowed by the complexity. 

We have two choices.

1. Presume we are omniscient and act accordingly: use else: clauses as if we are incapable of error. Treat all complex if-elif chains as if they were trivial.
2. Act more humbly and try to detect our failure to be omniscient.

If we acknowledge the possibility of a design error, what exception class should we use?

• RuntimeError. In a sense, it's an error which didn't occur until we ran the application and some edge case cropped up. However. The error was *always* present. It was a design error, a failure to be truly omniscient and properly prove all of our if-elif branches were complete.
• DesignError. We didn't think this would happen. But it did. And we need debugging information to see what exact confluence of variables caused the problem.

I submit that DesignError be added to the pantheon of Python exceptions. I'm wondering if I should make an attempt to write and submit a PEP on this. Thoughts?

Eager and Lazy Properties

See this

Dear Pythonista lazyweb: If I have a property spam and the attribute that backs it is _spam, what do we call that? The "backing attribute"? The "original property"? Something else? Is there an official term for this?

— Al Sweigart (@AlSweigart) January 29, 2019

My answer was -- frankly -- vague. Twitter being what it is, I should have written the blog post first and linked to it.

The use case is this.

class X:
    def __init__(self, x):
        self._value = f(x)
    @property
    def value(self):
        return self._value

We've got a property where the returned value is already an instance variable.

I'm not a fan.

This reflects an eager computation strategy. f(x) was computed eagerly and the value made available via a property. One can justify the use of a property to make the value read-only, but... still nope.

There are a lot of alternatives that make more sense to me.

Option 1. We're All Adults Here.

Here's an approach I think is better.

class X:
    def __init__(self, x):
        self.value = f(x)

It's read-only because -- really -- if you change it, you break the class. So don't change it.

This is my favorite. Read-onlyness is sometimes described has a way protect utter idiots from breaking a library they don't seem to understand. Or. It's described as a way to prevent some Evil Genius Programmer (EGP) from doing something intentionally malicious and breaking things.

Bah.

It's Python. They have access to the source. Why mess around breaking things this way?

Option 2. Lazyiness

Here's an approach that hits at the essential feature.

class X:
    def __init__(self, x):
        self.x = x
    @property
    @lru_cache(None)
    def value(self):
        return f(self.x)

This seems to hit at the original intent without an explicit cached variable. Instead the caching is pushed off into another space. (I'm writing a chapter on decorators, so this may be a bit much.)

The idea, though, is to make properties lazy. If they're expensive, then the result should be cached.

There may be other choices, but I think lazy and eager cover the bases. I don't think eager is wrong, but I don't see the need for a property to hide the attribute created by an eager computation.

Misunderstanding OO Programming

Read this. Goodbye, Object Oriented Programming

I like this because parts of it are wrong, and parts are based on peculiarities of specific languages which aren’t problems in other languages.

The “wrong” things are on a spectrum. At one end are things almost right. The other end is hoped-for things which — frankly — were never true.

The most important piece of nonsense is class-level reuse across projects. Class-level reuse in a new project was not a thing in OO programming. The monkey-banana-jungle “problem” only exists in a strange world were someone made up the idea of single classes being reused in isolation. The rest of us knew the scope of reuse was within a project or a narrow family of projects aimed at a single problem domain.

"Utility" classes that could be reused and generic data structures were always available as frameworks and libraries. Things built to solve a specific problem were going to be tailored to the problem. Most OO designers knew this and knew that making something generic would be hard. Making something reusable and installable by others was even harder. (Especially in compiled languages where you wanted to hide intellectual property by keeping the source secret.)

The "OO promised me reuse and lied" is a misstatement. Please rephrase this is "I imagined there could be class-level reuse and discovered it was hard."

Multiple inheritance does work in a number of languages, so I’ll skip the complaints centered on single inheritance.

I don't fully understand the complained about encapsulation. There are lots of books on separating interface from implementation to more fully isolate implementation details. If references need to be treated more opaquely, there are lots of techniques for this. It’s not broken. Indeed, it’s really well understood. ("But I won't want to introduce wrapper classes to insulate the references." Sigh. That's how it's done.)

I think the "references leak details about encapsulation" requires rephrasing as "I imagined some kind of perfectly isolated programming where references were not usable in spite of me making them usable." Or perhaps "I wish references had special treatment to make them not work as references except in a limited context which I get to imagine."

The polymorphism complaint appears to be “okay, this actually works.”  I guess. Or. “There are other ways to do this in other languages.” I'm sure it's an important point, but I can't quite discern what OO principle is allegedly broken here.

tl;dr

No one was lied to. If someone was "burned" by some OO hype, I’d like to see the actual quote of the actual hype. The “I was told there would be X”, requires some substantiation.

And. Stop griping about encapsulation. When the source is available (as it is in many languages) there's no enforcement other than public shaming.

Also. Use Python. Most of the original post seems to be complaints about C++ weirdness.

Lamenting the Death of Object-Oriented Programming. (Sigh) Again?

See Goodbye, Object Oriented Programming.

I don't want to say that the entire article is bunk. It's not. It raises a few good points. Points which I thought were pretty well known.

What's aggravating is that this lamentation is overly broad.  It treats all languages as if they're Java or C++. That's not true, and as a consequence, the article is less useful than it could be.

Banana Monkey Jungle Problem. Only true if you are sadly mistaken about the unit of reuse. The class as unit of reuse -- across projects -- is false, has been false, and will always be false. The idea of class inheritance for reuse makes perfect sense. Sharing individual classes between projects has never (as far as I know) been a promise of OO programming. Maybe I read the wrong books and missed that promise.

The Triangle Problem. Isn't actually a problem. Python has a defined method resolution order.

The Fragile Base Class Problem. This points out the well known issue with having concrete classes depend on other concrete classes. The SOLID design principles suggest concrete classes should depend on abstractions. Abstractions do not suffer (as much) from the fragile base class problem.

The Hierarchy Problem. I guess the idea that the real world is multi-dimensional can be confusing. If everything has to be force-fit into single inheritance, this would create the hierarchy problem. If we allow multiple inheritance, this problem evaporates.

The Reference Problem. Even C++ has "smart" pointer packages. Java has garbage collection. Python does reference counting. This is only a problem if you go out of your way to deal with pointers in a primitive way.

The part on Polymorphism didn't make any sense. There didn't seem to be a tidy problem. Just a confusingly vague statement that "Interfaces will give you [polymorphism?]. And without all of the baggage of OO". I don't get how interfaces are necessary without the baggage of OO. So, I can't really try to refute this.

In the long run, I guess this was a way to introduce some of the benefits of a functional approach. I'm not sure that this kind of criticism of object-oriented programming is very helpful. It doesn't apply to all OO languages, so it's misleading at best. (At worst, it's simply wrong.)

I think these problems are interesting and can be used to show the benefits of functional programming. But without the actual functional programming examples, this isn't very useful.

Getting Rid of the Gang-of-Four Design Patterns is Nonsense

Someone found Yet Another Post (YAP™) insisting that the Gang of Four (GOF™) patterns were on their last legs. The email was misleading, because this is not precisely what the article said. The bottom-line was that Design Patterns in general are merely a response to gaps in the underlying programming language. A position that's nonsense at its very foundation.

The lexicon of design patterns varies from language to language. GoF patterns aren't "going away." They're part of the Java/C++ world. They don't apply quite the same way to Python or functional languages.

There's a more serious issue, though: Language Mapping. First some background.

Design Patterns

Design Patterns will always exist. They're an artifact of how we process the world. We tend to classify individual objects so that we don't have to deal with each object as a separate wonder of nature.

It's Just Another Brick In The Wall.

We don't have to examine each rectangular solid of ceramic and understand the wonderfulness of it. We can group and summarize. Classify. Brick is a design pattern. So is masonry. So is wall. They're all patterns. It's how we think.

Design Patterns and Language Gaps

There's a claim that moving toward functional languages will kill design patterns. This presumes (partly) that non-OO languages magically don't have design patterns. This is (see above) kind of insane. Languages have design patterns. We recognize these patterns all the time.

A functional language has a common technique (or pattern) for visiting nodes in a hierarchy. We don't dwell on the wonderfulness of the code as if we'd never seen it before. Instead, we classify it based on the design pattern, and leverage this higher-level understanding to figure out why we're walking a hierarchy.

Sounding the death knell for design patterns also presumes (partly) that functional languages are magically more complete that OO languages. In this newer better language, we don't need patterns because there are no gaps. This is pretty much nutso, too. The Patterns Fill Language Gaps school of thought ignores the fact that there are many ways to implement these "gaps". We can use GoF design patterns, or we can use other software designs that don't fit the GoF design patterns. Both work.

The patterns aren't filling a "gap." They're providing guidance on how to implement something. That's all. Nothing more. Guidance.

"But wait," you say, "since I needed to write code, that's evidence that there's a gap."

"What?" I ask, incredulous. "Are you claiming that any code is evidence of a language gap? Does that mean all application software is just a language gap?"

"Let's not be silly," you say. "I can split a hair and create a tiny distinction between software I shouldn't have to write and software I should have to write."

I remain incredulous.

Design Patterns as Damage

The idea that somehow the GoF design patterns are a problem is also goofy. The GoF design patterns are pretty slick. They solve a fairly broad suite of problems in an elegant and consistent manner.

They're just good design.

Yes, they can be complex. Sorry about that. Software can be complex if you want really excellent flexibility and extensibility.

AND.

Bonus.

Software can be complex when you have to work around the problems of "compiler" and "locked libraries" and "no source." That is, the GoF patterns apply in full force for C++ and Java where you're trying to protect your intellectual property by disclosing only headers and obfuscated implementation details. Indeed, there are few alternatives to the GoF patterns if you're going to distribute a framework that has no visible source and needs to leave extension points for users.

If you don't have Locked-NoSource-Compiled code as a backdrop, the GoF patterns can be simplified a little. But some of the patterns are essential. And remain essential. There are some really great ideas there.

In Python world, we rely on a modified subset of the GoF patterns. They work extremely well.

When writing functional-style Python using immutable data structures (to the extent possible), we use a different set of design patterns. Not so many GoF patterns when we're trying to avoid stateful objects. But some patterns (like the Abstract Factory) are really very helpful even in a largely functional context. It morphs from an abstract factory class to a factory function, and it loses the "abstract" concept that's part of C++ and Java, but the core Factory design pattern remains.

The Serious Issue

The serious issue that is surfaced by the email is Language Mapping. We cannot (and must not) try to map languages to each other. What is true for Java design is emphatically not true for Python design. And it doesn't apply to assembly languages, FORTRAN, FORTH, or COBOL.

Languages are different.

There. I said it.

If there was an underlying "universal deep structure" behind all programming languages, the surface features would be merely syntax, and we'd have automated translation among languages. The universal deep structure (the underlying Turing Machine that does computations) appears to be too abstract to map well among programming languages. Hence the lack of translators.

When switching among languages, it's important to leave all baggage behind.

When moving from Java < 8 to Java >= 8<8 java="" to=""> (i.e., non-functional Java to more functional Java) we can't trivially map all design patterns among the language features. It's a new language with new features that happens to be compatible with the old language.

Attempting to trivially map concepts between non-functional (or strictly-OO Java) and more functional Java leads to dumb conclusions. Like the GoF patterns are dying. Or the GoF patterns represent damage or something else equally goofy.

The language changes lead to design pattern changes.

Language change doesn't deserve an gleeful/anguished blog post celebrating/lamenting the differences. It's a consequence of learning a new language, or new features of an existing language.

Please avoid mapping languages to each other.

On Waiting to Write "Serious Code"

Someone told me they weren't yet ready to write "serious code." They needed to spend more time doing something that's not coding.

I'm unclear on what they were doing. It appears they have some barriers that I can't see.

They had sample data. They had a problem statement. They had an existing solution that was not very good. I couldn't see any reason for waiting. Indeed, I can't figure out what "serious" code is. Does that mean there's frivolous code?

Because there was a previous solution, they had a minimum viable product already defined: it has to do what the previous version did, only be better in some way. One could trivially transform the previous product into unit test cases and an acceptance test case. Few things could be more amenable to coding than having test cases.

Since everything necessary seemed to be in place, I had a complete brain cramp when they mentioned they weren't yet ready to write "serious" code. "Serious?" Seriously?

It appears that this developer suffers from a bad case of Fear of Code™. I know some common sources of this fear.

1. Waterfall Project Experience (WPE™.) Old people (like me,) who started in Waterfall World, were told that we had to produce mountains of design before we produced any code. No one knew why in any precise way. Indeed, there's ample evidence that too much design is simply a way to introduce noise into the process. In spite of real questions, some folks think that you can write a design so detailed that a coder can just type in the code from the design. (This level of design is isomorphic to code; to avoid ambiguity it must be written as code.)
2. Relational Database Hegemony (RDH™.) Folks (like me,) who were DBA's, know that databases require a lot of design and a lot of review before they can be created. Writing stored procedures requires even more design and review time. You don't just slap an SP out there. It might be "bad" or "create problems." Also, when you insist on DBA's writing application code, it takes super-detailed, code-level design details. In effect, you must write the code for the DBA to write your code back to you. 
3. One and Done (OAD™.) Some people like to feel that they can write code once and it can be a thing of beauty and a joy forever. The idea of a rewrite is anathema to these people. While this is obviously silly, people still like the conceit that they can produce some prototype code that will be a proper part of every future release forever and always. It's not possible to make all of the decisions the first time regarding adoption and scaling and user preferences. Your prototype code will get replaced eventually: get over it. Write the prototype, get funding, move forward. Don't dither trying to make a bunch of future-oriented decisions based on a future you cannot actually foresee. You can't "future-proof" your code.
4. Learnings are Expensive (LAE™.) You can find people that think that the sequence of (spike, POC, version 0, version 1) is too expensive. They are sure that learning is a project drag, since no "tangible" results are created by learning. This means that they don't value intellectual property or knowledge work, either; an attitude is actually destructive to the organization. Knowledge is everything: software captures knowledge: a spike followed by a POC followed by version zero will arrive on the scene more quickly than any alternative strategy. Don't waste time trying to write version 1 from a position of ignorance.
5. Tools are Expensive (TAE™.) Some people feel that -- since tools are expensive -- they should be used rarely. Back in the olden days, when a compiler took many minutes to produce an error report, you had to be sure the code was good. (I'm old enough that I remember when compiles took hours. Really.) Those days are gone. Most compilers today work at the "speed of light" -- if they were any faster, you couldn't tell, because you can't click any faster. For dynamic languages, like Python, the speed with which code can be emitted makes all tool considerations quaint and silly.
6. Diagram it to Death (DTD™.) Rather than write code, some folks would rather talk about writing code. To them, email, powerpoint, and whiteboard are cheaper than coding. This is a false economy. Nothing is saved by avoiding code. Time is wasted drawing diagrams of things at a level of detail that mirrors the code. Pictures aren't bad in general. Detailed pictures are simply a stalling tactic.

I find it frustrating when people search for excuses to avoid simply creating code. While I see a number of sources, there are many counter-arguments available. 

1. Waterfall is dead. Make something minimal that works for this sprint. Call it a "spike" if that makes you happier. Clean it up in the next sprint. Create value early. Expand on the features later.
2. Databases are free now. SQLite and similar products mean that we can prototype a database without waiting around for DBA's to give us permission to make progress. Build the database now, get something that works. Rework the database as your understanding of the problem matures. Rework the database as the problem itself matures and morphs. Nothing is static; the universe is expanding; do something now.
3. No code lasts forever. Waiting around to create some kind of perfect value one time only is perfect silliness. Create value early and often. Discarding code means you're making progress. If you think it's important, write "draft" on every electronic document which might get changed. (Hint: version numbers are smarter than putting "draft" everywhere.)
4. A spike and code happens more quickly than code. It's a matter of technical risk: unfinished work is an "exposure" -- an unrealized investment. Failing soon is better than researching extensively in an effort prevent a failure that could have been found quickly.
5. Use a dynamic language and avoid all overheads.
6. Keep the diagrams high-level. Code is the only way to meaningfully capture details. Code endures better than some out-of-date Visio file that's in Sharepoint completely disconnected from GitHub.

It's imperative to break down the roadblocks. All "pre-coding" activities are little more than emotional props: knock them down and start coding.

Class Design Strategies -- analysis vs. synthesis

The conventional wisdom on class design is to model real-world things. This is the Domain-Driven Design Approach. It's what we used to teach as Rumbaugh's OMT (prior to the Three Amigos creating UML.)

The idea is simple: Look at the real world objects. Describe them.

Classes will have attributes and behaviors. They will have relationships. Rumbaugh was very careful about keeping object and class separate. A class had associations, and object had links. The association was the abstraction, the link was a concrete implementation. A class offered an operation, an object provided a method as an implementation.

As powerful as this is, I'm not sure it's the final word.

The only problem it has is that people often get confused by "real world objects." I've seen a number of places where folks completely fail to distinguish Enterprise IT implementation choices from actual things that reflect actual objects in the actual world.

Users and Credentials, for example. Users are real human beings. You find them in the hallways, standing around. They take up space in conference rooms. Credentials are a peculiar security-focused way to summarize a person as something public (username) and something private (password.) You don't find a stack of credentials tying up a room at the end of the hallway. Indeed, you can't physically stack credentials. While something a user knows is important, it isn't the entirety of a User. The attributes and behaviors of credentials aren't a good model for a User. But you still have this argument periodically when developing a class model or a noSQL database document model.

I'd like to emphasize that this is -- as far as I can tell -- the only problem with domain driven modeling. Some people don't see the domain very clearly because they tend to stick to a technology-driven world view.

However. That doesn't mean that drawing on the white board is the only way to discover the domain.

Building Classes from Functions

As a heretical alternative, allow me to suggest an alternative to the whiteboard.

Once upon a time, the whiteboard was the only way to do object modeling. The successor to the whiteboard (I use Argo UML as well as Pocketworks yuml) is a diagramming tool that -- ideally -- helps you understand the domain before committing to the cost and complexity of writing code.

Wait a second, though.

The "cost and complexity of writing code"? Java programmers know what I mean. If you don't have your classes understood, you should not start slapping code together.

Python programmers have no idea what "cost and complexity of writing code" means. They slap classes together faster than I can draw the damn pictures on Argo.

Indeed, the pictures can become a kind of burden. The picture shows "x.X", therefore, the module must include "x.X". Even though there might be a better way using classes in separate modules "a.Y" and "b.Z". But changing the cluster of pictures that comprises a fairly complete UML diagram isn't easy.

[Clearly, this depends on how much you tried to show. If your diagrams are really spare, refactoring is no problem. If you include parts of the object model in the component diagram or activity diagram, you're in trouble.]

This leads to an alternative to the whiteboard. And the diagramming tool.

Code. [Cue Orchestral Hit: Ta-daaa!]

Yes. Code. [Cue Orchestral Hit: Ta-ta-ta-daaa!]

When you can slap together a spike solution in Python you have a sensible alternative to the whiteboard.

You can build some classes, write some demonstration code to show how they work together. Don't like it? Start again from another base of classes. You can do this as a Mob Programming exercise. It fits somewhere between grooming a story and finishing an MVP release. Indeed, it may be a good way to do specific, concrete grooming.

In some cases, though, you can't build classes. You don't really know (or can't agree) on what the real world things are.

Rather than debate, shift the focus. Just write functions.

In Python, this is easy, since functions are first-class inhabitants of the programming model. In Java, this isn't easy at all. Functions aren't proper things; they must be part of a class; and you can't agree on what the classes are; the Java stalemate. [Yes, Java 8 introduces free-standing functions.]

How This Works In Practice

In many cases, it makes sense to punt on the "big picture." You're not really sure what you even have.  Yes, you know you have eight individual CSV files that reflect events that happened somewhere in cyberspace. (Let's just say they were the output from stored procedure triggers; the only record of changes made to crucial data.)

You can wring your hands over the eight-table join required to reconstruct the pre-change and post-change views of the objects. You can wring your hands over the way it's really three (or is it four?) different navigation paths from I to II to IV to (V to VI to VII) union I to II to IV to (V to VI to VII) union I to III to IV to oh my god I'm so confused.

Or.

You can get the sample data.  You can read it using the CSV module.

DictReader can awkward. It can be fixed, however. If your column titles are legal Python variables, you can use this to create a namespace reader from a DictReader. This allows you to say row.ATTRIBUTE instead of row['ATTRIBUTE'].

def nsreader(dictreader):
    return (SimpleNamespace(**row) for row in dictreader)

We can then turn to working out the various join algorithms on real data. Each step builds objects based on types.SimpleNamespace.

You start with simplest possible join algorithm: load everything into dictionaries based on their keys.

I_map = { row.KEY: row for row in nsreader(table_I_dict_reader) }
II_map = { row.SOMETHING: row for row in nsreader(table_II_dict_reader) }

Once you have sample data in memory, you can figure out what the actual, working relationships are. You can tinker with navigation paths through the tangled mess of tables. You can explore that data. You can do data profiling to find out how many misses there are.

If the tables are smallish (10,000 rows each) it all fits in memory nicely. No need for fancy database connections and no need to reason out join algorithms that don't tie up too much memory. You're not writing a database server. You're writing an application.

Look For Common Features

The design issue for classes is locating common features: common attributes or common methods. We often start down the road of common attributes. Because. Well... it seems logical.

Focus on attributes is a bias.

Classification of objects isn't based mostly on attributes. It's not 50-50 objects vs. attributes.

We tend to focus on attributes -- I think -- out of habit. Data structures mean "common data", right? Databases include tables of commonly-structured data.

But this isn't a requirement -- nor is it even important. It's just a habit.

We can conceive of a class hierarchy based around common behavior, too. This may require a very flexible collection of attributes. On the other hand, there's no a priori reason not to define classes based on their behavior.

That's why the idea of building functions first doesn't seem too far-fetched.

First, we can build working functions.  We can have test cases and everything.

Then we can look for commonality. We can refactor into classes. We can start with a Flyweight design pattern. As common attribute emerge, we can refactor to store more state in the class, and less state somewhere else. The API changes while we do this.

Then we examine it for the "is this a thing" criteria. Last, not first. We may need to make a few more tweaks to reflect the thing we discovered scattered around the functions. The thing may be a checklist or a recipe or a procedure: something active instead of simply stateful.

This tends to make RESTful web services a bit of a head scratcher. If we have an active thing, what is the state that we can represent and transfer? The state may be very small; the active agency may be quite sophisticated. This shouldn't be too baffling, but it can be confusing when the GET request response is either 200 or 403/409: OK or Forbidden/Conflict. Or there are multiple shades of 200: 200 OK with a body that indicates success, vs. 200 OK with a body that indicates something more needs to be done, vs. warnings, vs. exceptions, vs. other nuanced results.

Summary -- tl;dr

I think there's a place for code-first design. Build something to explore the space and learn about the problem domain. Refactor. Or Delete and Start Again. In modern languages (i.e., Python) code is cheap. Whiteboard design may not actually save effort.

I think there's a place for building functions and refactoring them into classes. I think the Java pre-8 "All Classes or Burn In Hell" approach is misleading. Functional programming and languages like Python show that functions should be a first-class part of programming.

I think there's too much emphasis on stateful objects. The DDD warnings about "anemic" classes seems to come from a habitual over-emphasis on state and an under-emphsis on operations. I think that active classes (as much as they push the REST envelope) might be a good thing.

Building Skills in Object-Oriented Design

New Kindle Edition of Building Skills in Object-Oriented Design.

It seems to work okay in my Kindle Readers.

I'm not sure it's really formatted completely appropriately. I'm not a book designer. But before I fuss around with font sizes, I think I need to spend some time on several more valuable aspects of a rewrite:

1. Updating the text and revising for Python 3.
2. Removing the (complex) parallels between Python and Java. The Java edition can be forked as a separate text. 
3. Reducing some of the up-front sermonizing and other non-coding nonsense.
4. Moving the unit testing and other "fit-and-finish" considerations forward.
5. Looking more closely at the Sphinx epub features and how those work (or don't work) with the KindleGen application which transforms .html to .mobi. This is the last step of technical production, once the content is right.

I have two other titles to finish for Packt publishing.

Maybe I should pitch this to Packt, since there seems to be interest in the topic? A skilled technical editor from Packt and some reviewers are likely to improve the quality.

The question, though, will be how to fit this approach to programming into their product offerings? Since I have two other titles to finish for them, perhaps I'll just set this aside for now.

Python Package Design, Refactoring and the Stingray Reader Project

We'll be digging into Mastering Object-Oriented Python. Chapter 17, specifically.

We'll also be looking at a big refactoring of the Stingray Schema-Based File Reader.

We can identify three species of packages.

One common design is a Simple Package. A directory with an empty __init__.py file. This package name becomes a qualifier for the internal module names. The package is simply a namespace for modules. We’ll use the package with something like this:

import package.module

Another common design is the Module-Package. This is a package which appears to be a module.  It will have a larger and more sophisticated __init__.py that is a effectively, a  module definition. There are two variations on this theme. Sometimes we'll use this during the early stages of development because we don't know if the package will get really big or stay small. If we start out with a package and all the code is in the __init__.py, we can refactor down to a module.

The more common use for a module-package is to have the __init__.py import objects or other modules from the package directory. Or, it can stand as a part of a larger design that includes the top-level module and the qualified sub-modules. We’ll use the package with something like this:

import package

or perhaps

from package import thing

The third common pattern is a package where the __init__.py selects among alternative implementations. The os module is a good example of this. We’ll use the package with something like this:

import package

Knowing that it did something roughly like the following for us.

import package.implementation as package

Refactoring Module to Package

The Stingray angle on this is the need to add iWork '13 numbers to the collection of spreadsheets which it can parse. The iWork '13 format is unique.

Previously, all of the spreadsheets fell into three families:

• CSV or CSV-like. Simple text.
• XLS. We relied on https://pypi.python.org/pypi/xlrd/0.9.3 to make sense of .XLS files.
• XML-based. We parsed the XML and located sheets, rows and cells.

iWork '13 uses Snappy compress and Protobuf Serialization. Without some documentation, the files would be incomprehensible.  Read this: See https://github.com/obriensp/iWorkFileFormat. Brilliant. 

The previous releases of Stingray had a single, large module to handle a variety of workbook formats. Folding iWork '13 into this module would have been lunacy. It was already large to the point of being painful to understand.

The original module will be transparently turned into Module-Package. The API (import stingray.workbook or from stingray.workbook import SomeClass) will remain the same.

However.

The implementation will involve a package with each workbook format as a separate module inside that package. At the top, the __init__.py will include code like the following.

    from stingray.workbook.csv import CSV_Workbook
    from stingray.workbook.xls import XLS_Workbook
    from stingray.workbook.xlsx import XLSX_Workbook
    from stingray.workbook.ods import ODS_Workbook
    from stingray.workbook.numbers_09 import Numbers09_Workbook
    from stingray.workbook.numbers_13 import Numbers13_Workbook
    from stingray.workbook.fixed import Fixed_Workbook

This has the advantage of allowing us to include additional parsing cruft in each module that's not part of the exposed API in the workbook package.

The Mastering Object-Oriented Python book has more details on this kind of design.

The SortedContainers Package for Python

See this: SortedContainers — sortedcontainers 0.6.0 documentation

Here's some text from the invitation.

You may find the the performance comparison and implementation details interesting because it doesn't use any sophisticated tree data structure or balancing algorithms. It's a great example of taking advantage of what processors are good at rather than what theory says should be fast.

The documentation is extensive. The implementation details are interesting. The claim of faster is supported nicely. I have two quibbles.

1. It actually does use a sophisticated tree data structure. A list of lists really is a kind of tree.
2. "rather than what theory says should be fast" doesn't make any sense to me at all. 

A claim that Computer Science theory isn't right bothers me. If theory says some algorithm is fast, there are only two possibilities: (1) theory is actually right and it really is fast and the demonstration was incomplete or (2) the theory is incomplete, and the implementation extends (or replaces) the old theory; the implementation is new theory.

It's never the case that theory is "wrong." That fails to understand the role of theory.

It's always the case that an implementation either confirms theory or extends theory with new results.

To me, this package demonstrates one of two things.

1. The theory was incomplete and this package is a new theory that replaces the old, wrong theory.
2. The theory was right and this package demonstrates that the theory was right by being a good, solid, usable implementation. 

I would suggest the second option here: this package shows the value of Python's list-of-lists as a high-performance technique for implementing sorted structures. It's not an example of "taking advantage of what processors are good at." This is an example of using Python properly to squeeze excellent performance out of the available structures.

The really important insight is this "The sorted container types are implemented based on a single observation: bisect.insort is fast, really fast."

This is a profound observation.  Read more here: http://www.grantjenks.com/docs/sortedcontainers/implementation.html

Mastering Object-Oriented Python

See http://www.packtpub.com/mastering-object-oriented-python/book

Coming soon.

This is relatively deep, under-the-hood stuff for folks who want to master the Python feature set.

Here's the overview of what you get:

• 0 Some Preliminaries 3 examples, 56 lines
• 1 The __init__() Method 55 examples, 351 lines
• 2 Integrating Seamlessly with Python: Basic Special Methods 92 examples, 558 lines
• 3 Attribute Access, Properties, and Descriptors 33 examples, 310 lines
• 4 The ABC's of Consistent Design 18 examples, 108 lines
• 5 Using Callables and Contexts 17 examples, 214 lines
• 6 Creating Containers and Collections 50 examples, 438 lines
• 7 Creating Numbers 12 examples, 232 lines
• 8 Decorators And Mixins – Cross Cutting Aspects 39 examples, 233 lines
• 9 Serializing and Saving: JSON, YAML, Pickle, CSV and XML 77 examples, 648 lines
• 10 Storing and Retrieving Objects via shelve 34 examples, 272 lines
• 11 Storing and Retrieving Objects via SQLite 45 examples, 410 lines
• 12 Transmitting and Sharing Objects 38 examples, 388 lines
• 13 Configuration Files and Persistence  59 examples, 490 lines
• 14 The Logging and Warning Modules 46 examples, 343 lines
• 15 Designing for Testability 38 examples, 393 lines
• 16 Coping With The Command Line  42 examples, 222 lines
• 17 Module and Package Design  31 examples, 93 lines
• 18 Quality and Documentation  42 examples, 269 lines
• Preface 3 examples, 12 lines
• Bonus Chapter 1 Archives and Directories  11 examples, 119 lines
• Bonus Chapter 2 Case Study: Document Analysis  39 examples, 308 lines

824 examples, 6467 lines

Yes. That's a lot of code. It's relentless.

Preconceived Notions, Perceptual Narrowing, The Einstellung Effect

Read this http://en.wikipedia.org/wiki/Einstellung_effect

Great article in Scientific American on this.

I didn't realize that sometimes I do spend time trying to defeat the Einstellung effect. Not a lot of time. But some time.

When confronted with gnarly design problems, I have the same bad habits as many other programmers. I reach for algorithms or data structures that I'm familiar with, even if they're not optimal. Sometimes I'll use algorithms that are not even appropriate to the problem domain.

However.

In working on a book on Advanced Object-Oriented Python, I realized that one habit I have is -- perhaps -- actually helpful.  It's this.

I can -- if I'm careful -- enumerate the alternatives. It's challenging to exhaustively enumerate design choices. It seems to help to have a list of things that clearly aren't optimal or aren't workable or aren't elegant. After pruning away the bad ideas, sometimes a good idea remains.

I'm not often good at this. Sometimes I dive in early, make choices, learn from my failures, and am forced to refactor.

The "enumeration" isn't literally every possibility. Sometimes, it's the types of possibilities or the strategies involved. Sometimes it's the patterns that the possibilities fulfill.

Example 1. When looking at Python data structures, the ABC's of Sequence, Mapping and Set provide a big-picture way to identify places to look. Once we've narrowed the field of view, we can look at kinds of sequences of kinds of mappings. We can also look at the generator expression alternative to a sequence object.

Example 2. There are often three design strategies: inheritance, composition (or wrapping) and invent-from-scratch. It's sometimes helpful to actually put together a technical spike of a subclass, a wrapper class and the outline of a de novo class definition. Bad ideas usually surface quickly when actual code is involved.

I thought I was being fussy. Or I was just stalling to avoid starting to write bad code too early. Or I was wasting time obsessing over performance issues.

No. I was preventing Einstellung. Avoiding Perceptual Narrowing.

Avoiding "Calling a problems nails because I'm wielding the hammer."

The Relational Database as Hammer

I feel obligated to note that the relational database often becomes the hammer and all problems are then reduced to RDBMS/SQL nails. No matter what the problem is.

One of the most amazing of these problems was an inquiry about "the top n rows query". It was the DBA's sense that getting the "top n rows" using some selection and ordering criteria was a really standard problem that everyone had confronted. The problem was so common there just had to be a standard, widely-adopted high-performance solution.

When getting the top 100 rows out of 40,000, there will be performance issues. The filtering and sorting (and any joins) will take time and DB resources. My question was "why?"

The answer was appalling. The database was being used as a message queue. The top 100 rows out of 40,000 was being doing to pick the next few items out of the queue for processing. The non-top-100 rows were merely lower priority items in the queue.

Wouldn't a proper message queue have been cheaper and simpler?'

Apparently not. Einstellung had set in. They had data. They had a database. What more is there?

The Visitor Design Pattern and Python

Epiphany.

In Python, with iterators, the Visitor design pattern is useless. And a strongly-ingrained habit. Which I'm trying to break.

Here's a common Visitor approach:

class Visitor:
    def __init__( self ): ...
    def visit( self, some_target_thing ): ...
    def all_done( self ): ...

v = Visitor()
for thing in some_iterator():
    v.visit(thing)
v.all_done()

If we refactor the for statement into the Visitor, then it's just a Command or something.

Here's the refactored Iterating Visitor:

class Command:
    def __init__( self ): ...
    def process_all( self, iterable ):
        for thing in iterable:
            self.visit( thing )
    def visit( self, thing ): ...
    def all_done( self ): ...

c=Command()
c.process_all( some_iterator() )
c.all_done()

Possible Objection

The one possible objection is this: "What if our data structure is so hellishly complex that we can't reduce it to a simple iterator?"

That's perfectly silly. Any hyper-complex algorithm to walk any hyper-complex data structure, no matter how hyper complex, can always be recast into a generator function which uses yield to iterate over the objects.

Better Design

Once we start down this road, we can generally simplify processing into a kind of Command that looks something like this.

class Command:
    def __init__( self ): ...
    def run( self ): 
        for thing in self.iterable:
            ....

c= Command()
c.iterable= some_iterator()
c.run()

I find that this interface is somewhat easier to deal with when composing large commands from individual small commands. It follows a Create-Configure-Run pattern that seems to work out well. I just wish I would start with this rather than start with a Visitor, refactor, and end up with this.

Third Time's the Charm: the version 3.0 phenomenon

Somewhere, I have a vague recollection of reading advice from someone (Bill Gates?) that it takes three versions to get things right. The context may have been a justification of the wild success of Windows 3.0.

Or, I could be just making it up.

But one thing I have noticed is that there's a definite bias toward looking at software three times.

I worked (briefly) with an agile project management group that suggested that everything will be released three times, called the "Good", "Better", "Best" releases.

• The good release passed the unit tests.
• The better release included any non-functional (performance, auditability, maintainability, etc.) improvements required.
• The best implementation possible.

Not everything required three releases. Simpler components can merge better and best. Some components simply start out in really, really good shape.

Teaching Moment

What I've also noticed is that the explanation of the component -- writing documentation, presenting to peers in a walkthrough -- leads to profound rethinking. 

May things may appear to be better or best in the sense above. Until we have to explain them. Then they're no longer "best" but merely "better" or perhaps even "good." 

A few minutes spent hand-waving through a design often points to things that aren't quite to easy to explain. A walkthrough is very beneficial to the person doing the presentation.

But, not too early.

When I made military software, we had Preliminary Design Reviews that were done before coding begins. The idea was to surround the difficult coding work with yet more process steps and yet more deliverable intermediate results. 

The intent was noble: if a walkthrough reveals so much, then do the walkthroughs early and often.

However. I'm beginning to think that early isn't ideal.

I think that the design walkthrough should be delayed until after minimally working code exists. Once there's code -- with automated unit tests -- then refactoring to meet non-functional quality factors (like performance) is easier and more likely to be successful.

Also, refactoring to make the software clear, simple, and elegant should probably wait until it works and has a complete suite of automated unit tests. 

Hacker Monthly

Check out this month's Hacker Monthly.

One of my Stackoverflow answers was reswizzled into a short article on class design.

That was gratifying.