Code Dojo and OO Design -- OO Design Dojo

Code Dojo, to an extent, includes a fair amount of OO Design.

I've been pondering ways to help folks who clearly have no design skills at all. I've read their code. It's appalling.

Toward that end, I looked at some of the Code Kata links: the CodeKata page, Mark Needham's blog posting on code-kata, Rizky Farhan's Collection of Software Projects, jp.hamilton's Code Kata Resources.

They asked for code samples to act as best practices. I suggested to our sales folks that code samples and simple code "best practices" were completely inadequate. They need serious remedial skill-building in programming.

What started to percolate was organizing a periodic "code dojo" meeting to help them build skills without the onerous "teaching" (or worse, "lecturing") mode. Teaching OO design to working programmers is generally hard. Many programmers seem to have a starting point that isn't based on the requirements or any kind of rational design. It appears that many programmers start with a pretty random boilerplate program.

Teaching Java to COBOL Programmers

I remember struggling with COBOL programmers. Back in '02 (before Code Dojo existed), I had no real way to educate folks except a lot of one-on-one conversations. I tried to schedule code walkthroughs, but the project manager didn't like the idea, and cancelled them.

I was allowed a quick overview of J2EE concepts and how the web side of our application was going to be assembled, but that was it.

Even covering basic J2EE servlet concepts become a FAIL because the legacy web framework was a JSP hack-around. It didn't work well, couldn't easily be explained (or used). But it was entrenched, and therefore, had priority in everyone's mind.

No matter how many times I tried to review basic OO concepts, and some design approaches, there were problems.

Everyone wanted to start from "the top", with a "main program" that "simply read and wrote files." COBOL concepts. Java File I/O has a subtle complexity with lots of nested constructors. No one likes to see that as a beginner. Also, file parsing is -- in reality -- fairly hard, but COBOL provides a handy optimization via a fixed format record layout and lots of implicit conversions.

We're writing servlets that query a database. There was no mapping to the COBOL concepts everyone wanted to start with. A few lectures and presentations aren't helpful. Had I but known about Code Dojo, I would have suggested that. It might have worked.

The "Getting Started" Problem

Some Stack Overflow questions on design are really questions about "getting started". These cause me to wonder how to help people who are sure they know the language and syntax, but can't seem to get started writing anything useful "from scratch" (or de novo.)

I've heard from people have have UML class diagrams and still claim they don't know what to do next. They can't -- for some reason -- get started.

I think this is related. They have a limited, fixed set of programming templates. Learning a new language does not fit their limited set of templates. Perhaps Code Dojo could help these folks gain a new set of templates.

No Brown M&M's -- A Brilliant Compliance Test

My son's a musician, and one of the standard jokes boils down to the phrase "no brown M&M's". They use it as a catch-all phrase for people being fussy to a level that's senseless.

Then BoingBoing pointed me to the Van Halen Brown M&M story in snopes.com.

The brown M&M's was actually a compliance test. If you read the contract rider, and complied with all the terms and conditions, you'd filter the brown M&M's off the buffet.

If you were not prepared for the technical requirements for the Van Halen show, you'd book them, skip reading the rider, hope it went off well, and -- generally -- fail to filter the M&M's.

Statements of Work

We write a lot of Statements of Work (SOW's) with lots of "assumptions". A common assumption is that deliverables will be approved within three days. We often write it, suspecting that the client can't actually take action in three days.

But we never really know until we start work. Once we're there, we discover that it takes them a month to get ready to spend five days reviewing a document and wondering what to do. Now we're five weeks behind the original schedule, and the customer will blame us for "springing" the 3-day decision window on them.

If we had a "No Brown M&M's" assumption, perhaps we'd have an earlier indication that things weren't going to work out well.

Use Cases

I'm also wondering if there needs to be a "Brown M&M" use case. Something egregious, but small. Something that should lead to client confusion. If they approve the scope of work, including the Brown M&M use case, we know they didn't really read or review the scope of work.

Perhaps there should be a "Brown M&M" in the architecture. Perhaps an irrelevant component that we insist must be downloaded and installed in the development environment. We simply check that it's there. If not -- well -- what else will be wrong?

The E. W. Dijkstra Archive (Update)

The E. W. Dijkstra Archive is a collection of over 1,000 manuscripts that EWD sent around during his career.

This Stack Overflow question ("explaining software development to management") had a really brilliant comment on one of the answers.

Analogies are always leaky, and you will end up with proposed solutions that solve the analogy, not your actual problem. Just explain the problem in simple terms without comparing it to anything physical. Read Edsger Dijkstra's famous 1036 and 854 for an insight into the horrors thinkign by analogy is inflicting upon us

Follow-up Reading

EWD 854, "The fruits of misunderstanding". "when faced with something new and unfamiliar we try to relate it to what we are familiar with". "a program is an abstract mechanism". "the mechanism being abstract, its production is subsumed in its design. In this respect a program is like a poem: you cannot write a poem without writing it. Yet people talk about programming as if it were a production process and measure 'programmer productivity' in terms of 'number of lines of code produced'."

Aha. Software production is subsumed in its design. There is no "production" other than design. We design at a high level. We design code. When we've designed the code, we're done. There is no further development effort.

EWD 1036, "On the cruelty of really teaching computing science". "From a bit to a few hundred megabytes, from a microsecond to a half an hour of computing confronts us with completely baffling ratio of 10⁹"

Love that cautionary note. Computer science forces us to confront layers of meaning that have a huge scope.

I'll have to work my way through the archive. It will probably take years to read through all the manuscripts.

Wait -- I've Got One Of Those

Back in the 70's, the undergrads at Syracuse University were given copies of EDW316. At the time, it was just a paper on the art of programming. It sat in a file drawer for decades. I unearthed it recently.

At the time, EDW316 looked like a course number. I assumed -- wrongly -- that it was notes from some other school of computer science. Turns out, it was one of those EDW missives. It had found it's way into the hands of the CS Faculty at SU. From there, a copy fell into my hands. Not until recently (30+ years later) did I realize exactly what it was a copy of.

It's densely-packed stuff. I think A Discipline of Programming is a little bit easier to work with. Further, Gries' The Science of Programming is easier still.

Wishful Thinking -- An Accident Waiting To Happen

Some assumptions are really hard to identify as "assumptions". Some assumptions are more "wishful thinking" than "assumption".

We process a lot of spreadsheets. As far as I'm concerned, the Spreadsheet User Interface (SUI) is a first-class part of any application. Users understand them, and you don't have to code as much.

We have a library that wraps XLRD, csv, zipfile and ElementTree XML parsing to read a wide variety of spreadsheet formats.

However, we were recently stabbed by an assumption. I had to spend over 40 hours restructuring our workbook library and application code.

Go With What You Know

The point of an Agile approach is to build high-value things first. In the olden days, we would have spent months (really) writing a sophisticated set of hypothetical use cases for the workbook library and then designing something that would cover all possible bases.

Rather than spend endless hours on the potential workbook features, I wrote what we needed to read the workbook files we actually had. We had a mixture of XLS, XLS in ZIP files, and CSV files. So we unified those with a fairly simple model of a "Row Source" that provided information on sheets, and provided the sequence of rows.

However, all those spreadsheets had a common feature. They were built by people with a strong IT background, people who -- even if they couldn't define "Normalization" -- knew what normalized data looked like. They provided everything as proper columns.

Recently, we got some data for a new customer pilot that was just enough different that it was a costly problem.

What Changed?

The change was the use of the sheet tab names to carry meaningful key information.

Every previous example either had sheets with names like "sheet1", "sheet2" and "sheet3", or the sheet name was something we could filter on.

This workbook had the time dimension coded in the sheet names, not a column of data on each sheet. Suddenly, the worksheet name was significant. And that's not all.

How Bad Can It Be?

The extensive breakage came from a bad design decision buried in the workbook library and all application layers that depend on the workbook libraries. Assuming that data was in columns -- instead of sheet names -- didn't create a big problem. Unwinding that assumption was an easy to fix.

What was bad was a design that permitted the various mappings to be independent of each other. The "operation" classes that stepped through rows were designed so that a simple list of independent mappings could be used to extract relevant columns from a row and process them. Each independent mapping created a Python object from columns.

It turns out that each mapping needed a context (with worksheet name). Also, it turns out that some mappings actually depend on other mappings.

When the mappings are picking up columns, having several mappings depending on a single column is easy. Having several mappings depend on the context, wasn't too bad. Having one mapping that parsed the sheet name, exposed our wishful thinking.

We needed to have mappings that depended on each other. When we map the sheet name to a Python object, we did parsing and database lookups. Other mappings now must be "aware" of this mapping so they don't redo the parsing and database lookups.

Lessons Learned

The trivial (and wrong) lesson learned could be "don't make so many assumptions". That's silly. We didn't casually make assumptions. We had example data; the sample data was biased and didn't show all conceivable permutations.

Another trivial (and wrong) lesson could be "document all your assumptions". That's silly, too. We did document them. That doesn't make the breakage significantly easier to fix.

The real lesson is to avoid wishful thinking . We'd tried too hard to make all of the mappings into independent objects. The phrase "shared nothing" is our mantra. While shared nothing gave us a very composable design, it wasn't actually correct.

Object Models and Relational Joins -- Endless Confusion

Check out this list of questions from Stack Overflow: [Django] join.

These are all folks trying to do joins or outer joins even though they have objects fetched through the ORM.

How does this confusion arise? Easy. Folks work with SQL as if the relational world-view is Important and Universal. It isn't. SQL isn't even a programming language, per se.

Here's the important thing for Django developers to know: SQL is a Hack; Leave it Behind.

The bad news is that all those years spent mastering the ins and outs of the SELECT statement doesn't have as much enduring value as I'd hoped it would have. [Yes, I was a DBA in Ingres and Oracle. I know my SQL.]

The good news is that Object Navigation replaces much of the hideousness of SQL. To an extent. Let's look at some cases.

Joins in General

SQL SELECT statements are an algebraic specification of a result set. The database is free to use any algorithm to build the required set.

SQL imposes the Join hack because SQL is a completely consistent set algebra system. A simple SELECT returns a row-column set of data. A join between tables has to construct a fake row-column set so that everything is consistent.

A Join is nothing more than navigation from an object to associated objects. In OO world, this is simply object containment; the navigation is simply the name of a related object. Nothing more.

Master-Detail (1:m) Joins

A master-detail join in SQL works with a foreign key reference on the children.

In Django, this has to be declared in a SQL-friendly way so that the ORM will work.

class Master( models.Model ):

class Detail( models.Model ):
   master= models.ForeignKey( Master )

The "Join" query is simply this. The "detail_set" name is deduced by Django from the class that contains the foreign key.

for m in Master.objects.filter():
   process m
       for d in m.detail_set.all():
           process d

"But wait!" the SQL purist cries, "isn't that inefficient?" The answer is "rarely". It's possible that the RDBMS, doing a "merge-join" algorithm to build the entire result set might be quicker than this.

As practical matter, however, the rest of the web transaction -- including the painfully slow download -- will dominate the timeline.

Association (m:m) Joins

An association in SQL requires an intermediate table to carry the combinations of foreign keys.

In Django, this has to be declared in a SQL-friendly way so that the ORM will work.

class This( models.Model ):

class That( models.Model ):
   these = models.ManyToManyField( This )

The navigation, however, is simply following the relationships. There's no complicated SQL join required.

for this in This.objects.filter():
   for that in this.that_set.all():
       process this and that

Here's the other side of the navigation.

for that in That.objects.filter():
  for this in that.these:
      process this and that

Outer Joins

An Outer Join is a "Join with Null for Missing Relationships". It's navigation with an if-statement or an exception clause.

for that in That.objects.filter():
   try:
       this = that.this_set.get()
   except This.DoesNotExist:
       this = None
   process this and that

There isn't any "join" in object-oriented programming. The ORM layer removes the need.

Python and UML

Searched for this the other day. Came up empty. Clearly, didn't search hard enough.

Automatically Generated Python Code from an UML diagram?

So far, there are three separate answers listing a total of five separate products.

I really want to annotate the UML with docstrings. If that was supported, it would be cool. Then my Sphinx documentation would include UML images pulled from the code itself.

Programming Language Popularity -- Update

I used to rely on the TIOBE Software Programming Community Index.

Today, I learned about the langpop site. The context was following an SD Times blog posting on Haskell. But I got distracted looking at language rankings and what it means to consulting companies.

[Until corrected, I was] particularly drawn to langpop's Amazon listing. It seems like everyone wants books on C, C++ and C#. All other languages are also-rans. Why? [I guessed] that it's because those languages are so (a) hard to work with and (b) are perceived as "old school" where print resources are more prevalent than on-line resources. [Turns out the rankings are screwed up. Dang.]

Marketplace

The langpop aggregate ranking is C, Java, C++, PHP, JavaScript, C# and Python. The TIOBE Community Index is Java, C, C++, PHP, VB, C#, Python.

Similar enough to confirm that a half-dozen languages dominate with two others fighting for position.

Years ago, one of our senior consultants made the case that we really have three fundamental tech-stacks into which our web services had to fit: VB/.Net, C#/.Net, and Java. Corporate IT doesn't demand a lot of PHP or Python from us.

Perhaps those are opportunities for growth. Or perhaps there isn't enough corporate IT demand for those frameworks.