Working Definitions of Complexity

Software developers get so used to their Culture of Complexity, they hardly notice it.

See Asshole-Driven Development for more thoughts on this. The comments add lots and lots of examples of dysfunctional development. Many of these are additional examples of complexity run amok.

My personal addition is Next Year's Dollars are Cheaper (NYDC): next year's dollars are less valuable than this year's dollars. So technical debt can be accrued without any long-term consequences. Dumb, bad design can be forced into production because maintenance is cheaper than getting something done by a fantasy-based deadline date. Never mind the fact that maintenance goes on -- effectively -- forever, and technical debt accrues real cost at an exponential rate. Complexity is free? Apparently so.

Recently I heard of the "DIMY" development. DIMY is Do It My Way. The specific war story was a PL/SQL stored procedure that was somehow "too complex" because all it did was call 7 other stored procedures. The business process had 7 steps; the parallelism between procedure and use case was an important part of the design. Yet, since some folks would prefer a Monolithic Stored Procedure (MSP), they saw fit to complain.

Asserting that a MSP is "less complex" is a mirror image our normal understanding of complexity creating cost. It's a mirror-image because the debits and credits are reversed. In the DIMY world, measurable complexity is valued as an asset and real simplicity is viewed as a cost.

Working Definitions

Based in the war story, we can identify several aspects of this working definition of complexity.

First, it appears that a monolithic piece of software (no matter how poorly it matches the use case) is "less complex" than a relatively simple piece of software that better matches the use case. So software that actually matches the use case is "complex".

It also appears that a simple count of stored procedure names is a measure of complexity. So any effort at doing any meaningful, "high-level" chunking of meaning is accused of creating complexity. So chunking is "complex".

[I am forced to agree that "more names" is a larger number; I can't agree that more names is more complex, because I find that chunking works. Note that refusing to engage in mental chunking is absurd. Claiming that multiple named stored procedures "obscures the details" is silly. Why stop at named procedures? Why not claim that PL/SQL -- by it's very nature -- obscures the details? Why not claim that an RDBMS obscures the details? Why not claim that the OS API's obscure details? Why not claim that compiled languages obscure the details? Chunking is essential.]

Finally, it appears that real measure of complexity, like Cyclomatic Complexity are irrelevant. So a monolith, with lots of loops and ifs is somehow less "complex" and more desirable.

Perfect Code

Clearly, then, for some folks, high quality code involves (1) no match against the use case, (2) a single name, and (3) no limit on the loops and if-statements. In order to achieve this, we need a very simple use case (real simplicity -- low cyclomatic complexity -- a sequence of decision-free steps) for which we can write an immense, possibly irrelevant pile of code.

What's wrong with the MSP?

Given a monolithic piece of code that doesn't match the use case sequence of steps, how could we construct unit tests? I don't see how. Since we can't decompose the problem into meaningful chunks, we can't test the thing in pieces. All we can do is write overall end-to-end functional tests. And hope.

Given this MSP, how would we debug problems? I don't see how. We'd be confronted with "it's broken" almost every time something went wrong. Pin-pointing the root cause seems like it would be impossible.

DIMY development -- and the associated complexity -- does mean one thing. It means job security: no one will ever be able to understand, maintain or adapt this software. Write once, maintain forever. If you're patient, you have a job for life. At some point, managers will realize it's too expensive to maintain and -- because you're the only expert -- you can rewrite it, continuing the cycle of complexity.

Ways to Complicate Use Case Analysis

I sat through a great use case analysis session recently.

"Great" because I saw lots of ways to derail a simple process. Eventually, we did identify a couple of actors and a couple of use cases. But it took hours and hours.

Bonus: this was the third go-round on these exact use cases.

Requirements Gathering

The first go-round of requirements gathering was a conference call. We produced some nice notes. Very good stuff.

The various whiteboarding tools available with Skype work pretty well. We could sketch stuff, and collect notes, and digest the conversation down to a tidy document.

The actual work went pretty well. Stuff got built. The principal users need -- of course -- to review the preliminary stuff. There's a first sprint to build stuff, followed by some chance to comment, followed by a second sprint to finalize.

Focus was lost two ways. One, the technical folks were derailed by other, more important projects. Customer pilots, reinstallation on a new host, and an unrelated project threw down trump cards. Second -- and more important -- the principal users simply could not find time or interest to review the preliminary stuff.

Trying Again

For no sensible reason, we had a second go-round of requirements gathering. The core problem is that the users simply won't take the time to try something out on their own. They're sales folks, without an actual customer in the room, they seem incapable of doing anything. This kind of world view takes some getting used to.

Instead of previewing what was available, they insisted on more requirements gathering. What we got was a random document that purported to describe what the actors might do. It was intended to repeat the initial phone call with more focus. Instead it had lots of "We'll need to talk about this" parenthetical comments.

In short, it was impossible for them to even set down a coherent idea on paper. All they could do was talk about it. There was no alternative to a conversation.

Round Three

In order to make some progress, our Adobe FLEX developers were brought in to create something a little snappier than the simplistic HTML interface we had been working on. We redid the entire requirements gathering for them.

The principle users -- sales folks -- did not want any of the previous requirements gathering results brought in. We had to have a "conversation", repeating the entire previous effort from the ground up.

And -- of course -- all of the previous dead-ends, bad ideas, logical impossibilities and business impossibilities had to be repeated yet again. Data that's never been available was still spoken of as "required". The conversation on why the data cannot possibly exist had to be rehashed.

Derailing

What's derailing the process is simple. The sales folks cannot work independently. Each interaction must be a hands-on, guided tour of the software in which the sales folks say random things that must be ignored.

At some point in the future, there's a remote possibility that someone will login on their own and actually run the demo that's been in place for months.

As developers, we have been remiss in not catering to their learning style. They cannot think without talking, and they cannot take action unless they're influencing someone. Asking them to test the demo site is unproductive because they simply wont. Asking them for "comments" can be troublesome because they're job is influence, not simply provide a simplified "ok/not ok" feedback on specific features of the implementation of their use case.

Broad Not Deep

Additionally, we have folks that keep trying to define the requirements in broad, sweeping terms. They're uncomfortable with an end-to-end use case. At each step in the interaction they want to define all the possible future courses of events and interactions and consequences of each alternative course.

Folks with the big-picture view have a hard time writing a use case that describes a task from beginning to the end. Even basic concepts like "Creating Business Value" can be elusive since the possibilities are limitless.

As developers, we were successful at focusing down on a complete use case. With some effort we got from the user's initial interaction to the final bit of business value.

Learning Styles -- The Astonishment Response

We're not really talking about "Learning Styles" as much as "Denial Styles". This is a list of responses to "Astonishment" I've seen.

We're not talking about the Kübler-Ross model of grief, although this is similar.

However, the response to astonishment isn't a "progression" toward acceptance. Some folks simply don't like to learn and are perfectly capable of arguing down the facts because they don't fit with assumptions and preconceived notions.

When faced with new information, some folks seem to have a consistent response to astonishment. Other folks seem to jump around among a few preferred responses. Additionally there are people who seem to prefer to escalate things to a crisis level because learning seems to require adrenaline.

Oh. Classic acceptance. Many folks start here; which is pleasant. It saves a lot of email traffic. When astonished, they assimilate the information without really fighting against it.

That Can't Be True. Classic denial. It's surprising how often this happens. Even when confronted with facts supplied by the learner themself.

Example. The DBA says stored procedures are a maintenance problem. You say, "Correct, perhaps they shouldn't be used to heavily".

The DBA asks "Why reduce dependence on stored procedures?" You say that, amongst other things, "they're a maintenance nightmare."

And the DBA says, "That can't be true; it's just a management problem." WTF -- Wasn't That Funny -- the DBA is going to deny their own facts in order to avoid learning something knew.

I Wasn't Told. This is a kind of grudging, negotiated acceptance. "What you say about bubble sort being inefficient may be true, but I wasn't told." Okay. You weren't told. Does that mean that I have to email all of Don Knuth to you so that you will have been "told"?

I'll have to see it. This is really just a basic denial wrapped in the terms of settlement. In short, the learner is saying, "I still disagree with all your facts." I'm not sure what "I'll have to see it" means when we have working implementations of something "new" or "different".

Example. A: "RESTful web services are simpler." B: "No." A: "No SOAP, no WSDL; seems simpler." B: "Perhaps, but I'll have to see it." See what? How can you "see" the absence of complex WSDL?

This project is out of control. This is a somewhere near grudging acceptance. It might also be a form of reneging or repudiation of acceptance. It's hard to say.

Example. Manager: "The Ontology has thousands of objects with dozens of properties and the SPARQL processing is slow." Architect. "Replace it with a relational database derived from the ontology." Manager: "Okay".

Four Weeks Later. Manager: "This Project is Out of Control."

Right. We're making a disruptive change to the architecture. What did you expect? Non-disruptive change? How is it change if it doesn't disrupt something?

Does Everyone Know This? This is a form of "I wasn't told". It's my favorite because it projects one's own knowledge-gathering onto a mysterious "everyone". I'm not sure why some folks say this. To me, it seems a pretty bold statement about the mental states of other folks on the team.

That's Non-Standard. More properly this should be "That's atypical" or "That's unconventional". This is another negotiated, grudging acceptance. But it's a pretty complex deal. The first part is to establish a convention. Sometimes a legacy usage needs to be elevated to "typical" or "conventional"; other times legacy usage already is conventional. The second part is to realize that the new thing is different from the convention. The third part -- which is subtle -- is to deprecate something new because it is unconventional.

Example. Architect: "You should use a HashMap for those dimensional conformance lookups." Programmer: "Not everyone understands those fancy collection classes, so we use primitive arrays." Architect: "That's amazingly slow. It's less code to build and lookup a HashMap, and it runs faster."

Programmer: "That would be non-standard". Architect: "There's no applicable ISO standard. Perhaps you mean 'unconventional'." Programmer: "Right, unconventional. And we can't change now because it would disrupt the established code base."

Architect: "It will be less code and run faster." Programmer: "I'll have to see it."

Doesn't That Contradict Something? This is best nit-picky form of denial ever. Step one is to analyze each word of the suggested change; in some cases, using the level of care appropriate to studying the Talmud. Step two is to locate something that could be construed as contradictory. The third step is to deprecate the new thing because it can be linked to something that can be seen as contradictory.

Architect: "Can we add some formal assert statements in the tricky actuarial scoring algorithm. It involves non-obvious assumptions about NULL's and ranges of values." Programmer: "No. That contradicts your earlier advice to unit test all those corner and edge cases."

Architect: "Contradict? Perhaps you mean it's redundant." Programmer: "No, it's clearly contradictory; one never needs both assertions and unit tests. You demanded unit tests, that means that assertions are a contradiction."

Summary

Other than patience, it's hard to provide any other advice on how to work through these things. Mostly, these are fact-free positions. In some cases, even facts don't help the learning process.

I think the only way to cope with a fundamental refusal to learn is to ask what it takes to convince them. In many cases, the answer amounts to "Do the entire implementation two ways and then micro-examine each nuance of performance, maintainability, adaptability and cost to the organization over a period of a decade before I'll consider your worthless opinion."

I remember once being asked -- seriously -- how I can possibly claim one implementation is higher performance than another. The question was asked as if "measurement" didn't apply to software performance. At the time, I couldn't figure out why "measurement" wasn't the obvious fact gathering technique. Now I realize that they were simply refusing to learn and didn't care about evidence; they simply didn't want to change to a more efficient implementation.

iPad Thoughts -- Fashion Accessory?

From a Blog that's inside a company's firewall, so this had to be heavily edited.

"The instant ON is a relief. The full page touch screen works just like on the iPhone - only better. Web pages look great.. Photographs and Movies are fabulous. The screen resolution is fantastic. Sharing pictures makes it clear that the photo album is history. Tough times for Kindle. Email - much better than on the Blackberry. The things we like on the PDAs are all more attractive - and more usable! Almost like on a laptop."

Also.

"I did not have an easy way to view Excel & Powerpoint. 3G is not available for another month. ... No Adobe Flash. For some, the one big 'flaw' will be the lack of a 'file system'."

Finally, emphasis mine.

"The iPad is not a big leap, it is just a step, a big iTouch. But this is the last step that brings a whole new vision home. While not quite ready for Enterprise deployment, it gives us time to get going. And this may be the Tablet that makes it acceptable for men to carry handbags"

Okay. Time to start shopping for a nice Timbuktu messenger bag.

Fancy Literate Programming

My bias is toward "printable" documents. I like the idea of an HTML document that is directly printable. I've used tools like Flying Saucer (and appropriate CSS) to guarantee printability.

I like using ReStructured Text to create HTML and LaTeX that match precisely.

When I think of Literate Programming, I'm biased toward print.

However, HTML has a lot of power. Taking off the blinkers for a moment, one can see that rich HTML and Javascript may be a really workable approach.

Take a look at the Literate Molly Module: another tool for Literate Programming. This is an example of using rich HTML as a vehicle for literate programming.

Getting Started Creating Web Pages

Got this question recently.

I’m looking for an HTML editor that fits into my price range (free of course). I don’t need to do anything fancy, just vanilla HTML to run on an Apache server ..., and maybe some PHP down the line. Can you recommend any open source or shareware software that would run on Windows?

What to do?

First, civilized folks don’t edit HTML any more. That’s so 1999.

You have a spectrum of choices if you want to try and edit HTML.

• General-purpose text editors. Good ones do HTML syntax coloring. This is the hardy, forge-through-the-forest way to go. Raw text editing. Like when we were kids. http://en.wikipedia.org/wiki/List_of_text_editors. In Windows world, I use Notepad++.
• HTML-specific editors. http://en.wikipedia.org/wiki/List_of_HTML_editors. Note that WYSIWYG HTML Editing is more trouble than you’d believe possible. It’s always fun for the first few months, but then you try to do something that confuses the GUI interface and you wind up with an entire paragraph in italics and can’t figure out why. Or you want to move a punctuation outside a link and discover that the editor just can’t figure out where the tag is supposed to fall and puts everything inside it. Most of us do not try to use WYSIWYG HTML editors because it slowly becomes annoying once you get beyond the trivial basics.
• IDE’s. To produce HTML sensibly, you have to also write .CSS style sheets, and you often have a number of related pages. Essentially, a “project”. An IDE is usually a better choice than an editor. All the good IDE’s are free: Eclipse, NetBeans and Komodo Edit. I use ActiveState Komodo Edit heavily.

While NetBeans or Komodo Edit seems like overkill, it will (eventually) pay out as you move into developing more than static HTML pages.

Better Than HTML

Instead of creating HTML, many of us use “Lightweight Markup” which is much, much easier to cope with and simple tools to produce HTML from the markup. http://en.wikipedia.org/wiki/Lightweight_markup_language

I use reStructuredText instead of HTML. I use the DocUtils project, which has an rst2html.py tool that converts my RST into HTML for me. I also use rst2s5.py to create power-point-like presentations from my reStructuredText. If you want to see the power of RST, you can look at my personal site and my books: and. 100% RST. No manual HTML anywhere. I use Sphinx to create really complex docments like the books.

For some tasks, I use HTML templates and simple scripts to process data and create static HTML from the data. You’d be surprised how effective this is. Few things require up-to-the-second web applications. Many things can be done as nightly batch programs that emit static HTML and FTP the HTML up to the web page. No PHP.

Application Development

For web development, PHP is fine. It will – before long – create holes in your head because it’s so badly thought out. But for getting started, it’s fun. Real companies (like Google) don’t waste their time with it because of the numerous problems PHP causes.

“Problems?” you say. “What problems?”

PHP’s world view (HTML + code in a single package) is a terrible architecture. It’s horribly slow and leads to very muddled, inflexible designs. Everyone who tries to make a global change to their site's “look and feel” finds that PHP is inflexible and a regrettable platform. Even folks who simply want consistency among several different pages within their site find that the PHP world view is more headache than solution.

But it’s fun when you first build a site that works.

Frameworks

Generally, most folks find that a “framework” is absolutely essential for debugging, consistency and separating Content, Processing and Presentation. Even a simple Blog or Forum or Visitor Registration has separate Content, Processing and Presentation; PHP muddles these. A framework can help unmuddle them.

I use Django as framework and Python as programming language. Your hosting site may not support this, in which case you may be in trouble.

The Web Frameworks list on Wikipedia is good. Zend and CodeIgniter are highly recommended in places like StackOverflow. However, here's a good Django vs. PHP comparison: The Onion Uses Django, And Why It Matters To Us.

"Because

Cleaner. Much cleaner. Proper unit testing. Real reusable components across applications. An ORM rather than a just a series of functional query helpers...."

Summary

1. Get an IDE to edit your pages. Komodo Edit.
2. Consider using RST and tools instead of raw HTML. Installing Python + DocUtils and using rst2html.py is easier than learning HTML.
3. Try to avoid PHP’s numerous pitfalls; ideally by avoiding PHP. Use Django + Python and create a real application that clearly separates the content (data model) from processing (view functions) from presentation (HTML templates)

The Final Design Review

Today, we're reviewing the final and only code in the application. It's just that simple. We'll start with the data model.

CREATE TABLE STUFF(

COLUMN1 TEXT,

COLUMN2 TEXT,

COLUMN3 TEXT

);

As you can see from the enclosed table design, we have generalized the general triple-store to make it more general by removing all type restrictions on the RDF triple.

This can be used to construct any relations by the convention of having COLUMN1 be the database, schema and table, separated with dots. COLUMN2 is the column or attribute name. COLUMN3 is -- again, by convention only -- the target data type (integer, string, date, etc.) and the quoted value.

Any Questions?

The stunned silence is -- I'm sure -- due to the glittering brilliance of this design. Why this hasn't been more widely used, I have no idea.

The code is equally simple. We don't need to get into the details, but we have

"a function called do_stuff. ... not to worry because the method does more than one thing. ... don't worry because the method is overloaded many different ways."

Great. That ends this design review. With this kind of obvious simplification, we don't need any more design reviews, this one covers all possible bases.

Seriously

I've seen the first discussed seriously in the context of "why don't we simply...". The answer ("it doesn't perform well") is always a surprise to people who pitch the triple-store solution to a problem they've managed to get wrong.

I recently received an email bemoaning a real code review in which someone seriously tried to put a function into production named "do_stuff".

Equally bad is this "question" on Stack Overflow. "Using table-of-contents in code?". (It's not really a question, it's a blog post in the rhetorical form of a question.) The money quote: "I know that alternative to that kind of listing would be to split up big files into smaller classes/files, so that their class declaration would be self-explanatory enough.. but some complex tasks require a lot of code".

It appears that there are programmers who have done too little maintenance and adaptation.