Moved

Moved. See https://slott56.github.io. All new content goes to the new site. This is a legacy, and will likely be dropped five years after the last post in Jan 2023.

Thursday, March 13, 2014

The Visitor Design Pattern and Python

Epiphany.

In Python, with iterators, the Visitor design pattern is useless. And a strongly-ingrained habit. Which I'm trying to break.

Here's a common Visitor approach:

class Visitor:
    def __init__( self ): ...
    def visit( self, some_target_thing ): ...
    def all_done( self ): ...

v = Visitor()
for thing in some_iterator():
    v.visit(thing)
v.all_done()

If we refactor the for statement into the Visitor, then it's just a Command or something.

Here's the refactored Iterating Visitor:

class Command:
    def __init__( self ): ...
    def process_all( self, iterable ):
        for thing in iterable:
            self.visit( thing )
    def visit( self, thing ): ...
    def all_done( self ): ...

c=Command()
c.process_all( some_iterator() )
c.all_done()

Possible Objection

The one possible objection is this: "What if our data structure is so hellishly complex that we can't reduce it to a simple iterator?"

That's perfectly silly. Any hyper-complex algorithm to walk any hyper-complex data structure, no matter how hyper complex, can always be recast into a generator function which uses yield to iterate over the objects.

Better Design

Once we start down this road, we can generally simplify processing into a kind of Command that looks something like this.


class Command:
    def __init__( self ): ...
    def run( self ): 
        for thing in self.iterable:
            ....

c= Command()
c.iterable= some_iterator()
c.run()


I find that this interface is somewhat easier to deal with when composing large commands from individual small commands. It follows a Create-Configure-Run pattern that seems to work out well. I just wish I would start with this rather than start with a Visitor, refactor, and end up with this.

Thursday, March 6, 2014

Enterprise JavaScript -- Not the best idea

See this:

The article lists reasons why Enterprise JavaScript is a recipe for disaster. "Finally, there's legacy integration..." This is the point.

In particular, JavaScript needs to get the data from somewhere: a backend process. If we push business knowledge into the front-end, even if we're assiduous about code libraries and sharing, we still have to fight with the "Out-Of-Date JS Library" issue. Server-side business knowledge is inherently consistent and sharable.

The big reason JavaScript feels good is because it's seems productive. Java is complex. C++, C#, and Objective C are Very Complex.

And.

Backend programming doesn't allow you to see finished-looking stuff right away. When you're fooling around with JavaScript you feel like you're doing real work. You're moving data around on the HTML page, that's productivity, right?

A spreadsheet is just as productive as JavaScript presentation.  Almost exactly as productive. The underlying data and processing still originates somewhere else. That's where the real value lies. In the data. In the backend.

Thursday, February 27, 2014

Django and REST -- Tastypie vs. Django REST

Ouch. What a difficult question.

This isn't easy.

Comparing http://django-tastypie.readthedocs.org/en/latest/ with http://www.django-rest-framework.org is hard. They're both outstanding projects with a long history.

Trivial Follow-up Question 1: What are the requirements?

I happen to know, however, a bit about the context, so I suspect that the requirements center around super-flexible data access and numerous serialization formats.

History

My initial reaction is "Django-REST" of course. Mostly because I started with this several years ago and spent some time tweaking and adjusting my local copy. Our requirements involved adapting Django (and Django-REST) to use Forge Rock Open AM for authentication.

One feature that we didn't need was a sophisticated set of built-in transactions that covered the full REST spectrum of GET, PUT, POST and DELETE. 90% of our processing was GET with an occasional POST.

The other feature we didn't need was a trivial mapping from the Django object model. Our GET processing required view functions as mediation between our database models and the "published" model available through the RESTful API.

Since we needed so little, we hacked out the essential serialization feature set to support our GET operations.

Serialization

Considering the context of the initial question, I think that serialization is the deciding factor. Comparing the serialization features seems to indicate that the following summary may be relevant.

Tastypie serialization is simpler. The support for XML, YAML, JSON, etc., is simple.

Django-REST serialization+render is quite a bit more sophisticated and more flexible. The process is explicitly decomposed into serialization (for breaking down the model objects) and rendering in some external representation like XML, JSON, YAML, etc.

This two-step breakdown in Django-REST seems to make an open data project work out nicely. The developers should find it easier to integrate and publish data from a variety of sources.

Thursday, February 20, 2014

Third Time's the Charm: the version 3.0 phenomenon

Somewhere, I have a vague recollection of reading advice from someone (Bill Gates?) that it takes three versions to get things right. The context may have been a justification of the wild success of Windows 3.0.

Or, I could be just making it up.

But one thing I have noticed is that there's a definite bias toward looking at software three times.

I worked (briefly) with an agile project management group that suggested that everything will be released three times, called the "Good", "Better", "Best" releases.

  • The good release passed the unit tests.
  • The better release included any non-functional (performance, auditability, maintainability, etc.) improvements required.
  • The best implementation possible.
Not everything required three releases. Simpler components can merge better and best. Some components simply start out in really, really good shape.

Teaching Moment

What I've also noticed is that the explanation of the component -- writing documentation, presenting to peers in a walkthrough -- leads to profound rethinking. 

May things may appear to be better or best in the sense above. Until we have to explain them. Then they're no longer "best" but merely "better" or perhaps even "good." 

A few minutes spent hand-waving through a design often points to things that aren't quite to easy to explain. A walkthrough is very beneficial to the person doing the presentation.

But, not too early.

When I made military software, we had Preliminary Design Reviews that were done before coding begins. The idea was to surround the difficult coding work with yet more process steps and yet more deliverable intermediate results. 

The intent was noble: if a walkthrough reveals so much, then do the walkthroughs early and often.

However. I'm beginning to think that early isn't ideal.

I think that the design walkthrough should be delayed until after minimally working code exists. Once there's code -- with automated unit tests -- then refactoring to meet non-functional quality factors (like performance) is easier and more likely to be successful.

Also, refactoring to make the software clear, simple, and elegant should probably wait until it works and has a complete suite of automated unit tests. 

Thursday, February 13, 2014

TCP/IP Mysteries and user support

It's not clear, actually, if this involves a TCP/IP "Mystery". What it may involve is a simple lack of ability to communicate. Or something.

I got this question:
"Request help w/ finding a reference or you can post a blog about how you can you have 2 oracle servers or for that matter any 2 servers listening in on different sockets on the same unix box."
And this background. Such as it is.
"They are going to ask, how can this work? My lame explanation is that on a unix box you can have multiple servers listening in on different ports. I tried Googling around but couldn’t find anything good."
It appears that the DBA provided a TNSNAMES.ORA. And some desktop tool user was not happy with the TNSNAMES.ORA that was provided.

The saga is long and sad.

It amounts to something like this.

DBA: Here's the TNSNAMES.ORA.
User: That didn't work.
DBA: Yes, it did.
User: No, it didn't. You're an idiot.
DBA: I know you are but what am I?

And it devolved from there into a request to help use Google to locate a tutorial on TCP/IP address and port numbers.

I'll repeat that: a request to help use Google.

Apparently, the desktop user had done something in database A and couldn't find the results in database B. And didn't understand what was going on.

And this lead to the DBA asking me to help with Google to prove that the DBA's TNSNAMES.ORA worked.

How does that help the user?



Thursday, February 6, 2014

Hacker Monthly

Check out this month's Hacker Monthly.

One of my Stackoverflow answers was reswizzled into a short article on class design.

That was gratifying.

Thursday, January 23, 2014

Manual Testing -- Bad Idea

The question of testing came up recently. The description of the process sounded like manually "testing" some complex web application.

When trying to work out manual "testing", I find it necessary to use scare quotes. I'm not sure there's a place for "manual testing" of any software.

I know that some folks use Selenium to created automated test scripts for interactive applications. That may be a helpful technology. I prefer automated test scripts over manual testing. Consequently, I'm not too interested in helping out with testing -- other than perhaps coaching developers to write automated test scripts.


To continue this rant.

I've seen the suggestion that having a person do some manual "testing" will permit them to notice things that are "broken" but not a formal requirement in a test script. This seems to require some willing misuse of words. A person who's supposed to be noticing stuff isn't testing: they're exploring or demonstrating or thinking. They're not testing. Tests are -- by definition -- pass-fail. This is a very narrow definition: if there's no failure mode, it's not a test, it's something else. Lots of words are available, use any word except "test." Please.

Reading about exploratory "testing" leads to profound questions about the nature of failing an exploratory "test." When did the failure mode become a requirement? During the exploration? Not prior to the actual development? 

When an explorer finds a use case that was never previously part of a user story, then it's really an update to a user story. It's a new requirement; a requirement defined by a test which fails. It's a really high-quality requirement. More importantly, exploratory "testing" is clearly design. It's product ownership. 

This kind of exploration/thinking/playing/experiencing is valuable stuff. It needs to be done. But it's not testing.

Developers create the test scripts: unit tests of various kinds. Back-end tests. Front-end tests. Lots of testing. All automated. All.

Other experienced people -- e.g., a product owner -- can also play with the released software and create informed, insightful user stories and user story modifications that may lead to revisions to test cases. They're not testing. They're exploring. They're writing new requirements, updating user stories, and putting work into the backlog.

Putting work into the backlog

An exploratory "test" should not be allowed to gum up a release. To do that breaks the essential work cycle of picking a story with fixed boundaries and getting it to work.  Or picking a story with nebulous boundaries and grooming it to have fixed boundaries. Once you think you're doing exploratory "testing" on a release that's in progress, then the user stories no longer have a fixed boundary, and the idea of a fixed release cycle is damaged. It becomes impossible to make predictions, since the stories are no longer fixed.

For a startup development effort, the automated test scripts will grow in complexity very quickly. In many cases, the test scripts will grow considerably faster than the product code. This is good.

It's perfectly normal for a product owner to find behaviors that aren't being tested properly by the initial set of automated test scripts. This is good, too. As the product matures, the test scripts expand. The product owner should have increasing difficulty locating features which are untested.

Management Support

What I've found is that some developers object to writing test scripts. One possible reason is because the test scripts don't seem to be as much "fun" as playing with GUI development tools. 

I think the more important reason is that developers in larger organizations are not rewarded for software which is complete, but are rewarded for new features no matter what level of quality they achieve. This seems to happens when software development is mismanaged using a faulty schedules and a faulty idea of the rate of delivery of working software.

If the schedule -- not working features -- dominates management thinking, then time spent writing tests to show precisely how well a feature works is treated as waste. Managers will ask if a developer is just "gold plating" or "polishing a BB" or some other way of discrediting automated test case development.

If the features dominate the discussion, then test development should be the management focus. A new feature without a sufficiently robust suite of automated tests is just a technology spike, not something which can be released.

Manual "testing" and exploratory "testing" seem to allow managers to claim that they're testing without actually automating the tests. It appears that some managers feel that reproducible, automated test take longer and cost more than having someone play with the release to see if it appears to work.

But What About...

The most common complaint about automated GUI testing isn't a proper pass-fail test issue at all.

Folks will insist that somehow font choice, color, position or other net effects of CSS properties must be "tested." Generally, they seem to be conflating two related (but different) things.

1. Design. This is the position/color/font issue. These are design features of a GUI page or JavaScript window or HTML document. Design. The design may need to be reviewed by a person. But no testing makes sense here. The design isn't a "pass-fail" item. Someone may say it's ugly, but that's a far cry from not working. CSS design (especially for people like me who don't really understand it) sure feels like hacking out code. That doesn't mean the design gets tested.

2. Implementation. This is the "does every element use the correct CSS class or id?" question. This is automated testing. And it has nothing to do with looking at a page. It has everything to do with an automated test to be sure HTML tags are generated properly. It has nothing to do with the choice of packing algorithm in a widget, and everything to do with elements simply making the correct API calls to assure that they're properly packed.

For people like me who don't fully get CSS, lots of pages need to be reviewed to make sure the CSS "worked". But that's a design review. It's not a part of automated testing.

Here's the rule: Ugly and Not Working have nothing to do with each other. You can automate tests for "works" -- that's objective. You can't automated the test for "ugly" -- that's subjective.

Here's how some developers get confused.  A bug report that amounts to "ugly" is fixed by making a  change to a GUI element. This is a valid kind of bug-to-change. But how can the change have an automated test? You must have a person confirm that the GUI is no longer ugly. Right?

Wrong.

The confusion stems from conflating design (change to reduce the ugliness) and implementation (some API change to the offending element.) The design change isn't subject to automated testing. Indeed, it passed the unit tests before and after because it worked.

No design can have automated testing. We don't test algorithm design, either. We test algorithm implementations.

Compare it with class design vs. implementation. We don't check every possible aspect of a class to be sure it follows a design. We check some external-facing features. We don't retest the entire library, compiler, OS and toolset, do we? We presume that design is implemented more-or-less properly, and seek to confirm that the edges and corners work.

Compare it with database design vs. implementation. We don't check every bit on the database. We check that -- across the API -- the application uses the database properly.

There's no reason to test every pixel of an implementation if the design was reviewed and approved and the GUI elements use the design properly.