TCP/IP Mysteries and user support

It's not clear, actually, if this involves a TCP/IP "Mystery". What it may involve is a simple lack of ability to communicate. Or something.

I got this question:

"Request help w/ finding a reference or you can post a blog about how you can you have 2 oracle servers or for that matter any 2 servers listening in on different sockets on the same unix box."

And this background. Such as it is.

"They are going to ask, how can this work? My lame explanation is that on a unix box you can have multiple servers listening in on different ports. I tried Googling around but couldn’t find anything good."

It appears that the DBA provided a TNSNAMES.ORA. And some desktop tool user was not happy with the TNSNAMES.ORA that was provided.

The saga is long and sad.

It amounts to something like this.

DBA: Here's the TNSNAMES.ORA.
User: That didn't work.
DBA: Yes, it did.
User: No, it didn't. You're an idiot.
DBA: I know you are but what am I?

And it devolved from there into a request to help use Google to locate a tutorial on TCP/IP address and port numbers.

I'll repeat that: a request to help use Google.

Apparently, the desktop user had done something in database A and couldn't find the results in database B. And didn't understand what was going on.

And this lead to the DBA asking me to help with Google to prove that the DBA's TNSNAMES.ORA worked.

How does that help the user?

Negative Requirements

Nothing is funnier than requirements which state things that are not to happen.

An actual quote.

... don't screw up cutting and pasting and the "/" vs "\" depending on unix / windows.

Why not list everything that's not supposed to happen?

• No fire in the server room.
• No anthrax outbreak.
• No Zombie apocalypse.

The list could go on. I wonder why it doesn't.

Automated Code Modernization: Don't Pave the Cowpaths

After talking about some experience with legacy modernization (or migration), I received information from Blue Phoenix about their approach to modernization.

Before talking about modernization, it's important to think about the following issue from two points of view.

Modernization can amount to nothing more than Paving the Cowpaths.

From a user viewpoint, "paving the cowpaths" means that the legacy usability issues have now been modernized without being fixed. The issues remain. A dumb business process is now implemented in a modern programming language. It's still a dumb business process. The modernization was strictly technical with no user-focused "value-add".

From a technical viewpoint, "paving the cowpaths" means that bad legacy design, bad legacy implementation and legacy platform quirks have now been modernized. A poorly-designed application in a legacy language has been modernized into a poorly-designed application in yet another language. Because of language differences, it may go from poorly-designed to really-poorly-designed.

The real underlying issue is how to avoid low-value modernization. How to avoid merely converting bad design and bad UX from one language to another.

Consider that it's possible to actually reduce the value of a legacy application through poorly-planned modernization. Converting quirks and bad design from one language to another will not magically make a legacy application "better". Converting quirky code to Java will merely canonize the quirks, obscuring the essential business value that was also encoded in the quirky legacy code.

Focus on Value

The fundamental modernization question is "Where's the Value?" Or, more specifically, "What part of this legacy is worth preserving?"

In some cases, it's not even completely clear what the legacy software really is. Old COBOL mainframe systems may contain hundreds (or thousands) of application programs, each of which does some very small thing.

While "Focus on Value" is essential, it's not clear how one achieves this. Here's a process I've used.

Step 1. Create a code and data inventory. 

This is essential for determine what parts of the legacy system have value. Blue Phoenix has "Legacy Indexing" for determine the current state of the application portfolio. Bravo. This is important.

I've done this analysis with Python. It's not difficult. Many organizations can provide a ZIP file with all of the legacy source and and all of the legacy JCL (Z/OS shell scripts). A few days of scanning can produce inventory summaries showing programs, files, inputs and outputs.

A suite of tools would probably be simpler than writing a JCL parser in Python

A large commercial operation will have all kinds of source checked into the repository. Some will be inexplicable. Some will have never been used. In some cases, there will be executable code that was not actually built from the source in the master source repository.

A recreational project (like HamCalc) reveals the same patterns of confusion as large multi-million dollar efforts. There are mystery programs which are probably never used; the code is available, but they don't appear in shell scripts or interactive menus. There are programs which have clear bugs and (apparently) never worked. There are programs with quirks; programs that work because of an undocumented "feature" of the language or platform.

Step 2. Capture the Data.

In most cases, the data is central: the legacy files or databases need to be preserved. The application code is often secondary. In most cases, the application code is almost worthless, and only the data matters. The application programs serve only as a definition of how to interpret and decode the data.

Blue Phoenix has Transition Bridge Services. Bravo. You'll be moving data from legacy to new (and the reverse, also.) We'll return to this "Build Bridges" below.

Regarding the data vs. application programming distinction, I need to repeat my observation: Legacy Code Is Largely Worthless. Some folks are married to legacy application code. The legacy code does stuff to the legacy files. It must be important, right?

"That's simple logic, you idiot," they say to me. "It's only logical that we need to preserve all the code to process all the data."

That's actually false. It's not simple logic. It's just wishful thinking.

When you actually read legacy code, you find that a significant fraction (something like 30%) is trivial recapitulation of SQL's "set" operations: SQL DML statements have an implied loop that operates on a set of data. Large amounts of legacy code merely recapitulates the implied loop. This is trivially true of legacy SQL applications with embedded SQL; explicit FETCH loops are very wordy. There's no sense in preserving this overhead if it can be avoided.

Programs which work with flat files always have long stretches of code that models SQL loops or Map-Reduce loops. There's no value in the loop management parts of these programs.

Another significant fraction is "utility" code that is not application-specific in any way. It's an application program that merely does a "CREATE TABLE XYZ(...) AS SELECT ....": a single line of SQL. There's no sense in preserving this through an "automated" tool, since it doesn't really do anything of value.

Also. The legacy code has usability issues. It doesn't precisely fit the business use cases. (Indeed, it probably hasn't fit the business use cases for decades.) Some parts of the legacy code base are more liability than asset and should be discarded in order to simplify, streamline or improve operations.

What's left?

The high value processing.

Step 3. Extract the Business Rules.

Once we've disposed of overheads, utility code, quirks, bad design, and wrong use cases, what's left are a the real brass tacks. A few lines of code here and there will decode a one-character flag or indicator and determine the processing. This code is of value.

Note that this code will be disappointingly small compared to the total inventory. It will often be widely scattered. Bad copy-and-paste programming will lead to exact copies as well as near-miss copies. It may be opaque.

IF FLAG-2 IS "B" THEN MOVE "R" TO FLAG-BC.

Seriously. What does this mean? This may turn out to be the secret behind paying bonus commissions to highly-valued sales associates. If this isn't preserved, the good folks will all quit en masse.

This is the "Business Rules" layer of a modern application design. These are the nuggets of high-value coding that we need to preserve.

These are things that must be redesigned when moving from the old database (or flat files) to the new database. These one character flag fields should not simply be preserved as a single character. They need to be understood.

The business rules should never be subject to automated translation. These bits of business-specific processing must always be reviewed by the users (or business owners) to be absolutely sure that it's (a) relevant and (b) has a complete suite of unit test cases.

The unique processing rules need to have modern, formal documentation. Minimally, the documentation must be in the form of unit test cases; English as a backup can be helpful.

Step 4. Build Bridges.

A modernization project is not a once-and-done operation.

I've been told that the IT department goal is to pick a long weekend, preferably a federal Monday holiday weekend (Labor Day is always popular), and do a massive one-time-only conversion on that weekend.

This is a terrible plan. It is doomed to failure.

A better plan is a phased coexistence. If a vendor (like Blue Phoenix) offers bridge services, then it's smarter and less risky to convert back and forth between legacy and new over and over again.

The policy is to convert early and convert often.

A good plan is the following.

1. Modernize some set of features in the legacy quagmire of code. This should be a simple rewrite from scratch using the legacy code as a specification and the legacy files (or database) as an interface.
2. Run in parallel to be sure the modern version works. Do frequent data conversions from old to new as part of this parallel test.
3. At some point, simply stop converting from old to new and start using the new because it passes all the tests. Often, the new will have additional features or remove old bugs, so the users will be clamoring for it.

For particularly large and gnarly systems, all features cannot be modernized at once. There will be features that have not yet been modernized. This means that some portion of new data will be converted back to the legacy for processing.

The feature sets are prioritized by value. What's most important to the users? As each feature set is modernized, the remaining bits become less and less valuable. As some point, you get to the situation where you have a portfolio of unconverted code but no missing features. Since there are no more desirable legacy features to convert, the remaining code is -- by definition -- worthless.

The unconverted code is a net cost savings.

Automated Translation

Note that there is very little emphasis on automated translation of legacy code. The important work is uncovering the data and the processing rules that make the data usable. The important tools are inventory tools and data bridging tools.

Language survey tools will be helpful. Tools to look for file operations. Tools to look for places where a particular field of a record is used.

Automated translation will tend to pave all the cowpaths: good, bad and indifferent. Once the good features are located, a manual rewrite is just as efficient as automated translation.

Automated translation cannot capture meaning, identify use cases or write unit test cases. Thoughtful manual analysis of meaning, usability and unit tests is how the value of legacy code and data is preserved.

How to Write Crummy Requirements

Here's an object lesson in bad requirements writing.

"Good" is defined as a nice simple and intuitive GUI interface. I would be able to just pick symbol from a pallette and put it somewhere and the software would automatically adjust the spacing.

Some problems.

1. Noise words.  Phrases like "'Good' is defined as" don't provide any meaning.  The word "just" and "automatically" are approximately useless.  Here is the two-step test for noise words.  1.  Remove the word and see if the meaning changed.  2.  Put the opposite and see if the meaning changed.  If you can't find a simple opposite, it's noise of some kind.  Often it's an empty tautology, but sometimes it's platitudinous buzzwords.
2. Untestable requirements.  "nice, simple and intuitive" are unqualified and possible untestable.  If it's untestable, then everything meets the criteria (or nothing does.)  Elegant.  State of the art.  Again, apply the reverse test:  try "horrid, complex and counter-intuitive" and see if you can find that component.  No?  Then it's untestable and has no place.
3. Silliness.  "GUI".  It's 2012.  What non-GUI interfaces are left?  Oh right.  The GNU/Linux command line.  Apply the reverse test: try "non-GUI" and see if you can even locate a product.  Can't find the opposite?  Don't waste time writing it down.

What's left?  

pick symbol from a palette ... the software would ... adjust the spacing.

That's it.  That's the requirement.  35 words that mean "Drag-n-Drop equation editing".

I have other issues with requirements this poorly done.  One of my standard complaints is that no one has actually talked to actual users about their actual use cases.  In this case, I happen to know that the user did provide input.

Which brings up another important suggestion.

• Don't listen to the users.

By that I mean "Don't passively listen to the users and blindly write down all the words they use.  They're often uninformed."  It's important to understand what they're talking about.  The best way to do this is to actually do their job briefly.  It's also important to provide demos, samples, mock-ups, prototypes or concrete examples.  It's 2012.  These things are inexpensive nowadays. 

In the olden days we used to carefully write down all the users words because it would take months to locate a module, negotiate a contract, take delivery, install, customize, integrate, configure and debug.  With that kind of overhead, all we could do was write down the words and hope we had a mutual understanding of the use case.  [That's a big reason for Agile methods, BTW:  writing down all the user's words and hoping just doesn't work.]

In 2012, you should be able to download, install and play with candidate modules in less time than it takes to write down all the user's words.  Often much less time.  In some cases, you can install something that works before you can get the users to schedule a meeting.

And that leads to another important suggestion.

• Don't fantasize.

Some "Drag-n-Drop" requirements are simple fantasies that ignore the underlying (and complex) semantic issues.  In this specific example, equations aren't random piles of mathematical symbols.  They're fairly complex and have an important semantic structure.  Dragging a ∑ or a √ from a palette will be unsatisfying because the symbol's semantics are essential to how it's placed in the final typeset equation.

I've worked recently with some folks that are starting to look at Hypervideo.  This is often unpleasantly difficult to write requirements around because it seems like simple graphic tools would be all that's required.  A lesson learned from Hypertext editors (even good ones like XXE) is that "WYSIWYG" doesn't apply to semantically rich markup.  There are nesting and association relationships that are no fun to attempt to show visually.  At some point, you just want to edit the XML and be done with it.

Math typesetting is has deep semantics.  LaTeX captures that semantic richness.  

It's often best to use something like LaTeXiT rather than waste time struggling with finding a Drag-n-Drop tool that has appropriate visual cues for the semantics.  The textual rules for LaTeX are simple and—most importantly—fit the mathematical meanings nicely.  It was invented by mathematicians for mathematicians.