Not in HamCalc -- But perhaps should be

This is the kind of little program that would be in HamCalc. But doesn't appear to be.

Looking at the Airfoil web page, specifically, this one: http://airfoiltools.com/airfoil/details?airfoil=ls013-il.

The measurements are all given in fractions of the depth of the airfoil. So you have to scale them. I was working with what may turn out to be a 48" rudder for a boat based on this design. I'm waiting for some details from the engineer who really knows this stuff.

How do we turn these fractions into measurements for folks that work in feet and inches?

We can use a spreadsheet -- and I suspect many folks would be successful spreadsheeting this data. For some reason, that's not my first choice. I worry about accidental copy and paste errors or some other fat-finger blunder in a spreadsheet. With code, it's easy to reproduce the results from the source as needed.

Here's the raw data.

http://airfoiltools.com/airfoil/seligdatfile?airfoil=ls013-il

Part 1. Fractions.

from fractions import Fraction

class Improper(Fraction):

    def __str__( self ):

        whole= int(self)

        fract= self-whole

        if fract == 0: return '{0}'.format(whole)

        if whole == 0: return '{0}'.format(fract)

        return '{0} {1}'.format(whole,fract)

The idea is to be able to produce improper fractions like 47 ½" or 24" or ¾".  My Macintosh magically rewrites fractions into a better-looking Unicode sequence. I didn't include that feature in the above version of Improper. Mostly because in Courier, the generic fractions look kind of icky.

The raw data is readable as a kind of CSV file.

import csv

def get_data( source ):

    rdr= csv.reader( source, delimiter=' ', skipinitialspace=True )

    heading= next(rdr)

    print( heading )

    for row in rdr:

        yield float(row[0]), float(row[1])

That saves fooling around with parsing -- we get the profile numbers from the raw data as a pair of floats. 

Finally, the report.

def report( seligdatfile, depth, unit ):

    scale=16 #th of an inch

    for d, t in get_data( source ):

        d_in, t_in = d*depth, t*depth

        d_scale = Improper( int(d_in*scale), scale )

        t_scale = Improper( int(t_in*scale), scale )

        print( '{0:6.2f} {unit} {1:6.2f} {unit} | {2:>8s} {unit} {3:>8s} {unit}'.format(

            d_in, t_in, d_scale, t_scale, unit=unit) )

This gives us a pleasant-enough table of the measurements in decimal places and fractions.

We can use this for any of the variant airfoils available.  Here's the top-level script.

import urllib.request

with urllib.request.urlopen( "http://airfoiltools.com/airfoil/seligdatfile?airfoil=ls013-il" ) as source:

    seligdatfile= source.read().decode("ASCII")

import io

with io.StringIO( seligdatfile ) as source:

    report( source, depth=48, unit="in." )

I'm guessing the data files are ASCII encoded, not UTF-8. It doesn't appear to matter too much, and it's an easy change to make if they track down an airfoil data file what has a character that's not ASCII, but UTF-8.

HamCalc Quirk of the Week

The HamCalc program binhop is one of those little nuggets of beauty that might be helpful or might be useless. Or. Perhaps there's some useful stuff commingled with quirky stuff.

For folks in agriculture or manufacturing, I'm hoping that the calculation could be helpful. Although it's also likely that folks don't spend much time designing hoppered bins; they just buy something out of a catalog.

The binhop program is 151 lines long. Of those lines, 13 lines appear to be some orphaned code that doesn't belong there. They appear to be part of a hoppered bin design program that appears to have been split off into binvol.

It's easy to grep for all 46 instances of (?:GOTO|GOSUB|THEN)\s+\d\d\d0 to locate references to line numbers. We're only interested in line numbers from 1050 to 1170. Of course, there are none.

But. 

This is GW-Basic. Maybe there's a language or implementation quirk that makes this code somehow get executed? It seems highly doubtful. After all, it's only 151 lines of code. It's relatively easy to read and understand what's there. 

This kind of quirk demonstrates that an "automated" code conversion is rarely going to be helpful. An automated conversion of orphaned code means that the good stuff is diluted by the bad stuff.

I spent the most time with this fumbling through the alternative use cases to see what the program does. It doesn't do much. But it's important to be sure that this calculation isn't part of the use cases. It doesn't get a unit test.

More Oddness

This program has another cute quirks that is less brain-scrambling than orphaned code.


1020 :REM'
1030 IF C$="SIDE" THEN N=ATN(H/(F-D))*180/PI:RETURN
1040 IF C$="CENTER" THEN N=ATN(2*H/(F-D))*180/PI:RETURN

What if C$ is neither SIDE nor CENTER? There's no "otherwise" case expressed or implied. It just "falls through" to the next line of code.

In this case, the next line of code just happens to be the orphaned code starting on line 1050. This will do some calculations on variables which merely have their GW-Basic default values of 0. Since line 1170 ends with a RETURN, the program will appear to "work". It won't crash outright. It just executes a bunch of useless statements.

In other programs with similar structure, the following line of code is a RETURN (or a STOP or even an END in one case.)

Since C$ is only referenced in five lines of code, it's easy to be certain that it can only have one of the two values. Of the five references, one is a PRINT statement. Two are the above IF statements. The other two are assignment statements. 

Here are the two assignment statements:

650 Y$=INKEY$:IF Y$=""THEN 650

660 IF Y$="1"THEN GOSUB 1340:C$="SIDE":R=1:RETURN

670 IF Y$="2"THEN GOSUB 1340:C$="CENTER":R=0:RETURN

680 BEEP

690 GOTO 650

It's clear that C$ is restricted to a domain of just two values. The lack of a formal "otherwise" case in the 1020-1040 block of code is just a weird little quirk. 

Implementation

Here are the relevant calculation bits: https://github.com/slott56/HamCalc-2.1/blob/master/python/hamcalc/construction/binhop/__init__.py

By adding documentation and test cases, we've bloated the good bits up to 336 lines of code.

Here are the user interaction bits: https://github.com/slott56/HamCalc-2.1/blob/master/python/hamcalc/stdio/binhop.py

This reflects two decisions. It seemed sensible to (nearly) duplicate some similar blocks of code rather than try to use a single block of code peppered with IF-statements. IF-statements raise the cyclomatic complexity. Lines of code just make it longer. Also, we've combined all of binvol into binhop.

In the long run, I know nothing about the subject matter. Nor did I even do the minimal amount of online research to confirm the formulae in the program. I'm secretly hoping that someone who actually understands this subject area will revise and correct the code to make it more useful and complete.

HamCalc and Quirks

Careful study of the HamCalc shows a number of quirks. Some are funny, some are just examples of the need for unit test frameworks.

The Wikispaces for the modernization project is here: http://hamcalc.wikispaces.com/home

For example, the following line of code, in GW-Basic, will (usually) set Y to zero.

Y = O

Yes. That's the variable "O", not the number 0.

Why does this work? Why can we use "O" instead of 0?

Most programmers avoid using the variable "O", since it's hard to read.  GW-Basic provides default values of 0 for almost all variables. So, "Y=O" works as well as "Y=0" most of the time. The only time is doesn't work is if the program happens to have "O" used as a variable.

This is one of the examples where people start shouting that a compiled language is so obviously superior that the rest of us must be brain-damaged to use a dynamic language like Python.

This isn't a very compelling argument for the overhead of a compiler. It's a more compelling argument for avoiding languages with default values. Python, for example, would throw an exception if the variable "O" had no value.

This isn't common (so far, I've only found one example) but it's amusing.

Another amusing quirk is the occasional tangle of GOTO/GOSUB logic that defies analysis. There are several examples of GOSUB/RETURN logic that is totally circumvented by a GOTO that circumvents the return. This should (eventually) lead to some kind of stack overflow. But GW-Basic doesn't really handle recursion well, so it would probably just be ignored.

One of my favorites is this.


    730 FOR N=A TO T STEP B
    750 IF T/N=INT(T/N)THEN X=X+1:PN(X)=N:T=T/N:GOTO 730
    760 A=3:B=2
    770 NEXT N

What does the GOTO on line 750 mean? Since GW-Basic doesn't use a stack of any kind, it doesn't create recursion or stack overflow. It appears to "restart" the loop with a new value of T. I think.

HamCalc -- Preserving the Legacy

Wow.

The response to modernizing HamCalc was overwhelming.

Apparently there are a fair number of people who also think that HamCalc is a treasure to be preserved, improved and added-to.

If you're interested, start here: http://hamcalc.wikispaces.com/home

You can ask to be a member of this wiki, and I can add you. This gives us a place to share ideas.

I think the most important aspect of this project is to be welcoming and supportive of recreational and amateur programmers. The explanatory notes and documentation need to be clear, with proper references and footnotes. The final application programs need to be simple.

The point of HamCalc is not to build a large, complex application. The point of HamCalc is to be a repository of small, simple solutions that can be easily understood and repackaged. In my estimation, no one simply "uses" HamCalc. They read, extend, modify and make local copies of their private extensions to HamCalc.

Toward this end, we need to allow for alternate implementations. Two people should be able to carve out the same piece of code and provide different implementations that provide consistent results but reflect different optimizations for speed and resource use.

Here's the code repository. https://github.com/slott56/HamCalc-2.1

This will let folks who are interested get started. What I think might happen is that people will carve out pieces that interest them and convert bits of HamCalc. There are 449 programs, each one of which is a potential nugget of goodness.

Legacy Code Preservation: Language Incompatibility and Technology Evolution

It's important to address language or platform incompatibility as consequences of technology modernization. The reason why we have to do manual conversions of software is because of the language incompatibility issue. We must convert manually when no tool can do the conversion.

There are several layers to this.

• Platform Incompatibility. This means that the supporting libraries for the language are incompatible between versions. This is relatively rare; language libraries are almost always backward compatible. When they aren't, the problem can often be masked with a "shim" or little bit software to "wrap" the new libraries to make them work like the old libraries. Adding the shim is -- generally -- a terrible idea. Why preserve the old version's weird features and quirks? Why add the complexity (and bugs and quirks) of the shim?
• Support or Framework Incompatibility. A common "Support" incompatibility is a database; there are many other examples. SQL, for example, has a standardized core, but is not consistently implemented and vendor extensions are common. Any large framework will have compatibility issues among versions and platforms.
• OS Incompatibility. Most POSIX-compliant OS's (Linux, Mac OS X, etc.) are reasonably compatible. Windows throws a monkey-wrench into the works. In some cases, a language offers a library to make the programs in that language OS-agnostic. An OS-unique feature of an application is a disturbing thing to convert. Is the OS-unique feature an essential feature of the application? In some cases, the OS-unique feature stems from specialized drivers for media support (sound, images, video, etc.) This media compatibility issue leads to complex OS-agnostic support or leads to the use of third-party OS-agnostic libraries.
• Language Incompatibility. This is usually an absolute block to automated conversion. Languages are designed not to be compatible at a conceptual or semantic level. Automated translation from one programming language to another is difficult and in some cases essentially impossible without some kind of supremely sophisticated artificial intelligence effort. If languages were compatible at a conceptual level, we'd have universal translation among programming languages.

When we look at our case studies, we see the following:

• What's the Story? OS conversion; the language remained more-or-less the same.
• Are There Quirks? OS and Language conversion: Fortran to PL/1.
• What's the Cost? Language conversion: JOVIAL to Fortran.
• Paving the Cowpaths. Persistence framework conversion: flat files to RDBMS.
• Data Warehouse and Legacy Operations. This often involves OS, language and persistence conversion.
• The Bugs are the Features. This was a mental problem, not a technical one.
• Why Preserve An Abomination? OS, language and persistence conversion: Basic to Java.
• How Do We Manage This? OS, language and persistence conversion: COBOL to Java.
• Why Preserve the DSL? Language and persistence conversion: C to Java.

In the case where there's a language conversion, the effort simply becomes new development. The "conversion" or "modernization" concept is there merely to make managers feel that value is being preserved.
In the rare case where the language was not converted, deep questions about user stories vs. technical implementation needed to be asked and answered clearly and completely. When they were not asked (or answered) the conversion did not go well.

Automated Language Translation

The idea of automated language conversion is an "attractive nuisance". (http://en.wikipedia.org/wiki/Attractive_nuisance_doctrine). Not only is it generally impossible, it reduces or eliminates the value of the captured knowledge.

Assume you have some program P_1 in language L_1. It captures some knowledge, K, about the problem domain, and encodes that knowledge in a more-or-less readable and meaningful format.
We want to 'automagically' create a new program P_2 in language L_2. Since the two languages employ different concepts, different data structures, different programming paradigms, the conversion doesn't happen at a "high level". This is not a matter of changing the print statement to the print() function. This is a matter of "understanding" the program, P_1 and then creating a new program, P_2. that performs the "same" functions from the user's point of view.

Choice #1 is to create a very high-level technical "specification" that's language-independent. Then, a translator compiles that high-level specification into the new language. In essence, we've "decompiled" from P_1 to P_S and then compiled P_S to P_2, using an intermediate specification language, L_S. The high-level specification language L_S must contain both languages, L_1 and L_2, as features.
There are examples of elements of this. C++ is compiled to C. Eiffel is often compiled to C. We can think of C++ as a specification language that's translated to C.

Further, we know that "control structure" (IF-THEN-ELSE, WHILE, GOTO) can all be mapped to each other. There's an elegant graph-theoretic proof that a program which is a morass of GOTO's can be revised into IF-THEN-ELSE and WHILE loops. Clearly, then, the converse is possible.

While we can go from C++ to C, can we go from C to C++? At least superficially, yes. But that's only true because C++ is defined to be a superset of C. So that example is really poor. We'll ignore C++ as a higher-level language.

Let's look at Eiffel. We can go from Eiffel to C. Can we go from C to Eiffel? Not really. Eiffel lacks the GOTO, which C supports. Also, C has unconstrained pointer coercion (or casting) which Eiffel lacks. In order to "decompile" C to Eiffel, we'd need to "understand" the C programming and essentially rewrite it into a neutral version in Eiffel which could be then translated to another implementation language.

Making the problem worse, C has murky semantics for some constructs. a[i++]= i; for example, is poorly-defined and can do a wide variety of things.

Semantic Loss

Choice #1--to create a very high-level technical "specification"--can't be done automatically.

Choice #2 is to create a very low-level implementation of the program P_1 by compiling it into machine instructions (or JVM instructions or Python byte codes or Forth words). This low-level language is L_M. Given a program in L_M, we want to restructure those machine instructions into a new program, P_2, in the new language, L_2.

It's important to observe that the translation from P_1 to machine code L_M may involve some loss of semantic information. A machine-language "AND" instruction might be part of a P_1 logical "and" operation or part of a P_1 bit mask operation. The context and semantic background is lost.
Without the semantic information, P_2 may not reflect the original knowledge captured in P_1.
Note that this difficulty is the same as choice #1--creating a higher-level specification.

We can't easily "decompile" code into a summary or understanding or description. Indeed, for some languages, we're pretty sure we don't want to try to automatically decompile it. Some legacy C code is so obscure and riddled with potential confusion that it probably should be rewritten rather than decompiled.

Here's a concrete example from HamCalc.

700 A=2:B=1:T=P:X=0
730 FOR N=A TO T STEP B
750 IF T/N=INT(T/N)THEN X=X+1:PN(X)=N:T=T/N:GOTO 730
760 A=3:B=2
770 NEXT N

The point is to find prime factors of P, building the array PN with the X factors.

Note that line 750 executes a GOTO back to the FOR statement. What -- precisely -- does this mean? And how can be be automagically decompiled into a specification suitable for compilation into another language?

This, it turns out, is also an example of a place where HamCalc is not a repository of profoundly useful programming. See http://en.wikipedia.org/wiki/Integer_factorization for more sophisticated algorithms.

Knowledge Capture

It appears that knowledge capture requires thinking.

There's no automatic translation among programming languages, data structures or programming paradigms.

The only viable translation method is manual conversion:

1. Understand the source program.
2. Create unit test cases.
3. Develop a new program that passes the unit test cases.

Legacy Code Preservation: Some Patterns

After looking at this suite of examples, we can see some patterns emerging. There seem to be several operating principles.

1. The Data Matters. In many cases, the data is the only thing that actually matters. The legacy application knowledge may be obsolete, or so riddled with quirks as to be useless. The legacy knowledge may involve so much technical detail---no user story---that it's irrelevant when a newer, better technology is used.
2. User Stories Matter; Legacy Technology Doesn't Matter. It is essential to distinguish the legacy technology from the meaningful user stories. Once the two are teased apart, the technology can be replaced and the user stories preserved. A cool DSL may be helpful, and needs to be preserved, or may be a distraction from the real solution to the real problem.
3. Understanding the New Technology Is Central. Misusing the new technology simply creates another horrifying legacy.
4. Testing is Essential. Legacy code cannot be preserved without test cases. Any effort that doesn't include automated comparisons between legacy and converted is just new development.
5. Discarding is Acceptable. Unless the legacy code has a seriously brilliant and unique algorithm, most business applications are largely disposable. It may be less expensive to simply do new development using the legacy code as a kind of overly-detailed specification. Calling it "conversion" to make managers feel good about "preserving" an "asset" is acceptable. The project is the same as new development, only the words change to protect the egos of those not really involved.
6. Quirks are Painful. They might be bugs or they might be features. It will be difficult to tell.

How do these principles match against our various case study projects?

• What's the Story? The applications were technical, and could be discarded.
• Are There Quirks? Without a test case, we could not be sure of our conversion. So we accepted the quirks.
• What's the Cost? The application was technical, and could be discarded.
• Paving the Cowpaths. New Technology was misused, the result was a disaster.
• Data Warehouse and Legacy Operations. The legacy software encoding knowledge can be split haphazardly into database and application software buckets. The user stories matter. The technology doesn't matter.
• The Bugs are the Features. The user stories matter. If you can't articulate them, you're going to struggle doing your conversion.
• Why Preserve An Abomination? When the code is shabby and has bugs, you have to sort out the quirks that will be carried forward and the junk that will be discarded.
• How Do We Manage This? The user stories matter. The data matters. Focus on these two can help prioritize.
• Why Preserve the DSL? The user stories and test cases lead to a successful outcome. While the customer may feel like a conversion was being performed, it was really new development using legacy code as a specification.

With modern languages and tools, legacy code conversion is quite simple. The impediments are simply managerial in nature. No one wants to have a carefully maintained piece of software declared a liability and discarded. Everyone wants to think of it as an asset that will be carefully preserved.

Legacy Code Preservation: Why Preserve the DSL?

\A Domain-Specific Language (DSL) can provide some intellectual leverage. We can always write long and convoluted programs in a general-purpose programming language (like Python, Java or C). Sometimes it can make more sense to invent a new domain-specific language and implement the solution in that language.

Sometimes, even well-written, highly portable programming becomes a legacy. I once converted a large, well-written program from C to Java. The organization had no skills in C and didn't want to build these skills.

They wanted their legacy C program rewritten into Java and then extended to cover some additional features.

The timeframe for this exercise is the sometime after 2010. This is important because automated unit test and test driven development are common knowledge. We're not fighting an uphill battle just to figure out how to compare legacy with new.

In essence, we're going to be doing "Test Driven Reverse Engineering." Creating test cases, seeing what the legacy software does and then creating new software that does the same thing.
We're preserving exactly those features which we can encode as unit test cases.

The Code Base

In this case, there was an interesting wrinkle in the code base.

The application included a small Domain-Specific Language (DSL) that was used to define processing rules.

There were only a dozen or so rules. The idea was the various combinations of features could be included or excluded via this little DSL. The application included a simple parser that validated the DSL and configured the rest of the application to do the actual work.

The DSL itself is of no value. No one in the user organization knew it was there. The file hadn't been touched in years. It was just a configuration that could have been meaningfully built as source code in C.

The dozen or so rules are extremely important. But the syntax of the DSL was disposable.
It was relatively simple to create class definitions that -- in a limited way -- mirrored the DSL. The configuration could then be translated into first-class Java.

package com.whatever.app.config;
import com.whatever.app.defs;
class Configuration {
List theRules= new LinkedList();
Configuration() {
theRules.add(
        new Simple_Rule( this_condition, SomeOption, AnotherOption );
theRules.add(
        new Simple_Rule( that_condition, SomeOption );
theRules.add(
        new Exception_Rule( some_condition, some_exception );
}
}

Things that had been keywords in the old DSL became objects or classes full of static declarations that could be used like objects.

By making the configuration a separate module, and using a property file to name that module, alternate configurations could be created. As Java code in Java Syntax, validated by the Java compiler.

The Unit Tests

The bulk of the code was reasonably clear C programming. Reasonably. Not completely.

However, I still insisted on a pair of examples of each of the different transactions this program performed. These mostly paralleled the DSL configuration.

Having this suite of test cases made it easy to confirm that the essential features really had been preserved.

The user acceptance testing, however, revealed some failures that were not part of the unit test suite. Since TDD was new to this customer, there was some fumbling while they created new examples that I could turn into proper unit test cases.

The next round of acceptance testing revealed another few cases not covered by the examples they had supplied. By now, the users were in on the joke, and immediately supplied examples. And they also revised a existing examples to correct an error in their test cases.

What Was Preserved

Of the original C software, very little actually remained. The broad outline of processing was all.
The tiny details were defined by the DSL. This was entirely rewritten to be proper Java classes.

The C data structures where entirely replaced by Java classes.

All of the original SQL database access was replaced with an ORM layer.

Further, all of the testing was done with an entirely fresh set of unit tests.

The project was -- actually -- new development. There was no "conversion" going on. The customer felt good about doing a conversion and "preserving" an "asset". However, nothing much was actually preserved.

Legacy Code Preservation: How Do We Manage This?

At an insurance company, I encountered an application that had been in place for thirty years.
Classic flat-file, mainframe COBOL. And decades old.

It had never been replaced with a packaged solution. It had never been converted to a SQL database. It had never been rewritten in VB to run on a desktop.

What had happened is that it had grown and morphed organically. Pieces the original application it had been subsumed by other applications. Additional functionality had been grafted on.

After a few decades of staff turnover, no single person could summarize what the applications did. There was no executive overview. No pithy summary. No elevator pitch.

The company had, further, spent money to have consultants "reverse engineer" the COBOL. This meant that the consultants created narrative English-language versions of the COBOL code.

This reverse engineering replaced detailed, disorganized COBOL with detailed, disorganized English. No summaries were produced that could serve as an explanation of the actual valuable parts of the program.

The question of scope and duration was daunting. The conversion would take years to complete. the central question become "How to manage the conversion?"

The Goal

The goal was to preserve the valuable features while migrating the data out of flat files into a proper SQL database. The focus on the data was important.

The technical obstacle was the hellish complexity of the applications and their various shell scripts ("JCL" in the Z/OS mainframe world.)

One approach to overcoming the complexity is to break the overall collection of applications down into just those applications that write to any of the central "master" files. Other applications that read master files or do other processing are less important than those which update the master files.

The master files themselves are easy to identify. The JCL that references these files is easy to identify.

The programs run by those JCL scripts give us clusters of related functionality.

We want to rank the master files by business value. The one with the most valuable data is something we tackle first. The least valuable data we leave for last.

In some cases, we'll identify programs that work with relatively low-value data; programs which are not actually assets. They don't encode any new, useful knowledge. A wise manager can elect to remove them from the software inventory rather than convert them.

Since the conversion can't happen overnight, there needs to be a period of coexistence between the first conversion and the last. And this coexistence means that database tables will get extracted back to flat files so that legacy programs can continue to operate.

Another component of the plan for this conversion was the assembly of test cases. This is critical when refactoring code.

The idea here is to preserve selected files and run them through the application software to create repeatable test cases. One of the existing file-compare utilities can be used to validate the output.

Other Barriers

The human obstacle was triumphant here. People who had worked with this software for their entire career in IT would rather quit than help with the conversion.

Experienced mainframe people could not see how a "little" Linux processor could ever provide the amazing feature set and performance of their beloved mainframe. They made the case that CICS was higher performance and more scalable than any web-based application platform.

And that lead to an impasse where one camp refused to consider any migration except to COBOL and CICS. The other camp simply wanted to write a web application and be done with it.

The COBOL/CICS group were either confused on in denial of how quick and simple to can be to build default web applications around stable data models. In this case, the relational database version of the flat files would not be difficult to concoct. Once built, 80% of the application programming would be default add-change-delete transactions. The other 20% would be transactions that included the remnants of useful knowledge encoded in the legacy COBOL.

More time would be spent "studying the alternatives" than were required to build working prototypes.

Preservation

The real question is one of what needs to be preserved.

Clearly, the data is central to the business.

The larger question, then, is how much of the COBOL processing was really essential processing?

How much of the COBOL was technical workaround to implement things that are one-liners in SQL?
How much of the COBOL is workaround for other bugs in other applications? How many programs fix broken interface files? How many programs provide data quality inspections?

How much of the COBOL is actually unique? A substantial fraction of the legacy code was irrelevant because a package replaced it. Another substantial fraction implemented a "Customer Relationship Management" (CRM) application for which a package might have been a better choice than a software conversion.

How much of the legacy code contain quirks? How much code would we would have to understand and consider repairing because it actually contains a long-standing bug?

Perhaps the only thing of value was the data.

And perhaps the reason for the human obstacle was an realization that the cost to convert exceeded the value being preserved. It's difficult to have your life's work simply discarded.

Legacy Code Preservation: Why Preserve An Abomination?

By the early aughts (2001-2005) Visual Basic had gone from state of the art to a legacy application language. Code written in VB was being replaced with something more modern (generally Java.)

Having worked with COBOL and Fortran legacy programs, it's easy to describe this legacy VB code as an abomination. Several customers. Several applications. Not a simple size of one. Abomination.

The VB programming I've been involved in preserving has been uniformly shabby. It seems reflect an IT department that threw warm bodies at a problem until it appeared solved and did nothing more. No code reviews. No cleanup. Just random acts of maintenance.

I'm sure there's are many good VB programmers. But I haven't seen their work product yet.
And now, a customer is paying consultants like me to clean up their shabby VB and replace it with Java. Or a web site. Or both.

A Code Base

One particular example of abominable VB was a hodgepodge of copy-and-paste programming, GOTO's and other poorly-used features.

The application printed insurance-like summaries of benefits. In order to do this, it extracted a great deal of data from a database. It relied on a collection of stored procedures and an intimate connection with a massive "calculation" module that derived the actual benefits, which the VB application summarized and reported.

The new application architecture was designed to separate the database from the calculations. The printing of letters to summarize benefits would be yet another separate part of a web site.

Instead of being done on one specific model of dot-matrix printer, the letter would be generated as a PDF that could be displayed or downloaded or printed. Pretty conventional stuff by modern standards. A huge revision considering the legacy programming.

The intimate connections between the database, the calculation module and the letter-writing module would have to be narrowed considerably. A formal list of specific pieces of information would have to replace the no-holds-barred access in the VB modules and database.

Preservation

Essentially, the VB code that produced the letters encoded some business knowledge.

Much of knowledge was encoded in the calculation module, also.

This needed to be refactored so that the business knowledge was focused on the calculation module.
The letter writing had to be stripped down to something that worked like this.

1. Query some initial stuff from the database.
2. Determine which letter template to use.
3. Query the rest of the stuff from the database based on the specific situation and template letter.
4. Fill in the blanks in the template. Generate the PDF.

The legacy VB code, of course, had to be studied carefully to locate all of the business process and all of the data sources.

And all of the quirks had to be explained. In this case, there are unfixed bugs that had to be preserved in order to say the new output matched the old output. In a few cases, we had to report that the old application was clearly wrong and actually fix the bugs. Interestingly, there was no arguing about this. The bugs we found were all pretty well-known bugs.

An "Interface" module was defined. All of the processing from the VB letter-writing programs was pushed into the interface. The letter-writing was refactored down to template fill-ins.

The interface became the subject of some architectural debate. It was the essential encoded knowledge from the original VB programs. Is it part of letter-writing? Or is it really part of the core calculation module?

Eventually, it was pushed into the calculation module and the "interface" could be removed, leaving a very clean interface between letter-writing and calculation.

I suspect (but I don't know) that the Java calculation module was quite the mishmash of stuff extracted from numerous VB programs. Hopefully, those programmers had proper unit test cases and could refactor the calculation module to get it into some sensible, maintainable form.

Legacy Code Preservation: The Bugs Are The Features

The extreme end of "paving the cowpaths" are people for whom the bug list is also the feature list.
This is a very strange phenomenon, rarely seen, but still relevant to this review.

In this particular case, the legacy application was some kind of publishing tool. It used MS-Word documents with appropriate style tags, and built documents in HTML and PDF formats from the MS-Word document. Badly.

To begin with, problems with MS-Word's style tags are very difficult to diagnose. Fail to put the proper style tag around a word or phrase and your MS-Word source file looks great, but it doesn't produce the right HTML or PDF.

More importantly, the PDF files that got created were somewhat broken, having bugs with embedded fonts and weirdness with downloading, saving locally and printing.

And--of course--the vendor was long out of business.

What to do?

Feature Review

In order to replace this broken publishing app, we need to identify what features are essential in the HTML and PDF output. This shouldn't be rocket science.

For instance, there may be inter-document links that should be magically revised when a document moves to a new URL. Or, there must be embedded spreadsheets. Or there have to be fill-in forms that can be printed.

The editor who worked with this tool could not---even after repeated requests---provide a list of features.

Could not or would not, it didn't matter.

Instead, they had a list of 20 or so specific bugs that needed to be fixed.

When we tried to talk about locating a better publishing package, all the editor could bring to the table was this list of bugs.

For example: "The downloaded document has to have the proper flags set so that it uses local fonts."
We suggested that perhaps this should include "Or it has all the fonts embedded correctly?"
We were told---firmly---"No. The local fonts must be used or the document can't be saved and won't print."

Trying to explain that this font bugs doesn't even exist in other packages that create proper PDF's went nowhere. There was a steadfast refusal to understand that the bugs were not timeless features of PDF creation. They were bugs.

Several meetings got sidelined with the Bug List Review.

Eventually, the editor had to be reassigned to something less relevant and visible. A more rational editor was put into the position to work with a technology team to bring in a new package and convert the legacy documents.

The Confusion

It's not easy to see where the confusion between feature and bug comes from.

Why did the desirable feature set become a murky unknown? An editor should be able to locate the list of styles actually used and what those styles did in the resulting documents.

The confusion and bizarre behavior could possibly stem from the stress or terror.

Perhaps learning a new package was too stressful.

Perhaps idea of converting several dozen complex documents from MS-Word to some other markup was terrifying. So terrifying that roadblocks needed to be put in place to prevent meaningful discussion of the conversion effort.

Where was the business knowledge encoded? What needed to be preserved?

Not the software. It was junk.

There was business knowledge represented in the documents in the obvious way. But there was also some business knowledge encoded in the markup that established links and references, emphasis, spreadsheets, forms or whatever other features were being used.

Any semantic markup encodes additional knowledge above and beyond the words being marked up. The semantic markup was what needed to be preserved during the conversion. Maybe this was the source of the terror and confusion.

Legacy Code Preservation: Data Warehouse and Legacy Operations

A data warehouse preserves data.

It can be argued that a data warehouse preserves only data. This, however, is false.

To an extent, a data warehouse must also preserve processing details.

Indeed, a data warehouse exemplifies knowledge capture because the data and its processing steps are both captured.

The ETL process that prepares data for loading into the warehouse is tied to specific source applications that provide data in a known form and a known processing state. A warehouse isn't populated with random data. It's populated with data that is at a known, consistent state.

For example, when loading financial data, the various accounting applications (like the General Ledger) must be updated with precisely the same data that's captured for data warehouse processing. Failure to assure consistency between ledger and warehouse makes it difficult to believe that the warehouse data is correct.

Preserving Details

In some cases, legacy applications have a tangled architecture. Code can be repeated because of copy-and-paste programming. This can make it difficult to be sure that a data warehouse properly captures data in a consistent state.

What's distressingly common is to have a "code" or "status" field where the first or last position has been co-opted to have additional meanings. A "9" in the last position of a product number may be a flag for special processing.

These cryptic flags and indicators are difficult to identify in the first place. They are often scattered throughout the application code base. Sometimes they reflect work-arounds to handle highly-specialized situations. Other times, they're pervasive changes that were done via cryptic flags rather than make a first-class change to a file format.

When populating a warehouse, these codes and flags and secret processing handshakes need to be found and properly normalized. This may mean that an ETL program will recapitulate different pieces of special-case logic that's scattered around a number of legacy programs.
This is the essence of knowledge capture.

It also drives up the cost and expense of maintaining the ETL pipeline that feeds the data warehouse.
After all, the source application can make processing changes that aren't properly reflected in the ETL processing pipeline.

As if this isn't bad enough, many organization permit technology that makes processing even more obscure.

The Evils of Stored Procedures

In far too many cases, software architectures place code into two locations.

• Application programs.
• Data bases.

Putting code into a database is simply a mistake. There's no rational justification. None.

The irrational justifications include the following farcical claims.

• Stored Procedures are faster.
  Not really. There's no reason why they should be faster, and simple benchmark measurements show that application programs outside the database will be as fast or faster than stored procedures. A process running outside the database doesn't compete for database resources the same way the stored procedure engine does.
• Some processing is essential to data integrity.
  This is absurd, since it presumes that the folks writing stored procedures are trustworthy and folks writing non-stored procedure applications are a lying bunch of thieving scoundrels who will break the data integrity rules if given half the chance.

Let's look at this second justification.

The argument has two variants.

1. Some logic is so essential to interpreting the contents of the database that it cannot meaningfully be packaged any other way.
   This makes the claim that all sharable programming technology (Java packages, Python modules, etc.) simply don't work, and the database is the only effective way to share code.
2. Some logic is so essential to correct status of the database, that no application developer can be trusted to touch it.
   This presumes that application developers are willing to cut corners and break rules and force bad data into an otherwise pristine database. Data integrity problems come from those "other" developers. The DBA's can't trust anyone except the stored procedure author.

When confronted with other ways to share logic, the stored procedure folks fall back on "faster" or possibly the "Us vs. Them" nature of the second variant.

Stored Procedure Consequences

Stored procedures really are code. They should not be separated from the rest of the code base.

Stored procedures are maintained with different tools and through different organizations and processes. This leads to conflict and confusion.

It can also lead to weird secrecy.

A stored procedure can be difficult to extract from the database. It may require privileges and help from DBA's to locate the unencrypted original source text.

In a huge organization, it can take weeks to track down the right DBA to reveal the content of the stored procedure.

Why the secrecy?

Once exposed, of course, the stored procedure can then be rewritten as proper code, eliminating the stored procedure.

The proper question to ask is "Why is critical business knowledge encoded in so many different places?" Why not just application code? Why also try to encode some knowledge in database stored procedures? How does this bifurcation help make the origination more efficient?

Legacy Code Preservation: Paving the Cowpaths

No discussion of legacy preservation is complete without some "Paving the Cowpaths" stories.

The phrase refers to the way cows tend to meander across the landscape in a remarkably consistent way. The cows reliably follow a consistent path. The paths tend to wander in ways that seem crazy to us.

Rather than do a survey and move some dirt to lay a straight, efficient road, paving the cowpaths refers to simply using the legacy path without consideration of more efficient alternatives.

There are two, nearly identical paving the cowpath stories, separated by three years. We'll only look at one in detail, since the other is simply a copy-and-paste clone.

The Code Base

In both cases, the code base was not something I saw in any detail. In one case, I saw a presentation, and I talked with the author in depth. In the other case, I had the customer assign a programmer to work with me.

In one case they had a fabulous application system that was the backbone of their business. It was lots and lots of VAX Fortran code that did simply everything they needed, and did it exactly the right way. It was highly optimized and encoded deep knowledge about the business.

[The other case wasn't so fabulous, but the outcome is the same.]

Sadly, each gem was entirely written to use flat files. It was relatively inflexible. A new field or new relationship required lots of tweaking of lots of programs to accommodate the revised file layout.

In 1991, the idea of SQL databases was gaining currency. Products like Oracle, Ingres, Informix and many others battled for market share. This particular customer had chosen Ingres as their RDBMS and had decided to convert their essential, foundational applications from flat file to relational database.

The Failure

There was a singular, and epic failure to understand relational database concepts.

A SQL table is not a file that's been tarted up with SQL access methods.

A foreign key, it turns out, is actually rather important. Not something to be brushed aside as "too much database mumbo-jumbo."

What they did was preserve all of their legacy processing. Including file operations. They replaced OPEN, CLOSE, READ and WRITE with CONNECT, DISCONNECT, SELECT and UPDATE in a remarkably unthinking way.

This also means that they preserved their legacy programs that made file copies. They rewrote a file copy as a table copy, using SELECT and INSERT loops.

Copying data from one file to another file can be a shabby way to implement a one-to-many relationship. It becomes a one-to-one with many copies. A file copy can be amazingly fast. A SQL table copy can never be as fast as a file copy.

They can, of course, easily compare the database results with the old flat file results. The structures are nearly identical. This, I think, creates a false sense of security.

My Condolences

In both cases, I was called in to help them "tune" the database to get it to run faster.

I asked about the longest-running parts of the application. I asked about the most business-critical parts of the application. "What's the most important thing that's being blocked by unacceptable slowness?"
It's not possible to get everything to be fast. It is, however, possible to get important things to be fast. Other, less important things, can be slow. That's okay.

They talked me though a particularly painful part of the application that was very important and unbelievably slow. It cloned a table making a small change to each row.

"Oh," I suggested, "you could have used an UPDATE statement with no WHERE clause to touch all rows."

That suggestion, it turns out, was wrong. The copying was essential because the keys were incomplete.

Then it began to dawn on me.

Their legacy application did file copies because they were almost instant. And the filename (and directory path) become part of the key structure.

They were shocked that a SQL table copy could be so amazingly slow. Somehow, the locking and logging that create transactional integrity wasn't visible enough.

The really hard part was trying to---gently---determine why they thought it necessary to clone tables.

The answer surfaced slowly. They had simply treated SQL as if it was a file access method. They had not redesigned their applications. They did not understand how primary key, foreign key relationships were supposed to work. They had, essentially, wasted a fair amount of time and money doing a very, very bad thing.

Preservation

They preserved the relevant business features.

They also preserved irrelevant technical implementation features.

They didn't understand the distinction between business process and technical implementation details.
In effect, they labored under the assumption that all code was precious, even code that was purely technical in nature.

Legacy Code Preservation: What's the Cost?

It's 1980-something. We're working on a fairly complex system that includes some big machines and three computers. One of the computers has a magnetic tape drive into which it writes a log of interesting events. In the 80's, this was a pretty big deal.

An operational run will produce a log; then we can use customized applications to analyze and reduce the log to something more useful and focused. The first step is to do some data extraction to get the relevant log entries off the tape and into a disk file that engineers can work with.

Recall that the spreadsheet has only been around for a few weeks at this point in the history of computing. Sums and counts require programs. In this case, they are written in Fortran.
So far, so good. My job is to add yet another feature to the data extraction program. It will pull some new different bits of data off the logs.

The log entries are, of course, fairly complex. This is not different from log scraping in a web server context. Some log entries have to be ignored, others have to be merged. Some have cryptic formats.

The Code Base

The extraction application has been in use (and heavily modified) for a couple of years. Many programmers have touched it. Many.

The data extractor is written in a language called JOVIAL. This is not a problem. It's the language of the large system being built. The engineers are happy to use Fortran for their off-line analysis of the files.

There's a subtlety that arises in this mixed language environment. Any engineer with Fortran skills can whip together an analysis program. But only the favored few programmers know enough JOVIAL to tweak the data extraction program. And they're all busy writing the real software, not supporting analysis and trouble-shooting.

This data extractor program suffers from a lot of "copy-and-paste" programming. Blocks of code are repeated with minor changes. Standard modules are repeated with differences from the official copy that the entire rest of the system uses. Block comments don't nest, so it's hard to remove a large chunk of code which contains a block comment.

Further, it suffers from "Don't Delete Diddly" programming. Large swaths of code are left in place, relegated to a subroutine that never gets used. Other blocks of code are circumvented with a GOTO statement to simply jump over the code.

And, it has a complex history and provenance. In order to debug anything on the complex target system, the logger had to be the first thing up and running. Therefore, the logger specifically predates all other features of the application. It doesn't involve any rational reuse of any other piece of software.
This is the 80's, so version control and forking a new version were simply not done.

My job was to make a minor revision and extract just one certain type of log entry. Effectively a "filter" applied to the log.

After several days of reading this mess, I voted with my feet. I wrote a brand-new, from-scratch, "de novo" program (in Fortran, not JOVIAL) which reads the tape and produces the required log entries.

Why?

It was cheaper than messing with the legacy code base. Less work. Less risk of breaking something. And less long-term cost from continuing to maintain the data extractor.

Grief and Consternation

Discarding the legacy JOVIAL analysis program was a kind of heresy. It was a Bad Thing To Do. It "Raised Questions".

Raised Questions? Really? About what?

Did it raise questions about the sanity of managers who preserved this beast? Or about the sanity of programmers doing copy-and-paste programming?

I had to endure a lengthy lecture on the history of the data extraction program. As if the history somehow made a bad program better.

I had to endure begging. The legacy program should be preserved precisely because it was a legacy. Really. It should be "grandfathered in" somehow. Whatever that means.

Preservation

The original Jovial data extractor program still existed. It still ran. It could still be used. The JOVIAL code base and tools (and skilled programmers) remained available.

No one had deleted anything. There was no actual problem.

We had just started to realize that it was time to move on.

I started with a clean, simple Fortran program that read the logs, extracted records, and created files that engineers could work with.

But, but doing that, I guess that I had called somebody's baby ugly.

This new Fortran program preserved the essential knowledge from the original JOVIAL program. Indeed, I think that one of the reasons for all the grief was that I had exposed relevant details of the implementation, stripped clean of the historical cruft.

The tape file format and the detailed information on the log file records had gone from closed and embedded in just one program to open and available to more than one program.

The Fortran program exposed the log file details so that anyone could write a short (and more widely readable) Fortran program. This allowed them to avoid the cost and complexity of waiting for someone like me to modify the JOVIAL extraction program.

The file format is merely a technical detail. It's the analyses that were of real value. And none of that was in the original JOVIAL program. They remained as separate Fortran programs.

Legacy Code Preservation: Are There Quirks?

Let's visit some other conversion activities in the 1970's. The gig was at a company implementing a customized insurance application. The actuaries used a PDP-10 (and Fortran) to compute their various tables and summaries.

I was roped into rewriting an actuarial Fortran programs into PL/1 for an IBM 370.

This program, clearly, encodes deep business knowledge. It must be preserved very precisely, since the actuarial calculations are directly tied to the financial expectations for the particular line of business.
The good news about Fortran to PL/1 conversion is that PL/1 offers features (and syntax) that are similar to Fortran. It's not an exact match, but it's close enough to make the conversion relatively risk-free.

There are, of course, issues.

In particular, Fortran IV was not big on the "structured if-then-else" features of Algol-like languages. PL/1, like Pascal, followed on the heels of Algol 60. Fortran didn't follow Algol; Fortran depended on GOTO statements instead of nested IF-THEN-ELSE statements.

This meant that some logic expressions were rather tangled and difficult to fully understand. Patience and and care were required to unwind the logic from it's tangled nest of Fortran GOTO's into neater PL/1 BEGIN-END blocks.

Test Case

Perhaps the most important gap here was the lack of any kind of definitive test case.

It was the 70's. Testing was---at best---primitive. The languages and tools didn't support very much in the way of automated testing.

Compounding the problem, IT management was so late in getting the project started that we had to do repeated overnighters to get things running. The fog of sleep deprivation doesn't facilitate high quality software.

Further compounding the problem, we don't really have access to the PDP-10 that the actuaries use. We can't run any controlled tests.

And. Bonus.

We were doing "test-in-production". As soon as it worked, that was the official production run. Everything prior to the one that worked was discard as a test run.

The test strategy was simply to do a side-by-side comparison with the legacy PDP-10 output. While it's tedious to read hundreds of pages of mainframe computer print-out, that was the job.

Results

For the first attempts, there were significant logic issues. Regions of IF-GOTO that hadn't been properly rewritten into IF-THEN-ELSE.

At some point, the output would disagree. The PDP-10 Fortran, of course, was deemed to be "right."

So it was a matter of discovering what was unique about the case where there was a difference. Lots of deduction and puzzle solving.

Finally, we got down to one really subtle issue.

The numbers were slightly different. Slightly.

What does this slight discrepancy mean?

Is it a bug? Do we have to chase down some math error? It's unlikely to be a math error, since the expressions convert trivially from Fortran to PL/1. And the numbers are close.

Is it a feature? Is there something in Fortran or PL/1 that we simply failed to understand? Unlikely.

Everything else works.

It's a "quirk". It's not a "bug" because it's not clearly wrong. It's not a feature, because we're not going to define it as being clearly right. It's in this middle realm of behavior best described as quirky.

Quirks

What we've uncovered, it turns out, is the difference between Fortran floating point calculations and PL/1's fixed-point decimal calculations. PL/1's compiler reasons out the proper number of decimal places in the intermediate results and generates fixed-point decimal code appropriately.

Decimal hardware, BTW, was part of the IBM 370 system. Decimal-mode arithmetic was often faster then floating-point.

The PL/1 rules have some odd features regarding division and multiplication. A*0.001 and A/1000 have different deduced number of decimal places. Other than that, the rules are obvious and mathematically sound.

The PL/1 version provides exact decimal answers. Lots of decimal places exact.

The Fortran version involved approximations. All floating-point calculation must be looked at as an approximation. Many numbers have an exact binary representation. But numbers without an exact binary representation will have tiny errors. The tiny errors are magnified through calculations. Generally, subtracting two nearly-equal floating-point values elevates the erroneous parts of the approximation to lofty heights of visibility.

Preservation

It was important to preserve the essential actuarial knowledge encoded in Fortran into PL/1.

It was not as important to preserve the quirks of single-precision floating-point math.

Clearly, we have to distinguish between three separate considerations.

1. Valuable Features: encoded business knowledge.
2. Implementation Details: technology knowledge.
3. Quirks. Aspects of the implementation that lead to low-value discrepancies in the output.

Legacy Code Preservation: What's the Story?

Wind back the clock to the late 1970's. Yes, there were computers in those days.

Some of my earliest billable gigs where conversions from old OS to new OS. (Specifically DOS/VSE to OS/370, now called Z/OS.) Back when a company owned exactly one computer, all of the in-house customized software had to be rewritten as part of the conversion.

For the most part, this was part of a corporate evolution from an IBM 360-series to 370-series computer. That included revisions of the operating system toward OS/370.

A company's custom software often encoded deep knowledge of business operations. Indeed, back in the day before desktop computers, that software was the business. There was no manual fallback if the one-and-only computer didn't work. Consequently, the entire IT department could be focused on converting the software from old operating system to new.

Every line of code was carefully preserved.

Not all software encoded uniquely valuable knowledge, however.

Flashback

In the days before relational databases, all data was in files. File access required a program. Often a customized piece of programming to extract or transform a file's content.

In old flat-file systems, programs would do the essential add-change-delete operation on a "master" file. In some cases programs would operate on multiple "master" files.

In this specific conversion effort, one program did a kind of join between two files. In effect, it was something like:

SELECT * FROM BIG_TABLE
JOIN OTHER_TABLE ON BIG_TABLE.CODE = OTHER_TABLE.CODE
...

What's interesting about this is the relative cost of access to OTHER_TABLE.

A small subset of OTHER_TABLE rows counts for most of the rows that join with BIG_TABLE. The rest of OTHER_TABLE rows were referenced once or a very few times in BIG_TABLE.

Clearly, a cache of the highly-used rows of OTHER_TABLE has a huge performance benefit. The question is, of course, what's the optimal cache from OTHER_TABLE? What keys in OTHER_TABLE are most used in BIG_TABLE?

Modern databases handle this caching seamlessly, silently and automatically. Back in the 70's, we had to tailor this cache to optimize performance on a computer that---by modern standards---was very small and slow.

The Code Base

In the course of the conversion, I was assigned a script ("JCL" is what they called a shell script on Z/OS) that ran two programs and some utility sort steps. Specifically, the sequence of programs did the following:

SELECT CODE, COUNT(*)
FROM BIG_TABLE
GROUP BY CODE
ORDER BY 2

Really? Two largish programs and two utility sort steps for the above bit of SQL?

Yes. In the days before SQL, the above kind of process was a rather complex extract from BIG_TABLE to get all the codes used. That file could be sorted into order by the codes. The codes could then be reduced to counts. The final reduction was then sorted into order by popularity.
This program did not encode "business knowledge." It's purely technical.

At the time, SQL was not the kind of commodity knowledge it is today. There was no easy way to articulate the fact that the program was purely technical and didn't encode something interesting. However, I eventually made the case that this pair of programs and sorts could be replaced with something simpler (and faster.)

I wrote a program that used a data structure like a Python defaultdict (or a Java TreeMap) and did the operation in one swoop.

Something like the following:

from collections import defaultdict
counts= defaultdict( int )
total= 0
with read( "big_table" ) as source:
    reader= BigTable_iter( source )
    for row in reader:
        counts[row.code] += 1
        total += 1
by_count= defaultdict( list )
for code,count in counts.items():
    by_count[count].append( code )
for frequency in sorted( by_count, reverse=True ):
    print( by_count[frequency] )

def BigTable_iter( source ):
    for line in source:
     yield BigTable( line[field0.start:field0:end], etc. )

Except, I did it in COBOL, so the code was much, much longer.

Preservation

This is one end of the spectrum of legacy code preservation.

What was preserved?

Not business knowledge, certainly.

What this example shows is that there are several kinds of software.

• Unique features of the business, or company or industry.
• Purely technical features of the implementation.

We need to examine each software conversion using a yardstick that measures the amount of unique business knowledge encoded.

Legacy Code Preservation

Rule One: Writing Software is Capturing Knowledge.

Consequence: Converting Software is Preserving Knowledge.

When software is revised for a new framework or operating system or database or when an algorithm is converted to a new language, then we're "converting" (or "migrating") software. We're preserving code, and preserving the knowledge encoded.

For the next few months, I'm going to post some examples of preserving legacy code and how this ties to the knowledge capture issue.

Once we've looked at some examples of business software, we can turn to something a little less concrete: HamCalc.

These examples are presented in historical order. Each example raises questions and outlines elements of a strategy for legacy code preservation.

• What's the Story? Late 1970's. What user story was encoded in the software?
• Are There Quirks? Late 1970's. Is the encoded knowledge really a useful feature? Or is it a bug? What if we can't be sure?
• What's the Cost? Early 1980's. What if the legacy code is complex and expensive? How can we be sure it doesn't encode some valuable knowledge?
• Paving the Cowpaths. Throughout the 80's. When converting from flat-file to database, how can we distinguish between encoded user stories and encoded technical details? Isn't all code equally valuable? There are several examples; I've combined them into one.
• Data Warehouse and Legacy Operations. This is a digression on how data warehouse implementation tends to preserve a great deal of legacy functionality. Some of that legacy functionality exists in stored procedures, a programming nightmare.
• The Bugs are the Features. Can you do software preservation when user doesn't seem to understand their own use cases?
• Why Preserve An Abomination? How do we preserve shabby code? How can we separating the user stories from the quirks and bugs? There are several instances, I've used one as an example.
• How Do We Manage This? The legacy code base was so old that no one could summarize it. It had devolved to a morass of details. With no intellectual handles, how can we talk about the process of converting and what needs to be preserved?
• Why Preserve the DSL? describes a modern instance of "Test-Driven Reverse Engineering" where the unit test cases were created from the user stories and the legacy code use merely as supporting details. An entirely new application was written which preserved very little of the legacy code, but met all the user's requirements.

These nine examples include some duplicates. It's really more like a dozen individual case studies. Some are simple duplicates; the name of the customer is changed, but little else.

This Seems Irrational... But... HamCalc

Step 1.  Look at the original HamCalc.  Even if you aren't interested in Ham radio, it's an epic, evolving achievement in a specialized kind of engineering support.  It's a repository of mountains of mathematical models, some published by the ARRL, others scattered around the internet.

Step 2.  Look closely at HamCalc.  It's all written in GW basic.  Really.  The more-or-less final update is from 2011 -- it's no longer an active project -- but it's a clever idea that suffers from a horrible constraint imposted by the implementation language.

A long time ago, I was captivated by the idea of rewriting HamCalc as Java Applets.  It seemed like a good idea at the time, but that's a lot of work: 449 programs, 85,000 lines of code.

Recently, I wanted to make some additional use of HamCalc's amazing collection of formulas.

However.  The distribution kit is rather hard to read.  The .BAS files are in the tokenized "binary" format.

I found a Python project to interpret the byte codes into a more useful format. See http://www.danvk.org/wp/gw-basic-program-decoder/  However, it wasn't terribly well written, and didn't prove completely useful.

GW Basic Bytes Codes

Look at http://www.chebucto.ns.ca/~af380/GW-BASIC-tokens.html for some basic rules on the file format.

See http://www.antonis.de/qbebooks/gwbasman/index.html for a reasonably clear definition of the language itself. Quirks are, of course, studiously ignored, so there's a lot of ambiguity on edge cases.

For simple bytes-to-text translation, this is pretty simple.  The next step -- interpreting GW Basic -- is a bit more complex.

Future

The irrational thing is that I'm captivated by the idea of preserving this legacy gift from the authors in another, more useful language. Indeed, the idea of a community of "HamCalc Ports to Other Languages" appeals to me. This base of knowledge is best preserved by being made open so that it can be rewritten into other commonly-used languages.

There's a subset version here: http://www.softpedia.com/get/Science-CAD/HamCalc.shtml and here http://www.dxzone.com/dx11432/hamcalc-v1-3.html. This is just a few of the calculations, carefully rebuilt to include nice versions of the ASCII-art graphics that are central to the original presentation.

The hard part of preserving HamCalc is the absolute lack of any test cases of any kind.

I think the project should work like this.

1. Publish the complete plain text source decoded from the tokenized binary format. It will likely be somewhere on http://www.itmaybeahack.com/ or perhaps a Dropbox.
2. Publish the index of programs and features as a cross-reference to the various programs. This should include the various links and references and documentation snippets that populate the code and output. This forms the backbone of the documentation as well as the unit testing.
3. Do a patient (and relatively lame) translation to Python3.2 to break HamCalc into two tiers. The calculation library and a simple UI veneer using stdio features of the print() and input() statements. The idea is to do a minimalist rewrite of the core feature set so that a GUI can be laminated onto a working calculation library.
4. Work out test cases for the initial suite of 449 legacy programs oriented toward the calculation layer, avoiding the UI behavior. The idea isn't 100% code coverage. The idea is to pick the relevant logic paths based on the more obvious use cases.

A sophisticated GUI is clearly something that was part of the original vision. But the limitations of GW Basic and tiny computers of that era assured that the UI and calculation were inextricably intertwingled.

If we can separate the two, we can provide a useful library that others can build on.

Maybe I should organize http://www.hamcalc.org/ as the jumping-off point for this effort?