Code Rewrites and How They Create Value -- Stop Fighting Against It

TL;DR -- To remove doubts and questions, rewrite it.

Many, many people are confronted with the request to maintain someone else's code.

Either it's open source, and you have to make formal PR's visible to the world.

Or it's "enterprise" in-house software, and you have to make PR's visible to a work team.

Or.

It's "work group" in-house software, and you have source that may not be under proper source-code control installed on a server where you're taking over someone else's carefully built structure of porcelain components.

In the first case -- public code -- a rewrite is a challenge. People depend on it having a well-known interface. A small change here could alienate large swaths of the user community. On the other hand. A small change here could make the project *more* useful to *more* people. This is challenging. My advice here is limited.

When we look at enterprise software, however, a rewrite won't have quite the same "blast radius."

When we look at work-group software, there is no blast radius.

Benefits of a Rewrite

Why rewrite? Three reasons.

1. You can understand it. (This is HUGE.)
2. You can make it objectively better (i.e., higher PyLint scores, better documentation coverage.)
3. You can add or expand the test cases. 

The first and foremost reason -- understanding -- can't be understated.

And.

It's a HUGE fight every time. The standard argument is "If It Ain't Broke, Don't Fix It."

This is, of course, based on misunderstanding the level of "broke." A delicately-balanced tower of porcelain components that worked once last month is -- in effect -- already pre-broken if any change will disturb the structure and ruin everything.

There are lots of examples.

• The app only works with Pandas 0.12.0 and will not work with 0.13.x or the 0.22.0 you have in your default Conda environment.
• The app only works when you provide --someoption=False, and no one can figure out why.
• Some test cases have @skip because they don't work. But should.
• The setup.py doesn't work and you can only run it using PYTHONPATH.
• The default logging initialization is "somehow" not right and requires a manual override in the app.

I know. I've created all of these problems.

On one hand, we have management: "It ain't broke."

On the other hand, we have everyone else: "It's a fragile nightmare of pre-broken components that cannot be touched."

The Script

Here's how it plays out. In the Real World.

Management: "It ain't broke. Don't fix it."
You: "I can't make it work."
Management: "It ran last month."
You: "I made one small change and it doesn't run this month."
Management: "Back out the change."
You: "The results are then useless."

After much Grrr and Gnashing...

Management: "Identify the problems and we'll prioritize."
You: "Here are a dozen things."
Management: "These are too vague. Be more specific."
You: "Here are a score of things."
Management: "You're in the weeds. Bring it up a level to where business people can understand it."

After more Grrr and Gnashing...

Management: "What's the smallest change we can get away with?"
You: "The one I made that broke everything."
Management: "Let's have lots of peer review and design walkthroughs."
You: "Cool, then you'll see how broken it is."
Management: "Okay. Let's not. Instead, make the smallest change you can."
You: "I made one small change and it doesn't run this month."
Management: "Back out the change."
You: "The results are then useless."

Can we break the cycle of uselessness?

Depends.

We may be struggling with management folks who are set against fixing what's obviously broken. They're living a rich fantasy life that you can't really change.

However.

The Grrr and Gnashing part of the dialog represents time in which useful stuff can be done. Specifically. Rewrites.

Rewriting Strategy

There are three important parts of rewriting.

• Understand what it's doing and why.
• Describe it with test cases.
• Make it objectively clear (i.e., high PyLint scores, complete documentation, etc.)

The effort often involves multiple passes. I like to describe it as Test-Driven Reverse Engineering (TDRE).

1. Create (or expand) the test cases.
2. Rewrite the code.
3. Repeat until it's better.

It's essential to do these in order. Without test cases, rewrites are only more breakage. With test cases, rewrites are guaranteed to produce the same results as the previous mess of horrible code.

Sometimes the test cases are really a kind of "system test" where the whole application is run against some known inputs to produce some expected outputs. This is better than nothing. It supports building fine-grained unit test cases that conform to the system test case.

Other times, the test cases may be proper unit tests and the rewrites can be at a finer level of granularity. In this case, test coverage may have to be expanded to include the fragile bits. In some cases, the rewrites may be necessary to make the code testable in the first place.

Adding test cases is objectively valuable work.

Even the dumbest of "It ain't broke" managers can recognize this value. The rewrites are a beneficial consequence of adding test cases. You may be able to achieve a goal of fixing something without ever being seen as "fixing" it. All you did was improve test case coverage and improve the "design for testability."

Costs and Benefits

Consider the cost of struggling vs. the cost of rewriting.

It's the same 80 hours of effort.

In one case, you struggled with something management insisted wasn't broken. Eventually, you found ways to make it work.

In the other case, you rewrite something management insisted wasn't broken. In the end, you actually understood it and created objective improvements in the code.

Which is better?

Explaining an Application

Some years ago--never mind how long precisely--having little or no money in my purse...  I had a great chance to do some Test-Driven Reverse Engineering on a rather complex C program. I extracted test cases. I worked with the users to gather test cases. And I rewrote their legacy app using Test-Driven Development. The legacy C code was more a hint than anything else.

I thought it went well. We uncovered some issues in the test cases. Uncovered a known issue in the legacy program. And added new features. All very nice. A solid success.

Years later, a developer from the organization had to make some more changes.

The client calls.

"No problem," I assure them, "I'm happy to answer any questions. With one provision. Questions have to be about specific code. I can't do 'overview' questions. Email me the code snippet and the question."

I never heard another word.  No question of any kind. Not a general question (that I find difficult to answer,) nor a specific question.

Why the provision?

I find it very hard to talk with someone who hasn't actually read the code yet. I have done far too many presentations to people who are sitting around a conference room table, nodding and looking at power-point slides.

I know the initial phone call focused on "an overview." But what counts as an overview? Use cases? Data model? Architectural layers? Test cases? Rather than waste time explaining something irrelevant, I figured if they asked anything -- anything at all -- I could focus on what they really wanted to know.

I know that I have never been able to understand people hand-waving at a picture of code. I have to actually read the code to see what the modules, classes and functions are and how they seem to work. I'm suspicious of graphics and diagrams.  I know that I can't read the code while someone is talking. If they insist on talking, I need to read the code in advance.

Perhaps I'm imposing too much on this customer. But. They're going to maintain the code -- that seems to mean they need to understand it. And they need to understand it their own way, without my babbling randomly about the bits that interested me. Maybe the part I found confusing is obvious to them, and doesn't bear repeating.

Perhaps raising the bar to "specific questions about specific code" forced them to read enough. Perhaps after some reading, they realized they didn't need to pay me to explain things. I certainly can't brag that the code explained itself.

Or. Perhaps they realized how the unit tests worked and realized that TestCases provide a roadmap of the API.

HamCalc and Quirks

Careful study of the HamCalc shows a number of quirks. Some are funny, some are just examples of the need for unit test frameworks.

The Wikispaces for the modernization project is here: http://hamcalc.wikispaces.com/home

For example, the following line of code, in GW-Basic, will (usually) set Y to zero.

Y = O

Yes. That's the variable "O", not the number 0.

Why does this work? Why can we use "O" instead of 0?

Most programmers avoid using the variable "O", since it's hard to read.  GW-Basic provides default values of 0 for almost all variables. So, "Y=O" works as well as "Y=0" most of the time. The only time is doesn't work is if the program happens to have "O" used as a variable.

This is one of the examples where people start shouting that a compiled language is so obviously superior that the rest of us must be brain-damaged to use a dynamic language like Python.

This isn't a very compelling argument for the overhead of a compiler. It's a more compelling argument for avoiding languages with default values. Python, for example, would throw an exception if the variable "O" had no value.

This isn't common (so far, I've only found one example) but it's amusing.

Another amusing quirk is the occasional tangle of GOTO/GOSUB logic that defies analysis. There are several examples of GOSUB/RETURN logic that is totally circumvented by a GOTO that circumvents the return. This should (eventually) lead to some kind of stack overflow. But GW-Basic doesn't really handle recursion well, so it would probably just be ignored.

One of my favorites is this.


    730 FOR N=A TO T STEP B
    750 IF T/N=INT(T/N)THEN X=X+1:PN(X)=N:T=T/N:GOTO 730
    760 A=3:B=2
    770 NEXT N

What does the GOTO on line 750 mean? Since GW-Basic doesn't use a stack of any kind, it doesn't create recursion or stack overflow. It appears to "restart" the loop with a new value of T. I think.

Legacy Preservation and "Code Modernization"

The correct marketing term is "Code Modernization".

There are a large number of companies in the Code Modernization business.

They appear to offer automated "modernization" of code.

I would suggest going slowly toward automated modernization. I'm not easily convinced that any automated tool can preserve what's meaningful and ignore the parts which are quirks, bugs or legacy cruft that needs to be disposed of.

Indeed, I'm rather sure that an automated modernization will actually be deleterious.

Without some care, cruft from the legacy code could be canonized into an incomprehensible "feature" in the modern implementation. This will eventually become it's own weird legacy quirk.

It's often best to rethink and rewrite the essential parts of the legacy. Why?


Rule One: Writing Software is Capturing Knowledge.

Consequence: Converting Software is Preserving Knowledge.

Knowledge is something that people use. As with modern Agile software development, we need to focus on the connections among people. We need to use code as part of the channels of communication.

Read the Agile Manifesto: http://agilemanifesto.org

There are four principles:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

A detailed investigation into the use cases of legacy software, the unique knowledge encoded, and the quirks and cruft requires thinking about who uses the software and what they expect.

Working software (i.e., readable code) is central. An automated modernization that doesn't properly handle quirks and cruft may create "working" software in the sense that it compiles. But software that people can't read and understand doesn't "work" in the more global sense of encoding captured knowledge.

Collaboration is defeated by automated modernization. The users may have features they don't like or additional features they need. Doing the functionality improvements side-by-side with code modernization makes the most sense. Indeed, it often leads to a proper rewrite, which is the best strategy.

It's difficult for users to envision new software that corrects long-standing quirks. When an Agile process makes software available incrementally (e.g., the release cycle of Scrum), then the users learn what they should have requested. An automated modernization cannot easily respond to the way that users learn through each release.

HamCalc -- Preserving the Legacy

Wow.

The response to modernizing HamCalc was overwhelming.

Apparently there are a fair number of people who also think that HamCalc is a treasure to be preserved, improved and added-to.

If you're interested, start here: http://hamcalc.wikispaces.com/home

You can ask to be a member of this wiki, and I can add you. This gives us a place to share ideas.

I think the most important aspect of this project is to be welcoming and supportive of recreational and amateur programmers. The explanatory notes and documentation need to be clear, with proper references and footnotes. The final application programs need to be simple.

The point of HamCalc is not to build a large, complex application. The point of HamCalc is to be a repository of small, simple solutions that can be easily understood and repackaged. In my estimation, no one simply "uses" HamCalc. They read, extend, modify and make local copies of their private extensions to HamCalc.

Toward this end, we need to allow for alternate implementations. Two people should be able to carve out the same piece of code and provide different implementations that provide consistent results but reflect different optimizations for speed and resource use.

Here's the code repository. https://github.com/slott56/HamCalc-2.1

This will let folks who are interested get started. What I think might happen is that people will carve out pieces that interest them and convert bits of HamCalc. There are 449 programs, each one of which is a potential nugget of goodness.

Legacy Code Preservation: Some Patterns

After looking at this suite of examples, we can see some patterns emerging. There seem to be several operating principles.

1. The Data Matters. In many cases, the data is the only thing that actually matters. The legacy application knowledge may be obsolete, or so riddled with quirks as to be useless. The legacy knowledge may involve so much technical detail---no user story---that it's irrelevant when a newer, better technology is used.
2. User Stories Matter; Legacy Technology Doesn't Matter. It is essential to distinguish the legacy technology from the meaningful user stories. Once the two are teased apart, the technology can be replaced and the user stories preserved. A cool DSL may be helpful, and needs to be preserved, or may be a distraction from the real solution to the real problem.
3. Understanding the New Technology Is Central. Misusing the new technology simply creates another horrifying legacy.
4. Testing is Essential. Legacy code cannot be preserved without test cases. Any effort that doesn't include automated comparisons between legacy and converted is just new development.
5. Discarding is Acceptable. Unless the legacy code has a seriously brilliant and unique algorithm, most business applications are largely disposable. It may be less expensive to simply do new development using the legacy code as a kind of overly-detailed specification. Calling it "conversion" to make managers feel good about "preserving" an "asset" is acceptable. The project is the same as new development, only the words change to protect the egos of those not really involved.
6. Quirks are Painful. They might be bugs or they might be features. It will be difficult to tell.

How do these principles match against our various case study projects?

• What's the Story? The applications were technical, and could be discarded.
• Are There Quirks? Without a test case, we could not be sure of our conversion. So we accepted the quirks.
• What's the Cost? The application was technical, and could be discarded.
• Paving the Cowpaths. New Technology was misused, the result was a disaster.
• Data Warehouse and Legacy Operations. The legacy software encoding knowledge can be split haphazardly into database and application software buckets. The user stories matter. The technology doesn't matter.
• The Bugs are the Features. The user stories matter. If you can't articulate them, you're going to struggle doing your conversion.
• Why Preserve An Abomination? When the code is shabby and has bugs, you have to sort out the quirks that will be carried forward and the junk that will be discarded.
• How Do We Manage This? The user stories matter. The data matters. Focus on these two can help prioritize.
• Why Preserve the DSL? The user stories and test cases lead to a successful outcome. While the customer may feel like a conversion was being performed, it was really new development using legacy code as a specification.

With modern languages and tools, legacy code conversion is quite simple. The impediments are simply managerial in nature. No one wants to have a carefully maintained piece of software declared a liability and discarded. Everyone wants to think of it as an asset that will be carefully preserved.

This Seems Irrational... But... HamCalc

Step 1.  Look at the original HamCalc.  Even if you aren't interested in Ham radio, it's an epic, evolving achievement in a specialized kind of engineering support.  It's a repository of mountains of mathematical models, some published by the ARRL, others scattered around the internet.

Step 2.  Look closely at HamCalc.  It's all written in GW basic.  Really.  The more-or-less final update is from 2011 -- it's no longer an active project -- but it's a clever idea that suffers from a horrible constraint imposted by the implementation language.

A long time ago, I was captivated by the idea of rewriting HamCalc as Java Applets.  It seemed like a good idea at the time, but that's a lot of work: 449 programs, 85,000 lines of code.

Recently, I wanted to make some additional use of HamCalc's amazing collection of formulas.

However.  The distribution kit is rather hard to read.  The .BAS files are in the tokenized "binary" format.

I found a Python project to interpret the byte codes into a more useful format. See http://www.danvk.org/wp/gw-basic-program-decoder/  However, it wasn't terribly well written, and didn't prove completely useful.

GW Basic Bytes Codes

Look at http://www.chebucto.ns.ca/~af380/GW-BASIC-tokens.html for some basic rules on the file format.

See http://www.antonis.de/qbebooks/gwbasman/index.html for a reasonably clear definition of the language itself. Quirks are, of course, studiously ignored, so there's a lot of ambiguity on edge cases.

For simple bytes-to-text translation, this is pretty simple.  The next step -- interpreting GW Basic -- is a bit more complex.

Future

The irrational thing is that I'm captivated by the idea of preserving this legacy gift from the authors in another, more useful language. Indeed, the idea of a community of "HamCalc Ports to Other Languages" appeals to me. This base of knowledge is best preserved by being made open so that it can be rewritten into other commonly-used languages.

There's a subset version here: http://www.softpedia.com/get/Science-CAD/HamCalc.shtml and here http://www.dxzone.com/dx11432/hamcalc-v1-3.html. This is just a few of the calculations, carefully rebuilt to include nice versions of the ASCII-art graphics that are central to the original presentation.

The hard part of preserving HamCalc is the absolute lack of any test cases of any kind.

I think the project should work like this.

1. Publish the complete plain text source decoded from the tokenized binary format. It will likely be somewhere on http://www.itmaybeahack.com/ or perhaps a Dropbox.
2. Publish the index of programs and features as a cross-reference to the various programs. This should include the various links and references and documentation snippets that populate the code and output. This forms the backbone of the documentation as well as the unit testing.
3. Do a patient (and relatively lame) translation to Python3.2 to break HamCalc into two tiers. The calculation library and a simple UI veneer using stdio features of the print() and input() statements. The idea is to do a minimalist rewrite of the core feature set so that a GUI can be laminated onto a working calculation library.
4. Work out test cases for the initial suite of 449 legacy programs oriented toward the calculation layer, avoiding the UI behavior. The idea isn't 100% code coverage. The idea is to pick the relevant logic paths based on the more obvious use cases.

A sophisticated GUI is clearly something that was part of the original vision. But the limitations of GW Basic and tiny computers of that era assured that the UI and calculation were inextricably intertwingled.

If we can separate the two, we can provide a useful library that others can build on.

Maybe I should organize http://www.hamcalc.org/ as the jumping-off point for this effort?

Performance "Tuning": running in 1/100th the time

For the 757 Python Meetup group, someone proposed looking at some Python code they had which was slow.  The code implemented a variation on the Elo chess rating system.  It applied the ratings to other sports, and used the points scored as well as basic win/lose/tie to work out a ranking for sports teams.  Very clever stuff.

But.

It was described as horribly slow. Since it was only 400 lines of code, it was a great subject for review in a Python meetup.  I would be able to show some tweaks and performance tips.

Hypothetical Step 1:  Profile

My first thought was to run the profiler and see what popped up as a possible root cause of slowness.

Generally, there are three kinds of performance problems.

• Wrong data structure.  Or wrong algorithm.  (They're simply opposite sides of the same coin.)  This generally leads to dramatic, earth-shattering improvements.
• Piss-Poor Programming Practices.  This kind of fine-tuning yields minuscule improvements.  In some cases, no measurable change at all.
• Bugs.  Every time I've been asked to improve "working" code, I find that it has bugs.  Every time.  These can exacerbate performance problems.  Interestingly, most of these are readily apparent during an initial survey:  i.e., while simply reading the code to see how it works.  Trying to create unit tests for the purposes of refactoring often reveals additional bugs.

I didn't know what I was going to see in this 400-line program.

Hypothetically, profiling will help show what kind of problems we have.

Prior to profiling, of course, we need to run the darn thing.  Thankfully, they provided some typical input files.  For example, 1979 High School Football season results.  159 lines of patiently gathered teams and scores.

It Didn't Run

When we tried to run it with the profiler, we found that we had a bug.  Tracking down the bug, revealed the essential performance problem, also.

The core problem was a failure to make use of Python data structures.  

This manifested itself in a number of really bad design errors.  Plus, it presented as the actual, show-stopping, serious bug.

In order to work out team rankings, the program kept a list of teams.

I'll repeat that for folks who are skimming.

In order to work out team rankings, the program kept a list of teams.

Not a dict mapping from team name to team details.  But a simple list.  Locating a team in the list meant iterating through the list, looking for the matching team.  Really.

for t in team_list:

    if t['name'] == target_name:

        process( t )

This kind of thing was repeated in more than one place.

And one of those repeats had the bug in it.

What we have here is "wrong data structure".  Replacing a list with a dict will have earth-shattering impact on performance.

The Bug

The bug, BTW, was a lookup loop which had the additional requirement of adding missing teams.  It tried to use the for-else structure.  This was the intended code (not the actual code).

for t in team_list:

    if t['name'] == target_name:

        return

else:

    t['name']= init_new_team(target_name)

    

This is a first-class bit of Python.  An else clause on a for statement is a real feature of the language.  However, it's obscure enough that it's easy to get wrong.

However, it's also unique to Python, and the kind of thing that can lead to confusion.  I discourage it's use.  

Test-Driven Reverse Engineering

We're doing TDRE on a little piece of recreational programming.  This means that we need to fabricate unit tests for code that is purported to work.  Some folks like to call this "exploratory testing" (a phrase which is contradictory, and consequently meaningless.)  We're "exploring".  We're not "testing".

Once the core bug is fixed, we can run sample data through the application to get "big picture" results.  We can extract bits from the sample data and exercise various functions in isolation to determine what they actually do now, bugs included.

Since this is so simple--and we're going to review it during a 2-hour meet up--and there's nothing billable going on--we can get by with a few really simple unit tests.  We'll run the darn thing once to create some expected output.  We save the output and use that single output log as the expected results from each refactoring step.

More complex applications require more unit tests.  For a billable TDRE effort last year, I had to create 122 unit tests to successfully refactor a program of over 6,000 lines of code.  It took several weeks.

Real Step 1: Fix The Data Structure

Before profiling (but after running to capture some output) we have to fix the essential data structure.  Repeated scanning of a list is never going to perform well.

The whole point of the study of "Data Structures" is to prevent (or optimize) search.

In this case, we can prevent linear search of a list by using a dictionary.  That, clearly, is job one.

It was a pervasive rewrite.   Essentially, everything in this little program included a goofy loop to lookup a team in the list.  The Elo algorithm itself, which is already O(n²), is dragged down by using the linear search for the opposing team four more times, making it O(n³).

Cyclomatic Complexity

One of the big "issues" is the use of if statements throughout the scoring algorithm.  An if statement creates Cyclomatic Complexity and can lead to performance problems.  Generally, if statements should be avoided.

This algorithm applies some normalization factors to reconcile scoring with win/loss numbers in different sports.  Basketball, specifically, involves generally high scores.  Since there are 2-point and 3-point scoring opportunities, a factor is used to normalize the points into "goals".  Football, similarly, has numerous scoring opportunities with values of 1, 2, 3 and 6 points; the scores here are also normalized.

This normalization was done with an if statement that was evaluated deep inside the Elo algorithm.  Repeatedly. Evaluated.

The two functions that handled the normalizations, plus the normalization factors, are ideal candidates for OO design.  There's a clear hierarchy of classes here.  A superclass handles most sports, and two polymorphic subclasses handle football and basketball normalization.

The if statement is now "pushed up" to the very beginning of the program where an instance of the sports normalization object is created.  This object's methods are then used by the Elo algorithm to normalize scores.

Icing on the Cake

Once we've fixed the bug and replaced a list with a dict, everything else is merely icing.
Some other OO changes.

1. The "Team" information should not be a flat, anonymous dictionary.  It should be a proper class definition with proper attributes.  There aren't many methods, so it's easy to create. 
2. The "Game" information is read by csv.DictReader.  However, it should not remain a simple, anonymous dict.  As with a Team, a simple class can be created to handle Game.
3. The overall structure of the application needs to be broken into two sections.  The command-line interface parses options, opens files, and generally gets everything set up.  The actual ranking algorithm should be a function that is given an open file-like object plus the Sport object for normalization.  This allows the ranking algorithm to be reused in other contexts than the command-line (i.e. a web service).

A more subtle OO design point is the question of "mutability".  A Team in this application is little more than a name.   There are also numerous "stateful" values that are part of the Elo algorithm.   A Game, similarly, is an immutable pair of teams and scores.  However, it has some mutable values that are part of the Elo algorithm.  

Really, we have immutable Team and GameHistory objects, plus a few values that are used as part of the Elo calculation.  I'm a big fan of disentangling these mutable and immutable objects from each other.  

I suspect that the Elo algorithm doesn't really need to update the "state" of an object.  I suspect that it actually creates (and discards) a number of immutable candidate ranking objects.  The iteration that leads to convergence might be a matter of object creation rather than update.  I didn't make this change, since it required real work and we were out of time.

Bottom Line

The more times I do TDRE to improve performance, the more I realize that it's all about bugs and data structures.

This recreational application took 45-60 seconds to process one year's record of games for a given league.  It now takes less than 0.2 seconds to do the same volume of work.  Two test cases involving a complete run of 159 records runs in 0.411 seconds.  That's 1/100th the time simply from switching data structures.

The idea of "tweaking" a working program to improve performance is generally misleading.  It might happen, but the impact is minuscule at best.  

Here's the checklist for getting 100:1 improvements.

• Remove searches.  
• Remove deeply-nested if statements.  

Generally, reduce Cyclomatic Complexity.

COBOL Rework

See this article: "The COBOL Brain Drain" in ComputerWorld.  This article is very, very well written and repeats a number of common fallacies.

The fallacies lead to expensive, labor-intensive in-house software development and maintenance.  Either there's a lot of effort poking at the COBOL code.  Or there's a lot of extra code to "wrap" the COBOL code so that it's never touched.

"Migrating large-scale systems built in Cobol is costly and risky."  A popular position.  But the risks are actually quite small; possibly non-existent.  The risks of not converting are usually higher than the risk of conversion.

The perception of the COBOL code is that it's filled with decades of tricky, nuanced legacy details that are hard to fully understand.  This is only partially true.

A great deal of the tricky code is simply redundant.  COBOL is often written with copy-and-paste programming and blocks of code are simply repeated.  It's also important to note that some of the code is no longer exercised in the first place.

Mythical Risk

The "risk" comes from the perceived opacity of  the tricky, nuanced legacy details.  It doesn't appear to be clear what they mean.  How can a project be started when the requirements aren't fully understood?

What appears to be the case in reality is is that this tricky code isn't very interesting.  Most COBOL programs don't do much.  They can often be summarized in a few sentences and bullet points.

Detailed analysis (641,000 lines of code, 933 programs) reveals that COBOL programs often contain several commingled feature sets.

• The actual business rules.  These are often easy to find in the code and can also be articulated by key users.  The real work is usually quite simple.  
• A bunch of hackarounds.  These are hacks to work around bugs that occur elsewhere in the processing.  Sometimes a hackaround introduces additional problems which require yet more hackarounds.  All of this can be ignored.  
• Solutions to COBOL data representation issues.  Most of these seem to be "subtype" issues: a flag or indicator is introduced to distinguish between subtypes.  Often, these are extensions.  A field that has a defined range of values ("A", "C" or "D") has a few instances of "*" to indicated another subclass that was introduced with a non-standard code for peculiar historical reasons.

Once we separate the real code from the hackarounds and the representation issues, we find that most COBOL programs are trivial.  About 46% of the lines of code (74% of the distinct programs) involves programs that read one to four files to write a report.  In effect, these programs do a simple "relational join" or query.  These programs have single-sentence summaries.  

The hackaround problem is profound.  When looking at COBOL code, there may be endless copy-and-paste IF-statements to handle some issue.  There may be whole suites of programs designed to work around some issue with a third-party package.  There may be suites of programs to deal with odd product offerings or special customer relationships.  

The remaining 26% of the non-trivial programs follow a normal distribution of 1/4 simple, 1/2 moderately complex, and 1/4 regrettably and stupidly complex.  That final 5% of the programs will also be a whopping 20% of the lines of code.  They are the few big programs that really matter.  

Risk Mitigation

The risk mitigation strategy involves several parts.  

1. Data Profiling.  The COBOL data may have considerable value.  The processing is what we're trying to rewrite into a modern language.  A profile of every field of every file will reveal that (a) most of the data is usable and (b) the unusable portion of the data isn't very valuable.   
2. Triage.  We can summarize 80% of the code in simple sentences.  46% of the code is single-sentence summaries.  34% of the code has multiple sentence summaries.  The remaining 20% requires in-depth analysis because the programs are large; they average of 2400 lines of code each.
3. Test-Driven Reverse Engineering.  Since a 5% of the programs do the real updates, it's important to collect the inputs and outputs of these few programs.  This forms a core test suite.
4. Agile Methods.  Find the user stories which really matter.  Find the COBOL programs that implement those user stories.

The most important risk reduction strategy is to take an Agile approach.  We need to prioritize based on value creation.  All COBOL programs are not created equal.  Some fraction can simply be ignored.  Another fraction produces relatively low value, and can be converted last.

The second most important risk mitigation is to build conversions from legacy COBOL files to the new, preferred databases.  The data is preserved.  

There's almost no risk in rewriting the 46% of low-complexity COBOL lines of code.  This code is trivial.  Note that some of this code actually has no business value.  Simply ignoring the no-value code also reduces risk.  Since we're using live files to drive testing, we can easily prove that our new programs work.

It's not risky to focus on the 20% of high-value COBOL lines of code.  This contains most (or all) of the processing that the business needs to have preserved.  They can articulate the user stories; it's easy to confirm that the COBOL does what the business needs.  It's easy to use live data to drive the reverse engineering.

The remaining 34% of the code base may actually involve a small amount of overlooked complexity.  There may be a nuance here that really matters.    

This overlooked "nuance" is something that the users didn't articulate, but it showed up in our unit testing.  We couldn't reproduce an expected output because we didn't correctly locate all the processing steps.  It wasn't in our summary of the 80% of moderate-to-low complexity programs.  It wasn't in our detailed analysis of the 20% subset of hyper-complex, large programs.  

We've reduced the risk by reducing the volume of code that needs to be examined under the microscope. We've reduced the risk by using live files for parallel testing.  We've reduced the risk by collecting user stories.

The remaining risks are ordinary project risks, unrelated to COBOL or legacy data.

The Other Great Lie

Another popular fallacy is this:

"The business wants us to make investments in programming that buys them new revenue. Rewriting an application doesn't buy them any value-add".

The value-add is to create a nimble business.  Purging the COBOL has a lot of advantages.

• It reduces the total number of lines of code.  Reducing costs.  Improving time-to-market for an IT solution to a business problem.
• It reduces the number of technologies competing for mind-share.  Less thinking about the legacy applications is less time wasted solving problems.
• It reduces the architectural complexity.  If the architecture is a spaghetti-bowl of interconnections between Web and Legacy COBOL and Desktop, then following the spaghetti-like connections is simply a kind of intellectual friction.

The COBOL does not need to be purged all at once through a magical "big-bang" replacement.  It needs to be phased out.  

Agile techniques need to be applied.  A simple backlog of high-value COBOL-based user stories is the place to start.  The prioritization of these stories needs to then be clustered around the data files. 

Ideally all of the programs which create or update a given file (or related group of files) can be rewritten in a tidy package.  The old files can be used for Test-Driven Reverse Engineering.  Once the programs have been rewritten, the remaining COBOL legacy applications can continue to operate, using a file created by a non-COBOL application.

Each file (and related cluster of programs) is replaced from high-value down to low-value.  Each step creates a more nimble organization.

TDRE - Test Driven Reverse Engineering Case Study

Background

Read up on compass variation or declination.  For example, this NOAA site provides some useful information.

Mariners use the magnetic variation to compute the difference between True north (i.e., aligned with the grid on the chart) and Magnetic north (i.e., aligned with the compass.)

The essential use case here is "What's the compass variation at a given point?"  The information is printed on paper charts, but it's more useful to simply calculate it.

There are two magnetic models: the US Department of Defense World Magnetic Model (WMM) and the International Association of Geomagnetism and Aeronomy (IAGA) International Geomagnetic Reference Field (IGRF).

A packaged solution is geomag7.0.  This includes both the WMM and the IGRF models.  This is quite complex.  However, it does have "sample output", which amount to unit test cases.

The essential spherical harmonic model is available separately as a small Fortran program, igrf11.f.

Which leads us to reverse engineering this program into Python.

TDRE Approach

The TDRE approach requires having some test cases to drive the reverse engineering process toward some kind of useful results.

The geomag7.0 package includes two "Sample Output" files that have the relevant unit test cases.  The file has column headings and 16 test cases.  This leads us to the following outline for the unit test application.

    class Test_Geomag( unittest.TestCase ):
        def __init__( self, row ):
            super( Test_Geomag, self ).__init__()
            self.row= row
        def runTest( self ):
            row= self.row
            if details: 
                print( "Source: {0:10s} {1} {2:7s} {3:10s} {4:10s} {5:5s} {6:5s}".format( row['Date'], row['Coord-System'], row['Altitude'], row['Latitude'], row['Longitude'], row['D_deg'], row['D_min'] ),
                file=details )
            
            date= self.parse_date( row['Date'] )
            lat= self.parse_lat_lon( row['Latitude'] )
            lon= self.parse_lat_lon( row['Longitude'] )
            alt= self.parse_altitude(row['Altitude'] )
            
            x, y, z, f = igrf11syn( date, lat*math.pi/180, lon*math.pi/180, alt, coord=row['Coord-System'] )
            D = 180.0/math.pi*math.atan2(y, x) # Declination 

            deg, min = deg2dm( D )
            
            if details: 
                print( "Result: {0:10.5f} {1} K{2:<6.1f} {3:<10.3f} {4:<10.3f} {5:5s} {6:5s}".format( date, row['Coord-System'], alt, lat, lon, str(deg)+"d", str(min)+"m" ), 
                    file=details )
                print( file=details )
            
            self.assertEqual( row['D_deg'], "{0}d".format(deg) )
            self.assertEqual( row['D_min'], "{0}m".format(min) )

    def suite():
        s= unittest.TestSuite()
        with open(sample_output,"r") as expected:
            rdr= csv.DictReader( expected, delimiter=' ', skipinitialspace=True )
            for row in rdr:
                case= Test_Geomag( row )
                s.addTest( case )
        return s

    r = unittest.TextTestRunner(sys.stdout)
    result= r.run( suite() )
    sys.exit(not result.wasSuccessful())

The Test_Geomag class does two things.  First, it parses the source values to create a usable test case.  We've omitted the parsers to reduce clutter.  Second, it produces details to help with debugging.  This is reverse engineering, and there's lots of debugging.  It depends on a global variable, details, which is either set to sys.stderr or None.

This suite() function builds a suite of test cases from the input file.

The unit under test isn't obvious, but there's a call to the igrf11syn() function where the important work gets done.  We can start with this.

def  igrf11syn( date, nlat, elong, alt=0.0, coord='D' ):
    return None, None, None, None

This lets us run the tests and find that we have work to do.

Reverse Engineering

The IGRF11.F fortran code contains this IGRF11SYN "subroutine" that does the work we want.  The geomag 7.0 package has a function called shval3 which is essentially the same thing.

Both are implementations of the same underlying "13th order spherical harmonic series" or a "truncated series expansion".

The Fortran code contains numerous Fortran "optimizations".  These are irritating hackarounds because of actual (and perceived) limitations of Fortran.  They fall into two broad classes.

• Hand Optimizations.  All repeated expressions were manually hoisted out of their context.  This is clever but makes the code particularly obscure.  It doesn't help when local variables are named ONE, TWO and THREE.  Bad is it is, not much needs to be done about this.  Python code looks a bit like Fortran code, so very little needs to be done except add `math.` to the various function calls like sort, cos and sin.
• Sparse Array Chicanery.  There are actually two spherical harmonic series.  The older 10-order and the new 13-order.   Each model has two sets of coefficients: g and h.  These form two half-matrices plus a vector.  The old models have 55 g values in one matrix, 55 h values in second matrix, and a set of 10 more g values that form some kind of vector; 160 values.  The new models have 91 g, 91 h and 13 g in the extra vector; 195 values.  There are 23 sets of these coefficients (for 1900, 1905, ... 2015).  The worst case is 23×195=4,485 values.  This appears to be too much memory, so the two matrices and vectors are optimized into a single opaque collection of 3,256 numbers and delightfully complex set of index calculations.

Phase 1.  Do the smallest "literal" transformation of Fortran to Python.

This means things like this:

• Transforming the subroutine into a Python function with multiple return values.
• Reasoning out the overall "steps".  There's a bunch of setup followed by the essential series calculation followed by some final calculations.
• Locating and populating the global variables.
• Reformatting the if statements.
• Removing the GOTO's.  Either make them separate functions or properly nest the code.
• Reformatting the do loop.
• Handling the 1-based indexing.  In almost all cases, Fortran "arrays" are best handled as Python dictionaries (not lists).  

Once this is done, there are some remaining special-case discrepancies.  Most of these are tacit assumptions about the problem domain that turn out to be untrue.  For example, the Geodetic, Geocentric features seemed needless.  However, they're not handled trivially, and need to be left in place.  Also, conversion of signed values in radians to degrees and minutes isn't trivial. 

This leads to passing all 16 unit tests with the single opaque collection of 3,256 numbers and delightfully complex set of index calculations. 

Phase 2.  Optimize so that it makes some sense in Python.

This involves unwinding the index calculations to simplify the array.  The raw coefficients are available (igrf11coeffs.txt) and they have a sensible structure that separates the two matrices very cleanly.  The code uses the combined matrix (called gh) in a very few places.  The index calculations aren't obvious at all, but a few calls to print reveal how the matrix is accessed.  

Given (1) unit tests that already work and (2) the pattern of access, it's relatively easy to hypothesize a dictionary by year that contains a pair of simple dictionaries, g[n,m] and h[n,m], for the coefficients.

Cleanup and Packaging

Once the tests pass, the package -- as a whole -- needs to be made reasonably Pythonic.   In this case, it means a number of additional changes.  For example, converting the API from degrees to radians, supplying appropriate default values for parameters, providing convenience functions.

Additionally, there are Python ways to populate the coefficients neatly and eliminate global variables.  In this case, it seemed sensible to create a Callable class which could load the coefficients during construction.

Note that there's little point in profiling to apply further optimizations.  The legacy Fortran code was already meticulously hand optimized.  

Test-Driven Reverse Engineering (TDRE)

Another case study on TDRE.

Provided: 2,938 lines of Python code which process a handful of large files to create a number of outputs. [Details can't be disclosed.]

Objective: Refactor to distinguish between the overall sequence of transformational steps and the details of each individual step.

Observations

The code is almost purely procedural. There are 11 class definitions. 6 of these wrap built-in types with type conversion and null-handling. 1 is a new exception. 1 is a generic "table" that essentially duplicates features of SQLite. The remaining 3 are actually part of the problem domain.

One reason for reverse engineering is that the code has reached an intellectual limit. It's small, but "dense" with highly-optimized processing steps. The cohesion type is almost all "Temporal". Processing is grouped into successive processing loops; each loop contains a cluster of processing steps. Consequently, it's quite hard to tease apart the algorithm to get a "big picture" of what's going on. It's just a dense stand of trees. No forest.

Another reason for reverse engineering is to support the endless adaptation and modification of the code base. The program is a kind of "spreadsheet on steroids". This isn't a simplistic collection of cells and formulæ that permits simple what-if analysis. This is a more complex set of formulæ that would be challenging (but not impossible) to implement as a spreadsheet. The use case, however, is the spreasheet use case: think, tweak, create results, repeat.

TDRE Approach

Start with an Initial Survey of the legacy code base and sample files.

Create an Outline or "sketch" of the domain model and main program. This will be a modules (or a package) with comments and some preliminary class definitions. Little more.

Pick a processing Step in the legacy code. This often requires creating processing summaries of the legacy code. Most legacy code is procedural, so the processing tends to be sequential in nature.

Instrument the Legacy Code with print statements to gather data. This can be simple. The output can be challenging to interpret.

with open("tdre_results_1","w") as tdre:

# some legacy processing

print( "Case:", foo, bar, ", Expect:", baz, file=tdre )

From the output, Build Unit Test Cases. Fill in parts of the processing sequence and domain model. Debug code until the tests pass.

Initial Survey

The Initial Survey locates several things.

1. The usable, working modules. It appears that all reverse engineering involves a code base with dead or unused code. Even a small project (3,000 lines) will have a remarkable amount of dead code.
2. Priorities for the implemented functionality. Not every "main" module is relevant.
3. Example inputs and outputs.

If the software cannot be run (as is the case with organically developed systems that depend on large, complex corporate databases), then the example inputs and outputs may not actually match the software. If the software can be run, it should be run and the actuals compared against the samples to confirm that the code base supplied really produced the sample outputs.

Expect that the provided legacy code is slightly different from the code in production use. In some cases, this cannot be resolved; for example, when the executables are older than the source. In other cases, the code matches and no further work is required to establish the legacy baseline.

The sample outputs point in the direction of an acceptance test case. The sample output cannot be taken literally as the one-and-only acceptance test. While it's desirable for reverse engineering to reproduce the sample output, most reverse engineering will involve enhancements or bug fixes. Expect that errors will be found (or may be known to exist) in the sample output.

Create Outline

The outline is -- initially -- just generic MVP. There must be a domain model, some "presenter" that has the application logic, and some "view" for displaying the outputs.

In our case study, above, the "view" is a collection of (mostly text) output files. The model was undefined in the legacy code, which was all "presenter" application logic.

The goal was to extract the underlying model, break the application "presenter" logic into two layers (forest and trees) and build some views for each of the output files.

Pick a Processing Step

This can be challenging, depending on the legacy code base. There are two paths through a procedural code base.

• Back to Front. Start with the final results and unit test the final steps based on previous steps that will be defined later.
• Front to Back. Start with the first recognizable intermediate result based on the input files. Unit test the initial steps.

It's more rewarding to work front-to-back because progress can be shown a little more clearly.

A better architecture can be created by working back-to-front since dependencies are easier to understand.

Unit Test Volume, Edges and Corners

There are two unit test design challenges when doing reverse engineering.

• Volume. The sample data can be large. 100,000 rows of sample data is too many to test. Finding a "representative" subset is difficult. Generally, arbitrary subsets have to be used to get started. Once the application mostly works, more refined unit tests need to be created.
• Edge and Corner Cases. While the code may be riddled with if-statements, it can still be difficult to locate sample inputs that exercise the various conditions in the code. It's risky to create data -- we have to assume that the legacy code does unexpected things. In many cases, print statements have to be put into complex if statements to locate any actual data that exercises that logic path.

Once the unit tests are built, this is just Test-Driven Development (TDD).