Moved

Moved. See https://slott56.github.io. All new content goes to the new site. This is a legacy, and will likely be dropped five years after the last post in Jan 2023.

Tuesday, August 24, 2021

Spreadsheets, COBOL, and Schema-Driven File Processing

I need to rewrite Stingray Reader. This project handles a certain amount of file processing using a schema to assure the Logical Layout is understood.  It handles several common Physical Formats:

  • CSV files where the format is extended by the various dialects options.
  • COBOL files in ASCII or EBCDIC.

The project's code can be applied to text files where a regular expression can yield a row-level dictionary object. Web server log files, for example, are in first normal form, but have irregular punctuation that CSV can't handle. 

It can also be applied to NDJSON files (see http://ndjson.org or https://jsonlines.org) without too much work. This also means it can be applied to YAML files. I suspect it can also be applied to TOML files as a distinct physical format.

The complication in the Singran Reader is that COBOL files aren't really in first normal form. They can have repeating groups of fields that CSV files don't (generally) have. And the initial data model in the project wasn't really up to handling this cleanly. The repeating group logic was patched in.

Further complicating this particular project was the history of its evolution. It started as a way to grub through hellishly complex CSV files. You know, the files where there are no headings, or the headings are 8 lines long, or the files where there are a lot of lines before the proper headings for the data. It handled all of those not-first-normal-form issues that arise in CSV world.

I didn't (initially) understand JSON Schema (https://json-schema.org) and did not leverage it properly as an intermediate representation for CSV as well as COBOL layouts. It arose as a kind of after-thought. There are a lot of todo's related to applying JSON Schema to the problem. 

Recently, I learned about Lowrance USR files. See https://github.com/slott56/navtools in general and https://github.com/slott56/navtools/blob/master/navtools/lowrance_usr.py for details. 

It turns out that the USR file could be described, reasonably well, with a Stingray schema. More to the point, it should be describable by a Stingray schema, and the application to extract waypoints or routes should look a lot like a CSV reader.

Consequences

There are a bunch of things I need to do.

First, and foremost, I need to unwind some of the COBOL field extraction logic. It's a right awful mess because of the way I hacked in OCCURS DEPENDING ON. The USR files also have numerous instances of arrays with a boundary defined by other content of the file. This is a JSON Schema Extension (not a weird COBOL special case) and I need to use proper JSON schema extensions and attribute cross-references.

Of course, the OCCURS DEPENDING ON clauses can nest, leading to quite complex navigation through a dynamically-sized collection of bytes. This is not done terribly well in the current version, and involves leaving little state reminders around to "simplify" some of the coding.

The field extractions for COBOL apply to binary files and should be able to leverage the Python struct module to decode individual fields. We should be able to also extract data from USR files. The schema can be in pure JSON or it can be in Python as an internal data structure. This is a new feature and (in principle) can be applied to a variety of binary files that are in (approximately) first normal form. 

(It may also be sensible to extend the struct module to handle some EBCDIC conversions: int, float, packed-decimal, numeric string, and alphanumeric string.)

Once we can handle COBOL and USR file occurs-depending-on with some JSON Schema extensions, we can then work on ways to convert source material (including JSON Schema) to the internal representation of a schema.

  1. CSV headers -> JSON Schema has an API that has worked in the past. The trivial case of first-line-is-degenerate-schema and schema-in-a-separate-file are pleasant. The more complex cases of skip-a-bunch-of-prefix-lines is a bit more complex, but isn't much of a rewrite. This recovers the original feature of handling CSV files in all their various incarnations and dialects with more formally defined schema. It means that CSV with type conversions can be handled.
  2. Parse COBOL DDE  -> JSON Schema. The COBOL parser is a bit of a hacky mess. A better lexical scanner would simplify things slightly. Because the field extraction logic will be rebuilt, we'll also have the original feature of being able to directly decode Z/OS EBCDIC files in Python.
This feels ambitious because the original design was so weak.

Tuesday, August 17, 2021

I Have Code That Didn't Work. What Now?

I don't get many of these "I have code that doesn't work" requests. But I do see them once in a great while.

It might be something like the following two-part explanation with a following question.

Tuesday, August 10, 2021

Why Python Is Weird For C++ Developers -- Some Thoughts

See 9 Reasons Why Python Is Weird For C++ Developers.

I'm often bothered by inter-language comparisons. Mostly because programming languages -- except in the most abstract way -- aren't really very comparable. At the Turing Machine level the finite state automata are comparable, but that reductionist view (intentionally) eliminates all the expressive power from a given language.

Let's look at the reasons in some detail. A few of them actually are interesting.

  1. Whitespace. I'm dismissive of this as an interesting difference. When I read code in EVERY other programming language, I'm immediately aware that programmers can indent. Indeed, I've seen C and C++ code were {}'s were omitted, but the code was indented properly, making it devilishly hard to debug. My experience is that folks get the indentation right BEFORE the get the {}'s right.
  2. Syntax. In this article, it's the lack of {}'s. Again, I'm dismissive because I've actually helped folks learning C++ who had the indentation right and the {}'s wrong. This is ony "weird" if you're absolutely and completely convinced that {}'s are somehow a divine requirement that transcends all human attempts at interpretation. With Unicode, we're in a position to separate set membership from block-of-code and start using multiple variants on {}'s.  I'd vote for if a > b 【m = a】else 【m = b】 using 【】for code blocks.
  3. Class Variables. This points out an inherent ambiguity of C++. Most of the time, most things are not "static". They're "automatic" that is, associated with the instance. The auto keyword, however, is rarely used, and is mostly assumed. Python (outside dataclasses) is more consistent. All things inside the class statement are "static": part of the class. In the case of dataclasses, this simple rule is broken, which can be confusing. But. This wasn't mentioned.
  4. Pointer and Reference Transparency. This is simple confusion. All Python is handled by reference all the time. C++ is an absolute mess of "primitive" types that don't use references and objects that do use references. Java is just as bad. And I want to emphasize bad. Python is perfectly consistent, and -- I would suggest -- the opposite of weird. But. The article is describing things from a C++ perspective, as if C++ were somehow not weird. I suggest this isn't a great approach.
  5. Private Class Members. This is summarized as "better encapsulation and control" without a concrete example. It's hard to provide a concrete example because the Pythonic approach works so well. The only use case for "private" that I've been able to understand is when you're concealing the entire implementation from all scrutiny. That is, you have a proprietary implementation with an encrypted JAR file and you want to avoid revealing it to protect some intellectual property. Since Python is source, this can't happen, and we say "We're all adults here." Flag it with a leading _ and we'll recognize it as part of an implementation detail that might change. 
  6. Self vs. this. Not sure what this is but the phrase "only major programming language" is something that relies on Java and C++ being near the top of the TIOBE index. I suspect we can find a lot of languages that use neither "self" nor "this". I'm not sure exactly how this is weird, but, I get that it's different.
  7. Multiple Return Values. This seems like an intentional refusal to understand how tuples and tuple unpacking work. Again, this seems to make C++ the yardstick when C++ is clearly kind of weird here.
  8. No Strong Data Types. This seems like another refusal to understand Python. In this case, it feels like it's a refusal understand that objects are strongly typed in Python and variables are transient labels attached to objects. The mypy tool will try to associate a type with a variable and will warn you about a = "string" followed by a = 42. Perhaps I'm not understanding, but the portrayal of C++ rules as "not weird" seems like it's being taken too far. 
  9. No Constants. This isn't completely true. Some folks use enums to provide enumerated numeric constant values in the rare cases where this might matter. Using global variables as constants actually works out fine in practice. Most tools will look for ALL_CAPS names on the left of an = sign; and if this occurs more than once will raise a warning. If you have really stupid fellow programmers who can't understand how some variables shouldn't be reused, you can easily write a script to walk the AST looking for references to global variables and warn your colleagues that there are rules and they're not following them. This is part and parcel of the "We're all adults here" approach. If folks can't figure out how constants work, you need to collaborate more fully with other developers to help them understand this.

I'm unhappy with lifting up C++ quirks as if they're somehow really important. I don't think C++ is a terribly helpful language. The need for explicit memory management, for example, is a terrible problem. The explicit distinction between primitives and objects is also terrible.

While compare-and-contrast with Python might be helpful for C++ expatriates, I think this article has it exactly backwards. I think the following list couuld be more useful.

  • Python frees you from counting {}'s. Just indent. It's easier.
  • Python has simple rules for class/instance variables (except in the case of dataclasses and named tuples.) Also: if it starts with self. it's an instance variable.
  • Python is all references without the horrifying complexity of primitive types.
  • We're all adults here. Don't stress yourself out over privacy or constants. Document your code, instead. Write a unit test case or two. Use mypy. Use black.
  • Tuple unpacking and the fact that tuples are often implied works out very nicely to create very clean code.
  • Data types are part of the object. There's no magical "cast" capability to process a block of bytes as if they're some other type. 
These are advantages of Python. And disadvantages of C++. I think it's better to talk about what Python has than what Python lacks when measured against a terribly complex language like C++.


Tuesday, August 3, 2021

Writing Interactive Compute-Intensive Programs for Web Browsers

Fascinating. The reference to the classic Mac OS with non-preemptive multi-tasking is quite cool. The concept fits nicely with Python's async/await coroutines that need to collaborate with a periodic OS request to permit interaction with streams of events from another source (i.e., a foreground window.) 



Tuesday, July 27, 2021

SOLID Coding in Python

SOLID Coding in Python by Mattia Cinelli.

Download Medium on the App Store or Play Store

This was fun to read. It has some nice examples. 

I submit that the order of presentation (S, O, L, I, D) is misleading. The acronym is fun, but awkward.

My LinkedIn Learning course covers these in (what I think is) a more useful order.

  1. Interface Segregation. I think this is the place to start: make your interfaces as small as possible.
  2. Liskov Substitution. Where necessary, leverage inheritance.
  3. Open/Closed. This is a good quality check to be sure you've followed the first two principles well.
  4. Dependency Injection. This is often about test design and future expansion. In Python, where everything really happens at run time, we often fail to parameterize a type properly. We often figure that out a test time, and need to revisit the Open/Closed principle to get things right.
  5. Single Responsibility is more of a summary of the previous principles than a distinct, new principle. I think it comes last and should be treated as a collection of good ideas, not a single idea.

I think time spent on the first three -- Interface Segregation, Liskov Substitution, and the Open/Closed principle -- pay off tremendously. The ILODS acronym, though, isn't as cool as SOLID.

The "Single Responsibility" suffers from an ambiguous context. At one level of abstraction, all classes have a single responsibility. As we dive into details, we uncover multiple responsibilities. The further we descend into implementation details the more responsibilities we uncover. I prefer to consider this a poetic summary, not the first step in reviewing a design.

Tuesday, July 20, 2021

How can people find inspiration at work? #CreateMeaning

What do I know about "inspiration" at work? I'm not sure I know much, but I think I may have some advice that could be useful.

I was in the high-tech write-software-every-day workplace since about 1976 or so. (The first two years were part-time while in college.) I use past perfect "was" because I'm old enough (and lucky enough) to be able to retire from daily work. I've switched from writing code to writing books about writing code.

For the math-impaired, my software career spanned 45 years.

Early in this career, the question of finding inspiration at work wasn't asked in the same stark way people discuss it nowadays. When I was younger, the idea of maintaining a work/life balance wasn't something we were asked or encouraged to consider. We did the best we could and tried to avoid getting replaced by someone who could do it better.

Which -- with the advantages of hind-sight -- was a terrible way to live and work. Simply awful. I was lucky enough to see hundreds of projects in my working career. I worked with scores of different organizations. There was a spectrum of bad behavior.

I did learn this: Fear is not Inspiring. I learned a few other things, but let's start with fear.

The Fear Factor

I want to dwell a bit on the fear factor in the workplace. I'm firmly convinced that some manager types suffer with a nagging background of essential fear for their own jobs. And they can project this fear onto folks around them.

Try these shoes on for a moment. The technology has moved on and you haven't kept up. You're trying to manage people, but you have a nagging suspicion your core managerial skills are weak enough that you could be replaced. Motivated by fear, you encourage "casual overtime" and "working weekends" and "meeting the committed schedule". Perhaps you feel it necessary to go so far as to demand these things.

Fear of getting fired creates an uninspiring place to work. It will be an incubator for burnout. 

Further, I suspect it can lead to worse situations than people quitting. I think the "work a little harder" folks plant seeds for various kinds of workplace abuses. 

I think there are a variety of fears. The fear of getting fired is at the bottom of Maslow's hierarchy of needs: we could get fired, and be unable to get another job. We're in the Physiological and Safety realm of the needs pyramid.

A fear of a project not working out means we'll tumble in the eyes of more senior management. This is somewhere in the higher level of Social Belonging and Esteem needs. The problem is that projects have a variety of metrics, and simply making the schedule is an easy metric and can seem to lead to immediate esteem.

What about higher level cognitive needs like Self-actualization and Transcendence? I strongly suspect fears related to these needs can color someone's workplace. I think these often show up as "Am I really going to be doing this for the Rest of My Life?" questions. This becomes an undercurrent of negativity stemming from fear of being trapped in unfulfilling work. 

We might see these fears in several places. We each harbor our own private fears. In any organization with a hierarchy, we'll have to deal with fears that trickle down to us from supervisors. In non-hierarchical organizations, we'll have to deal with fears of our peers and colleagues. We're surrounded by fears, and I think this can sap our inspiration.

What can we do to find inspiration in a work environment?

My Experience

I've worked on hundreds of projects. That means hundreds of jobs that came to an end. And when the project ended, I was no longer needed. 

In effect, I was fired hundreds of times.

This isn't a helpful thing. If Maslow's base physiological needs are met, then having a project end isn't too horrible. I was a contract programmer in the olden days when we were salaried, and the company would carry us from assignment to assignment. Being let go by a customer can be harsh, but getting paid in spite of being let go softens the blow.

I emphatically do not recommend this way of working as a source of inspiration. Some people like the constant changing gears and changing directions. Other people might find it terrifying: each project is a whole new group of people in a new organization. Potentially very unsettling.

I don't think the "get tougher" or "grow thicker skin" advice is good. I'm don't think it worked out for me. I think this kind of transience left me feeling isolated. I think it lead me to carry around a sense of superiority. So. Let's set aside any dumb-sounding advice based on a literal review of my experience. 

How Did I Cope?

Finding ways to cope, I think, is important, but it is also potentially misleading. The idea of coping with new projects, new organizations, travel, and getting fired all the time isn't inspiring. It's merely coping with an endless stream of loss and separation. 

Underpinning the idea of coping is a more foundational question. Where did I find the inspiration to keep on doing this contract programming thing for so many years? And the other question is how well my search for inspiration might apply to folks who aren't commuting computer programmers?

I think there's a first step that many people can take. It's this:

We can disentangle our self-worth from the work-place imposed sense of worth.

This may be overly glib. But. I think the things rewarded in the workplace aren't a good reflection of who we are and what we're capable of. While it's important to be confident in one's self, our confidence can be undermined by a toxic workplace. Having confidence can let us take our skills and abilities in a variety of directions. We might, for example, decide to find another workplace; one where our value is recognized. Or, we might decide to change our circumstances in the workplace we're currently inhabiting. In both cases, we're asserting our value. We're making a further claim about our value: we may not match the workplace's expectations of us. The workplace can change, or we can find another workplace.

We might see a mismatch in lousy performance reviews. These can can stem from many causes. Perhaps we're not suited for a job and need to find something else. Or, perhaps the person reviewing our work doesn't see what we could (or should) be doing. (They have their own fears, and they may not be willing to try to make the changes we'd like them to make.)

Looking back, I may have been doing this all along, without being clear or intentional about it. Perhaps I excelled at places that valued me, and failed to meet expectations at places that treated me poorly. Perhaps my job shifting was (in an indirect way) a search for a workplace that valued me, my unique experiences, and my distinctive voice.

I was not intentional about it. I stumbled from job to job, knowing the sales folks would find me a next assignment when the current assignment had run its course. I think a vague sense of self-worth is what lead me to locating inspiration in spite of a lot of change and disruption.

Finding Inspiration

When we think of inspiration, we think of a spiritual drive to do the work. This doesn't often parallel with working for pay to cover rent and expenses. 

A good manager, however, can create a cohesive team from a group of people. A group of peers can welcome a new colleague. This creates belonging and esteem: the middle levels of Maslow's hierarchy of needs. We may find that a team or a team's goal may be inspiring. This means that our own self-worth is recognized and valued by our co-workers. This can be a marvelous experience.

What about the bad manager or unhelpful group of colleagues? In this cases, we're forced to make the best of an awkward situation. I think we can do this:

We can search for inspiration at the margins of our work life. 

Can we find some side-bar aspect of the work that leads to some helpful insights? Perhaps there is a chronic problem we can take notes on and -- eventually -- fix. Perhaps someone is less helpful that others, and we can try to understand what would make them less toxic. Perhaps cleaning the break-room fridge is better than complaining about month-old food. (Yuck. But. If things are better, it may be worth it.)

For years, I had an aspiration to write about software development. To further this dream, I started taking more and more careful notes of projects I was on. In the era before the World Wide Web, publication was difficult, but not impossible. I wrote small articles for technical magazines; this effort was something that inspired me to work with customers who were inept and had horrible, horrible problems. I liked the awful customers because it provided me good examples of things that should not be done.

At the end of a horrible project, I'd have a good anecdote for what not to do.

I acknowledge my two ideas of self worth and inspiration isn't a dramatic, life-changing epiphany. I'm pretty sure the scales won't fall from anyone's eyes as they think about looking at sidebar topics as a source of inspiration.

Looking at the margins, edges, and corners of a job can help to reveal the whole job. The whole team. The whole goal. Finding this broader view might inspire us to look for a better team with better goals. In other cases, it might help us find the missing skills in the team we're on. In other cases, a better perspective might help us steer our supervisor toward doing something that's better than what they're doing right now.

There are very fine lines between toxic, poorly organized, poorly managed, confusing work places, and workplaces that are still trying to find a workable organization. Most places have a combination of good and bad, inept and well-done, confusing and sensible features. Indeed, these may all be different axes and an organization is really a multi-dimensional object with different kinds of overlaps and gaps. 

I believe the foundation for inspiration is a clear sense of self-worth. I think we create meaning in our workplace by knowing what we can contribute, what we want to contribute, and what the organization needs. Our unique contribution and what the organization needs may not overlap at all, or the organization may have always been searching for someone like us. Either way, our awareness of our skills, our experience, and our authentic voice is what lets us find inspiration.

Tuesday, July 13, 2021

What Books Should I Read? In What Order?

A fascinating question.

... I'm baffled by the amount of books you've published over the course of time. Currently Reddit suggests that I use Building Skills in Python under Beginner's section, but it looks quite outdated. So back and forth, I found your Building Skills in OO on GitHub Page and was quite happy with the read on the first 100 pages.

I searched for more info on the books you've published and wanted to know if you could sort them in ascending order of difficulty for me as I intend to purchase them slowly.

My main concern to learn Python is just to cross technical interviews and building applications that help with my workflow (they are in bash with around 200 functions, so I'm hoping to migrate them to something which is more robust). 

Currently the focus I intend to develop is on:
1) Strong Foundations of the Python Language.
2) Strong Foundations on the Basic Libraries for Data Structures and Algorithms (For example, bisect gives me insort(), calendar gives me isleap(), iter_tools gives me permutation(), etc).
3) Strong Foundations on the Design Patterns.

So could you please help me out and suggest your books?

This is challenging for a few reasons.

First, the "Building Skills" books have been reduced to only the Building Skills in OO Design. This can be found in GitHub. https://github.com/slott56/building-skills-oo-design-book.

That book is not really targeted to beginners, though. It presumes some core OO skills, and provides a (very) long series of exercises to build on those skills.

Second, I never really conceived of a beginner-to-expert sequence of books. From your letter, I see that I need to look at filling in some gaps in my sequence of books. I'll alert my editors at Packt, and we can consider this in the future.

Specific Needs

Let's look at your needs.

1. Foundations in the Python language.

This might be something you can learn from my Python Essentials. This isn't focused on complete n00bs. All of my books expect some programming background. Since you're an Android engineer and write code in C++ and Java, this may be helpful. This title is getting old, however, and needs a second edition.

For someone with core programming skills, I suspect that Mastering OO Python will be suitable. My Python 3 OO Programming (4th ed.) is similarly aimed at folks who can program and can learn a new language quickly.

A book like Martelli's Python in a Nutshell may provide a better foundation the way the language works than any of mine. Also Lutz's Learning Python is extremely popular.

2. Foundations in the Standard Library.

This is tricky. I touch on some of these topics in Functional Python Programming (2nd ed.) I also touch on some of these topics in the Modern Python Cookbook (2nd ed.)

I don't, however, cover very much of the library. I touch on a few really important modules. The library is vast. A book like Hellmann's The Python Standard Library by Example might be more suitable than one of mine.

3. Design Patterns.

This is central to Python 3 OO Programming (4th ed.) Dusty Phillips and I cover a number of popular design patterns in detail. 

There are -- of course -- a lot of very, very good books on Python. I'm honored you reached out to me.

Other Random Advice

Because Python is a relatively simple language (with a vast library) I have always suspected that language foundations don't really require a ton of explanation. Many languages (i.e., C++) are filled with odd details and weird features that are really unpleasantly complex. Many Java programmers get used to the distinction between the primitive int type and the Integer class type. While the Java and C++ approach can seem simple (after a while) it really isn't simple at all.

The standard library is vast, and it takes time to get used to how much is there. I would suggest having a browser tab open to https://docs.python.org/3/library/ at all times.

Design patterns, similarly, require some care. There are complex details around implementing the Singleton pattern in C++ and Java. Python class definitions and Python module definitions are Singletons, and using a class definition as a Singleton object is often far simpler than the commonly-used techniques for C++ and Java.