Coming Soon: Python for Secret Agents Part II

I guess it's like a movie franchise or a series of novels. The first one was popular. So, write a second story with similar characters.

You can check find part I here: http://www.amazon.com/gp/product/B00N2RWMMW/ref=dp-kindle-redirect?ie=UTF8&btkr=1 and here: https://www.packtpub.com/hardware-and-creative/python-secret-agents

Part II will be available soon. New missions. New ways to gather and analyze intelligence information assets.

I should probably read some Ian Fleming or Robert Ludlum boos to get some ideas for more exciting missions.

I'm more a fan of John le Carré stories which are less high-tech and more about ordinary selling out.

I'm also a fan of the history of Agent Garbo and Operation Mincement. These are things that are really interesting uses of data, intelligence, and misdirection.

Writing About Code -- Or -- Why I love RST

I blog. I write books. I write code. There are profound tool-chain issues in all three of these. Mostly, I'm tired of shabby "What You See Is All You Get" editing.

First. I use this blogger site as well as a Jive-based site at work. They're handy. But. There are a lot of issues. A lot. Web-based editing leaves a lot to be desired.

Second. Books. Packt requires MS-Word for drafts. The idea here is that authors, editors, and reviewers should all use a single tool. I push the boundaries by using Libre Office and Open Office. This works out most of the time, since these tools will absorb the MS-office style sheet that Packt uses. It doesn't work out well for typesetting math, but the technical editors are good about tracking down the formulae when they get lost in the conversions. These over-wrought do-too-much word processing nightmares leave a lot to be desired.

Third. Code. I use ActiveState Komodo Edit.  Both at work and outside of work. This rocks.

Web-Based Editing Fail

What's wrong with Jive or Blogger? The stark contrast between JavaScript-based text edit tools and HTML. It's either too little control or too much detail.

The JS-based editors are fine for simple, running text. They're actually kind of nice for that. Simple styles. Maybe a heading here or there.

Code? Ugh. Epic Fail.

It gets worse.

I've become a real fan of semantic markup. DocBook has a rich set of constructs available.  RST, similarly, has a short list of text roles that can be expanded to include the same kind of rich markup as DocBook. Sphinx leverages these roles to allow very sophisticated references to code from text. LaTeX has a great deal of semantic markup.

Web-based editors lack any of this. We have HTML. We have HTML microformats available. But. For a JavaScript web editor, we're really asking for a lot. More than seems possible for a quick download.

Desktop Tool Fail

What's wrong with desktop tools? We have very rich style sheets available. We should be able to define a useful set of styles and produce a useful document. Right?

Sadly, it's not easy.

First, the desktop tools are extremely tolerant of totally messed-up markup. They're focus is explicitly on making it look acceptable. It doesn't have to be well-structured. It just has to look good.

Second, and more important, the file formats are almost utterly opaque. Yes. There are standards now. Yes. It's all just XML. No. It's still nearly impossible to process. Try it.

Most word-processing documents feel like XML serializations of in-memory data structures. It's possible to locate the relevant document text in there somewhere. It's not like they're being intentionally obscure. But they're obscure.

Third, and most important, is the reliance on either complex GUI gestures (pointing and clicking and what-not) or complex keyboard "shortcuts" and stand-ins for GUI gestures. It might be possible to use that row of F-keys to define some kinds of short-cuts that might be helpful. But there's a lot of semantic markup and only a dozen keys, some of which have common interpretations for help, copy, paste, turn off the keyboard lights, play music, etc.

The Literate Programming ideal is to have the words and the code existing cheek by jowls. No big separation. No hyper-complex tooling. To me, this means sensible pure-text in-line markup.

Text Markup

I find that I really like RST markup. The more I write, the more I like it.

I really like the idea of writing code/documentation in a simple, uniform code-centric tooling. The pure-text world using RST pure-text markup is delightfully simple. 

1. Write stuff. Words. Code. Whatever. Use RST markup to segregate the formal language (e.g. Python) from the natural language (e.g., English in my case.)
2. Click on some icon the right side of the screen (or maybe use an F-key) to run the test suite.
3. Click on some icon (or hit a key) to produce prettified HTML page from python3 -m pylit3 doc.py doc.rst; rst2html.py doc.rst doc.html. Having a simple toolchain to emit doc from code (or emit code from doc) is a delight.

The genesis for this blog post was an at-work blog post (in Jive) that had a code error in it. Because of Jive's code markup features (using non-breaking spaces everywhere) there's no easy copy-and-paste to check syntax. It's nearly impossible to get the code off the web page in a form that's useful.

If people can't copy-and-paste the code, the blog posts are approximately worthless. Sigh.

If I rewrite the whole thing into RST, I lose the Jive-friendly markup. Now it looks out-of-place, but is technically correct.

Either. Or.

Exclusive Xor.

Ugh. Does this mean I have to think about gathering the Jive .CSS files, and create a version of those that's compatible with the classes and ID's that Docutils uses?  I have some doubts about making this work, since the classes and ID's might have overlaps that cause problems.

Or. Do I have to publish on some small web-server at work, and use the <iframe> tag to include RST-built content on the main intranet? This probably works the best. But it leads to a multi-step dance of writing, publishing on a private server, and then using a iframe on the main intranet site. It seems needlessly complex.

Lynda and Educational Content

Just found http://www.lynda.com.

Unlike random YouTube videos, these are professionally edited.

Not everything on YouTube is poorly edited. Some are really good.

Having done a few webcasts for O'Reilly (and I have another scheduled for January 2016,) I know that my "you knows" -- you know -- and my "umms" are -- umm -- annoying.

I know professionals -- actors, pastors, lawyers -- who can extemporize really well. And it raises the bar a lot.

But the idea of having an editor clean up the "you knows" is appealing.

Coding Camp vs. Computer Science

Step 1, read this: "Dear GeekWire: A coding bootcamp is not a replacement for a computer science degree".   It's short, it won't hurt.

I got this comment.

"The world runs in legacy code and the cs degrees focus on leading edge 

Most of what is learned in cs [is] never used in the mainstream of business 

Much of computer work is repetitive and uninviting to upwardly mobile people who generally are moving up not improving the breed"

I disagree.  A lot.

"The world runs in legacy code." First, this is reductionist: everything that's been pushed to GitHub is now a "legacy". 

• Does "legacy" mean "old, bad code?" If so, only CS grads will be equipped to make that judgement. 
• Does "legacy" mean "COBOL?" If so, only CS grads will be able to articulate the problems with COBOL and make a rational plan to replace it with Microservices. 
• Does "legacy" mean "not very interesting?" We'll return to this.

"CS degrees focus on leading edge." Not really true at all. The foundations of CS: data structures and algorithms, logic, and computability, haven't changed much since the days of Alan Turing and John von Neumann. They're highly relevant and form the core of a sensible curriculum.  

The "leading edge" would be some Java 1.8 nonsense or some Angular JS hokum. The kind of thing that comes and goes. The point of CS education is to make languages and language features just another thing, not something special and unique. A little CS background allows a programmer to lump all SQL databases into a broad category and deal with them sensibly. A Code Camp grad who only knows SQLite may have trouble seeing that Oracle is superficially different but fundamentally similar.

"cs is never used in the mainstream of business." True for some businesses. This is completely true for those businesses where "legacy" means "not very interesting." 

There is a great deal of not very interesting legacy code that fails to leverage a data structure more advanced than the flat file. This code is a liability, not an asset. The managers that let this happen probably didn't have a strong CS background and hired Code Camp graduates (because they're inexpensive) and created a huge pile of very bad code.

I've met these people and worked at these companies. It's a bad thing. The "leadership" that created such a huge pile of wasteful code needs to be fired. The "all that bad coded evolved during the 70's and 80's" isn't a very good excuse. A large amount of not interesting code can be replaced with a small amount of interesting code quickly and with almost zero risk.

Any company that's unable to pursue new lines of business because -- you know -- we've always done X and it's expensive to pivot to Y is deranged. They're merely holding onto their niche because they're paralyzed by fear of innovation=failure.

"Much of computer work is repetitive".  False. It's made repetitive by unimaginative management types who like to manage repetitive work. If you've done it twice, you need to be prepared to distinguish coincidence from pattern. When you've done it three times, that's a pattern, and you need to automate it. If you do it a fourth time, you're missing the opportunity to automate, wasting money instead of investing it.

"Much of computer work is ... uninviting to upwardly mobile people" Only in places where repetitive is permitted to exist.  If repetitive is not permitted, upward mobility will be the norm for the innovators.

"people who generally are moving up not improving the breed". I get this. The smart people move on. All we have left in this company are Code Camp graduates and their managers who value repetitive work and large volumes of not interesting code. 

Improving the Breed means what? 

Hiring CS graduates instead of Code Camp kiddies.

Navigation: Latitude, Longitude, Haversine, and all that

For a few years, I was a tech nomad. See Team Red Cruising for some stories of life on a sailboat. Warning: it's pretty dull.

As a tech nomad, I lived and died (literally) by my ability to navigate. Modern GPS devices make the dying part relatively unlikely. So, let's not oversell the danger aspect of this.

The prudent mariner plans a long voyage with a great deal of respect for the many things which can go wrong. One aspect of this is to create a "Float Plan". Read more about it here: http://floatplancentral.cgaux.org.

The idea is to create a summary of the voyage, provide that summary to trusted shore crew, and then check in periodically so that the shore crew can confirm that you're making progress safely. Failure to check in is an indicator of a problem, and action needs to be taken. We use a SPOT Messenger to check in at noon (and sometimes at waypoints.)

Creating a float plan involved an extract of the waypoints from our navigation software (GPS NavX). I would enrich the list of waypoints with estimated travel time between the points.  Folding in a departure time would lead to a schedule that could be tracked. I also include some navigation hints in the form of a bearing between waypoints so we know which way to steer to find the next point.

The travel time is the distance (in  nautical miles) coupled with an assumption about speed (5 knots.) It's a really simple thing. But the core haversine calculation is not a first-class part of any spreadsheet app. Because of the degrees-to-radians conversions required, and the common practice of annotating degrees with a lot of internal punctuation (38°54ʹ57″ 077°13ʹ36″), it becomes right awkward to simply implement this as a spreadsheet.

Some clever software has a good planning mode. The chartplotter on the boat can do a respectable job of estimating time between waypoints. But. It's not connected to a computer or the internet. So we can't upload that information in the form of a float plan. The idea of copying the data from the chart plotter to a spreadsheet is fraught with errors.

Navtools

Enter navtools. This is a library that I use to transform a route into a .csv with distances and bearings that I can use to create a useful float plan. I can add an estimated arrival time calculation so that a change to departure time creates the entire check-in schedule.

This isn't a sophisticated GUI app. It's just enough software to transform a GPS NavX extract file into a more useful form. The GUI was a spreadsheet (i.e., Numbers.) From this we created a PDF with the details.

Practically, we don't have good connectivity on the boat.  So we would create a number of alternative plans ("leave tomorrow", "leave the day after", "leave next Monday", etc.) we would go ashore, find a coffee shop, and email the various plans to ourselves. They could sit in our inbox, waiting for weather and tide to be favorable.

Then, when the weather and tides were finally aligned, we could forward the relevant details to our trusted shore crew. This was a quick spurt of cell phone connectivity to forward an email. It worked out well. When the scheduled departure time arrived, we'd coax Mr. Lehman to life, raise the anchor and away.

Literate Programming

This is an exercise in literate programming. The code that's executed and the HTML documentation are both derived from source ReStructured Text (RST) documents. The documentation for the navigation module includes the math along with the code that implements the math.

I have to say that I'm enthralled with the intimate connection between requirements, design, and implementation that literate programming embodies.

I'm excited to (finally) publish the thing to GitHub. See https://github.com/slott56/navtools.  I'm looking at some other projects that require the navtools module. What I wind up doing is copying and pasting the navigation calculation module into other projects. I had something like three separate copies on my laptop. It was time to fold all of the features together, delete the clones, and focus on one authoritative copy going forward.

I still have to remove some crufty old code. One step at a time. First, get all the tests to pass. Then expunge the old code. Then make progress on the other projects that leverage the navtools.navigation module.

Events: PyCon 2016, OSCon 2016

Many years ago ('07?) I went to my first PyCon. My situation changed and I didn't get to another PyCon until last year.

The story is a kind of major dumbosity. In '07 I could expense the trip as education. In '08, I'd lost that feature of my employment. After that I was actively figuring out how to be self-employed as a writer and technomad, and completely took my eye off the various kinds of tax deductions and sponsorship opportunities that I might have leveraged. It was too complex, arbitrary, and bewildering for me.

PyCon is an energizing event.  I can't say enough good things about attending session after session on Python and the Python-related ecosystem. In particular, it's a joy to see people pitching their solutions to complex problems.

Here's a reminder: https://us.pycon.org/2016/

Since I do some work for O'Reilly media -- if a pair of webcasts count as work -- I think I want to see if I can finagle my way into OSCon, also.

Here's the reminder: http://conferences.oreilly.com/oscon/open-source

I think I can leverage some material from Functional Python Programming to create an interesting tutorial.  My webcast on the five kinds of Python functions can expand into a bunch of hands-on-keyboard exercises to build examples of each kind of callable thingy. 

Proposals are in. Waiting for comments. Fingers crossed.

Formatting Strings and the str.format() family of functions -- Python 3.4 Notes

I have to be clear that I am obsessed with the str.format() family of functions. I've happily left the string % operator behind. I recently re-discovered the vars() function.

My current go-to technique for providing debugging information is this:

print( "note: local={local!r}, this={this!r}, that={that!r}".format_map(vars)) )

I find this to be handy and expressive. It can be replaced with logging.debug() without a second thought. I can readily expand what's being dumped because all locals are provided by vars().

I also like this as a quick and dirty starting point for a class:

def __repr__(self):
    return "{__class__.__name__}(**{state!r})".format(__class__=self.__class__, state=vars(self))

This captures the name and state. But. There are nicer things we can do. One of the easiest is to use a helper function to reformat the current state in keyword parameter syntax, like this:

def args(obj):
    return ", ".join( "{k}={v!r}".format(k=k,v=v) for k,v in vars(obj).items())

This allows us to dump an object's state in a slightly nicer format. We can replace vars(self) with args(self) in our __repr__ method. We've dumped the state of an object with very little class-specific code. We can focus on the problem domain without having to wrestle with Python considerations.

Format Specifications

The use of !r for formatting is important. I've (frequently) messed up and used things like :s where data might be None. I've discovered that -- starting in Python 3.4 -- the :s format is unhappy with None objects. Here's the exhaustive enumeration of cases. 

>>> "{0} {1}".format("s",None)
's None'
>>> "{0:s} {1:s}".format("s",None)
Traceback (most recent call last):
  File "", line 1, in 
    "{0:s} {1:s}".format("s",None)
TypeError: non-empty format string passed to object.__format__
>>> "{0!s} {1!s}".format("s",None)
's None'
>>> "{0!r} {1!r}".format("s",None)
"'s' None"

Many things are implicitly converted to strings. This happens in a lot of places. Python is riddled with str() function evaluations. But they aren't everywhere. Python 3.3 had one that was removed for Python 3.4 and up.

Bottom Line: be careful where you use :s formatting.  It may do less than you think it should do.