Tragedy Averted

I almost made a terrible blunder.

See https://github.com/slott56/py-web-tool for some background. This is a "Literate Programming" tool. I started fooling around with this kind of thing back in '05 (maybe even earlier.) This is not the blunder. The whole idea of literate programming is not very popular. I'm a fan of Jupyter{Book} as the state of the art in sophisticated literate programming, if you're interested in it.

In my case, I started this project so long ago, I used docutils. This was long before Sphinx arrived on the scene. I never updated my little project to use Sphinx. The point was to have a kind of pure literate programming tool that could work with a variety of markup languages, including (but not limited to) RST.

Recently, I learned about PlantUML. The idea of a text description of a diagram is appealing. I don't really need to draw it; I just need to specify what's in it and let graphviz do the rest. This tool is very, very cool. You can capture ideas quickly. You can refine and expand on ideas until you reach a point where code makes more sense than a picture of code. 

For some things, you can gather data and draw a picture of things *as they are*. This is particularly valuable for cloud-based infrastructure where a few queries leads to PlantUML source that is depicted very nicely.

Which leads to the idea of Literate Programming including UML diagrams. 

Doesn't sound too difficult. I can create an extension to docutils to introduce a UML directive. The resulting RST would look like this:

..  uml::

    left to right direction
    skinparam actorStyle awesome

    actor "Developer" as Dev
    rectangle PyWeb {
        usecase "Tangle Source" as UC_Tangle
        usecase "Weave Document" as UC_Weave
    }
    rectangle IDE {
        usecase "Create WEB" as UC_Create
        usecase "Run Tests" as UC_Test
    }
    Dev --> UC_Tangle
    Dev --> UC_Weave
    Dev --> UC_Create
    Dev --> UC_Test

    UC_Test --> UC_Tangle

This could be handy to have the diagrams as part of the documentation that tangles the working the code. One source for all of it. 

I started down the path of researching docutils extensions. Got pretty far. Far enough that I had an empty repository and everything. I was about ready to start creating spike solutions.

Then.

[music cue] *duh duh duuuuuuh*

I found that Sphinx already has an extension for PlantUML. I almost started reading the code to see how it worked.

Then I realized how dumb that was. It already works. Why read the code? Why not install it?

I had a choice to make.

1. Continue building my own docutils plug-in.
2. Switch to Sphinx.

Some complications:

• My Literate Programming tool produces RST that *may* not be compatible with Sphinx.
• It's yet another dependency in a tool that started out with zero dependencies. I've added pytest and tox. What next? 

What to do?

I have to say that Git is amazing. I can make a branch for the spike. If it works, pull request. If it doesn't work, delete the branch. This continues to be game-changing to me. I'm old. I remember when we had to back up the whole project directory tree before making this kind of change.

It worked. My tool's RST (with one exception) worked perfectly with Sphinx. The one exception was an obscure directive, .. class:: name, used to provide an HTML class name for the following block. This always should have been the docutils .. container:: name directive. With this fix, we're good to go.

I'm happy I avoided the trap of reimplementing something. Instead of that, I upgraded from "bare" docutils with my own CSS to Sphinx with it's sophisticated templates and HTML Themes.

Writing Tools

Read this: http://thesweetsetup.com/apps/our-favorite-pro-writing-app-for-mac/

What I have been doing instead of using these sophisticated, integrated writing tools?

I use OmniOutliner. https://www.omnigroup.com/omnioutliner I've used it for years. It does a lot of things. Most notably, I can create multiple columns so that I can create page budgets for outlines. Acquisition Editors like this. Except, of course, they like it as an DOCX file, which requires a bit of manual juggling to produce.

I use BBEdit and KomodoEdit for a the bulk of my writing. http://www.barebones.com/products/bbedit/index.html
https://www.activestate.com/komodo-ide/downloads/edit

"But wait," you say, "those are text editors."

(Or, more dismissively, "there are merely text editors.")

Correct.  I use RST markup and write in Unicode text.  I use tools to convert the RST text to a variety of other binary formats. See http://docutils.sourceforge.net/docs/user/tools.html for a list of tools. This is what I often use:

• rst2odt creates ODT files which can be then converted to DOCX.
• rst2html creates web pages.
• rst2epub creates ePub files. From there, Kindlegen can convert to Kindle-unique format. https://www.amazon.com/gp/feature.html?docId=1000765211 

How is this better than a tool like Scrivener? It depends -- as always -- on what you're trying to optimize. My pipeline has the (dubious) advantage of being very inexpensive. Except for OmniOutliner and BBEdit, it's all community-edition, free software. If cheap is your goal, I've got cheap.

The cool part is this.

The Mac OS X desktop is an integrated writing environment. I have browser, outliner, writing tool, publishing tool, etc., etc., all readily and immediately available. The "look and feel" isn't consistent, but I'm not sure that's a show-stopper.

The biggest difficulty?

BBEdit doesn't enable the Mac OS X grammar checker. Really. It's switched off. The grammar checker is sometimes handy for preventing a large number of common, dumb writing mistakes. BBEdit shows the word count, which is very helpful for some kinds of writing. I wind up using a second app (i.e. the built-in Mac OS X TextEdit) to make a grammar check pass.

I think, however, the hacker-friendly free-and-open-source tool chain may have reached the end of its service life.

Why Not Use Word?

"After all," you say, "MS-Word does everything."

Agreed. It does everything badly and confusingly. (1) The outliner is hard to use and is firmly tied to the text in a way that breaks outlines all the time. (What's that paragraph doing there? Why is it the wrong outline level?) (2) There are too many useless features. The presence of "advanced" mode is a UX nightmare come true. (3) The character-mode and paragraph-mode formatting rules are baffling (and break the outlining.) (4) The styles are essentially invisible: you have to click on the text and check the style side-bar to be sure that the (invisible) markup is actually right.

The worst thing is that publishers have house style sheets for MS-Word that drive the publishing pipeline. This means that writing involves a weird step where I have to apply the publishers styles to things that are **very** clearly annotated with RST markup. You have to review each word. The words may look right, but have the wrong style applied. This is extremely tiresome to get right.

I intend to stick with plain-text markup. Scrivener supports MultiMarkdown. It's not RST, but it seems to be as rich with built-in semantic categories.

That Feeling When... You're reading your own documentation because it's useful and (mostly) correct

I'm looking at code (as a man does) and I can't remember if there's a class that does X. There's a lot of code. I wrote almost all of it. And -- maybe it's the gin -- but I just can't recall if there's an X. It seems like there should be.

Scan. Scan. Scroll. Scroll.

Read. Read.

Wait!

I have a pretty good gh-pages branch for this. Sphinx-based. Mostly up-to-date. Let's look there.

Ahhh. So much nicer than scrolling through code. Indexes work.

This whole "documentation" thing is pretty cool. Now I'm actually happy that other people guilted me into doing it.

Using SCons

In looking at Application Lifecycle Management (see "ALM Tools"), I had found that SCons appears to be pretty popular. It's not as famous as all the make variants, or Apache Ant or Apache Maven, but it seems to have a niche in the forest of Build Automation Software.

While it looks nice, parts of SCons are confusing. I struggled until I found a simple use case.

More: Empirical Comparison of SCons and GNU Make.

"SCons proved to be more accurate, mostly due to its stateful, content-based signature model.

On the other hand, GNU Make proved to be more resource friendly, especially regard- ing the memory footprint. SCons needs to address this problem to be a viable alternative to Make when building large software projects."

[Also, it appears that a lot of build and test automation have been reframed as "Continuous Integration". Which isn't really a bad thing. But it can be confusing because there are too many categories into which general-purpose tools can be fit.]

While SCons looks cool, I haven't had a huge need for it at work. Working in Python, there's no real "build". Instead our continuous integration boils down to unit testing. Our "build" is an SVN checkin. Our deployment is an SVN checkout and `python setup.py install`.

At some point, I would like to create an SConstruct file that runs our integration test suite. But it's trapped at a low priority.

SCons and Sphinx

I did find an SConscript example that automated a document build using Sphinx. This

sphinx-scons was quite cool. However, it was challenging to customize. The SCons documentation requires real work to understand. I could see the value, but it was a lot of work.

I'm hoping that No Starch Press finds someone to write a tidy introduction to SCons.

SCons and RST and LaTeX (oh, my!)

Sphinx has made me a total fanboi of ReStructured Text. While I know MS Word and iWorks Pages quite well, I have no patience with all the pointing and clicking. Getting consistency is requires consistent pointing and clicking; some people can do it, but some of us find that manual pointing and clicking is sometimes irreproducible. Semantic markup is a huge pain in the neck because we have to stop typing to click on the proper style hint for the various words.

I also know DocBook XML and LaTeX quite well. I've used very cool XML editors including XML Mind XML Editor (which is very nice.) I no longer have any patience with any of these tools because there's too much GUI.

RST is fun because you write in plain text. There are a few directives and a few bits of inline roles for semantic markup. But your work can focus on the content, leaving presentation aside. A command-line tool (with templates) emits HTML or LaTeX or whatever. The style considerations can be (a) secondary and (b) completely consistent.

RST will easily produce complex LaTeX from plain text. What a joy. LaTeX, of course, isn't the goal, it's just an intermediate result that leads to DVI which leads -- eventually -- to a PDF.

Because of the Unicode and font selection on the Mac, I'm a user of XeTeX and XeLaTeX. I have some problems with getting my copy of Blackadder ITC to work, but generally I'm able to write without much fussing around.

SCons has a great deal of the TeX/DVI/PDF tool chain already installed. However, it doesn't have either the rst2latex script or the XeTeX tools.

An SConscript

While my first attempts to understand SCons didn't work out well, looking at RST and XeLaTex was a much better use case.

I wound up with this.

rst2latex = Builder( action="rst2latex.py $SOURCES >$TARGET",

suffix='.tex', src_suffix='.rst',

)

xelatex = Builder( action=["xelatex $SOURCES", "xelatex $SOURCES"],

suffix='.pdf', src_suffix='.tex',

)

env = Environment(ENV=os.environ,

BUILDERS = { 'rst2latex' : rst2latex, 'xelatex':xelatex }

)

env.rst2latex('someDoc')

env.xelatex('someDoc' )

env.rst2latex('anotherDoc')

env.xelatex('anotherDoc')

Getting this to work was quite pleasant. I can see how I could further optimize the document production pipeline by combining the two Builders.

[And yes, the xelatex step is run twice to guarantee that the references are correct.]

Now, I can get away with write, run `scons` and review the resulting PDF. It's fast and it produces a nice-looking PDF with very little work and no irreproducible pointing and clicking.

Given this baseline, I can now dig into SCons for ways to make this slightly simpler.