Tuesday, September 29, 2015

Python 3.5 and the Upgrade Strategy

Start here: https://docs.python.org/3/whatsnew/3.5.html#whatsnew-pep-484

While new syntax is important, remember your audience in pitching the upgrade from Python 2.7. You may need to pander to people who aren't programmers or don't really know Python.

When selling the upgrade, it can help to focus on the objective measures.
  1. Performance. When anyone asks why we should disturb our precious Python 2 ecosystem, point out the performance improvements. Begin with Python 3.2, 3.3, 3.4, and then 3.5 improvements. The union of these is an impressive list. Faster is better, right?
  2. New Libraries. For some folks who don't know Python well, it helps to give them a concrete list of features you absolutely require. Seriously. Enumerate all the new libraries from Python 3.2, ..., 3.5. It's a big list. Some of them have been backported, so this list isn't a a complete win. You may not really need all of them, but use them to bolster your case.
  3. Other Cleanups. These are important for folks who use Python daily, but aren't too impressive to manager types who aren't deeply into the language details.
    1. The fact that Python 3 handles class/type better than Python 2 isn't impressive to anyone who hasn't dealt with it. 
    2. The fact that Python 3 handles Unicode better than Python 2 isn't going to impress too many people, either. 
    3. The print statement issue will cause some managers to claim that the upgrade is "risky". 
    4. The division issue is a complete win. Weirdly, nay-sayers will claim (a) just use float() a lot, (b) just add +0.0 a lot, or (c) just add from __future__ import division a lot.  How is this workaround better? No clue. Be prepared to make the case that the dumb workarounds are... well... dumb.
You can lift up the type definition and http://mypy-lang.org. If you do, be prepared for snark from the Java/Scala crowd. These folks will (wrongly) claim that a partial type proof is useless, and static type checking is mandatory. This is a difficult discussion to have because the "type safety is important" crowd don't seem to recognize the awful gyrations they're forced into so they can write generic code that's type-agnostic. All Python code is type-agnostic; the type checking just confirms some design constraints. The presence of differing strategies -- type-specific code vs. generic type-agnostic code -- means that neither is right, and the argument is moot. 

Don't focus on async/await. Yes, it's first on the Python web site, but, it can be a tough sell.


The easy sell is this impressive list of optimizations.

  • Peephole optimizer improvements
  • Serializing and unserializing data using the pickle module is now several times faster.
  • The Timsort algorithm used in list.sort() and sorted() now runs faster and uses less memory when called with a key function. 
  • JSON decoding performance is improved and memory consumption is reduced whenever the same string is repeated for multiple keys. 
  • Recursive locks (created with the threading.RLock() API) now benefit from a C implementation which makes them as fast as regular locks, and between 10x and 15x faster than their previous pure Python implementation.
  • The fast-search algorithm in stringlib is now used by the split(), splitlines() and replace() methods on bytes, bytearray and str objects. Likewise, the algorithm is also used by rfind(), rindex(), rsplit() and rpartition().
  • Integer to string conversions now work two “digits” at a time, reducing the number of division and modulo operations.
  • Several other minor optimizations. 
    • Set differencing now runs faster when one operand is much larger than the other
    • The array.repeat() method has a faster implementation
    • The BaseHTTPRequestHandler has more efficient buffering
    • The operator.attrgetter() function has been sped-up
    • ConfigParser loads multi-line arguments a bit faster
  • Some operations on Unicode strings have been optimized
  • UTF-8 is now 2x to 4x faster. UTF-16 encoding is now up to 10x faster.
  • The UTF-32 decoder is now 3x to 4x faster.
  • The cost of hash collisions for sets is now reduced. 
  • The interpreter starts about 30% faster. 
  • bz2.BZ2File is now as fast or faster than the Python2 version for most cases. lzma.LZMAFile has also been optimized.
  • random.getrandbits() is 20%-40% faster for small integers.
  • By taking advantage of the new storage format for strings, pickling of strings is now significantly faster.
  • A performance issue in io.FileIO.readall() has been solved. 
  • html.escape() is now 10x faster.
  • The os.walk() function has been sped up by 3 to 5 times on POSIX systems, and by 7 to 20 times on Windows. 
  • Construction of bytes(int) (filled by zero bytes) is faster and uses less memory for large objects.
  • Some operations on ipaddress IPv4Network and IPv6Network have been massively sped up,
  • Pickling of ipaddress objects was optimized to produce significantly smaller output. 
  • Many operations on io.BytesIO are now 50% to 100% faster.
  • The marshal.dumps() function is now faster: 65-85% with versions 3 and 4, 20-25% with versions 0 to 2 on typical data, and up to 5 times in best cases. 
  • The UTF-32 encoder is now 3 to 7 times faster. 
  • Regular expressions are now parsed up to 10% faster.
  • The json.dumps() function was optimized.
  • The PyObject_IsInstance() and PyObject_IsSubclass() functions have been sped up.
  • Method caching was slightly improved, yielding up to 5% performance improvement in some benchmarks. 
  • Objects from random module now use two times less memory on 64-bit builds. 
  • The property() getter calls are up to 25% faster.
  • Instantiation of fractions.Fraction is now up to 30% faster.
  • String methods find(), rfind(), split(), partition() and in string operator are now significantly faster for searching 1-character substrings.
I think this list can help move an organization away from Python 2 and toward Python 3. This list and a lot of lobbying from folks who know what the improvements are.


Here's the library upgrade list, FWIW.
The details of the improvements can be overwhelming.

The dozen new modules, however, might help overcome organizational inertia to make progress on ditching Python2. I've been making heavy use of statistics. I need to make better use of pathlib in future projects.

Tuesday, September 22, 2015

Python Tutor

Read This: http://radar.oreilly.com/2015/08/learning-programming-at-scale.html

The core visualization tool (pythontutor.com) can be helpful for many people. The shared environments seem like a cool idea, also, but I don't have any specific comments on the other tools.

While this looks very cool, I'm not a huge fan of this kind of step-by-step visualization. This uses very clear graphics, and looks very clever, it has some limitations. I think that some aspects of "visualization" can be misleading. Following an execution path for a specific initial condition can obscure events and conditions that aren't on the happy path. It's not clear how a group of statements establish a more general condition.

I'm a fan of formal post-conditions. From these, we can postulate a statement, and work out the weakest precondition for the statement. As we work through this exercise, we create a formal proof and a program. It's very elegant. And it covers the general case, not specific examples.

Most importantly, this effort depends on having formal semantics for each statement. To write code, we need to have a concise definition of the general state change is made by each statement in a language. We're looking at the general case for each statement rather than following a specific initial condition through a statement.

In C, what does this do? a[i++] = ++i; There is no formal definition. The statement has three state changes stated. But how are they ordered? No matter what initial values for a[] and i we provide, this is still pretty murky. A debugger only reveals the specific implementation being debugged.
Visualization may help some people understand the state change created by a statement. Some people do learn things by watching this kind of "debugger" mode. In particular, this may help because it has much better graphics than the built-in character-mode debugger.

This idea works best with programs that already make sense: programs that are well designed. Programs that make orderly progress from some initial state to the desired final state.

Programs written by learners may not be all that clean. Realistically, they may be inept. They may even reach the far end of the spectrum and be downright bad.

While this tool is graphically gorgeous, it's still a debugger. It wallows around in an internal world in which the formal semantics can get obscured. The general case can't easily be shown.

We have a forest and trees problem here. A debugger (or other statement-by-statement visualization tool) emphasizes each individual tree. The larger structures of glades, thickets, groves, stands, brakes, and coppices are lost to view.

The humble while statement (especially one with an internal if-break) can be extremely difficult to understand as a single statement. If we break down the statement-by-statement execution, the presence of two termination conditions (one on the while clause and one on the if clause) can be obscured because a visualization must follow a specific initial condition.

With really well-written tutorials -- and some necessary metadata -- a super-visualizer might be able to highlight the non-happy-path logic that exists.  This alternate path viewing could be helpful for showing how complex logic works (and doesn't work.)

With programs written by learners -- programs which are inept and won't have appropriate metadata -- a super-visualizer would need to reason very carefully about the code to determine what happy path and non-happy-path kinds of logic are present. It would have to locate and highlight

  • contradictory elif clauses, 
  • gaps among elif clauses, 
  • missing else clauses, 
  • hellishly complex else clauses, 
  • break conditions, 
  • continue conditions, as well as 
  • exception handling.

For truly bad programs, the super-visualizer may be stumped as to what is intended. Indeed, it may be impossible to determine how it can be displayed meaningfully to show alternatives and show how the specific code generalizes into a final outcome.

def this_program_terminates(some_code):
    # details omitted

def demo():
    while this_program_terminates(demo):
        print("w00t w00t")

What does this do? How can any visualizer aid the student to show problems?

To take this one step further, I think this kind of thing might also be hazardous to learning how the functional programming feature of Python work.  I think that exposing the underlying mechanics of a generator expression might be more confusing than simply treating it as a "lazy list."

It's very nice. But it isn't perfect. Something that supports reasoning about the general post-conditions established by a statement would be more useful than a step-by-step debugger with great graphics.

Tuesday, September 15, 2015

Exploratory Data Analysis in Functional-Style Python

Here are some tricks to working with log file extracts. We're looking at some Enterprise Splunk extracts. We can fiddle around with Splunk, trying to explore the data. Or we can get a simple extract and fiddle around with the data in Python.

Running different experiments in Python seems to be more effective than trying to do this kind of exploratory fiddling in Splunk. Primarily because there aren't any boundaries on what we can do with the data. We can create very sophisticated statistical models all in one place.

Theoretically, we can do a lot of exploration in Splunk. It has a variety of reporting and analytical features.


Using Splunk presumes we know what we're looking for. In many cases, we don't know what we're looking for: we're exploring. We may have some indication that a few RESTful API transactions are slow, but little more than that. How do we proceed?

Step one is to get raw data in CSV format. Now what?

Reading Raw Data

We'll start by wrapping a CSV.DictReader object with some additional functions.

Object-Oriented Purists will object to this strategy. "Why not just extend DictReader?" they ask. I don't have a great answer. I lean toward functional programming and the resulting orthogonality of components. With a purely OO approach, we have to use more complex-seeming mixins to achieve this.

Our general framework for processing logs is this.

with open("somefile.csv") as source:
    rdr = csv.DictReader(source)

This allows us to read the CSV-formatted Splunk extract. We can iterate through rows in the reader. Here's trick #1. It's not really very tricky, but I like it.

with open("somefile.csv") as source:
    rdr = csv.DictReader(source)
    for row in rdr:
        print( "{host} {ResponseTime} {source} {Service}".format_map(row) )

We can -- to a limited extent -- report raw data in a helpful format. If we want to dress up the output, we can change the format string. Maybe "{host:30s} {ReponseTime:8s} {source:s}" or something like that.


A common situation is that we've extracted too much, and only need to see a subset. We can change the Splunk filter, but, we hate to overcommit before we've finished our exploration. It's far easier to filter in Python. Once we've learned what we need, we can finalize in Splunk.

with open("somefile.csv") as source:
    rdr = csv.DictReader(source)
    rdr_perf_log = (row for row in rdr if row['source'] == 'perf_log')
    for row in rdr_perf_log:
        print( "{host} {ResponseTime} {Service}".format_map(row) )

We've injected a generator expression that will filter the source rows, allowing us to work with a meaningful subset.


In some cases, we'll have additional columns of source data that we don't really want to use. We'll eliminate this data by making a projection of each row.

In principle, Splunk never produces an empty column. However, RESTful API logs may lead to data sets with a huge number of unique column titles based on surrogate keys that are part of request URI's. These columns will have one row of data from the one request that used that surrogate key. For every other row, there's nothing useful in that column. Life is much simpler if we remove the empty columns from each row.

We can do this with a generator expression, also, but it gets a bit long. A generator function is somewhat easier to read.

def project(reader):
    for row in reader:
        yield {k:v for k,v in row.items() if v}

We've built a new row dictionary from a subset of the items in the original reader. We can use this to wrap the output of our filter.

with open("somefile.csv") as source:
    rdr = csv.DictReader(source)
    rdr_perf_log = (row for row in rdr if row['source'] == 'perf_log')
    for row in project(rdr_perf_log):
        print( "{host} {ResponseTime} {Service}".format_map(row) )

This will reduce the unused columns that are visible in the inside of the for statement.

Notation Change

The row['source'] notation will get clunky. It's much nicer to work with a types.SimpleNamespace than a dictionary. This allows us to use row.source.

Here's a cool trick to create something more useful.

rdr_ns = (types.SimpleNamespace(**row) for row in reader)

We can fold this into our sequence of steps like this.

with open("somefile.csv") as source:
    rdr = csv.DictReader(source)
    rdr_perf_log = (row for row in rdr if row['source'] == 'perf_log')
    rdr_proj = project(rdr_perf_log)
    rdr_ns = (types.SimpleNamespace(**row) for row in rdr_proj)
    for row in rdr_ns:
        print( "{host} {ResponseTime} {Service}".format_map(vars(row)) )

Note the small change to our format_map() method. We've added the vars() function to extract a dictionary from the attributes of a SimpleNamespace.

We could write this as a function to preserve syntactic symmetry with other functions.

def ns_reader(reader):
    return (types.SimpleNamespace(**row) for row in reader)

Indeed, we could write this as a lambda construct which is used like a function.

ns_reader = lambda reader: (types.SimpleNamespace(**row) for row in reader)

While the ns_reader() function and ns_reader() lambda are used the same way, it's slightly harder to write a document string and doctest unit test for a lambda. For this reason, a lambda should probably be avoided.

We can use map(lambda row: types.SimpleNamespace(**row), reader). Some folks prefer this over the generator expression.

We could use a proper for statement with an internal yield statement, but there doesn't seem to be any benefit from making a big statement out of a small thing.

We have a lot of choices because Python offers so many functional programming features. We don't often see Python touted as a functional language. Yet, we have a variety of ways to handle a simple mapping.

Mappings: Conversions and Derived Data

We'll often have a list of data conversions that are pretty obvious. Plus, we'll have a growing list of derived data items. The derived items will be dynamic and are based on different hypotheses we're testing. Each time we have an experiment or question, we might change the derived data.

Each of these steps: filtering, projection, conversions, and derivation, are stages in the "map" portion of a map-reduce pipeline. We could create a number of smaller functions and apply them with map(). Because we're updating a stateful object, we can't use the general map() function.  If we wanted to achieve a more pure functional programming style, we'd use an immutable namedtuple instead of a mutable SimpleNamespace.

def convert(reader):
    for row in reader:
        row._time = datetime.datetime.strptime(row.Time, "%Y-%m-%dT%H:%M:%S.%F%Z")
        row.response_time = float(row.ResponseTime)
        yield row

As we explore, we'll adjust the body of this conversion function. Perhaps we'll start with some minimal set of conversions and derivations. We'll extend this with some "are these right?" kind of things. We'll take some out when we discover that the don't work.

Our overall processing looks like this:

with open("somefile.csv") as source:
    rdr = csv.DictReader(source)
    rdr_perf_log = (row for row in rdr if row['source'] == 'perf_log')
    rdr_proj = project(rdr_perf_log)
    rdr_ns = (types.SimpleNamespace(**row) for row in rdr_proj)
    rdr_converted = convert(rdr_ns)
    for row in rdr_converted:
        row.start_time = row._time - datetime.timedelta(seconds=row.response_time)
        row.service = some_mapping(row.Service)
        print( "{host:30s} {start_time:%H:%M:%S} {response_time:6.3f} {service}".format_map(vars(row)) )

Note that change in the body of our for statement. Our convert() function produces values we're sure of. We've added some additional variables inside the for loop that we're not 100% sure of. We'll see if they're helpful (or even correct) before updating the convert() function.


When it comes to reductions, we can adopt a slightly different style of processing. We need to refactor our previous example, and turn it into a generator function.

def converted_log(some_file):
    with open(some_file) as source:
        rdr = csv.DictReader(source)
        rdr_perf_log = (row for row in rdr if row['source'] == 'perf_log')
        rdr_proj = project(rdr_perf_log)
        rdr_ns = (types.SimpleNamespace(**row) for row in rdr_proj)
        rdr_converted = convert(rdr_ns)
        for row in rdr_converted:
            row.start_time = row._time - datetime.timedelta(seconds=row.response_time)
            row.service = some_mapping(row.Service)
            yield row

We've replace the print() with a yield.

Here's the other part of this refactoring.

for row in converted_log("somefile.csv"):
    print( "{host:30s} {start_time:%H:%M:%S} {response_time:6.3f} {service}".format_map(vars(row)) )

Ideally, all of our programming looks like this. We use a generator function to produce data. The final display of the data is kept entirely separate. This allows us to refactor and change the processing much more freely.

Now we can do things like collect rows into Counter() objects, or perhaps compute some statistics. We might use a defaultdict(list) to group rows by service.

by_service= defaultdict(list)
for row in converted_log("somefile.csv"):
    by_service[row.service] = row.response_time
for svc in sorted(by_service):
    m = statistics.mean( by_service[svc] )
    print( "{svc:15s} {m:.2f}".format_map(vars()) )

We've decided to create concrete list objects here. We can use itertools to group the response times by service. It looks like proper functional programming, but the implementation points up some  limitations in the Pythonic form of functional programming. Either we have to sort the data (creating a list object) or we have to create lists as we group the data. In order to do several different statistics, it's often easier to group data by creating concrete lists.

Rather than simply printing a row object, we're now doing two things.
  1. Create some local variables, like svc and m. We can easily add variance or other measures.
  2. Use the vars() function with no arguments, which creates a dictionary out of the local variables.
This use of vars() with no arguments -- which behaves like locals() -- is a handy trick. It allows us to simply create any local variables we want and include them in the formatted output. We can hack in as many different kinds of statistical measures as we think might be relevant.

Now that our essential processing loop is for row in converted_log("somefile.csv"), we can explore a lot of processing alternatives in a tiny, easy-to-modify script. We can explore a number of hypotheses to determine why a some RESTful API transactions are slow and others are fast. 

Tuesday, September 1, 2015

Audio Synth in Python 3.4, Part II

See Audio Synth.

At first, I imagined the problem was going to be PyAudio. This package has a bunch of installers. But the installers don't recognize Python 3.4, so none of them work for me. The common fallback plan is to install from source, but, I couldn't find the source. That looks like a problem.

Once I spotted this: "% git clone http://people.csail.mit.edu/hubert/git/pyaudio.git", things were much better.  I built the PortAudio library. I installed PyAudio for Python3.4. Things are working. Noises are happening.

Next step is actual synth.

In the past, I have played with pysynth because it has some examples of wave-table additive synth. That's very handy. The examples are hard to follow because a lot of the synth ideas are conflated into small functions.

Complication: The pysynth package is Python2. It lacks even the simple from __future__ import print_function to make it attempt Python3 compatibility.

The pysynth.play_wav module could be a handy wrapper around various audio playback technologies, include pyaudio. It has to be tweaked, however, to make it work with Python3.4. I really need to clone the project, make the changes, and put in a pull request.

The pysynth.pysynth and pysynth.pysynth_beeper modules are helpful for seeing how wave tables work.  How much rework to make these work with Python3.4? And how much reverse engineering to understand the math?

I've since found pyo. Which is also Python 2. See the AjaxSoundStudio pages for details. This may be a better example of wave tables. But it's still Python2. More investigation to follow.

The good news is that there's some forward motion.

Tuesday, August 25, 2015

Visual studio and Python

Why write Python in Visual Studio?

That what I want to know, too.

IntelliSense? ActiveState Komodo does this. And it does it very well considering the potential complexity of trying to determine what identifiers are possibly valid in a dynamic language.

Debugger? No thanks. I haven't used it yet. [I should probably blog on the perils of debuggers.]

Project Management? GitHub seems to be it. Some IDE integration might be helpful, but the three common command-line operations -- git pull, git commit, and git push -- seem to cover an awful lot of bases.

I've been asked about Python IDEs -- more than once -- and my answer remains the same:
The IDE Doesn't Matter. 

One of the more shocking tech decisions I've seen is the development manager who bragged on the benefits of VB. The entire benefit was this: Visual Studio made the otherwise awful VB language acceptable.

The Visual Studio IDE was great. And it made up for the awful language.


The development manager went to to claim that until Eclipse had all the features of Visual Studio, they were sure that Java was not usable. To them, the IDE was the only decision criteria. As though code somehow doesn't have a long tail of support, analysis, and reverse engineering. 

Tuesday, August 18, 2015

Audio Synth [Updated]

I learned about synthesizers in the '70's using a Moog analog device. Epic coolness.

Nowadays, everything is digital. We use wave tables and (relatively) simple additive synth techniques.

I made the mistake of reading about Arduino wave table synthesis:




The idea of an Arduino alarm that uses a chime instead of a harsh buzz is exciting. The tough part about this is building the wave tables.

What a perfect place to use Python: we can build wave tables that can be pushed down to the Arduino. And test them in the Python world to adjust the frequency spectrum and the complex envelope issues around the various partials.

See http://computermusicresource.com/Simple.bell.tutorial.html


Python3.4 doesn't have PyAudio support.


Sigh. Before I can work with Arduino wave tables, I'll have to start by figuring out how to build PyAudio for Python 3.4 on Mac OS X.

Look here: http://people.csail.mit.edu/hubert/git/pyaudio.git for the code.

Look here for the secret to building this on Mac OS X: https://stackoverflow.com/questions/2893193/building-portaudio-and-pyaudio-on-mac-running-snow-leopard-arch-issues/2906040#2906040.


  1. Get pyaudio source.
  2. Inside pyaudio create a portaudio-v19. Get the portaudio source and put it here.
  3. Inside pyaudio/pyaudio, do ./config; make and sudo make install
  4. Inside pyaudio, do python3.4 setup.py install --static-link

Tuesday, August 4, 2015

Mocking and Unit Testing and Test-Driven Development

Mocking is essential to unit testing.


It's also annoyingly difficult to get right.

If we aren't 100% perfectly clear on what we're mocking, we will merely canonize any dumb assumptions into mock objects that don't really work. They work in the sense that they don't crash, but they don't properly test the application objects since they repeat some (bad) assumptions.

When there are doubts, it seems like we have to proceed cautiously. And act like we're breaking some of the test-first test-driven-development rules.

Note. We're not really breaking the rules. Some folks, however, will argue that test-driven development means literally every action you take should be driven by tests. Does this include morning coffee or rotating your monitor into portrait mode? Clearly not. What about technical spikes?

Our position is this.
  1. Set a spike early and often. 
  2. Once you have reason to believe that this crazy thing might work, you can formalize the spike with tests. And mock objects.
  3. Now you can write the rest of the app by creating tests and fitting code around those tests.
The import part here is not to create mocks until you really understand what you're doing.

Book Examples

Now comes the tricky part: Writing a book.

Clearly every example must have a unit test of some kind. I use doctest heavily for this. Each example is in a doctest test string.

The code for a chapter might look like this.

test_hello_world = '''
>>> print( 'hello world')
'hello world'

__test__ = { n:v for n,v in vars().items() 
    if n.startswith('test_') }

if __name__ == '__main__':
    import doctest

We've used the doctest feature that looks for a dictionary assigned to a variable named __test__. The values from this dictionary are tests that get run as if they were docstrings found inside modules, functions, or classes.

This is delightfully simple. Expostulate. Exemplify. Copy and Paste the example into a script for test purposes and Exhibit in the text.

Until we get to external services. And RESTful API requests, and the like. These are right awkward to mock. Mostly because a mocked unittest is singularly uninformative.

Let's say we're writing about making a RESTful API request to http://www.data.gov. The results of the request are very interesting. The mechanics of making the request are an important example of how REST API's work. And how CKAN-powered web sites work in general.

But if we replace urrlib.request with a mock urllib, the unit test amounts to a check that we called urlopen() with the proper parameters. Important for a lot of practical software development, but also uniformative for folks who download the code associated with the book.

It appears that I have four options:

  1. Grin and bear it. Not all examples have to be wonderfully detailed.
  2. Stick with the spike version. Don't mock things. The results may vary and some of the tests might fail on the editor's desktop.
  3. Skip the test.
  4. Write multiple versions of the test: a "with real internet" version and a "with corporate firewall proxy blockers in place" version that uses mocks and works everywhere.
So far, I've leveraged the first three heavily. The fourth is awkward. We wind up with code like this:

class Test_get_whois(unittest.TestCase):
    def test_should_get_subprocess(self):
        subprocess = MagicMock()
        with patch.dict('sys.modules', subprocess=subprocess):
            import subprocess
            from ch_2_ex_4 import get_whois
            result= get_whois('')
        self.assertEquals( result, ['', 'words'] )
        subprocess.check_output.assert_called_with(['whois', ''])

This is not a lot of code for enterprise software development purposes. It's a bit weak, in fact, since it only tests the Happy Path.

But for a book example, it seems to be heavy on the mock module and light on the subject of interest.
Indeed, I defy anyone to figure out what the expository value of this is, since it has only 2 lines of relevant code wrapped in 8 lines of boilerplate required to mock a module successfully.

I'm not unhappy with the unitest.mock module in any way. It's great for mocking modules; I think the boilerplate is acceptable considering what kind of a wrenching change we're making to the runtime environment for the unit under test.

This fails at explication.

I'm waffling over how to handle some of these more complex test cases. In the past, I've skipped cases, and used the doctest Ellipsis feature to work through variant outputs. I think I'll continue to do that, since the mocking code seems to be less helpful for the readers, and too focused on purely technical need of proving that all the code is perfectly correct.