Friday, July 5, 2019

Mastering Object-Oriented Python 2nd ed


The book https://www.packtpub.com/programming/mastering-object-oriented-python-second-edition

Some new chapters. 

Type hints almost everywhere.


If you want to write a review, DM me on twitter @s_lott I can add you to the list for freebies in exchange for a review.

Thursday, June 20, 2019

HumbleBundle -- Functional Python Programming -- Through July 1


See this https://www.humblebundle.com

This is amazing to me. 
Humble Bundle sells games, ebooks, software, and other digital content. Our mission is to support charity while providing awesome content to customers at great prices. We launched in 2010 with a single two-week Humble Indie Bundle, but we have humbly grown into a store full of games and bundles, a subscription service, a game publisher, and more.
Currently, Packt is offering some of my books.

Want a *ton* of technical books and donate to charity?

Click Now. Thank me later.

Tuesday, June 18, 2019

The Pythonista app for iPad

Let me start my review with "wow!"

Python 3.6 on the iPad. Works. Nicely. Easy to use. Reliable. Rock-Solid.

I'm not switching to iPad as my primary platform any time in the near future.  But. For certain kinds of small and tightly focused hackery, this is really nice.

I use a bracket to hold the iPad up and an external keyboard. I can be used with the on-screen keyboard, but, that's slow-going for me.

Here's the thing that was exquisitely simple in Pythonista:



I'm able to draw a hex grid ("Flat Top", "Double Height") in a few dozen lines of code. This includes a bunch of geometry rules like adjacency and directional movement.

The Pythonista package includes a super-easy-to-use canvas module that's a tiny bit simpler than turtle graphics. It takes a bit of getting used to, but it has enough graphics primitives to make it easy to create hexagons and tile the surface.

Given a HexGrid instance, I can then create "cities" and their surrounding territories in an "empire". I've tried a few organic growth algorithms, and I like the look of these maps. They provide a lot of avenues for conflict for writing fiction or playing role-playing games.

Some of the algorithmic foundations: https://www.redblobgames.com/grids/hexagons/.

Fun Hackery

This is fun hackery because I can change the code, click the run icon, and watch the consequence of the changes. A traceback is highlighted in the original file. Easy. Fun.

It's pretty slow. No surprise there. It's running on an iPad.

It's pretty easy to work on. Whip out the iPad and start coding.

The super-easy, built-in canvas module means feedback is instant and gratifying.

I can see having an intro to programming class where the fee includes an extra $800 for the iPad you take home along with your new-found skills in basic coding. (This is still a *lot* of money, but it's less than a full laptop.)

Filling in the Holes

Looking at the output, you can see the growth algorithm left some unfilled holes. A later version examines all unfilled spaces to see if they're entirely surrounded by one color and fills them. This is a fun algorithm because it works in a simple way with the adjacency iterator and the set of locations covered by a city. Locations 12L and 17K are these "Simply Surrounded Single Holes."

However, there are still some "Edge" cases that are challenging.

Location 12D reflects a hole on a border. These are interesting, and could be the seed for epic wargaming, role-playing game, novel-writing drama. A simple algorithm can find these and assign a random owner... But... They really need a special "On Fire" color scheme to show the potential for drama.

There's a subtlety in the upper-left corner (5W and 6V) between Blue and Green. While these seem like simple border holes, each hole as only five of six required neighbors.

Compare these with 16P in the upper-right corner. This also has five of six neighbors. However, this space looks like it could be a bay leading up to a river and the river is a natural border between nations.

The head of the bay at 16P has 5 neighbors of two colors, similar to 5W and 6V. The difference can be detected by a recursive walk to see if a hole is connected to other holes and the composite is actually surrounded. There are lots of *edge* cases, but the (5W, 6V) pair seems to embody the next stage of surround detection.

This more nuanced algorithm design doesn't work out well in the Pythonista environment. This algorithm design requires careful unit tests, not the code-and-run cycle of hackery. For this kind of careful design, we'd need to leverage doctest (or unittest) for testing. While I'd like pytest, that's a lot to ask for. For these kinds of apps, doctest is more than adequate, and a simple import doctest; doctest.testmod() in a scrip can help be sure things work as expected.

tl;dr


If you're an iPad user, consider adding Pythonista. You can really write real Python. It's a useful environment. It's fun for teaching.

Tuesday, June 11, 2019

Circuit Python on the Gemma M0 -- The Red Ranger Beacon

PyCon 2018 Swag included a Gemma m0. (https://www.adafruit.com/product/3501)

PyCon 2019 Swag included a Circuit Playground Express. See https://learn.adafruit.com/adafruit-circuit-playground-express/circuitpython-quickstart.

Both of these are (to me) amazing. They mount as USB devices; there’s a code.py file that’s automatically run when the board restarts.

The Gemma has fairly few pins and does some real simple things.  The CPX has a bunch of pins and ton of hardware on the board. Buttons, Switches, LED’s, motion sensor, temperature, brightness... I’ve lost count.

Step 1 -- Get Organized

Create a proper project directory on your local machine. Yes, you can hack the code.py file immediately, but you should consider making a backup before you start making changes.

Few things are more frustrating than making a mistake and being unable to restore the original functionality as a check to be sure things are still working.

Also. At some point, you'll want to upgrade the OS on the chip. This will require you to have a bootable image. The process isn't complex, but it does require some care. See https://learn.adafruit.com/welcome-to-circuitpython/installing-circuitpython#download-the-latest-version-3-4 for downloading a new OS.

So. Step 1a. Create a local folder for your projects. Within that folder, create a folder for each project. Put the relevant code.py into the sub-folder. Like this

gemma
 ┣━━ baseline
 ┃   ┗━━ code.py
 ┣━━ my-first-project
 ┃   ┗━━ code.py
 ┣━━ another-project
 ┃   ┗━━ code.py
 ┗━━ os-upgrade
     ┗━━ other files...

See https://learn.adafruit.com/adafruit-gemma-m0/troubleshooting for additional help if you have a Windows PC.

Step 2 -- Start Small

Tweak a few things in the supplied code.py if you're new to IoT stuff.

A lot of folks like the Mu editor for this. https://codewith.mu.

I like using BBEdit. https://www.barebones.com/products/bbedit/.  To make this work I *also* need to have

  • a terminal window open so I can use the Mac OS screen application, and
  • a finder window open to copy the code.py from my PC to the Gemma M0.

This is a lot of busy screen real-estate with three separate apps open. It's not for everyone. I like it because there's little hand-holding. You may prefer Mu.

It's important to go through the edit/download/play cycle many times to be sure you're clear on what code's on your PC and what code's on your board.

It's even more important to see how you're forced to debug syntax errors using the screen app until you invent a suitable mock library for off-line unit testing.

Step 3 -- Plan Carefully

The version of Python is remarkably complete.

However.

It's also a very small processor, with very few pins, so you can't do anything super elaborate.  You can, however, do quite a bit.

See Nina Zakharenko - Keynote - PyCon 2019 for some inspiration

Step 4 -- Check This Out

https://github.com/slott56/gemma-boat-beacon

Many thanks to @nnja for showing us some elegant, inspirational ideas.


Tuesday, June 4, 2019

Probabilistic Data Structures

Interesting data structures with O(n) performance. This can help to unscramble O(n²) problems allowing progress.

https://pdsa.readthedocs.io/en/latest/index.html

Tuesday, May 28, 2019

Rules for Debugging

Here's the situation.

Someone wrote code. It didn't do what they assumed it would do.

They come to me for help.

Here are my rules for debugging. All of them.

1. Try something else.



I don't have any other or more clever advice. When I look at someone's broken code, I'm going to suggest the only thing I know. Try something else.

I can't magically make broken code work. Code's not like that. If it doesn't work, it's wrong, usually in a foundational way. Usually, the author made an assumption, followed through on that assumption, and was astonished it didn't work.

A consequence of this will be massive changes to the broken code. Foundational changes.

When you make an assumption, you make an "ass" out of "u" and "mption".

Debugging is a matter of finding and discarding assumptions. This can be hard. Some people think their assumptions are solid gold and write long angry blog posts about how a language or platform or library is "broken."

The rest of us try something different.

My personal technique is to cite evidence for everything I think I'm observing. Sometimes, I actually write it down -- on paper -- next to the computer. (Sometimes I use the Mac OS Notes app.) Which lines of code. Which library links. Sometimes, i'll include in the code as references to documentation pages.

Evidence is required to destroy assumptions. Destroying assumptions is the essence of debugging.

Sources of Assumptions

I'm often amazed at how many people don't read the "But on Windows..." parts of the Python documentation. Somehow, they're willing to assume -- without evidence -- that Windows is POSIX-compliant and behaves like Linux. When things don't follow their assumed behavior, and they're using Windows, it often seems like they've compounded a bunch of personal assumptions. I don't have too much patience at this point: the debugging is going to be hard.

I'm often amazed that someone can try to use multiprocessing.apply_async() without reading any of the example code. What I'm guessing is that assumptions trump research, making them solid gold, and not subject to questioning or locating evidence. In the case of multiprocessing, it's important to look at code which appears broken and compare it, line-by-line with example code that works.

Comparing broken code with relevant examples is -- in effect -- trying something else. The original didn't work. So... Go to something that does work and compare the two to identify the differences.

Tuesday, May 14, 2019

PyCon 2019

There are some things I could say.

But.

You can come to understand it yourself, also.

Go here: https://www.youtube.com/channel/UCxs2IIVXaEHHA4BtTiWZ2mQ

Start with the keynotes.  https://www.youtube.com/channel/UCxs2IIVXaEHHA4BtTiWZ2mQ/search?query=keynote

For me, one of the top presentations was this https://www.youtube.com/watch?v=9G2s1TN9QQY
There are several closely related, but I found this very helpful.

Tuesday, May 7, 2019

Fiction Writers and Query Letters

See http://flstevens.itmaybeahack.com/writing-world-building-and/ for some back-story on F. L. Stevens and the need to write a *lot* of query letters to agents for fiction. (The non-fiction industry is entirely different; there are acquisition editors who look for technical contributors.)

There's a tiny possibility of a Query Manager Tool (of some kind) on a writer's desktop.

Inputs include:
  • Template letter.
  • A table of customizations per Agent. The intent is to allow more than simple name and pronoun changes. This includes Agent-specific content requirements. These vary, and can include the synopsis, first chapter, first 10 pages, first 50 pages. They're not email attachments; they have to be part of the main body of the email, so they're easy to prepare.
  • Another table of variant pitches to plug into the template. There are a lot of common variations on this theme. Sizes vary from as few as 50 words to almost 300 words. Summaries of published works seem to have a median size of 140 words. A writer may have several (or several dozen) variants to try out.
This can't become a spam engine (agents don't like the idea of an impersonal letter.) 

Also. A stateful list of agents, queries, and responses is important. Some Agents don't necessarily respond to each query; they often offer a blanket apology along the lines of "if you haven't heard back in sixty days, consider it a rejection." So. You want to try again. Eventually. Without being rude. And if you requery, you have to send a different variant on the basic pitch.

Some Agents give a crisp "nope" and you can update your list to avoid requerying.

For new authors (like F. L. Stevens,) there's a kind of manual query tracking mess. It's not really horrible. But it is annoying. Keeping the database up-to-date with responses is about as hard as a tracking spreadsheet, so, there's little value to a lot of fancy Python software.

The csv, string.Temple, email and smtplib components of Python make this really easy to do.  While I think it would be fun to write, creating this would be work avoidance. 

Tuesday, April 9, 2019

PyLit-3 Maintenance, Love and Care

The PyLit tool dates from 2009. Here's a historical reference: http://wiki.c2.com/?PyLit

It was Python 2. It had some minor problems. I forked it and cleaned it up for Python 3.

Then I set it aside for a few years (six or so.)

Dusting it off. Rearranging things. The legacy Python 2 version -- it appears -- is gone forever.

The current thing available in PyPI doesn't even download and install on a modern Python because the metadata makes it look like it won't be compatible with a Python 3.7 world. So. That needs to be fixed. And while I'm at it...

- Add tox support for Python 3.5, 3.6, and 3.7 properly.
- Restructure the docs to use Github Pages from master/docs.
- Get the download squared away so pip install will work.
- Use pathlib.
- Start down the road toward type hinting. Which will likely exclude py35 support.

I may, as part of type hinting, be forced to make some more changes to the essential structure of the app(s).

For now, I simply need to get it to be pip installable.

Tuesday, March 26, 2019

Python and pathlib and Windows -- this problem has been solved -- and yet...

The Passive-Aggressive Programmer strikes again. A sad story of sadness.

I tell everyone to stop using os.path and use pathlib. Everyone. Here's the link: https://docs.python.org/3/library/pathlib.html

It's essential to realize the semantic richness of OS filesystem paths. They're not simply strings. They have a string representation, but there's quite a bit going on there that is not captured trivially by strings and string parsing.  "path/basename.extension" is more than just slashes and dots.

Windows users, of course, have a nightmarish problem. Actually many nightmarish problems, one of which is pathnames.

I tell Windows developers to use pathlib, it will make their life somewhat more bearable.

And Yet. The Passive-Aggressive Programmer insists on using Windows as if it doesn't have a problem with \ in path strings.

Line 110 has a literal r"C:\windows\is\xtreme\evil" Note the \x in the path. Without the r"" string, this literal raises a SyntaxError.

Line 50 had subprocess.run(r"C:\path\to\xectuable -option" + " " + options + " " + filename). Note the r"" string. Note the Linux-style -option, too. They're wrapping an open source app in a Python shell.

You're with me so far? They're Windows devs. They've managed to use raw strings in two places. Right?

But. They're Passive-Aggressive. They don't like PR comments of any kind. They'll "agree" to a change, but then... This...

Line 50 should change. It's needs to use a list.

The Passive Aggressive Programmer can't make a list work.

list_of_arguments = ["C:\path\to\xectuable -option"] + options_list + ["C:\windows\is\xtreme\evil"]

See what they did there? They didn't want to change from string to list. (We had to go over it more than once.) They wanted to leave it alone. Grudgingly, they agreed to change from string to list.

But.

SyntaxError. See? The list just doesn't work. Python is weird. It's an undocumented WAT.

[Yes, some of us know the r's vanished. They author couldn't figure that out, though.]

And the pathlib suggestion?

Since the strings are now a SyntaxError, they need me to fix that for them. I made them change to a list, therefore, I caused the SyntaxError. It would be a distraction to spend tome researching pathlib. "I need to Google and think about how to handle the Unicode error" was the response.

Using Path("C:/path/to/xecutable") to avoid any Window-ism of any kind is an impossible burden. Impossible. It requires them to Google. Instead, the SyntxError is all my fault.

The previous examples of the use of raw strings?  Don't know why they're not helpful, but I'm not the one who's struggling to implement a change.

Tuesday, March 19, 2019

Don't Solve My Problem.

Two and a half examples of "Don't solve the problem I described. Provide the implementation I dream about."

Can't Use Enums for Constants

I was asked to see this because sometimes there's just too much abstraction https://stackoverflow.com/questions/2668355/how-much-abstraction-is-too-much

The accepted answer links to some useful design principles. Read the answer. It's useful.

The question objects to abstract superclasses without much (or any) actual implementation.  I've seen folks toy around with frameworks where there are classes that introduce a name, but little else. So I understand the complaint. I once tried to use Python classes as surrogates for Java interfaces. It was a bad idea. And irrelevant to solving the underlying problem.

The problem that lead to the "yet another abstraction" complaint, however, was not related to a design with empty layers of framework abstractions. It was not related to classes used to define an interface-like feature in Python. It wasn't related to *anything* in the Stack Overflow question or answer.

The "yet another abstraction" complaint was based on Python not having constants. Seriously. How did we get here? Right. They don't want a solution. They want to complain.

I lift this situation up to folks who are trapped in conversations where things devolve into bizarro-world like "Yet Another Abstraction is bad" when we're not talking about abstractions. The solution is simple, but, it's not what they wanted so, it's labeled as bad in some way.

The solution is bad because it's unexpected. Consequently peripheral, tangential, weird-ass nonsense will show up in trying to avoid an unexpected solution.

Can't Assign Numbers

There's an API to load some data.  They have 100's of clients happily loading data. In some cases, the clients must assign numbers in addition to names; it's a disambiguation thing. Most of the time, the name is good. In a few cases, (name, number) is a two-part key because they have multiple instances with the same name.

We're good here. The data structure's key can be (name, number) and the default number is zero. Works for almost everyone.

Almost everyone. Exception they have one client who cannot count or enumerate their data.

Really.

The client can't even pre-process the data to add numbers because reasons.

The stated reason is "the data originates off-line and the numbers might be inconsistent." The key needs a number. It doesn't need to consistent. The point is asking the client to own the identity -- a name and a number.

The solution seemed easy. Assign a number. If your data comes from a spread-sheet, use the row number. The =row() function works. Use that.  If your data doesn't come from a spread-sheet write a tiny utility to laminate a number into the data. This doesn't seem hard. And then the client owns the object identity.

Nope.

Can't do it. The web service will have to assign the number for them.

It's not a difficult feature to add. It's a complicated, stateful default value. This will turn into trouble tickets in the future when the numbers are unacceptable because they change with each load or something even more obscure than that.

Can't Fork a Repo

This isn't recent, and I may not have the details right. But.

The team had evolved an approach where they had several different pieces of software spread among multiple branches in a single Git repository.

This was weird. And they were -- of course -- about to start having CI/CD problems as they moved away from manual builds into a world of git commit hooks and relatively fixed CI/CD pipelines.

And they were really unhappy. They liked having multiple branches in a single repo. The idea of forking this into separate repos was unacceptable. Unworkable. Breaks everything. (Breaks everything they had. Everything they needed to replace. Or so they claimed.)

They had some vision of having the CI/CD jobs all reworked to move beyond the common dev/master world into their uniquely odd world of lots of parallel branches, each it's own private "master". But all in one repo.

They seemed to have locked into a strange world view, and weren't happy discarding it. The circular discussions of how multiple repos would break something something in their something was more examples of tangential, irrelevant discussion to cloak empty whining.

tl;dr

I think there are people who don't really want a "solution." They want something else.

There are people who have a vision: How Things Should Be (HTSB™,) They seem to be utterly unwilling to consider something that is not literally their (narrow) vision of HTSB.

It's very much as "Don't confuse me with facts, my mind is made up" situation. It's exasperating to me because of the irrelevant side-channel discussions they use to avoid confronting (or even stating) the actual problem.

Tuesday, March 12, 2019

Python's multi-threading and the GIL

Got this in an email.
"Python's multi-threading module seems not efficient because of the global interpreter lock?" 
Yep.
Is the trick is to use "Thread-Local Data"?
Nope.

It Gets Worse

Interestingly, there was no further ask. The questioner had decided on thread-local data because the questioner had decided to focus on threads. And they were done making choices at that point.

Sigh.

No question on "What was recommended?" or "What's a common solution?" or "What is Dask?" Nothing other than "confirm my assumptions."

This is swirling around a bunch of emails on trying to determine the maximum number of concurrent threads or processes based on the number of cores or CPU's or something.

Maximum.

I'll repeat that for those who skim.

They think there's a maximum number of concurrent threads or processes.

If you have some computation which (1) makes zero OS requests and (2) is never interrupted, I can imagine you'd like to have all of the cores fully committed to executing that theoretical stream of instructions. You might even be able to split that theoretical workload up based on the number of cores.

Practically, however, that stream of uninterrupted computing rarely exists.

Maybe. Maybe you've got some basin-hopping or gradient-following or random forest ML algorithm which is going to do a burst of computation on an in-memory data structure. In that (rare) case, Dask is still ideal for exploiting all of the cores on your processor.

The upper-bound idea bugs me a lot.

  • Any OS request leads to a context switch. Any context switch leads to waiting. Any waiting means you can have more threads than you have cores. 
  • AFAIK, any memory write outside the local cache will lead to a stall in the pipeline. Another thread can (and should) leap in to the core's processing stream. The only way you can create the "all-computing" sequence of instructions bounded by the number of cores is to *also* be sure the entire thing fits in cache. Hahahaha.

What's the maximum number of threads or processes? It depends on the wait times. It depends on memory writes. It depends on the size of the data structure, the size of cache, and the size of the instruction stream.

Because it depends on a lot of things, it's rather difficult to predict. And that makes it rather difficult to determine a maximum.

Replying about the uselessness of trying to establish a maximum, of course, does nothing. AFAIK, they're still assiduously trying to use os.cpu_count() and os.sched_getaffinity() to put an upper bound on the size of a thread pool.

Acting as if Dask doesn't exist.

Solution

Use Dask.

Or

Use a multiprocessing pool.

These are simple things. They don't require a lot of hand-wringing over the GIL and Thread Local Data. They're built. They work. They're simple and effective solutions.

Side-bar Nonsense

From "a really smart guy. He got his PhD in quantum mechanics and he got major money to actually go build … . He initially worked for ... and now he is working for .... So, when he says something or asks a question, I listen very carefully."
The laudatory blah-blah-blah doesn't really change the argument. It can be omitted. It is an "Appeal to Authority" fallacy, and the Highest Paid Person's Opinion (HIPPO) organizational pattern. Spare me.

Indeed. Asking for my confirmation of using Thread-Local Data to avoid the GIL is -- effectively -- yet another Appeal to Authority. Don't ask me if you have a good idea. An appeal to me as an authority is exactly as bad as appeal to some other authority to convince me you've found a corner case that no one has ever seen before.

Worse is to ask me and then blah-blah-blah Steve Lott says blah-blah-blah. Please don't.

I can be (and often am) wrong.

Write your code. Measure your code's performance. Tell me your results. Explore *all* the alternatives while you're at it.

Tuesday, March 5, 2019

Python exceptions considered an anti-pattern

https://sobolevn.me/2019/02/python-exceptions-considered-an-antipattern

While eloquent and thorough, I remain unconvinced that this is a significant improvement over try/except.

It's common enough in some functional languages to have strong support and a long, successful history.

I think it replaces one problem with another. It's not a "solution". It's an alternative. Instead of exceptions being raised, they're returned. Cool.

Tuesday, February 12, 2019

On the uselessness of Enum -- wait, what?

Had a question about an enumerated set of constant values.

"Where do I put these constants?" they asked. It was clear what they wanted. This is another variation on their personal quest which can be called "I want Python to have CONST or Final." It's kind of tedious when a person asks -- repeatedly -- for a feature that's not present in the form they want it.

"Use Enum," I said.

"Nah," they replied. "It's Yet Another Abstraction."

Wait, what?

This is what I learned from rest of their nonsensical response: There's an absolute upper bound on abstractions, and Enum is one abstraction too many. Go ahead count them. This is too many.

Or.

They simply rejected the entire idea of learning something new. They wanted CONST or Final or some such. And until I provide it, Python is garbage because it doesn't have constants. (They're the kind of person that needs to see CONST minutes_per_hour = 60 in every program. When I ask why they don't also insist on seeing CONST one = 1 they seem shocked I would be so flippant.)

YAA. Seriously. Too many layers.

As if all of computing wasn't a stack of abstractions on top of stateful electronic circuits.

Tuesday, February 5, 2019

Python Enhancement Proposal -- Floating an Idea

Consider the following code

def max(m: int, n: int) -> int:
    if m >= n:
        return m
    elif n >= m:
        return n
    else:
        raise Exception(f"Design Error: {vars()}")

There's a question about else: clause and the exception raised there.
  • It's impossible. In this specific case, a little algebra can provide that it's impossible. In more complex cases, the algebra can be challenging. In some cases, external dependencies may make the algebra impossible.
  • It's needless in general. An else: would have been better than the elif n >= m:.  The problem with else: is that a poor design, or poor coordination with the external dependencies, can lead to undetectable errors.
Let's look at something a little more complex.

def ackermann(m: int, n: int) -> int:
    if m < 0 or n < 0:
        raise ValueError(f"{m} and {n} must be non-negative")
    if m == 0:
        return n + 1
    elif m > 0 and n == 0:
        return ackermann(m - 1, 1)
    elif m > 0 and n > 0:
        return ackermann(m - 1, ackermann(m, n - 1))
    else:
        raise Exception(f"Design Error: {vars()}")

It's somewhat less clear in this case that the else: is impossible. A little more algebra is required to create a necessary proof.

The core argument here is Edge Cases Are Inevitable. While we can try very assiduously to prevent them, they seem to be an emergent feature of complex software. There are two arguments that seem to indicate the inevitability of edge and corner cases:

  • Scale. For simple cases, with not too many branches and not too many variables, the algebra is manageable. As the branches and variables grow, the analysis becomes more difficult and more subject to error. 
  • Dependencies. For some cases, this kind of branching can be refactored into a polymorphic class hierarchy, and the decision-making superficially simplified. In other cases, there are multiple, disjoint states and multiple conditions related to those states, and the reasoning becomes more prone to errors.
The noble path is to use abstraction techniques to eliminate them. This is aspirational in some cases. While it's always the right thing to do, we need to check our work. And testing isn't always sufficient.

The noble path is subject to simple errors. While we can be very, very, very, very careful in our design, there will still be obscure cases which are very, very, very, very, very subtle. We can omit a condition from our analysis, and our unit tests, and all of our colleagues and everyone reviewing the pull request can be equally snowed by the complexity. 

We have two choices.
  1. Presume we are omniscient and act accordingly: use else: clauses as if we are incapable of error. Treat all complex if-elif chains as if they were trivial.
  2. Act more humbly and try to detect our failure to be omniscient.
If we acknowledge the possibility of a design error, what exception class should we use?
  • RuntimeError. In a sense, it's an error which didn't occur until we ran the application and some edge case cropped up. However. The error was *always* present. It was a design error, a failure to be truly omniscient and properly prove all of our if-elif branches were complete.
  • DesignError. We didn't think this would happen. But it did. And we need debugging information to see what exact confluence of variables caused the problem.
I submit that DesignError be added to the pantheon of Python exceptions. I'm wondering if I should make an attempt to write and submit a PEP on this. Thoughts?

Tuesday, January 29, 2019

Eager and Lazy Properties

See this


My answer was -- frankly -- vague. Twitter being what it is, I should have written the blog post first and linked to it.

The use case is this.

class X:
    def __init__(self, x):
        self._value = f(x)
    @property
    def value(self):
        return self._value

We've got a property where the returned value is already an instance variable.

I'm not a fan.

This reflects an eager computation strategy. f(x) was computed eagerly and the value made available via a property. One can justify the use of a property to make the value read-only, but... still nope.

There are a lot of alternatives that make more sense to me.

Option 1. We're All Adults Here.

Here's an approach I think is better.

class X:
    def __init__(self, x):
        self.value = f(x)

It's read-only because -- really -- if you change it, you break the class. So don't change it.

This is my favorite. Read-onlyness is sometimes described has a way protect utter idiots from breaking a library they don't seem to understand. Or. It's described as a way to prevent some Evil Genius Programmer (EGP) from doing something intentionally malicious and breaking things.

Bah.

It's Python. They have access to the source. Why mess around breaking things this way?

Option 2. Lazyiness

Here's an approach that hits at the essential feature.

class X:
    def __init__(self, x):
        self.x = x
    @property
    @lru_cache(None)
    def value(self):
        return f(self.x)

This seems to hit at the original intent without an explicit cached variable. Instead the caching is pushed off into another space. (I'm writing a chapter on decorators, so this may be a bit much.)

The idea, though, is to make properties lazy. If they're expensive, then the result should be cached.

There may be other choices, but I think lazy and eager cover the bases. I don't think eager is wrong, but I don't see the need for a property to hide the attribute created by an eager computation.


Things that start badly

Today's Example of Starting Badly: Building HTML.

The code has a super-simple email message with f"<html><body><p>stuff {data}</p></body></html>". It was jammed into an email object along with the text version. All very nice.

For a moment, I considered suggesting that f-string substitution wasn't a good long-term solution, since it doesn't cover anything more than the most trivial case.

Two things stopped me from complaining:
  • The case really was trivial.
  • It's administrative code: it sends naggy reminder emails periodically. Why over-engineer it?
What an idiot I was.

Today, the {data} has been replaced with a complex table instead of a summary. (Why? The user story evolved. And we needed to replace the summary with details.)

The engineer was pretty sure they could use htmlify(data) or data.htmlify() to transform the data into an HTML structure without seriously breaking the f-string nature of the app.

I should have commented "Don't build HTML that way, it's a bad way to start" on the previous release.

The f-string solution turns rapidly into complexities layered on complexities dusted over the top with sprinkles of NOPE.

This is a job for Jinja2 or Mako or something similar. 

There's a step function change in the app's perceived "complexity". Instead of a simple f-string, we now have to populate a template. It goes from one line of code to more than one (three seems typical.) And. The file-system loader for templates seems more appropriate rather than hard-coding the template in the body of the code. So there are now more files in the app with the HTML templates in them.

However. The Jinja {{variable|round(2)}} was an immediate victory. The use of {%for%} to build the HTML table was the goal, and that simplification was worth the price of entry. Now we're arguing over CSS nuances to get the columns to look "right."

Lessons learned.

Don't let the currently superficial trivial case slide past without at least a warning. Make the suggestion that functions like "get template" and "populate template" will be necessary even for trivial f-string or string.Template processing.

HTML isn't a first-class part of anything. It's external serialization.  Yes, it's for people, but it's only serialization. Serialization has to be separated from the other aspects of the data gathering, map-reduce summarization, and email distribution. There's a pipeline of steps there and the final app should reflect the complete separation of these concerns. Even if it is admin overhead.

Tuesday, January 22, 2019

Read this: 10 Reasons to Learn Python in 2019

See 10 Reasons to Learn Python in 2019.

I want to dwell on #4 for a moment.

The Python community actually has a Code of Conduct. We try to stick by it and conferences will have reporting mechanisms in place so we can educate folks who are being inconsiderate or disrespectful.

The consequence of this is a welcoming and intentionally helpful community. It's hard to emphasize the "intentionally helpful" enough. We don't have much patience for snark. And we're willing to call each other out on being unhelpful.

In my Day Job, we have an in-house Slack channel with well over 1,000 Python folks. The single most common class of questions is a variation on "My Corporate Firewall Setup Doesn't Let Me Use PIP." This is ubiquitous. And confusing. And frustrating for folks who are surprised there is a corporate firewall.

We have a number of pinned answers in Slack for this. And -- perhaps once a week -- someone will patiently repeated the pinned answers for someone who's truly and deeply in over their head trying to get pip to work. (We have an in-house PyPI, also, but it requires doing something in addition to typing `pip install whatever` at the command line, and that can require hand-holding.)

As Python2 winds to a close, and we uncover folks working with Python 2, we have to issue guidance. I've switched tone from "please consider rewriting your app/tool/framework" to "we strongly recommend you start using Python 3." In June, I'm plan to switch to "You have only six months to convert whatever you're working on."

We've had some sidebar conversations on making sure I'm being properly positive, supportive, considerate, and respectful of the folks who think Python2 might be useful.

The point of the Python community is to help each other. We're actively and intentionally trying to be helpful and inclusive.

Technical Sidebar -- Conda and Virtual Environments 

What about the trickle of people trying to make use of the built-in Python2 in Mac OS X or the Python2 that comes with Linux on a cloud-based server?

Some important coaching: Don't Use The OS Default Python.

This is kind of negative. It helps to state this positively. Always Use A Virtual Environment.

Because we have a *large* community of data scientists, this becomes: Always Use A Conda Environment.

And yes, there are some packages that also require a pip install. And yes, XKCD 1987 describes the consequence of the rapid growth of Python and the variety of ways it can be made to work. (While all the strands of spaghetti look like a negative, they reflect the variety of clever solutions, all of which work without any problems.)

Therefore. This.

1. conda create --name=working python=3.7 --file conda_install.txt
2. conda activate working
3. python3 -m pip install pip_install.txt

The absolute worst case is a project with two lists of requirements, one in a conda conda_install.txt and some extra stuff in a pip conda_install.txt. We're able to use `python3 -m pip install requirements.txt` for almost everything.

If you're just starting out, you can use miniconda to bootstrap everything else you might need.

Tuesday, January 15, 2019

Super-picky Writing Advice

There are patterns to bad writing. I'll give some examples based on a blog post I was sent. It's also based -- indirectly -- on some of proposals I saw for PyCon and PyDataDC.

For the conference calls for papers, I can ask a few questions of the author, but that's about it.

For the blog post, I suggested a bunch of changes.

They balked.

Why ask for advice and then refuse to do anything?  (We can conjecture they wanted a "good job" pat on the head. They didn't want to actually have me give them a list of errors to fix.)

One of the points of contention was "Not everyone has your depth of expertise."

Sigh.

The blog post was on Ubuntu admin: something I know approximately nothing about. Let me step away from being an expert, while sticking closely with being able to write. I work with editors who -- similarly -- can write without being deep technology experts. I'm trying to learn what and how they do it.

In this case, my editing was based on general patterns of weak writing.

  1. Contradictions
  2. Redundancy.
  3. Waffling.
  4. Special-Casing.

Any blog post on Ubuntu admin that starts with "This is not just another blog post..." has started off with a flat-out contradiction. It emphatically is another blog post. You can't rise above the background noise of blog posts by writing a blog post that claims it's not a blog post. Sheesh. Find a better hook.

Any blog post on Ubuntu admin that includes "This blog post assumes the reader is familiar with linux sys administration." As if -- somehow -- a reader interested in Ubuntu admin could be confused by the required skills. It's clearly redundant. Cut it.

(This led to an immense back-and-forth with repeated insistence that *somehow* someone once got confused and *something* bad happened to someone. Once. My response was adamant. "It's redundant. The title says Ubuntu. That covered it. More repetition is an insult to the reader.")

Anything that has "this may nor may not work depending on your filesystem" is flat-out confusing: it covers both bases. Does it work or does it not work? Which is it? Clearly, there's some kind of precondition -- "must be this file system" or "most not be this file system" -- buried under "may or may not work." It's not that I know anything about Ubuntu file systems. But I can spot waffling.

Indeed, when you look at it, this is a "hook" to make the blog post useful and interesting. Some advice doesn't work. This advice always works. Simple statements of fact are better than contradictions and waffling.

Finally, there was a cautionary note that replacing "/swapfile" with "/ swapfile" would brick your OS. Which. Was. Crazy. It's really difficult to arbitrarily introduce spaces into shell commands and still have proper syntax. Sometimes a shell command with rando spaces may have proper syntax and may work. Most commands won't work at all with a rando space added. Try some and see.

What's more important is only one random punctuation error was listed. The special-case nature of this was a tip-off that something was not right with this advice.

What about a random ">" in the command? Or a random "|"? Not covered. A single space was considered worthy of mention. The rest. Meh.

(If you want details, it's was `dd -of=/swapfile ...`. If asked, I'd guess `dd -of=/ swapfile` is a syntax error because `swapfile` isn't a valid operand to `dd`. Not an expert. I didn't check. AFAIK, the author didn't actually check, either.)

None of this editing was based on any vast expertise. It was simple editorial work looking for some common problems with hasty writing.

  1. Avoid clear contractions.
  2. Avoid redundancy.
  3. Don't waffle.
  4. Special cases are instances of a more general pattern. The pattern is more important than the case.

One of the contentious back-and-forth issues was "I'm not writing a book, it's only a blog post."

Wait. What? Blog posts are often more widely-read than books. Look at Stack Overflow. If one of my books had the kind of readership my Stack Overflow answers have, I'd be living on royalty payments alone.

A blog post requires the same depth of care as a book.

This applies to proposed talks at conferences, also. The proposal needs to be carefully edited to reflect the final presentation's care and quality.

Thank goodness, most of the 100's of proposals I've looked at rarely have the four obvious problems listed above.

The problems I see in conference proposals are minor.

  1. Incompleteness. A 45 minute talk boiled down to 4 bullet points doesn't give us any confidence. It's hard to imagine filling the whole time with useful content when looking at a four-sentence outline. Will it be rambling digressions? Or will we have disgruntled attendees who had hoped for more?
  2. Weirdly cute style. Things like "This is where I jokingly outline something something and the real fun begins." We assume everyone is witty and charming, you don't need to tell us. We assume all talks will be fun. Can we move on to the Python (or PyData) topics you'll cover?
  3. Sales Pitches. "[Speaker's name] is a respected industry expert who delivers exciting and transformative keynote addresses and will dynamically cover the state-of-the-art blah blah blah..." Please. What Python topic is this? We like to review the outlines without speaker information; we need to focus on the content. Subverting this by including the speaker's name in the description or outline is irritating.
I'm really pleased we see very few PyCon Code-of-Conduct problems in the calls for proposals. That is a delight.

Editing is hard work. I'm sorry to report that editing means making changes. If you ask for editorial advice, it helps to listen.

Tuesday, January 8, 2019

Tuesday, January 1, 2019

PyCon Call for Proposals

PyCon CFP closes in a day or so. See https://us.pycon.org/2019/speaking/ for details.

After looking at 100's of proposals there are several broad categories.

  • Some cool stuff I wrote (or helped write, or contributed to)
  • Some cool aspect of Python the language
  • Some cool thing happening in the community

These are -- of course -- refined into separate tracks by the program committee. But as a reader of outlines, there are a few BIG buckets I tend to throw things into.

There are some tangentially Python things.

  • Using k8s (or some other tool or framework) with Python. The focus is often the tool, not Python.
  • Doing some machine learning with Python. The focus is often on the ML application domain or the cool model that got produced. Again, Python sometimes feels secondary.
  • Optimizing a personal (or enterprise) workflow or CI/CD pipeline. This can be helpful. But it can sometimes skip past Python and focus on IDE or something not solidly Python.
One of the most challenging are the personal journeys. These are often interesting stories. We like to see how others have succeeded. But. (And this is important.) Some of these cool personal journeys are only tangentially Pythonic, which can be disappointing. Some people can make good things happen in spite of big obstacles.

A personal journey can happen with a lot of technologies. When the proposal doesn't seem to show how Python was a unique enabler of the journey, then it's a cool talk for SXSW or OSCON, but might not be ideal for PyCon.

It's an honor to see everyone's ideas streaming through my in-box.