My algorithm performs badly, do I need asyncio?

Real Question (somewhat abbreviated): "My algorithm performs badly, do I need asyncio?"

Short answer: No.

Long answer: Sigh. No. Do you need a slap upside the head?

Here's how it plays out:

Q: "We figured that if we 'parallelize' it, then we can apply multiple cores, and it will run 4x as fast."

Me: "What kind of I/O are you doing?"

Q: "None, really. It's compute-intensive."

Me: "Async is for I/O. A function can be computing while other functions are waiting for I/O to complete."

Q: "Right. We can have lots of them, so they each get a core."

Me: "Listen, please. A function can be computing. That's "A". Singular. One. Take a step back from the asyncio package. What are you trying to do?"

Q: "Make things faster."

Me: "Take a breath. Make what faster?"

Q: "A slow algorithm."

Me: 

Q: "Do you want to know what we're trying do?"

Me: 

Q: "First, we query the database to get categories. Then we query the database to get details for the categories. Then we query the database to organize the categories into a hierarchy. Except for certain categories which are special. So we have if-statements to handle the special cases."

Me: "That's I/O intensive."

Q: "That's not the part that's slow."

Me: 

Q: "Context is important. I feel the need to describe all of the background."

Me: "That's trivia. It's as important as your mother's maiden name. What's the problem?"

Q: "The problem is we don't know how to use asyncio to use multiple cores."

Me: "Do you know how to divide by zero?"

Q: "No. It's absurd."

Me: "We already talked about asyncio for compute-intensive processing. Same level of absurd as dividing by zero. What are you trying to do?"

Q: "We have some for loops that compute a result slowly. We want to parallelize them."

Me: "Every for statement that computes a collection is a generator expression. Every generator expression can be made into a list, set, or dictionary comprehension. Start there."

Q: "But what if the for statement has a super-complex body with lots of conditions?"

Me: "Then you might have to take a step back and redesign the algorithm. What does it do?"

Q: <code> "See all these for statements and if-statements?"

Me: "What does it do? What's the final condition?"

Q: "A set of valid answers."

Me: "Define valid."

Q: "What do you mean? 'Define valid?' It's a set that's valid!"

Me: "Write a condition that defines whether or not a result set is valid. Don't hand-wave, write the condition."

Q: "That's impossible. The algorithm is too complex."

Me: "How do you test this process? How do you create test data? How do you know an answer it produces is correct?"

Q:

Me: "That's the fundamental problem. You need to have a well-defined post-condition. Logic. An assert statement that defines all correct answers. From that you can work backwards into an algorithm. You may not need parallelism; you may simply have a wrong data structure somewhere in <code>."

Q: "Can you point out the wrong data structure?"

Me: 

Q: "What? Why won't you? You read the code, you can point out the problems."

Me: 

Q: "Do I have to do all the work?"

Me:

Needless Complexity -- Creating Havoc Leads to Mistakes

I received the worst code example ever. The. Worst.

Here's the email.

I have hurriedly created a blog post titled [omitted] at the url below
[also omitted]
...
Unfortunately, I am neither an algorithm expert or a Python expert. However, I am willing to jump in. 

Please review the Python code snippets. I know that they work because I ran them using Python 2.7.6. It was the environment available on my work PC. Speed to get something to the group so that it does not disband outweighs spending time on environments. The goal is not to be Pythonic but to have anyone that has written any code follow the logic.

Also, please review the logic. Somehow, I managed to get the wrong answer. The entire blog post is a build to provide a solution to CLRS exercise 2.3-7. My analysis gave me the answer of O( {n [log n]}**2 ) and the CLRS answer is O(n [log n] ). Where did I screw up my logic?

CLRS is https://mitpress.mit.edu/books/introduction-algorithms.

The referenced blog post is shocking. The "neither an algorithm expert or a Python expert" is an understatement. The "willing to jump in" is perhaps a bad thing. I sent several comments. They were all ignored. I asked for changes a second time. That was also ignored. Eventually, changes were made reluctantly and only after a distressing amount of back-and-forth.

Havoc was created through a process of casually misstating just about everything that can possibly be  misstated. It transcended mere "wrong" and enters that space where the whole thing can't even be falsified. It was beyond simply wrong.

My point here is (partially) to heap ridicule on the author. More importantly, want to isolate a number of issues to show how simple things can become needlessly complex and create havoc.

The Problem

The definition of the problem 2.3-7 seems so straightforward.

"Describe a $\Theta(n \log n)$-time algorithm that, given a set $S$ of $n$ integers and another integer $x$, determines whether or not there exist two elements in $S$ whose sum is exactly $x$."

This brings us to problem #1. The blog post is unable to actually repeat the problem. Here's the quote:

x + y = x0
where
x ∈ N and y ∈ N for some finite set of integer values N
x0: the integer to which x and y sum
x != y

It's not at all clear what's going on here. What's x0? What's N? Why is x!=y even introduced?

This is how havoc starts. The requirements have been restated in a way that makes them more confusing. The original terminology was dropped in favor of random new terminology. There's no reason for restating the problem. The consequence of the bad restatement is to introduce needless features and create confusion.

The Python Nonsense

A quote:

Concepts are demonstrated via code snippets. They code snippets were executed using Python 2.7.6. They were written in such a way that anyone with basic coding skills could read the code. In other words, the goal was not to be Pythonic.

Python 2.7.6 has been obsolete since May of 2014. At the very least, use a current release.

The goal of using Python without being Pythonic seems to be -- well -- schizophrenic. Or it's intentional troll-bait. Hard to say. 

Another paragraph says this.

The Python community will be annoyed because I am using Python 2 and not 3. Their annoyance is appropriate. Unfortunately, I only have Windows machines and can't afford to screw them up at this point in time.

What? That makes no sense at all. It's trivial to install Python 3.6 side-by-side with Python 2. Everyone should. Right now. I'll wait. See https://conda.io/docs/. Start here: https://conda.io/miniconda.html.

If you're going to insist on using the quirky and slow Python 2, you absolutely must use this in all of your code:

from __future__ import print_function, division, absolute_import, unicode_literals

Python 2 code without this is wrong. If you're still using Python 2, add this to all your code, right now. Please. You'll have to fix stuff that breaks; but we'll all thank you for it. pylint --py3k will help you locate and fix this.

The code with a -2/10 pylint score

I'm trying to reproduce this faithfully. It's hard, because the original blog post has issues with layout.

SomeIntegerList = [1, 2, 3, 4, 5, 6]
DesiredSumOfIntegers = 11
for SomeIntegerA in SomeIntegerList:
 for SomeIntegerB in SomeIntegerList:
  if SomeIntegerA == SomeIntegerB: continue
        SumOfIntegers = SomeIntegerA + SomeIntegerB
        print "SomeInteger A = ", SomeIntegerA, ", SomeInteger B = ", SomeIntegerB, ", Sum of Integers = ", SumOfIntegers
  if DesiredSumOfIntegers == SumOfIntegers:
      print "DesiredSumOfIntegers = ", DesiredSumOfIntegers, " was found"

(The original really could not be copied and pasted to create code that could even be parsed. I may have accidentally fixed that. I hope not.)

Almost every line of code has a problem. It gets worse, of course.

There's output in the original blog post that provides a hint as to what's supposed to be happening here.

Addition is Commutative

Yes. There is an entire paragraph plus a spreadsheet which proves that addition is commutative. An. Entire. Paragraph. Plus. A. Spreadsheet.

Meanwhile, factorial, multiplication, and division aren't mentioned. Why do we need a spreadsheet to show that addition is commutative, yet, all other operators are ignored? No clue. Moving on.

Permutations

A quote:

Now, let's talk about the number of computations involved in using nested for loops to examine all the possible addition permutations. Here I am using the term permutation as it is strictly defined in mathematics.

First. The algorithm uses all combinations. $\textbf{O}(n^2)$.

Second. "as it is strictly defined in mathematics" should go without saying. If you feel the need to say this, it calls the entire blog post into question.

It's like "honestly." Anyone who has to establish their honesty with "can I be honest with you?" is still lying.

If we're being strict here, are we not being strict elsewhere? If we're not being strict, why not?

The algorithm enumerates all combinations of n things taken 2 at a time without replacement. For reasons that aren't clear. The original problem statement permits replacement. The restatement of the problem doesn't permit replacement.

The n things taken r or 2 at a time problem

There's a table with values for $\frac{n!}{(n-r)!}$

No hint is given as to what this table is or why it's here.  I think it's supposed to be because of this:

$\frac{n!}{r!(n-r)!} \text{ with } r=2 \equiv \frac{n!}{2(n-2)!} \equiv \frac{n\times(n-1)}{2}$

It's hard to say why commutativity of addition gets a paragraph, but this gets no explanation at all. To me, it shows a disregard for the reader: the reader doesn't understand addition, but they totally get factorial.

Another Perspective

A quote

Another perspective is to note that the nested for loops result in O(n^2). Clearly, the above approach is not scalable.

That's not "another perspective." That's. The. Point. The entire point of the exercise is that the brute force algorithm isn't optimal.

The Worst Code Snippet Ever

This is truly and deeply shocking.

SomeIntegerList = [1, 2, 3, 4, 5, 6]
DesiredSumOfIntegers = 11
i = 0
for SomeIntegerA in SomeIntegerList:
    i = i + 1
    j = 0
 for SomeIntegerB in SomeIntegerList:
        j = j + 1
        if j > i:
            print "i = ", i, ", j = ", j
            SumOfIntegers = SomeIntegerA + SomeIntegerB
            print "SomeInteger A = ", SomeIntegerA, ", SomeInteger B = ", SomeIntegerB, ", Sum of Integers = ", SumOfIntegers
      if DesiredSumOfIntegers == SumOfIntegers:
          print "DesiredSumOfIntegers = ", DesiredSumOfIntegers, " was found"

This is what drove me over the edge. This is unconscionably evil programming. It transcends mere "non-Pythonic" and reaches a realm of hellish havoc that can barely be understood as rational. Seriously. This is evil incarnate.

This is the most baffling complex version of a half-matrix iteration that I think I've ever seen. I can only guess that this is written by someone uncomfortable with thinking. They copied and pasted a block of assembler code changing the syntax to Python. I can't discern any way to arrive at this code.

The Big-O Problem

This quote:

Even though the number of computations is cut in half

The rules for Big-O are in the cited CLRS book.  $\textbf{O}(\frac{n^2}{2}) = \textbf{O}(n^2)$.

The "cut in half" doesn't count when describing the overall worst-case complexity. It needs to be emphasized that "cut in half" doesn't matter. Over and over again.

This code doesn't solve the problem. It doesn't advance toward solving the problem. And it's unreadable. Maybe it's a counter-example? An elaborate "don't do this"?

The idea of for i in range(len(S)): for j in range(i): ... seems to be an inescapable approach to processing the upper half of a matrix, and it seems to be obviously $\textbf{O}(n^2)$.

The Binary Search

This quote is perhaps the only thing in the entire blog post that's not utterly wrong.

we can compute the integer value that we need to find. We can than do a search over an ordered list for the integer that we need to find.

Finally. Something sensible. Followed by more really bad code.

The code starts with this

def binarySearch(alist, item):

instead of this

from bisect import bisect

Why does anyone try to write code when Python already provides it?

There's more code, but it's just badly formatted and has a net pylint score that's below zero. We've seen enough.

There's some further analysis that doesn't make any sense at all:

Since the integers that sum must be distinct, the diagnol on the matrix have values of N/A

And this:

Secondly, we should remove the integer that we are on from the binary search

This is a consequence of the initial confusion that decided that $x \neq y$ was somehow part of the problem. When it wasn't. These two sentences indicate a level of profound confusion about the essential requirements. Which leads to havoc.

Added Complication

The whole story is pretty badly confused. Then this arrives.

Complicate Problem by Having Integer List Not Sorted

It's not clear what this is or why it's here. But there it is. 

It leads eventually to this, which also happens to be true. 

The total computation complexity is O(2 * n [log n] ) = O(n [log n] )

That's not bad. However. The email that asked for help claimed O( {n [log n]}**2 ). I have no idea what the email is talking about. Nor could I find out what any of this meant. 

The Kicker

The kicker is some code that solves the problem in $\textbf{O}(n)$ time. Without using a set, which is interesting.

This was not part of the CLRS exercise 2.3-7. I suppose it's just there to point out something about something. Maybe it's a "other people are smarter than CLRS"? Or maybe it's a "just google for the right answer without too much thinking"? Hard to say.

A sentence or two of introduction might be all that's required to see why the other result is there.

Lessons Learned

Some people like to add complexity to the problem. The $x \neq y$ business is fabricated from thin air. It adds to the code complexity, but is clearly not part of the problem space.

This creates havoc. Simple havoc.

Some people appear to act like they're asking for help. But they're not. They may only want affirmation. A nice pat on the head. "Yes, you've written a blog post." Actual criticism isn't expected or desired. This is easy to detect by the volume and vehemence of the replies.

Given a list of numbers, S, and a target, x, determine of two values exist in the set that sum to x.

>>> S = [1,2,3,4,5,6]
>>> x=11
>>> [(n, x-n) for n in S if (x-n) in S]
[(5, 6), (6, 5)]
>>> bool([(n, x-n) for n in S if (x-n) in S])
True

This follows directly from the analysis. It doesn't add anything new or different. It just uses Python code rather than indented assembler.

This first example is $\textbf{O}(n^2)$ because the in operator is applied to a list. We can, however, use bisect() instead of the in operator.

>>> [(n, x-n) for n in S if S[bisect(S, (x-n))-1] == x-n]
[(5, 6), (6, 5)]
>>> x=13
>>> [(n, x-n) for n in S if S[bisect(S, (x-n))-1] == x-n]
[]

This achieves the goal -- following the parts of the analysis that aren't riddled with errors -- without so much nonsensical code.

This does require some explanation for what bisect(S, i) does. It's important to note that the bisect() function returns the position at which we should insert a new value to maintain order. It doesn't return the location of a found item. Indeed, if the item isn't found, it will still return a position into which a new item should be inserted.

If we want this to be $\textbf{O}(n)$, we can use this:

>>> S = [1,2,3,4,5,6]
>>> S_set = set(S)
>>> x=11
>>> bool([(n, x-n) for n in S_set if (x-n) in S_set])
True

This replaces the linear list with a set, S_set. The (x-n) in S_set operation is $\textbf{O}(1)$, leading to the overall operation being $\textbf{O}(n)$.

If you want to shave a little time, you can use any() instead of bool([]). If you're not returning the pairs, you can reduce it to any(x-n in S_set for n in S_set). Try it with timeit to see what the impact is. It's surprisingly small.

Class Design Strategies -- analysis vs. synthesis

The conventional wisdom on class design is to model real-world things. This is the Domain-Driven Design Approach. It's what we used to teach as Rumbaugh's OMT (prior to the Three Amigos creating UML.)

The idea is simple: Look at the real world objects. Describe them.

Classes will have attributes and behaviors. They will have relationships. Rumbaugh was very careful about keeping object and class separate. A class had associations, and object had links. The association was the abstraction, the link was a concrete implementation. A class offered an operation, an object provided a method as an implementation.

As powerful as this is, I'm not sure it's the final word.

The only problem it has is that people often get confused by "real world objects." I've seen a number of places where folks completely fail to distinguish Enterprise IT implementation choices from actual things that reflect actual objects in the actual world.

Users and Credentials, for example. Users are real human beings. You find them in the hallways, standing around. They take up space in conference rooms. Credentials are a peculiar security-focused way to summarize a person as something public (username) and something private (password.) You don't find a stack of credentials tying up a room at the end of the hallway. Indeed, you can't physically stack credentials. While something a user knows is important, it isn't the entirety of a User. The attributes and behaviors of credentials aren't a good model for a User. But you still have this argument periodically when developing a class model or a noSQL database document model.

I'd like to emphasize that this is -- as far as I can tell -- the only problem with domain driven modeling. Some people don't see the domain very clearly because they tend to stick to a technology-driven world view.

However. That doesn't mean that drawing on the white board is the only way to discover the domain.

Building Classes from Functions

As a heretical alternative, allow me to suggest an alternative to the whiteboard.

Once upon a time, the whiteboard was the only way to do object modeling. The successor to the whiteboard (I use Argo UML as well as Pocketworks yuml) is a diagramming tool that -- ideally -- helps you understand the domain before committing to the cost and complexity of writing code.

Wait a second, though.

The "cost and complexity of writing code"? Java programmers know what I mean. If you don't have your classes understood, you should not start slapping code together.

Python programmers have no idea what "cost and complexity of writing code" means. They slap classes together faster than I can draw the damn pictures on Argo.

Indeed, the pictures can become a kind of burden. The picture shows "x.X", therefore, the module must include "x.X". Even though there might be a better way using classes in separate modules "a.Y" and "b.Z". But changing the cluster of pictures that comprises a fairly complete UML diagram isn't easy.

[Clearly, this depends on how much you tried to show. If your diagrams are really spare, refactoring is no problem. If you include parts of the object model in the component diagram or activity diagram, you're in trouble.]

This leads to an alternative to the whiteboard. And the diagramming tool.

Code. [Cue Orchestral Hit: Ta-daaa!]

Yes. Code. [Cue Orchestral Hit: Ta-ta-ta-daaa!]

When you can slap together a spike solution in Python you have a sensible alternative to the whiteboard.

You can build some classes, write some demonstration code to show how they work together. Don't like it? Start again from another base of classes. You can do this as a Mob Programming exercise. It fits somewhere between grooming a story and finishing an MVP release. Indeed, it may be a good way to do specific, concrete grooming.

In some cases, though, you can't build classes. You don't really know (or can't agree) on what the real world things are.

Rather than debate, shift the focus. Just write functions.

In Python, this is easy, since functions are first-class inhabitants of the programming model. In Java, this isn't easy at all. Functions aren't proper things; they must be part of a class; and you can't agree on what the classes are; the Java stalemate. [Yes, Java 8 introduces free-standing functions.]

How This Works In Practice

In many cases, it makes sense to punt on the "big picture." You're not really sure what you even have.  Yes, you know you have eight individual CSV files that reflect events that happened somewhere in cyberspace. (Let's just say they were the output from stored procedure triggers; the only record of changes made to crucial data.)

You can wring your hands over the eight-table join required to reconstruct the pre-change and post-change views of the objects. You can wring your hands over the way it's really three (or is it four?) different navigation paths from I to II to IV to (V to VI to VII) union I to II to IV to (V to VI to VII) union I to III to IV to oh my god I'm so confused.

Or.

You can get the sample data.  You can read it using the CSV module.

DictReader can awkward. It can be fixed, however. If your column titles are legal Python variables, you can use this to create a namespace reader from a DictReader. This allows you to say row.ATTRIBUTE instead of row['ATTRIBUTE'].

def nsreader(dictreader):
    return (SimpleNamespace(**row) for row in dictreader)

We can then turn to working out the various join algorithms on real data. Each step builds objects based on types.SimpleNamespace.

You start with simplest possible join algorithm: load everything into dictionaries based on their keys.

I_map = { row.KEY: row for row in nsreader(table_I_dict_reader) }
II_map = { row.SOMETHING: row for row in nsreader(table_II_dict_reader) }

Once you have sample data in memory, you can figure out what the actual, working relationships are. You can tinker with navigation paths through the tangled mess of tables. You can explore that data. You can do data profiling to find out how many misses there are.

If the tables are smallish (10,000 rows each) it all fits in memory nicely. No need for fancy database connections and no need to reason out join algorithms that don't tie up too much memory. You're not writing a database server. You're writing an application.

Look For Common Features

The design issue for classes is locating common features: common attributes or common methods. We often start down the road of common attributes. Because. Well... it seems logical.

Focus on attributes is a bias.

Classification of objects isn't based mostly on attributes. It's not 50-50 objects vs. attributes.

We tend to focus on attributes -- I think -- out of habit. Data structures mean "common data", right? Databases include tables of commonly-structured data.

But this isn't a requirement -- nor is it even important. It's just a habit.

We can conceive of a class hierarchy based around common behavior, too. This may require a very flexible collection of attributes. On the other hand, there's no a priori reason not to define classes based on their behavior.

That's why the idea of building functions first doesn't seem too far-fetched.

First, we can build working functions.  We can have test cases and everything.

Then we can look for commonality. We can refactor into classes. We can start with a Flyweight design pattern. As common attribute emerge, we can refactor to store more state in the class, and less state somewhere else. The API changes while we do this.

Then we examine it for the "is this a thing" criteria. Last, not first. We may need to make a few more tweaks to reflect the thing we discovered scattered around the functions. The thing may be a checklist or a recipe or a procedure: something active instead of simply stateful.

This tends to make RESTful web services a bit of a head scratcher. If we have an active thing, what is the state that we can represent and transfer? The state may be very small; the active agency may be quite sophisticated. This shouldn't be too baffling, but it can be confusing when the GET request response is either 200 or 403/409: OK or Forbidden/Conflict. Or there are multiple shades of 200: 200 OK with a body that indicates success, vs. 200 OK with a body that indicates something more needs to be done, vs. warnings, vs. exceptions, vs. other nuanced results.

Summary -- tl;dr

I think there's a place for code-first design. Build something to explore the space and learn about the problem domain. Refactor. Or Delete and Start Again. In modern languages (i.e., Python) code is cheap. Whiteboard design may not actually save effort.

I think there's a place for building functions and refactoring them into classes. I think the Java pre-8 "All Classes or Burn In Hell" approach is misleading. Functional programming and languages like Python show that functions should be a first-class part of programming.

I think there's too much emphasis on stateful objects. The DDD warnings about "anemic" classes seems to come from a habitual over-emphasis on state and an under-emphsis on operations. I think that active classes (as much as they push the REST envelope) might be a good thing.

Obstinate Idiocy, Expanded

See Obstinate Idiocy for some background.

Here are three warning signs I was able to deduce.

• No Rational Justification
• Ineffective Tool Choice
• Random Whining

To which I can now add two more.

Symptom 4 of Obstinate Idiocy is that all questions are rhetorical and they often come with pre-argued answers.

Actual email quote:

Me: ">Excel is almost the stupidest choice possible

OI: "What criteria are you using to make that statement?

My criteria was that I needed a way for non-tech people and non-programmers..."

And on the email spins, pre-arguing points and pre-justifying a bad answer. Since their argument is already presented (in mind-numbing detail), there's no effective way to answer the question they asked. Indeed, there's little point in trying to answer, since the pre-argued response is likely to be the final response.

In order to answer, we have to get past the pre-argued response. And this can be difficult because this devolves to "it's political, you don't need the details." So, if it's not technical, why am I involved?

Symptom 5 of Obstinate Idiocy is Learning is Impossible. This may actually be the root cause for Symptom 3, Ineffective Tool Choice. It now seems to me that the tool was chosen to minimize learning. I had suggested using Mathematica. I got this response: " I don't know Python or R or SAS." The answer seems like a non-sequitur because it is. It's justification for a bad decision.

The problem they're trying to solve is gnarly, perhaps it's time to consider learning a better toolset.

Excel has already failed the OI. They asked for an opinion ("Q2: What do you believe are the pros/cons of ... using Excel with "Excel Solver" ...?") that seems to ignore the fact that they already failed trying to use Excel. They already failed, and they followed up by asking for the pros and cons of a tool they already failed with. 

From this limited exchange it appears that they're so unwilling to learn that they can't gather data from their own experience and learn from it.

Obstinate Idiocy [Updated]

Once in a great while, you see someone engaging in Obstinate Idiocy.

Here's my recent example.

They're solving some kind of differential equation. Not sure why. Symptom 1 of Obstinate Idiocy is No Rational Justification. The explanation is often "that's not relevant, what's relevant is this other thing I want to focus on" or something equivalent to "never mind about that."

The equation is this:

\[y + 3 \ln \lvert y \rvert = \frac{1}{3}(x-1)^3-\frac{11}{3}\]

Pretty gnarly.

Apparently, they were so flummoxed by this that they immediately turned to Excel.  Really.  Excel.

Symptom 2 of Obstinate Idiocy is Random Tool Choice. Or perhaps Ineffective Tool Choice. A kind of weird, unthinking choice of tools.

Of course, Excel struggles with this sort of thing, since it appears to gnarly. I was told that there's an Excel Solver, but there was some problem with using it. It didn't scale, or it required some understanding of the shape of the equation or something.

Symptom 3 of Obstinate Idiocy is Seemingly Random Whining. It's random because there's no rational justification for what's going on and the tool was chosen apparently at random.

Ask a question like "why not use another tool?" and you don't get an answer. You get an argument about tool choice or the politics of the situation or "tool choice isn't the point" or some other dismissive non-answer.

Ask a question like "what are you really trying to do?" and you get user stories that make approximately no sense. We had to endure a long discussion on system-assigned surrogate keys as if that was somehow relevant to the graphing the equation shown above. See Symptom #1. There's just no reason for this. It's Very Important, but No One Else Can Understand The Reason Why.

How To Begin?

So, now we're at this weird impasse.

We have the obstinate idiot who won't discuss their tool choice. Somehow I'm supposed to sprinkle around some Faerie Dust and magically make Excel do something better or different than what it normally does. Indeed, I'm having trouble understanding any of the whining about Excel.

Clearly, they've never heard of MatLab or Mathematica or any commercial product that does this nicely. Apparently, they've never even seen the graph tool on Mac OS X which simply draws the graph with no effort of any kind on the part of the user.

Clearly, they've never seen Google and can't make it work.

They asked how a Pythonista would approach a problem this gnarly. I couldn't even properly understand that question, since they hadn't Googled anything and didn't really have a question that could be answered. As a Pythonista, I use Google. I wasn't sure how to approach an answer, since I couldn't really understand what their goal was or what their knowledge gap was.

Since their principal complaint was about Excel, asking a Python-related question didn't make much sense. Were they planning on dropping Excel? If so, why not use MatLab or Mathematica?

See Symptom 2. The tool choice was fixed. Other tools weren't on the table. If so, why ask about Python?

At this point, the place to begin seems to be this link: http://bit.ly/11usbtH And that's not going to be helpful with the Obstinate Idiot. They'll claim they already knew all of that, they just needed some additional or different help.

They specifically said they weren't going to use Python. Which raises the question "Why ask me anything, then?" To which there was no real answer, just sulking about me not being helpful.

Correct. I'm not being helpful. I can't figure out what the problem is. There's a gnarly formula and Excel somehow doesn't work in some optimal way. And database surrogate keys. And departmental politics.

Did You Try This?

The equation simplifies to

\[ x = (3y + 9 \ln \lvert y \rvert + 11)^{\frac{1}{3}} + 1 \]

Which is really easy to graph. \(x=f(y)\) is, of course, not the usual approach of \(y=f(x)\).

Apparently, the Obstinate Idiot had not actually applied algebra to the equation. Nor had they ever conceived of graphing \(x=f(y)\).

Which brings us to Symptom 4 of Obstinate Idiocy: Slow To Ask For Help.

And the variation on Symptom 1 of Obstinate Idiocy: Goal-Free Activity. By this I mean that the thrashing around with Excel and discussing Python was all just a long, drawn-out and utterly irrelevant side-bar from the real purpose, which apparently was to find something out related to a differential equation. It's still unclear what the equation is being used for and why the graph is helpful.

Python Approach

First: Differential Equations are hard. Nothing makes them easy.

Interactive Python, however, can be of some help after you've taken the first steps with pencil and paper.

Here's a console log of something I did to help the Obstinate Idiot.

>>> import math
>>> import pprint
>>> 
>>> def lde_1(y):
...     try:
...         x = (3*y+9*math.log(abs(y))+11)**(1/3)+1
...     except ValueError:
...         x = float("NaN")
...     return x
... 
>>> def eval(y_lo=-15, y_hi=15, y_step=0.5, f_y=lde_1):
...     # Next smaller power of 2: prettier numbers. Less noise.
...     step_2 = 2**math.floor(math.log(y_step, 2))
...     for t in range(int((y_hi-y_lo) // step_2)):
...         y = y_lo + step_2*t
...         x = f_y(y)
...         yield( x, y )
... 
>>> data1= list(eval())
>>> pprint.pprint(data1)

I'll leave out the dump of the data points. However, it's possible to see the asymptote at zero and the ranges where the results switch from real to complex numbers.

We can drill into the region around zero to see some details.

data2 = list(eval(-2, 2, .0625))
pprint.pprint(data2)

These are just numbers.  A picture is worth a thousand numbers.

We have lots of choices for graphic packages in Python. The point here, however, is that evaluating the gnarly equation required two preliminary steps that were far, far more important than choosing a graphic package.

1. Do some simple algebra.
2. Write a simple loop.

If output to Excel is somehow important, there's always this.

>>> import csv
>>> with open("data.csv","w") as target:
...    wtr= csv.writer(target)
...    wtr.writerows(data1)

That will produce a CSV that Excel will tolerate and display as an X-Y scatter plot.

A jupyter notebook with pyplot will knock out a picture directly, allowing visualization.

Enriched Details of Apache Common Log Format

See Apache Log Parsing for the background.

Here's a generator function which expands a simple Access to be a more detailed named tuple.

Access_Details= namedtuple( 'Access_Details',
    ['access', 'time', 'method', 'url', 'protocol'] )
            
def details_iter( iterable ):
    def parse_time_linux( ts ):
        return datetime.datetime.strptime( ts, "%d/%b/%Y:%H:%M:%S %z" )

    def parse_time_macos( ts ):
        dt= datetime.datetime.strptime( ts[:-6], "%d/%b/%Y:%H:%M:%S" )
        tz_text= ts[-6:]
        sign, hh, mm = tz_text[:1], int(tz_text[1:3]), int(tz_text[3:])
        minutes= (hh*60+mm) * (-1 if sign == '-' else +1)
        offset = datetime.timedelta(minutes = minutes)
        tz= datetime.timezone( offset, tz_text )
        return dt.replace(tzinfo=tz)
        return dt

    first, last = None, None
    for access in iterable:
        meth, uri, protocol = access.request.split()
        dt= parse_time_macos( access.time )
        first= min(dt,first) if first else dt
        last= max(dt,last) if last else dt
        yield Access_Details(
            access= access,
            time= dt,
            method= meth,
            url= urllib.parse.urlparse(uri),
            protocol= protocol )

    print( "Log Data from", first, "to", last, 'duration', last-first )

This "wraps" the original Access object with an Access_Details that includes information that isn't trivially parsed from the access row.

• The datetime object with the real timestamp.  Not the Mac OS subtlety.  Due to platform issues, the %z strptime format doesn't seem to work in Python 3.2
• The three fields from the request: method, URL and protocol.  
• The URL is parsed into its individual fields.

Note that the Access_Details object is pickle-able.  While seemingly irrelevant, it turns out that having something which can be pickled means that we can use multiprocessing to create a multi-staged concurrent pipeline of log analysis.

What's important here is that we're adding functionality without redefining the underlying Access class. Indeed, the underlying Access object is immutable.  The idea of stateless values comes from the functional programming crowd.  It seems to work out really well because the functionality seems to accrete in relatively simple layers.

Apache Log Parsing

How much do I love Python?  Consider this little snippet that parses Apache logs.

import re
from collections import defaultdict, named tuple

format_pat= re.compile( 
    r"(?P<host>[\d\.]+)\s" 
    r"(?P<identity>\S*)\s" 
    r"(?P<user>\S*)\s"
    r"\[(?P<time>.*?)\]\s"
    r'"(?P<request>.*?)"\s'
    r"(?P<status>\d+)\s"
    r"(?P<bytes>\S*)\s"
    r'"(?P<referer>.*?)"\s' # [SIC]
    r'"(?P<user_agent>.*?)"\s*' 
)

Access = namedtuple('Access',
    ['host', 'identity', 'user', 'time', 'request',
    'status', 'bytes', 'referer', 'user_agent'] )

def access_iter( source_iter ):
    for log in source_iter:
        for line in (l.rstrip() for l in log):
            match= format_pat.match(line)
            if match:
                yield Access( **match.groupdict() )

That's about it.  The access log rows are now first-class Access-class objects that can be processed pleasantly by high-level Python applications.

Cool things.

1. The adjacent string concatenation means that the regular expression can be broken up into bits to make it readable.
2. When the named tuple attributes match the regular expression names, we can trivially turn the match.groupdict() into a named tuple. 
3. By using a generator, the other parts of the application can simply loop through the results without tying up memory to create vast intermediate structures.

A couple of years back, a sysadmin was trying to justify spending money on a log analyzer product.  I suggested they (at the very least) get an open source log analyzer.

I also suggested that they learn Python and save themselves the pain of working with a (potentially) complex tool.  Given this as a common library module, log analysis applications are remarkably easy to write.

A Taxonomy of Use Case Errors

First, the definition. A use case describes an actor's interaction with a system to create business value. There are three parts: Actor, Interaction and Business Value.

1. Not Interactive.

1.1. The use case is just features and technical attributes with no actor interaction expressed.

1.2. The use case is just algorithms and processing with no connection to an actor or a goal.

2. No Business Value.

2.1. Incomplete

2.1.1. The use case focus on sequential operations with no value or goal.

2.1.3. The use case simply follows existing precedent without supporting actual business goals. It "paves the cow path".

2.2. Non-Specific

2.2.1. The use case is a result of free-running imagination; it conflates "possibly" vs. "required". It contains descriptions of interactions which could happen or would be nice to happen.

2.3. Covers the Technology Only

2.3.1. The solution technology is conflated with the business problem. Words like "database" or "foreign key" or "error log" or other solution technology are central.

2.4. Contradictory

2.4.1. The use case goal contradicts other goals.

2.4.2. The use case sequence is inconsistent with the stated goal.

3. No Actor.

Questions, or, How to Ask For Help

Half the fun on Stack Overflow is the endless use of closed-ended questions. "Can I do this in Python?" being so common and so hilarious.

The answer is "Yes." You can do it.

Perhaps that's not the question they really meant to ask.

See "Open versus Closed Ended Questions" for a great list of examples.

Closed-Ended Questions have short answers, essentially yes or no. Leading Questions and presuming questions are common variations on this theme. A closed-ended question is sometimes called "dichotomous" because there are only two choices. They can also be called "saturated", possibly because all the possible answers are laid out in the question.

Asking Questions

The most important part about asking questions is to go through a few steps of preparation.

1. Search. Use Google, use the Stack Overflow search. A huge number of people seem to bang questions into Stack Overflow without taking the time to see if it's been asked (and answered) already.
2. Define Your Goal. Seriously. Write down your objective. In words. Be sure the goal includes an active-voice verb -- something you want to be able to do. If you want to be able to write code, write down the words "I want to write code for [X]". If you want to be able to tell the difference between two nearly identical things, write down the words "I want to distinguish [Y] from [Z]". When in doubt, use active voice verbs to write down the thing you want to do. Focus on actions you want to take.
3. Frame Your Question. Rewrite your goal into a sentence by changing the fewest words. 90% of the time, you'll switch "I want to" to "How do I". The rest of the time, you'll have to think for a moment because your goal didn't make sense. If your goal is not an active-voice verb phrase (something you want to do) then you'll have trouble with the rewrite.

In some cases, folks will skip one or more steps. Hilarity Ensues.

Leading/Presuming Questions

Another form of closed-ended question is the veiled complaint. "Why doesn't Python do [X] the way Perl/PHP/Haskell/Java/C# does it?"

Essentially, this is "my favorite other language has a feature Python is missing." The question boils down to, "Why is [Y] not like [Z]?" Often it's qualified by some feature, but the question is the same: "Regarding [X], why is Python not like language [Z]?"

The answer is "Because they're different." The two languages are not the same, that's why there's a difference.

This leads to "probing" questions of no real value. "Why did Python designers decide to leave out [X]" and other variants on this theme.

If the answer was "Because they're evil gnomes" what does it matter? If the answer was "because it's inefficient" how does that help? Feature [X] is still missing, and all the "why?" questions won't really help add it back into the language.

It's possible that there's a legitimate question hidden under the invective. It might be "How do I implement [X] in Python? For examples, see Perl/PHP/Haskell/Java/C#." Notice that this question is transformed into an active-voice verb: "implement".

If we look at the three-step question approach above, there's no active-voice verb behind a "why question". What you "know" isn't really all that easy to provide answers for. Knowledge is simply hard to provide. Questions about what you want to do, are much, much easier to answer.

Probing/Confirming Questions

One other category are the "questions" that post a pile of details looking for confirmation. There are three common variations.

• tl;dr. The wealth of detail was overwhelming. I'm a big fan of the "detail beat-down". It seems like some folks don't need to summarize. There appear to be people with massive brains that don't need models, abstractions or summaries, but are perfectly capable of coping with endless details. It would be helpful if these folks could "write down" to those of us with small brains who need summaries.
• No question at all, or the question is a closed-ended "Do you agree?" An answer of "No." is probably not what they wanted. But what can you do? That's all they asked for.
• Sometimes the question is "Any comments?" This often stems from having no clear goal. Generally, if you've done a lot of research and you simply want confirmation, there's no question there. If you've got doubts, that means you need to do something to correct the problems.

Here's what is really important with tl;dr questions: What do you want to do?

80% of the time, it's "Fix my big, complex tl;dr proposal to correct problem [X]." [X] could be "security" or "deadlock" or "patent infringement" or "cost overrun" or "testability".

Here's how to adjust this question from something difficult to answer to something good.

You want to know if your tl;dr proposal have problem [X]. You're really looking for confirmation that your tl;dr proposal is free from problem [X]. This is something you want to know -- but knowledge is not a great goal. It's too hard to guess what you don't know; lots of answers can provide almost the right information.

Reframe your goal: drop knowledge and switch to action. What do you want to do? You want to show that your tl;dr proposal is free from problem [X]. So ask that: "How do I show my tl;dr proposal is free from problem [X]?"

Once you write that down, you now have to focus your tl;dr proposal to get the answer to this question. In many cases, you can pare things down to some relevant parts that can shown to be free from problem [X]. In most cases, you'll uncover the problem on your own. In other cases, you've got a good open-ended question to start a useful conversation that will give you something you can do.

Finding Simplicity

In Creating Complexity Where None Existed, I noted that it's possible to create complexity out of thin air.

Indeed, by wallowing in the supposed drama, one can turn the differences between sales and service delivery into a hopelessly complex situation. A focus on a manufactured "conflict" leads to the following question: "What are the standard techniques for conflict resolution ?"

Standard Techniques. Conflict Resolution.

First, there's no "conflict". Sales offers something. The customer may or may not understand that offer. The customer commits to something. Sales may or may not understand what the customer thinks they're buying. And delivery has to fill in these gaps between what sales offered and what the customer thought they were buying.

It's very simple. I called the lawn service, asking someone to mow my lawn. But they didn't trim my hedge. No one asked me if I had a hedge. I didn't have one when I called. I had it put in after I placed the order for the services. Is this "conflict"? Does it require "resolution"? Or, does it require that the folks selling the services on the phone and the folks delivering the service have some smarter way of coping with the inevitable differences of understanding?

Reality

The trick to avoiding 482 words of drama (using code names!) to describe sales and delivery is simple. Get Out Of Fantasy Land.

In the real world, sales has one view of the order, and delivery has a different view.

This is not news. Accounts cope with this variability all the time. They ask people to create a "budget" or a "plan". And then they measure actual expenditures against the budget. Budgets changes. Actuals don't match the budget. This is not "conflict". There's nothing to "resolve".

There's planned and actual and they're different.

The real sales promise, memorialized in an "order" is one thing. This order can change, of course, making things complex.

The delivery on that promise, memorialized in an "invoice" is another thing. The delivery may be done in "scramble" mode; folks struggling to balance ability to deliver against promises made. Or, the delivery may be done in a more leisurely pace; the work being fit in to a schedule as time and resources permit.

Ideally -- of course -- order and invoice match. In reality, they don't always match.

Before the invoice is sent, someone needs to be sure it matches reality; someone has to affirm that the work was actually done. It may not be what the customer ordered, in which case there will be issues to resolve.

But there is not "conflict". There's no "drama". We don't need to assign code names ("Flintstones", "Rubbles") to sales and delivery.

Above all, we don't need to impose a weird legacy software world-view on something as simple as order, delivery and invoice. Sales has orders. Delivery has invoices. Hopefully, sales order changes get to delivery in time to adjust what's really happening. Some has to look at the mismatches and exceptions to determine what the consequences are. Maybe the customer gets a credit. Maybe someone sales is over-promising. Maybe someone in delivery, is under-delivering.

How to Derail Use Case Analysis: Focus on the Processes

It's easy to prevent successful use case analysis: make it into an exercise of defining lots of "processes" in excruciating detail.

First, ignore all "objects" definition.  All business domain entities -- and actors -- must be treating as second class artifacts.

Second, define everything as a process.  A domain entity is just some stuff that must be mapped between processes.  Act like the entity doesn't really have independent existence.

Symptoms

You may be trying to do use case analysis, but if you have these symptoms, it might be time to step away from the process flows and ask what you're really doing.

There Are No Actors.  Well, actually, there's one actor: "user".  When all of your use cases have one actor, you've forgotten the users and their goals.  Stop writing the processes and take a step back.  Who are the users?  What are they trying to accomplish?  Where is their data?  When is it available?  What interactions with a system would make them happier and more productive?

Every Action Defines A New Class of Actors.  You have actors like content creators, content updater, content quality assurance, content refinement, content link checking, do this and do that.  Too many actors is easy to spot because the attributes and behaviors of all those actors are essentially identical.  In this example, they all edit content.

Each Use Case is a Wizard.  If each use case is a strictly sequential input of a data element followed by "click next to continue", you've taken away the actor's obligation to make decisions and take action on those decisions.  If you're lucky, you've got a use case for each individual goal the actor has.  More typically, you've overlooked a fair number of the actor's goals in your zeal of automating every step of one goal. 

You Need an "Overall Flow" or Sequence for the Use Cases.   If your use cases have to be exercised in one -- and only one -- order, you've taken away the actor's goals

Collaboration

Use Case analysis describes the collaboration between actors and a system to create something of value.  If the system is described by wizards or modal dialogs that completely constrain the conversation to one where the system asks the actor for information, something's terribly wrong.

The point is to describe the system as a series of "interfaces", each of which has a use case.  The actors interact with the system through those interfaces.    The actor is free to gather information from the system, make decisions, and take action via the system.

War Story

The users had a legacy "application" that was a pile of SAS code that did some processing on the source data before reporting.

The use cases were -- essentially -- "1.  Actor runs this program  2. System does all this stuff."  The "all this stuff" was usually a lengthy, complex reverse engineering exercise trying to discern what the SAS code did.  

No mention of the business value.  No reason why.  And no room to implement a better process.

War Story

Analyst is pretty sure the user wants collaborative editing.  The analyst has a pretty good "epic" (not a proper user story, but a summary of a number of user stories) that describes creating, modifying and extracting from a collaboratively edited document.

The initial discussion lead to every single verb somehow defining a separate actor.  In the original epic, there were exactly two actors, one who added or elided certain details for the benefit of another.

Later discussions lead to a single "User" actor and the craziest patchwork of use cases.  Random "might be nice to have"s crept in to the analysis, and the original "epic" was dropped.  No trace of it remained, making it very difficult to determine priorities.

War Story

Users had developed a complex work-around because they didn't have all the equipment they needed in their local office.  It involved mailing CD's from one office to another to prevent network bandwidth problems.  The business analysts wanted to capture this process, even though parts of it created no value.

It took a fair amount of work to get the analysts to stop documenting implementation details (mailing addresses, Fedex account numbers) and start documenting interactions and the business value that was created.  

Many process steps are physical moves and don't involve making information available for decision-making.  Those no-decision physical move steps should not be described in a use case.  Perhaps in an appendix, but their incidental because they're just the current implementation.  A use case should have the essence of the business value and how the actor uses the system to create that value.