Books! Books! More Channels!

I started with the Apple Books platform because it's an easy default for me. 

Pivot to Python

A Guide for professionals and skilled beginners

https://books.apple.com/us/book/pivot-to-python/id1586977675 

I've recently updated this to fix some cosmetic problems with title pages, the table of contents and stuff like that. The content hasn't changed. Yet. It's still an introduction to Python for folks who already know how to program, they want to pivot to programming in Python. Quickly.

But wait, there's more. 

Unlearning SQL

When your only tool is a hammer, every problem looks like a nail

https://books.apple.com/us/book/unlearning-sql/id6443164060

Many folks know some Python, but struggle with the architectural balance between writing bulk processing in SQL or writing it in Python. For too many developers, SQL is effectively the only tool they can use. With a variety of tools, it becomes easier to solve a wider variety of problems effectively.

Google Play

Now, I'm duplicating the books on Google Play. Here's Unlearning SQL:

https://play.google.com/store/books/details?id=23WAEAAAQBAJ

I've made a clone of Pivot to Python, also.

https://play.google.com/store/books/details/Steven_F_Lott_Unlearning_SQL?id=23WAEAAAQBAJ&hl=en_US&gl=US

Both books are (intentionally) short to help experts make rapid progress.

Books! Books! Books!

First, there's 

Pivot to Python

A Guide for professionals and skilled beginners

https://books.apple.com/us/book/pivot-to-python/id1586977675 

I've recently updated this to fix some cosmetic problems with title pages, the table of contents and stuff like that. The content hasn't changed. Yet. It's still an introduction to Python for folks who already know how to program, they want to pivot to programming in Python. Quickly.

But wait, there's more. 

Unlearning SQL

When your only tool is a hammer, every problem looks like a nail

https://books.apple.com/us/book/unlearning-sql/id6443164060

This is all new. It's written for folks who know Python, and are struggling with the architectural balance between writing bulk processing in SQL or writing it in Python. For too many developers, SQL is effectively the only tool they can use. With a variety of tools, it becomes easier to solve a wider variety of problems.

What Books Should I Read? In What Order?

A fascinating question.

... I'm baffled by the amount of books you've published over the course of time. Currently Reddit suggests that I use Building Skills in Python under Beginner's section, but it looks quite outdated. So back and forth, I found your Building Skills in OO on GitHub Page and was quite happy with the read on the first 100 pages.

I searched for more info on the books you've published and wanted to know if you could sort them in ascending order of difficulty for me as I intend to purchase them slowly.

My main concern to learn Python is just to cross technical interviews and building applications that help with my workflow (they are in bash with around 200 functions, so I'm hoping to migrate them to something which is more robust). 

Currently the focus I intend to develop is on:

1) Strong Foundations of the Python Language.

2) Strong Foundations on the Basic Libraries for Data Structures and Algorithms (For example, bisect gives me insort(), calendar gives me isleap(), iter_tools gives me permutation(), etc).

3) Strong Foundations on the Design Patterns.

So could you please help me out and suggest your books?

This is challenging for a few reasons.

First, the "Building Skills" books have been reduced to only the Building Skills in OO Design. This can be found in GitHub. https://github.com/slott56/building-skills-oo-design-book.

That book is not really targeted to beginners, though. It presumes some core OO skills, and provides a (very) long series of exercises to build on those skills.

Second, I never really conceived of a beginner-to-expert sequence of books. From your letter, I see that I need to look at filling in some gaps in my sequence of books. I'll alert my editors at Packt, and we can consider this in the future.

Specific Needs

Let's look at your needs.

1. Foundations in the Python language.

This might be something you can learn from my Python Essentials. This isn't focused on complete n00bs. All of my books expect some programming background. Since you're an Android engineer and write code in C++ and Java, this may be helpful. This title is getting old, however, and needs a second edition.

For someone with core programming skills, I suspect that Mastering OO Python will be suitable. My Python 3 OO Programming (4th ed.) is similarly aimed at folks who can program and can learn a new language quickly.

A book like Martelli's Python in a Nutshell may provide a better foundation the way the language works than any of mine. Also Lutz's Learning Python is extremely popular.

2. Foundations in the Standard Library.

This is tricky. I touch on some of these topics in Functional Python Programming (2nd ed.) I also touch on some of these topics in the Modern Python Cookbook (2nd ed.)

I don't, however, cover very much of the library. I touch on a few really important modules. The library is vast. A book like Hellmann's The Python Standard Library by Example might be more suitable than one of mine.

3. Design Patterns.

This is central to Python 3 OO Programming (4th ed.) Dusty Phillips and I cover a number of popular design patterns in detail. 

There are -- of course -- a lot of very, very good books on Python. I'm honored you reached out to me.

Other Random Advice

Because Python is a relatively simple language (with a vast library) I have always suspected that language foundations don't really require a ton of explanation. Many languages (i.e., C++) are filled with odd details and weird features that are really unpleasantly complex. Many Java programmers get used to the distinction between the primitive int type and the Integer class type. While the Java and C++ approach can seem simple (after a while) it really isn't simple at all.

The standard library is vast, and it takes time to get used to how much is there. I would suggest having a browser tab open to https://docs.python.org/3/library/ at all times.

Design patterns, similarly, require some care. There are complex details around implementing the Singleton pattern in C++ and Java. Python class definitions and Python module definitions are Singletons, and using a class definition as a Singleton object is often far simpler than the commonly-used techniques for C++ and Java.

Refactoring

Follow my Patreon: Become a Patron!

I'll try to focus on my Building Skills in OO Design book there. I'm thinking of adding some more code examples. Is that a good idea?

Maybe that should be the higher-level Patreon benefit?

Building Skills in OO Design

See https://www.patreon.com/posts/30995708

I've (finally) gotten the book content upgraded to Python 3.7.

I've also deleted all the previous versions of the book. I had been keeping them on my web server because -- well -- because I don't know why. They go back to at least 2011, some of the content may be even older than that.

I've also deleted some previous self-published content.

(I started writing about Python almost 20 years ago. Some of the content could have been that old. It deserves to be deleted.)

Finally Planning the Rewrite of Building Skills in Object-Oriented Design

See Building Skills in Object-Oriented Design for the old content, which has a number of features that hold up well over time.

1. A graduated series of exercises to build up large, complete applications is important. 
2. It covers a lot of skills essential to building real applications -- unit testing, integration, code reuse. I want to expand on this to include more testing strategies, and final documentation.
3. It's so popular, I've got enough donations to move forward on a rewrite.

Previously, it was hosted out of my ancient web site as HTML and PDF download. That hasn't aged well.

Also, it was originally Python 2, and that ship sailed years ago.

I'm leaning toward hosting the content on GitHub.

One idea is to have a complex project with the following top-level folders:

• A docs folder that has the HTML as well as PDF (and maybe an ebook format, too.)
• A src folder with seed files for the various packages and modules.
• A tests folder with seed tests.

Someone could fork and then build on the framework.

It's possible to put the exposition into the wiki pages associated with the repo. This has the advantage of keeping the meta-level documentation and individual project requirements separate from the project itself.

Before I go too far, I'll need to experiment a bit to see what the editing process is like. The Github wiki pages are their own git branch, and are easy to edit off-line and push to the repo. Some of the fancy Sphinx markup features vanish, replaced with basic RST. This may not be all bad, since the baseline content is not *very* complex.

Stand by for more.

This is awkward

Years ago -- many years ago -- I self-published three books on Python.

I also set up a Google Group, Building Skills Books, for a discussion area.

For a while, I check my download logs carefully to see how the book was being received.

Then I became a tech nomad, and commenced traveling and working from coffee shops and cellular network hotspots with my phone. The discussion group went from a luxury to a complexity to a near impossibility.

Now that I have steady connectivity, it's really kind of embarrassingly awkward that I ignored my readers like that.

For years.

I submit my apology to all of the folks who hoped for a more responsive author.

What can I do about it?

Nothing to fix the past, obviously. For the moment, while working for Packt, most of my writing goes to them.

However.

I hope to revisit these Building Skills Books in some depth in the not-too-distant future. I see three stories on my "As a Reader, I want..." storyboard.

1) Typos Fixed so the books are readable. Gather all the typos and corrections from the discussions.

2) Clarifications so the books are more useful. Gather all the questions, comments, suggestions. Fold those into the rewrites.

3) Python3. The beginner-level Intro to Programming book needs further revision. (I tried to publish an iBook for the Python3 rewrite and am not happy with the process or the results.) I think I'll revise it (yet again) and post it as GitHub pages.

4) Python3. The Intro to Python book needs to be rewritten. It's a HUGE project, but, I feel that it still has some value because it's chock-full of exercises and attempts to be really complete. I think I'll drop the reference material, however. Back in '02 -- when I started the first draft -- that seemed like it was beneficial. Now it's approximately worthless.

The OO Design book is a real hand-wringer. The approach of a strict parallelism between Python and Java can be seen as a disservice to both languages. On the other hand, I think it's good to focus on lowest-common denominator features that are common to all OO languages. I'm undecided on what to do here. I think I'd like to drop Java and add some additional refactoring exercises.

I need to pick one of these two:

4a) Python Focus for OO Design.

4b) Lowest Common Denominator Focus for OO Design.

There's no precise schedule for this; it's mostly a kind of placeholder and discussion jumping-off point. Maybe I should start a proper https://trello.com board for this.

Building Skills in Object-Oriented Design

New Kindle Edition of Building Skills in Object-Oriented Design.

It seems to work okay in my Kindle Readers.

I'm not sure it's really formatted completely appropriately. I'm not a book designer. But before I fuss around with font sizes, I think I need to spend some time on several more valuable aspects of a rewrite:

1. Updating the text and revising for Python 3.
2. Removing the (complex) parallels between Python and Java. The Java edition can be forked as a separate text. 
3. Reducing some of the up-front sermonizing and other non-coding nonsense.
4. Moving the unit testing and other "fit-and-finish" considerations forward.
5. Looking more closely at the Sphinx epub features and how those work (or don't work) with the KindleGen application which transforms .html to .mobi. This is the last step of technical production, once the content is right.

I have two other titles to finish for Packt publishing.

Maybe I should pitch this to Packt, since there seems to be interest in the topic? A skilled technical editor from Packt and some reviewers are likely to improve the quality.

The question, though, will be how to fit this approach to programming into their product offerings? Since I have two other titles to finish for them, perhaps I'll just set this aside for now.

The Getting Started Problem

How does one get started developing software? What's the first step?

When you come to this craft -- or sullen art -- without a background except as a user, how do you get started writing code?

It's not easy. Indeed, developing software may be one the hardest things there is. Really, really hard.

Why? Consider the orders of magnitude involved. From sub-microsecond clock speeds to software that's supposed to continue running for 8,763 hours a year without interruption. That's 31,547,269 seconds. Isn't that about 15 orders of magnitude?

Or consider scope of storage. We wrangle over bytes in a dataset that spans terabytes. That's 12 orders of magnitude.

When engineers build a 13,000' long bridge, are they looking at it from scales of 10^±5? Do they even care what's 21 miles away? They might care about things at the scale of 10^-5, since that's about an inch. But 10^-7? 100^th of an inch? I could be wrong, but I have doubts.

I won't go so far as to say bridge building is particularly easy. It's safety critical work. People die when things go wrong. Consequently, it's regulated by civil engineering standards. Bridge designs are limited to proven patterns. You can't spring something new on the world and expect anyone to pay money for it or trust their life to it.

If you're with me so far, you see my point: software is different. And that makes it particularly hard. People do learn elements of it. How does this happen?

Two Paths Diverge

I see two separate paths:

• More formal, and
• Less formal.

The more formal path includes the kind of curriculum you find at big CS schools. Formal treatment of algorithms and data structures. Logic and Computable Functions. The essentials of Turing Completeness.

Books like http://en.wikipedia.org/wiki/Structure_and_Interpretation_of_Computer_Programs. 

The less formal path starts with -- essentially -- random hacking around, trying to get stuff to work. Some folks argue that a curriculum of structured exercises isn't "random" hacking around. I suggest that a curriculum of structured exercises can be the formal path concealed under a patina of hackeriness. On the other hand, a set of exercises can be successful at training programmers; if it doesn't follow a formalized structure, it's merely a small step from random. 

[Random doesn't mean "bad;" it means "informal" and "unstructured."]

Some folks learn well in a formal, structured approach. They like axiomatic definitions of computability, and they can get a grip on how to map the abstractions of computing to specific languages and problem domains. They read content at http://www.algorist.com and see applications of principles.

Other folks can be shown the formal background that makes their random hacking fit into a larger pattern. When shown how some things fit a larger pattern, they're often happy work in a new context with an expanded repertoire of data structures and algorithms. They read content at http://www.algorist.com and look for solutions to problems; the formal patterns will emerge eventually.

Not all folks respond well to having their informal notions challenged. Some folks have ingrained bad habits and prefer to fight to the death to avoid change. A sad state of affairs, but remarkably common. They didn't understand linked lists at some point and steadfastly refuse to use the java.util.LinkedList class. This is what software religious wars are about. Some trolls truly and deeply love an uniformed religious war. 

Chickens and Eggs

Is this a chicken-and-egg problem? 

• You can't really appreciate the formal foundations until you have some hands-on coding experience.
• You shouldn't dirty your hands with implementation details until you have the proper theoretical foundations.

That seems potentially reductionist and uninformative. Or. Perhaps there is a nugget of truth in this. Perhaps one is actually foundational.

Eggs, to be specific, show the fresh mutations. The egg comes first from a chicken-like precursor that's not properly a chicken. 

What's that precursor to programming in Python? CS Fundamentals? Hacking around? I suggest that the way we acquire languages is important here.

Language Skills

Software languages are a small step from natural languages. As with learning natural languages, formal grammar may not be as helpful as engaging in conversations. Indeed, for natural languages, formal grammars are an afterthought. They're something we discover about a corpus. We impose the discovered grammar rules on ourselves (and others) to be understood in a context of other writing (and speaking.) 

Natural language grammar isn't timeless and immutable. People throw their hands up in despair at the erosion of grammar and language. They're -- of course -- just being reactionary. Language evolves. The loudest complainers are the ones who didn't pay attention for a long time and suddenly (somehow) realized the don't know what "WTF" means. LOL.

With an artificial language, the grammar is formalized. It has a first-class existence in compilers, interpreters and other tools. 

However, I think the bits of our brain that assimilate grammar work best from concrete examples. A formal grammar definition -- while helpful -- isn't the way to start. I think that a less formal, "try this" suite of exercises is perhaps the best way to learn to program.

As an author, I'm beholden to my publisher's notions of what sells. Examples sell. See almost everything from Packt. Working examples are solid gold. 

These are not necessarily problems for the reader to tackle and solve. They're examples to study.

The conundrum with attempting to solve problems is the attempting part. It's hard to set out a list of "solve these problems and master programming" problems and hope folks get through them. What if they fail? Clearly, you'd provide answers. In that case, you'd be back at examples to study. Hmm.

I have intermittent interest in my older Building Skills in Python book. Partly because it's got extensive exercises in each chapter. I get donations. I get inquiries. The exercises seem to resonate in a small way.

I've done about 22 levels of the Python Challenge (I'll write about that separately.) It's not a great way to learn from scratch. You need to know a lot. And you need a lot of hints. 

I've done almost 70 levels of Project Euler. It might be a better way to learn programming because the easy problems are really easy. No guesswork. No riddles. No steganography. The answers are totally cut-and-dried, unambiguous, and absolute. However, there's no easy guidance for learners. Either you have an answer, and want help on improving it, or ... well ... you're stuck and frustrated. 

Structured Sequence of Exercises

What strikes me as a possibility here is a structured series of exercises that lay out the foundations of computer science as realized in a specific programming language.

Puzzle-style. With extensive hints. Background readings, too. But with absolutely right answers. And a score-keeping system to show where you stand. 

No tricky riddles. No quizzes to proceed. You could go on to advanced material without mastering the foundations, if you wanted.

I've got a bunch of exercises and examples in my Building Skills books. Plus some of the examples in my Packt books can be modified and repurposed. Plus. Projects like HamCalc contain a wealth of simple applications that can be adjusted to show CS fundamentals.

Perhaps relevant is this: https://www.google.com/edu/programs/exploring-computational-thinking/.   I'm not sure precisely how it fits, since it seems to be more aimed at providing a general background, rather than teaching programming language skills. They decompose the skills into four specific techniques. Here are specific techniques.

• Decomposition: Breaking a task or problem into steps or parts.
• Pattern Recognition: Make predictions and models to test.
• Pattern Generalization and Abstraction: Discover the laws, or principles that cause these patterns.
• Algorithm Design: Develop the instructions to solve similar problems and repeat the process.

Perhaps this is relevant: http://interactivepython.org/courselib/static/pythonds/index.html.  I haven't read this carefully, but it seems to be expository rather than exploratory.  It's really thorough. It has quizzes and self-checks. 

I think there's a big space for publishing lots simple recreational programming exercises as teaching tools. 

Introduction to Programming: iBook Edition for Python 3.2

That was challenging.

I rewrote almost all of my Introduction to Programming book into an iBook. Trimmed it down. Refocused it. Changed from Python 2.6 to 3.2. A complete refactoring from which almost nothing of the original book survives except the goals.

Look for it October 1st in the iTunes bookstore. Programming for Absolute Beginners: Building Skills in Programming.

[My intent is to have several Building Skills titles. We'll see how far I get.]

The rewrite involved three substantial changes.

1. I removed all of the duplicative reference material. The Python library reference is (now)  utstandingly good. When I started using Python over ten years ago, it was not very good, and I started writing a Python reference of my own merely to organize the documentation. The books grew from there; the reference aspect is now useless.
2. I dropped almost all Python 2 references. There's a little bit of Python 2 history, but that's it. It's time to move forward, now that most of the major packages seem to have made the switch. 
3. I changed the focus from processing to data.

Processing vs. Data

When looking at a multi-faceted language like Python, it's difficult to know what's the most gentle introduction to software development.

Historically, the procedural, imperative style of programming appears the most appealing. The roots of Python come from procedural programming. It reaches back to Pascal (and even Algol 60) by elegantly restating the core principles of those languages with an easier-to-read syntax.

Indeed, if you read classic foundational CACM articles where essential algorithms were first formally described, they used a neatly typeset variant on Algol that (for the early years of my career) was the gold standard in how code should look. Python follows this tradition nicely.

But.

That doesn't mean that procedural programming is really the absolutely best way to introduce the language.

Data First

I think that it may be possible to introduce the language with a focus on data objects first and the procedural/imperative statements as a secondary consideration.

When it comes to anything beyond trivial Rate-Time-Distance calculations, the data structure matters more than almost any other aspect of the software. The objects, their relationships, their operations and their attributes are core to the problem. The presentation, user actions and persistent representation are secondary considerations after the structure of the data.

It seems like the data structures should "drive" the presentation. The outline of the book should be introductions of each of the important and visible builtin data structures. Additionally, the library extensions that are most often used can be introduced, also.

Definitional features (def, return, yield, class, and the ideas of module and package) are central, but a step behind the builtin data structures.

Procedural features (if, for, while, break, continue, with, etc.) are clearly second-class; they exist only to support access to the data structures. A for statement, makes a "for all" assertion about a data structure. A for with a break (or a while) makes a "there exists" assertion about a data structure. The data is central. The imperative statements are secondary.

Other features (global, nonlocal, del, raise, try, etc.) are tertiary, and exist to create more elegant programs that don't annoy the other developers as much. 

This also means that generator expressions and comprehensions are first-class, front-and-center features of the language. This parallels the approach in the NLTK Book, which puts the focus on generator expressions as a way to clearly state the processing.

Other Forms

Currently, I only have the iBook available.

The iBook Author application can (and does) produce a PDF. I think I may offer that separately through www.lulu.com. 

Python Big Picture -- What's the "roadmap"? [Revised]

Here's an interesting idea: http://www.xmind.net/m/WvfC/

This is associated with the following question: "I've had a hard time finding the Big Picture re: Python, and it makes it difficult ... to proceed and prioritize my efforts without one."

An interesting question: what is the overview or strategy for mastering Python?

In this case, the focus is on "Big Data", but I've found that to be merely a tangential. The application area has a small influence, and then only around the fringes of the language and libraries.

I'm going to disagree with several particulars on the mind map. I'll present an alternative, with a point-by-point commentary on the mind map. (And I'll eschew the graphics, I don't find them helpful.)

Foundation

The language itself is (duh) the foundation. I find it important to emphasize this because the Python universe is replete with a seemingly endless supply of packages and libraries that help solve nearly every problem a programmer might encounter.

This profusion of packages is -- in a way -- it's own problem.

It's obligatory to run the following interaction in Python. (Any Python 2.7 or 3.2 will work; older Pythons prior to 2.7 need to be upgraded.)

>>> import antigravity

Yes.

Everything you can imagine is an add-on package. Everything.

But.

That's not the starting point for learning how to solve problems with Python. That's merely one waypoint along the course. And it's not the most important waypoint.

Attractive Nuisance

We have to set the external libraries aside as an "attractive nuisance." They're a distraction, in fact. Let's focus on the stuff that comes with the installation kit: language and library.

When looking at the Language, we actually see two things: Data and Processing. The "Data" is the built-in data structures: bool, int, float, complex, exception, context, string, tuple, list, map, set, lambda, function, class, module and package. The "Processing" is the imperative programming features: the 21 (or so) statements that comprise the language.

Both facets are essential, but they're also (approximately) orthogonal to each other.

For years, I was convinced that the way to learn Python was to come to grips with most of the imperative statements and then apply these statements to the various data structures. The tidy orthogonality between many of the statements and some of the data structures makes this appealing. I wrote two Python tutorials based on this idea.

My approach was to echo the ancient Structured Concurrent Programming with Operating System Applications. They define a nested series of subsets of a hypothetical PL/I-like (or Pascal-like) programming language. While the details don't apply well to Python, the approach does make a lot of sense. Start with constants, expressions and output (i.e., print) as the minimal language. Then add state change via variables, assignment and input. Then add if/elif/else. Fold in for and while, and continue to add features in this careful progression: functions, exceptions, contexts, generators, etc.

I'm becoming less and less sure that the imperative, procedural statements should define the roadmap through the language.

It's true that computing is defined by number theory. The original Turing Machine theorem equates all of number theory to an imperative, procedural notion of computers and programming. While unconditionally true, it's not necessarily the most helpful strategy. We could, for example, start programming by covering Boolean Algebra and Set Theory first. But it would be a long dull slog before we got to anything that appeared "useful."

Data Is Central

I'm starting to see that the data structures are more helpful than imperative statements. This leads to a different approach to studying this language. Experienced programmers may feel that a list of fundamental language topics isn't too helpful.

However. I've noted that many experienced programmers tend to skip over the unique-to-Python features. This leads them to write clunky and awkward Python code because they missed something that would lead to simplicity and clarity.

1. int. Natural numbers are boring but necessary. The first explorations of Python can easily be simple expressions, output, variables and input using integers.
2. bool. Comparisons and logic allow introduction of the if, elif and else statements in a graceful way. 
3. str. Strings can be a gentle introduction to objects which are collections. Strings have methods, unlike integers and booleans. Strings introduce a number of conversion functions (int, float, str, hex, oct, etc.) This allows introduction of the for statement based on this simple collection.
4. float and complex. Floating point numbers are an important side-bar. They're not central. The notion of "approximation" can't be stressed enough, and pathological examples of noise bits at the end of floats is absolutely central. The math library is perhaps part of this. Also the decimal and rational modules.
5. Exception. For programmers who have a background in languages like C (without exceptions) the exception seems complex and mysterious. However, for Python they are absolutely central. And easy to play with by getting simple Value Errors. This introduces the try/except statements, also. While it's a little advanced, the class MyException( Exception ): pass is not a bad thing at this point. Yes, it's a bit of a "magical incantation." But so is len(string).
6. tuple and list. This is an extension to some of the discussion of string. It's also a time to introduce mutability and show some of the consequences of a mutable list. This introduces iterability, also.
7. dict and defaultdict. This introduces more loop constructs including list comprehensions and various kinds of generator expressions.
8. set and frozenset. This allows a review of mutability and the ways list and tuple differ.
9. function and lambda. The def and return statements, plus global. Additionally, the sort method of a list as well as the sorted iterator function can be looked at in some depth. 
10. file, open and context. This includes the with statement. This is a two-part or three-part exploration. It has to include some of numerous library packages for dealing with the file system. Plus data representation in CSV and JSON files. The way that a file is iterable is essential.
11. Iterators, generators and the itertools package. This includes techniques for implementing map-reduce algorithms using iterators and generators.
12. namedtuple. This is a small thing, but it can help to crystalize attribute access and some of the features that are part of a class.
13. class. This must include an multi-step excursion into special method names. 
14. module and package.  Note that these are different things. Java only offers "package". A Python module is a very, very important concept. The module (not the class) is the practical unit of reuse. Python is emphatically not written in the style of Java with one class per file. 

Class Definitions

The essential goal behind the first 14 topics is to get to the point where all the language features can be used to create workable class definitions.

1. Common object-oriented design patterns. Most of the "Gang-of-Four" suite of patterns is relevant to Python. A few changes to the textbook examples are required to remove the C++ and Java biases. Patterns like State, Strategy and Factory are central to good OO design. The Python version of Singleton has to be treated carefully; the Python Borg pattern is rarely useful; on the other hand the concept of module global variable is important and underpins some of the standard library
2. Above and beyond the common design patterns, Python has a number of unique design patterns. These are largely exemplified by the special method names. Attribute Access (properties and descriptors). This allows creation of simple collections.
3. Callable objects allows a review of functions and lambdas, also. The Abstract Base Class definitions must be emphasized for this to work out well in the long run.
4. Sequence Types expands simple collections to created ordered collections. 
5. Number Types. This allows a complete understanding of decimal and rational packages, also.
6. Some additional design patterns need to be added, also. Specifically, things like metaclass and classmethod are features of Python that are absent from Java or C++.

Programmers experienced in other languages might object to this depth in Python OO design techniques and design patterns.

What I find is that programmers who don't really "get" the Python design patterns (especially the ABC's) overwrite their programs. They needlessly reinvent methods that are already first-class features of the language, but weren't well understood. Properties and descriptors, for example, allow for a simpler and very clear syntax; it's often better than the endless parade of explicit getter and setter method calls that characterize Java Beans programming.

Additionally, bad habits from other languages need to be unlearned. For example, many Java (and C++) programmers are taught to overuse the private keyword. When they learn Python, they think that private is somehow really important.  When the find out about __ (double underscore) name mangling, they go off the deep end, using __ names everywhere. This is all bad.

Encapsulation has little to do with private. In Python, the _ (single underscore) is the convention for private. But it's not like Java's (or C++) compiler-enforced privacy, it's just a nodding understanding. As the creator of Python says "we're all adults here." An overused Java private is more of a problem for proper extension of a Java class than Python's casual "nudge-nudge-wink-wink-private".

The Standard Library

After looking at class definitions, it's important to look at the default library, subsection by subsection. There is a lot to the installed library.

For most Python programmers, sections 1 to 6 will have been covered by the previous material. Sections 26 and on to the end, also, are less important.

Sections 7 to 25 of the library reference contain the centrally important modules. A familiarity with the list of topics is essential before tackling "real" projects. This is so important, we'll use this set of topics as the basis for our point-by-point commentary on the mind-map linked above.

External Components and Downloads

One of the reasons why Python is a well-designed language is the way the principle of orthogonality is applied.

Most statements and data structures play well together. For example, all the built-in collections are sequences, so that they are iterable; the for statement works directly with collections.

Also, the external libraries themselves are all independent of the language, and the language exists without resorting to any of the external libraries.

Looking at the mind map, there are several interesting topics. And a few mysteries. And some unhelpful labels. Here's a quick commentary on the mind map.

• Basic Stack. I supposed these can be called "essential" external packages. This seems to be a way to emphasize other packages listed elsewhere on the diagram. I'm not sure why this topic is here.
• Newer Packages. This is a completely opaque label. Not helpful.
• Integrated Platforms. This isn't too helpful, either. I suppose one could make a guess based on the list of packages.
• Visualization. Ah. Now we're getting somewhere. These are some helpful visualization packages. PIL isn't listed, perhaps because it's too primitive.
• Data Formats. YAML isn't listed. The SQL and NoSQL categories make precious little sense. Those are all about persistence, not data formats. Data format and persistence are separate and unrelated. JSON, for example, is a data format. CouchDB is persistence.
• Packages. I suppose it's helpful to point out PyPi, but it doesn't make sense in this context. This is metadata and relatively unhelpful.
• Efficiency. Cython for "efficiency" makes precious little sense. Proper data structure and algorithm is the secret to efficiency. See my post on a 100:1 speedup in Python. For efficiency, it's sometimes necessary to drop out of Python and write the important 20% of the code in C++. 
• Parallel. A non-Windows OS handles parallelism gracefully. Process-level parallelism with pipelines is simple and efficient. Thread-level parallelism is often more trouble than benefit. 
• GPU. This is an example of where a little C++ code can go a long way to improving the 20% of the code that's the actual performance bottleneck.
• Glue. Interfaces to other applications or packages can be useful if the other package is actually a first-class part of the solution.
• MapReduce. This is essentially persistence, and goes with SQL database and noSQL databases. It's also a fundamental design pattern that can be exploited trivially in Python.

On this mind-map, there are a few topics that are really important. So important that the topics parallel the Python library.

• Data Persistence, chapter 11. Databases and Files. This includes SQL and noSQL databases as well as pickled data structures. Python comes with SQLite, allows SQL development without additional downloads. Postgres and MySQL libraries often popular because the price is right and the functionality is outstanding.
• Archive and Compressed Structures, chapter 12. ZIP, BZ2, etc. Compression is sometimes relevant for big data projects.
• Data Representation and File Formats, chapters 13, 18 and 19. CSV, JSON, YAML, XML, HTML, etc. It's important to note that JSON is more compact (and almost as expressive) as XML. While XML is popular, it's sometimes overused.
• OS Features, chapter 15 and 16. These are tools needed to build command-line applications. For Big Data applications, logging and command-line parameter parsing are essential.
• Multiprocessing. This is it's own design discipline. What's important here is that the OS process-level design is central. The queue and multiprocess packages are sufficient for this. There are some external multiprocessing packages, also, like Zero MQ.
• Internet Protocols, chapter 20. This is part of using RESTful web services, which is essential for making noSQL database (like CouchDB) work. For creating RESTful servers, the WSGI approach is essential.
• Unit Testing and Documentation, chapter 25. Sphinx is extremely important for creating useful documentation with minimal pain.
• Visualization. matplotlib, PIL are popular. The built-in turtle package is a bit primitive. However, it's also rather sophisticated, and a great deal can be done with it.
• Numeric Processing. numpy or scipy.

Note that the number of external packages on this list is rather small. Python comes with batteries included. 

Admittedly, it's hard to make general recommendations for external packages. But it's misleading to provide a huge list of external packages when the default suite of packages will solve a large number of problems gracefully.

Which Python Version?

Generally, everything should be done in Python3.2.

In some cases a crucial package hasn't been upgraded to Python 3.2. In these exceptions, Python 2.7 can be used.  For example, nltk is still focused on Python 2.7.

But.

Every Python2.7 program should always begin with

from __future__ import print_function, division

That's every and always. All new development should always be focused on Python3.2. There is no rational exception to this rule.

If there's any need to use the input() function, then the following line must be included, also.

input= raw_input

This will use the Python 3.2 version of the input() function.

Book Deal Fell Apart (sigh)

After spending a couple of years (really) working with a publisher, the deal has gone south.

The problem was—likely—all mine.  The book wasn't really what the publisher wanted.  Perhaps it was close and they thought they could edit it into shape.  And perhaps I wasn't responsive enough to criticism.

I like to think that I was responsive, since I made almost all of the suggested changes.

But the deal fell apart, so my subjective impression isn't too relevant.

What To Do?

I have two essential choices here.

1. Finish up.  Simply press on with the heavily-edited version.  The path of least resistance.
2. Roll back the content.  This involves going back to a less heavily edited version and merging in the relevant changes and corrections from the writing process.

Option 2 is on the table because my editor was unhappy with "digressive" material.  The editor had a vision of some kind of narrative arc that exposed Python in one smooth story line.  My attempted presentation was to expose language features and provide supporting details.

Perhaps I'm too deeply invested in the details of computer science.   Or.  Perhaps I'm just a lousy writer.  But I felt that the digressions were of some value because they could fill in some of the gaps I observed while coaching and teaching programmers over the last few decades.

In addition to the editorial challenge, there's technical challenge.  Do I step back from LaTeX?

• Use LaTeX.  This means that I would have to create the non-PDF version with a LaTeX to HTML translator.  Read Converting LaTeX to HTML in the Modern Age.  See Pandoc.  Or, it means that I don't offer an HTML version (a disservice, I think.)  Also, I need to unwind the publisher's LaTeX style libraries and revert to a plain LaTeX.  Since LaTeX is semantically poor, I need to rework a lot of RST markup.
• Revert to RST.  While RST tools can make both LaTeX and HTML from a common source, it does mean that some of the fancier LaTeX formatting has to go.  Specifically, I really got to know the algorithm and algorithmic packages.  I hate to give those up.  But maybe I can work something out with Sphinx 1.1.3's various features.

The problem is that three of the four combinations of paths have advantages.

• Finish up using LaTeX is easiest.  Remove the publisher's document style.  Use HeVeA or Pandoc to make HTML.  Move on.  
• Finish up but revert to RST.  This means conversion from the publisher's LaTeX to RST followed by some editing to fix the missing RST semantic markup and then some debugging to get the LaTeX and HTML to look good.
• Rollback the content using LaTeX.  This would be challenging because I would have to merge manually edited publisher-specific LaTeX from the heavily-edited version with the Sphinx-generated LaTeX from the less-heavily edited source.  Things like the publishers' style tags would complicate the merge. 
• Rollback the content and revert to RST. This means using Pandoc to convert the heavily-edited LaTeX to RST. Then merging the relevant edits into the RST original text.   This actually seems pretty clean, since the heavily-edited RST (converted from LaTeX) would be short.

Perhaps, if I was a better writer, I wouldn't have these problems.

It appears than the solution is MacFarlane's Pandoc.  This can reverse LaTeX to RST, allowing easy side-by-side merging of texts from various sources.  Or.  It can convert LaTeX to HTML, allowing easy work with the heavily edited LaTeX version.

Building Skills in Programming

I've revised (and streamlined) my Building Skills in Programming book.

The 2.6.2. edition will simply replace the 2.6.1. edition, leading to the possibility of broken bookmarks because of the changes.

Currently, the non-programmer book accounts for under 10% hits on the http://www.itmaybeaback.com/book site.  Consequently, I'm not very worried about the breakage.  I know someone will hate me for messing with the content just as they were starting to understand it.

I'm indebted to all my readers for the numerous suggestions, corrections and complements that I've received.  In order to simplify the correction process, I've put the source onto SourceForge.  See the Programming Book-Python 2.6 project.

The next step will be to add a PayPal donations button.

And... If I can get the PDF into really good shape, I may post it on Lulu for folks who really want a hardcopy.

Traffic

For a random 2-day period, the usage looks like this:

246 distinct "users" (really IP addresses).


{'html only': 161,
 ('both', 'oodesign-java-2.1'): 9,
 ('both', 'oodesign-python-2.1'): 5,
 ('both', 'programming-2.6'): 3,
 ('both', 'python-2.6'): 7,
 ('pdf only', 'oodesign-java-2.1'): 21,
 ('pdf only', 'oodesign-python-2.1'): 14,
 ('pdf only', 'programming-2.6'): 11,
 ('pdf only', 'python-2.6'): 37}

The "both" is a count of users reading HTML as well as the identified PDF editions.
For the "html only" and "both" users, there's a detailed list of particular books and sections.  Too large and boring to repeat here.

One interesting part is this detail:

82.128.23.63 {'programming-2.6': 37, 'oodesign-java-2.1': 28, 'python-2.6': 37, 'oodesign-python-2.1': 24} 

Apparently, Nigeria needs a lot of copies of the PDF.  I think I might want to block them, because this can't be anything sensible except endless polling by some botnet script.

Since we don't drop off cookies, we can't really identify user sessions.  Maybe in the future, I'll wrap the static content download with a simple WSGI application to drop off and collect cookies to track users instead of IP addresses.

How to Learn

A recent question.

i came up with two options.
 1.  building skills 1 (+ other references)... then algorithms & data
structures.... then your books 2 & 3

or

 2.  your three books 1,2 & 3... then algo & ds

kindly help me decide so i can start soon. 

I have two pieces of advice.

First.  Programming is a language skill.  Just like English.  If you can't get the English right, the odds of getting Python, Java, HTML or SQL right is considerably reduced.  Please, please, please take more care in grammar, syntax and punctuation.  Otherwise, your future as a programmer doesn't look very good.  For example, the personal pronoun is spelled "I".  In the 20th century, we spell out "and"; we stopped writing "&" as a stand-in for the Latin "et" centuries ago.  Also, ellipses ("...") shouldn't be used except when eliding part of a quote.  Clarity and precision actually matter.

Second, and more relevant, your two choices don't really amount to a significant difference.  If you're waiting around for advice, you're wasting your time.  Both sequences are good ideas. It's more important to get started than it is to carefully choose the precise and exact course of study. Just start doing something immediately.

Learning to program is a life-long exercise. There will always be more to learn. Start as soon as you can. The exact choices don't matter.  Why?  Because, eventually, you'll read all of those books plus  many, many others.

Spend less time waiting for advice and more time studying.

I almost wet myself

Someone sent me this: "“Building Skills in Python” – Steven F. Lott".

I had a vague idea that this book would get some traction.  This response was surprising.  I guess I should get to work on the upgrades.  And focus on the "no-nonsense" comment.

Intuition and Experience

First, read EWD800.

It has harsh things to say about relying on intuition in programming.

Stack Overflow is full of questions where someone takes their experience with one language and applies it incorrectly and inappropriately to another language.

I get email, periodically, also on this subject. I got one recently on the question of "cast", "coercion" and "conversion" which I found incomprehensible for a long time. I had to reread EWD800 to realize that someone was relying on some sort of vague intuition; it appears that they were desperate to map Java (or C++) concepts on Python.

Casting

In my Python 2.6 book, I use the word "cast" exactly twice. In the same paragraph. Here it is.

This also means the "casting" an object to match the declared type

of a variable isn't meaningful in Python. You don't use C++ or Java-style

casting.

I though that would be enough information to close the subject. I guess not. It appears that some folks have some intuition about type casting that they need to see reflected in other languages, no matter how inappropriate the concept is.

The email asked for a "a nice summary with a simple specific example to hit the point home."
It's quite hard to provide an example of something that doesn't exist. But, I guess, intuition provides a strong incentive to see things which aren't there. I'm not sure how to word it more strongly or clearly. I hate to devolve into blow-by-blow comparison between languages because there are concepts that don't map. I'll work on being more forceful on casting.

Coercion

The words coercion (and coerce) occur more often, since they're sensible Python concepts. After all, Python 2 has formal type coercion rules. See "Coercion Rules". I guess my summary ("Section 3.4.8 of the Python Language Reference covers this in more detail; along with the caveat that the Python 2 rules have gotten too complex.") wasn't detailed or explicit enough.

The relevant quote from the Language manual is this: "As the language has evolved, the coercion rules have become hard to document precisely; documenting what one version of one particular implementation does is undesirable. Instead, here are some informal guidelines regarding coercion. In Python 3.0, coercion will not be supported."

I guess I could provide examples of coercion. However, the fact that it is going to be expunged from the language seems to indicate that it isn't deeply relevant. It appears that some readers have an intuition about coercion that requires some kind of additional details. I guess I have to include the entire quote to dissuade people from relying on their intuition regarding coercion.

Further, the request for "a nice summary with a simple specific example to hit the point home" doesn't fit well with something that -- in the long run -- is going to be removed. Maybe I'm wrong, but omitting examples entirely seemed to hit the point home.

Conversion

Conversion gets it's own section, since it's sensible in a Python context. I kind of thought that a whole section on conversion would cement the concepts. Indeed, there are (IMO) too many examples of conversions in the conversion section. But I guess that showing all of the numeric conversions somehow wasn't enough. I have certainly failed at least one reader. However, I can't imagine what more could be helpful, since it is -- essentially -- an exhaustive enumeration of all conversions for all built-in numeric types.

What I'm guessing is that (a) there's some lurking intuition and (b) Python doesn't match that intuition. Hence the question -- in spite of exhaustively enumerating the conversions. I'm not sure what more can be done to make the concept clear.

It appears that all those examples weren't "nice", "simple" or "specific" enough. Okay. I'll work on that.

"Might Be Misleading" is misleading

My books (Building Skills in Programming, Building Skills in Python and Building Skills in OO Design) develop a steady stream of email. [Also, as a side note, I need to move them to the me.com server, Apple is decommissioning the homepage.mac.com domain.]

The mail falls into several buckets.

Thanks. Always a delight. Keep 'em coming.

Suggestions. These are suggestions for new topics. Recently, I've had a few requests for Python 3 coverage. I'm working with a publisher on this, and hope -- before too long -- to have real news.

Corrections. I get a lot of these. A lot. Keep 'em coming. I didn't pay a copy editor; I tried to do it myself. It's hard and I did a poor job. More valuable than spelling corrections are technical corrections. (I'm happy to report that I don't get as many of these.) Technical corrections are the most serious kind of correction and I try to fix those as quickly as possible.

Source Code Requests. No. I don't supply any source. If I send you the source, what skill did you build? Asking for source? Not a skill that has much value, IMHO. If you want to learn to program, you have to create the source yourself. That is the job. Sorry for making you work, but you have to actually do the work. There's no royal road to programming.

The "Other" Bucket

I get some emails that I file under "other" because they're so funny. They have the following boilerplate.

"Code fragment [X] might is misleading because [Y]."

First, it's a complaint, not a question. That's not very helpful. That's just complaining. Without a suggested improvement, it's the worst kind of bare negativity.

The best part is that — without exception — the person sending the email was not mislead. They correctly understood the code examples.

Clearly, the issue isn't that the code is "misleading" in the usual sense of "lying" or "mendacious". If it was actually misleading, then (a) they wouldn't have understood it and (b) there'd be a proper question instead of a complaint.

Since they correctly understood it, what's misleading?

User Interface Reviews

In software development, we used to go through the "misleading" crap-ola in user interface reviews. In non-Agile ("waterfall") development, we have to get every nuance, every aspect, every feature of the UI completely specified before we can move on. Everyone has to hand-wring over every word, every font choice, field order, button placement, blah, blah and blah.

It seems like 80% of the comments are "label [X] might be misleading". The least useful comment, of course, is this sort of comment with no suggestion. The least useful reviewer is the person who (1) provides a negative comment and, when asked for an improvement, (2) calls a meeting of random people to come up with replacement text.

[Hint: If you eventually understood the misleading label, please state your understanding in your own words. Often, hilarity ensues when their stated understanding cycles back to the original label.]

The "label [X] might be misleading" comment is — perhaps — the most expensive nuisance comment ever. Projects wind up spinning through warrens of rat-holes chasing down some verbiage that is acceptable. After all, you can't go over the waterfall until the entire UI is specified, right?

Worse, of course, the best sales people do not interpose themselves into the sales process. They connect prospective customers with products (or services). Really excellent sales people can have trouble making suggestions. Their transparency is what makes them good. It's not sensible demanding suggestions from them.

Underneath a "Might Be Misleading" comment, the person complaining completely understood the label. They were not actually mislead at all. If it was misleading, then (a) they wouldn't have understood it and (b) there'd be a proper question instead of a complaint.

Thank goodness for Agile product owners who can discard the bad kind of negativity. The right thing to do is put a UI in front of more than one user and bypass the negativity with a consensus that the UI actually is usable and isn't really misleading.

Might Be Misleading

The "Might be Misleading" comments are often code-speak for "I don't like it because..." And the reason why is often "because I had to think." I know that thinking is bad.

I understand that Krug's famous Don't Make me Think is the benchmark in usability. And I totally agree that some thinking is bad.

There are two levels of thinking.

• Thinking about the problem.
• Thinking about the UI and how the UI models the problem.

Krug's advice is clear. Don't make users think about the UI and how the UI models the problem. Users still have to think about the problem itself.

In the case of UI labels which "Might Be Misleading", we have to figure out if it's the problem or the UI that folks are complaining about. In many cases, parts of the problem are actually hard and no amount of UI fixup can ever make the problem easier.

Not Completely Accurate

One of the most common UI label complaints is that the label isn't "completely" accurate. They seem to object to fact that a UI label can only contain a few words and they have to actually understand the few words. I assume that folks who complain about UI labels also complain about light switches having just "on" and "off" as labels. Those labels aren't "completely" accurate. It should say "power on". Indeed it should say "110V AC power connected". Indeed it should say "110V AC power connected through load". Indeed it should say "110V AC 15 A power connected via circuit labeled #4 through load with ground".

Apparently this is news. Labels are Summaries.

No label can be "completely" accurate. You heard it here first. Now that you've been notified, you can stop complaining about labels which "might be misleading because they're not completely accurate." They can't be "completely" accurate unless the label recapitulates the entire problem domain description and all source code leading to the value.

Apologies

In too many cases of "Might Be Misleading," people are really complaining that they don't like the UI label (or the code example) because the problem itself is hard. I'm sympathetic that the problem domain is hard and requires thinking.

Please, however, don't complain about what "Might Be Misleading". Please try to focus on "Actually Is Misleading."

Before complaining, please clarify your understanding.

Here's the rule. If you eventually understood it, it may be that the problem itself is hard. If the problem is hard, fixing the label isn't going to help, is it?

If the problem is hard, you have to think. Some days are like that. The UI designer and I apologize for making you think. Can we move on now?

If the label (or example) really is wrong, and you can correct it, that's a good thing. Figure out what is actually misleading. Supply the correction. Try to escalate "Might Be Misleading" to "Actually Mislead Someone". Specifics matter.

Also, please remember that labels are summaries. At some point, details must be elided. If you have trouble with the concept of "summary", you can do this. (1) Write down all the details that you understand. Omit nothing. (2) Rank the details in order of importance. (3) Delete words to pare the description down to an appropriate length to fit in the UI. When you're done, you have a suggestion.

Is It Worth Describing Obscure Features?

I'm rewriting Building Skills in Python. 2.7 is out. As more libraries make the move, 3.1 is getting more and more viable.

I'm looking closely at the Decorators chapter (Part 3, Chapter 6).

And I'm struggling with classmethod. It's a first-class part of Python. And I'm sure there are folks who find it useful.

But I'm struggling to find a "simple" (i.e., under 12 lines of code) example where it might be useful.

Indeed, I'm starting to suffer from a growing feeling that this is one language feature that I can gracefully elide from in-depth to mere mention.

There is a relevant Stack Overflow question: What are Class methods in Python for? But the examples there aren't terribly compelling. Perhaps I have a blind-spot here because I never seen a big need for writing metaclasses. Or perhaps because I don't see a need for creating alternate constructors -- the options in __init__() seem to cover almost all my needs.

Building Skill Books -- Google Group

The readers of the Building Skills Books have a number of needs:

1. A way to post errors and corrections. I get a lot of these. Thanks!
2. A way to share comments and questions. I get a fair number of these.

A Google Group seems to be the best approach. We have pages, discussions, email notifications, a lot of features.

As long as we're opening the group, I figured I should actually make the dozens of corrections that have been sent in. Also, I'm looking closely at using Lulu to handle hard-copy production for the folks that have asked for that.

Dive in: Building Skills Books.

Building Skills in Object-Oriented Design

Completely revised Building Skills in Object-Oriented Design. Cleaned up many of the exercises to make them simpler and more sensible to the n00b designer.

Also, to make it easier to follow, I made use of the Sphinx "ifconfig" feature to separate the text into two parallel editions: a Python edition and a Java edition. A little language-specific focus may help.

Interestingly, I got an email recently from someone who wanted the entire source code for the projects described in the book. I was a little puzzled. Providing the source would completely defeat the purpose of the book, which is to build skills by actually doing the work. So, no, there is no source code available for these exercises. The point is to do the work, all the work, all by yourself.