Twelve Important Design Patterns

Read this: http://12factor.net/

Then. After reading it. Read it again to be sure you've got it. It's dense with best practices.

Now that you've read it, make yourself a Quality Engineering checklist.

I. Codebase: One codebase tracked in revision control, many deploys
II. Dependencies: Explicitly declare and isolate dependencies
III. Config: Store config in the environment
IV. Backing services: Treat backing services as attached resources
V. Build, release, run: Strictly separate build and run stages
VI. Processes: Execute the app as one or more stateless processes
VII. Port binding: Export services via port binding
VIII. Concurrency: Scale out via the process model
IX. Disposability: Maximize robustness with fast startup and graceful shutdown
X. Dev/prod parity: Keep development, staging, and production as similar as possible
XI. Logs: Treat logs as event streams
XII. Admin processes: Run admin/management tasks as one-off processes

If your app doesn't follow all of these patterns, you've got technical debt to work off. Start by posting the debt remediation stories in Jira (or whatever you're using.)

I've got config issues left, right, and center. Numerous assumptions include the URL's for RESTful services on which my RESTful services rely: this is not good.

Some of these things, however, are a done deed in the Python/Flask world with no real thinking required.

• Build, release, run - done
• Processes - done
• Port binding - done
• Disposability - done

Other things require some care. And the config is something that I've really got to fix.

Bottle vs. Flask vs. Django vs. a dozen others

There are times when a "micro framework" is actually useful. I wasn't easily convinced that this could be true. Big framework or die trying. Right?

Maybe not so right.

My primary example of a micro framework's value is a quick demo site to show how some API's are going to be consumed.

I've been avoiding an Angular.js app. We're going to be supporting the user experience with a sophisticated Angular.js app. But, as a back-end developer, I don't want to try to write demo pages in the proper app framework. There's too many ways to screw this up because I'll miss some key UX feature. Instead, I want to write fake pages that show a considerably simplified version of consuming an API. Sort of "suggestion" pages to clarify how the API's fit together.

To make it even more complex than necessary, I'm not interested in learning Angular.js, and I'm not interested in figuring out how it works. Running node.js, bower, npm, etc., etc., is too much.

[And Angular.js is just one version of the front-end. There's also mobile web, mobile native, tablet native, and God-alone-only-knows-what-all-else, to name a few. Each unique.]

Several sprints back, I slapped together two fake forms using bootstrap for layout and hosted them with Bottle. The navigation and framework look were simply copied from a screenshot and provided as static graphics. Lame, but acceptable. All that matters is getting the proper logo to show up.

The problem is that the sequence of API requests has grown since then. The demo grew to where we need a session so that alternative sequences will provide proper parameters to the APIs. We're beyond "Happy Path" interactions and into "what-if" demonstrations to show how to get (or avoid) a 404.

Bottle started with the significant advantage in fitting entirely into a single .py module. The barrier to entry was very low. But then the forms got awkwardly complex and Jinja2 was required. Now that sessions are required, the single module benefit isn't as enticing as it once was.

I've been forced to upgrade from Bottle to Flask. This exercise points out that I should have started with Flask in the first place. Few things are small enough for Bottle. In some ways, the two are vaguely similar. The @route() annotation being the biggest similarity. In other ways, of course, the two are wildly different. There's only a single Flask, but we can easily combine multiple Bottles into a larger, more comprehensive site. I like the composability of Bottles, and wish Flask had this.

The Flask Blueprints might be a good stand-in for composing multiple Bottles. Currently, though, each functional cluster of API's has their own unique feature set. The bigger issue is updating the configuration to track the API's through the testing and QA servers as they march toward completion. Since they don't move in lock-step, the configuration is complex and dynamic.

The transparent access to session information is a wonderful thing. I built a quick-and-dirty session management in Bottle. It used shelve and a "simple" cookie. But it rapidly devolved to a lot of code to check for the cookie and persist the cookie. Each superficially simple Bottle @route() needed a bunch of additional functionality.

The whole point was to quickly show how the API's fit together. Not reinvent Yet Another Web Framework Based On Jinja2 and Werkzeug.

Django seems like too much for this job. We don't have a model; and that's the exact point. Lacking a database model doesn't completely break Django, but it makes a large number of Django features moot. We just have some forms that get filled in for different kinds of events and transactions and searches and stuff. And we need a simple way to manage stateful sessions.

Omitted from consideration are the other dozen-or-so frameworks listed here: http://codecondo.com/14-minimal-web-frameworks-for-python/. This is a great resource for comparing and contrasting the various choices. Indeed, this was how I found Bottle to begin with.

Complexity and Simplicity

Here's an interesting -- and common -- question.

"... any tools that I could use to create a web scraper that I could use to interact with a .aspx website?

I want to build a tool that will read an input file (e.g. an excel spreadsheet) containing a list of property parcel numbers, and for each parcel number:

- connect to the property appraiser's website (which happens to be the .aspx page),

- enter the parcel number,

- scrape selected data (which is contained in a table on the search results page)

- store the scraped data in an output file (e.g. in the excel file that contains the input list)

- repeat the process for each parcel number"

The follow-on is interesting, also.

"I've created an excel macro which does the above with 'simple / plain vanilla html pages using the WebQuery feature, but it can't interact with an .aspx page."

Let's consider some of the complexities and simplicities that are present here.

Solution-Speak

First, and most important, this is written in solution-speak. It's an IT habit, and it's very hard to break. The input appears to be a spreadsheet. It may not actually be a spreadsheet, but this description essentially forces the solution to be built around the spreadsheet. The source may be another web page or some other file format. Since the problem is written in solution-speak, we don't know and can't -- easily -- explore the alternatives.

Let's assume that the source actually is a spreadsheet. And that this is the real source; it's maintained manually by the person who really "owns" the data.

The "update-in-place" nature of the question ("e.g. in the excel file that contains the input list") constrains the solution. This tends to add complexity because it somehow seems simpler to update a file in place.

What's actually simpler is often a process that creates a next revision of the file, leaving the first one intact and read-only. It's actually simpler because the "revert" strategy -- in case of problems -- is trivial. Simply delete the new file, fix the data (or the software) and run things again. Backup and history are simpler when creating a new file, also.

Technology Choices

Since it's written in solution speak, many technology choices have been made that might be inappropriate.

First, it appears that Excel is the "database" of choice. This is a terrible thing, but very, very common. The person has a problem. They tried to solve it with a spreadsheet. Now they have two problems.

A spreadsheet has a great GUI, but -- sadly -- leads to weird, inconsistent, undisciplined and generally "out-of-control" data. It doesn't have to create a mess, but it's hard to constrain it to prevent creating a mess.

Alternatives

This problem is ubiquitous and -- often -- trivial to solve if we cut Excel out of the picture.

Consider this workflow.

1. A small Python program uses xlrd to read this "list of property parcel numbers" and creates a simple CSV file. Excel is now officially out of the picture. If this process can't run (because the spreadsheet got tweaked) we can produce elegant reports with row and column information so that the person creating the spreadsheet can fix their problem. Let's say this is 20 lines of code, assuming the spreadsheet is hellishly complex.
2. Some small Python programs read a CSV file, uses urllib2 to "connect to the property appraiser's website (which happens to be the .aspx page), enter the parcel number", do the POST and retrieve the resulting page. This can be written to a file for future reference purposes. Numerous problems will be encountered here every time an appraiser's web site changes. It's best to keep this separate, since there may be several, each unique to an appraiser. There's no reason to generalize. Each of these is under 20 lines of code. Often under a dozen.
3. Some small Python program reads the resulting pages, uses Beautiful Soup to parse the resulting HTML. Again, numerous problems will be encountered here every time an appraiser's web site changes. It's pleasant to keep this separate from posting the query since this is just parsing result pages and doing nothing more. Easy to tweak and fix to keep up with changes. However, because of the potential complexity of each page, these might be complex. Let's pretend they're 20+ lines of code.
4. Some small Python program merges the original "list of property parcel numbers" and parsed results into a new .CSV file. With a double-click, this will be loaded back into Excel to make it look like the file was updated more-or-less in place. This should be about a dozen lines of code.

Since each step is separate, each can be written, tested and debugged separately. Once they work, some kind of master script can sequence through all four steps. That master script should be under a dozen lines of code.

Design Patterns

One important design pattern is to get out of "Office Product" mode as early as possible. Office Produces (like Excel) are fine for people, but dreadful for automation. They're too complex.

Another important design pattern is to decompose the problem into small scripts that can be run independently. Each step creates a work result that can be viewed and used for debugging. The files aren't big and can be deleted when the final work product is created. But an overly automated system is very, very hard to debug.

Another design pattern is to separate the various web services requests (in this case a form POST) by destination web site. Each site has unique security and validation considerations. It's too complex to write a super-universal, uber-form-filler-outer. It's easier to write a bunch of specific RESTful web services requests that are tailored to the unique problems present in each site.

Finally, it's important to avoid "update in place". It's hard to do well, and it's a pain in the neck when something goes wrong and you want to fall back to the previous version of the database.

REST and HTTP Digest Authentication

It seems so simple: use the HTTP Digest Authorization with the Quality of Protection set to "auth".

It's an easy algorithm. A nonce that encodes a timestamp can be used to be sure no one is attempting to cache credentials. It's potentially very, very nice.

Except for one thing: Apache.

Apache absorbs the Authorization header. And that's the end of that. It seems so simple, but I think I've been burned by it twice, now. I write unit tests that work with simplified Python wsgiref (or similar) servers. And I believe that those unit tests are equivalent to integration tests.

Ouch.

There's another reason why HTTP Digest authentication for RESTful services is a poor idea.

It involves too much traffic. HTTP authentication is usually a two-step dance to establish a session. Two steps in one too many, and RESTful services don't usually have any kind of session.

Schemes like this can actually work: http://broadcast.oreilly.com/2009/12/principles-for-standardized-rest-authentication.html

The comments on this post are almost as helpful as the post itself.

The three points are straight-forward.

1. Use SSL. Always.
2. Multiple Key/Secret credentials. Read this as username/password if that's helpful. We store hashes of "username:realm:password" as part of a RFC 2167 Digest Authentication. We plan to continue to use those hashes. This is a bit touchy, but we think we can handle this by a slight change to our user profile table.
3. The "signed query" principle requires some thought. We don't make heavy use of query strings. Indeed, we make almost no use of the query strings. So the hand-wringing over this is a bit silly. We simply ignore any query string when signing the request.

I just wish I did integration testing with Apache sooner, not later. Sigh.