When to choose Python over Java and vice versa ??: A Very Silly Question

The correct answer is: It Doesn't Matter.

In spite of this.

(A) The question gets asked.

And worse.

(B) It gets answered. And people take their answers seriously. As if there are Profound Differences among programming languages.

Among Turing Complete programming languages there are few Profound Differences.

The pragmatic differences are the relative ease (or pain) of expressing specific algorithms or data structures.

This means that there's no easy, blanket, one-size-fits-all answer to such a silly question.

You can have some code (or data) which is painful in Java and less painful in Python.

But.

You can also find an extension library that makes it much, much less painful.

This, alone, makes the question largely moot.

When it comes to a specific project, the question of the team's skills, the existing infrastructure, and any integration requirements are the driving considerations.

Because of this, an incumbent language has huge advantages.

If you've already got a dozen web sites in Java, there's no good reason to flip-flop between Java and Python.

If you're going to switch from some Java Framework to Django, however, you'd do this is part of a strategic commitment to drop Java and convert to Python.

To read the discussion, see LinkedIn Python Community. http://www.linkedin.com/groupItem?view=&gid=25827&type=member&item=5796513808700682243&qid=8612bee7-76e1-4a35-9c80-c163494191b0&trk=groups_most_popular-0-b-ttl&goback=%2Egmp_25827

Mac OS X 10.9 "Mavericks" Crashes Python -- Patch Available

I upgraded to Mac OS X 10.9 "Mavericks" today.

Python 3.2.4 crashed in odd places. Not all the time. But help() stopped working to give one example.

I upgraded to Python 3.3 (it's about time to do that, anyway.)

It crashed, also. Similar places.

Here's the workaround: http://bugs.python.org/issue18458

There's a script which you can download and run. This will tweak your installations until the next maintenance release of Python.

Whew. That was close.

Now I need to square away setuptools and "easy_install-3.3" all the missing pieces of my 3.2 environment.

Required Reading for everyone who said "If it ain't broke, don't fix it."

http://pythonsweetness.tumblr.com/post/64740079543/how-to-lose-172-222-a-second-for-45-minutes

Here an important lesson.

Code Rot is Real.  It Leads to Subtle and Expensive Bugs.

Claiming that code cleanup is just pointless "gold plating" is the kind of thing that can drive a company out of business.

Literate Programming: PyLit3

I've revised PyLit to work with Python3.

See https://github.com/slott56/PyLit-3

The code seems to pass all the unit tests.

The changes include Python3 revisions, plus a small change to handle trailing spaces in a sightly cleaner fashion. This was necessary because I have most of my editors set to remove trailing spaces from the files I create, and PyLit tended to create trailing spaces. This made the expected output from the unit tests not precisely match the actual output.

Literate Programming and PyLit

Even though I wrote a literate programming tool (PyWeb) I slowly came to realize that it's not very good.

Mostly, I followed the Web/Weave world view and cribbed their markup syntax. It's not bad, but, the PyWeb markup is based on some presumptions about literate programming that were, perhaps, true with some languages, but are not true at all when working with Python.

1. The source presentation order incomprehensible. To fix this, we create a literate programming document, and from that tangle the source into an order that's acceptable to the compiler, but perhaps hard to understand for people. We weave a document that's easy for people to understand.
2. The source syntax may be incomprehensible. To fix this, we have fine grained substitution. The target source can be built at any level of syntax (token, line, or higher-level language construct.) We can assure that the woven document for people is written using elegant symbols even if the tangled source code uses technical gibberish.
3. The woven documentation needs a lot of additional output markup. The original web/weave toolset create extensive TeX markup. Later tools reduced the markup to allow HTML or XML, minimizing the added markup in a woven document.

In Python, there's very little "boilerplate" or overhead in a module file. Also, because of very late binding, the presentation order of the source can better match reader expectations. For definitions, inter-class references mandate an order for the class statements in an inheritance hierarchy, but almost everything else is remarkably flexible.

Python syntax doesn't benefit from fine-grained web/weave techniques. It's pretty clear as written in it's normal form.

Finally, the presence of RST markup language means that a whole new meta-markup for literate programming isn't necessary. 

PyLit demonstrates that an additional markup language is not helpful. RST is sufficient. PyLit is an elegant parser of RST and Python. It can reshape RST into Python as well as reshape Python into RST. Do your literate programming in either language and produce the other easily.

Enter Python 3

The problem with PyLit is that it's oriented to Python 2.4 through 2.7. How can we use PyLit for Python 3?

• Use six.py to make a single version that covers both Python2 and Python3.
• Rewrite PyLit it for Python3 and move forward.

My preference is to move forward. The backward compatibility is helpful when there's a vast user base, lots of ongoing development, and the very real possibility of bug fixes that apply to Python2 as well as Python3.

PyLit has a small user base, no real development to speak of, and a very remote possibility of backward compatible bug fixes.

The rewrites are small. Here's the summary.

• Remove print statement and exec statements.
• Replace string formatting % with .format().
• Replace raise statements and except statements with Python3 (and Python2.7) syntax.
• Upgrade for dict method changes in Python3.
• Replace DefaultDict with collections.defaultdict.
• Replace optparse with argparse.

I've done this in my Python3.2 installation.

This doesn't address the Sphinx documentation, however, which should probably be tweaked to be the latest and greatest Sphinx version, also. Not much will change, there, however, since the RST remains compatible.

Also, it doesn't address the files with names that differ only in case. There are two graphics files in the /trunk/rstdocs/logo/ path that differ only in case of letters. Bad, but acceptable for Linux. Fatal for Mac OS X with the default filesystem.

The question is, what's the polite way to proceed? 

• Fork the PyLit 0.7.5 to create PyLit3 release 1.0? A whole, new project.
• Try to use six.py to create a 2-3 compatible source file and call this PyLit 0.8?

Adding six.py to a package that was a single module file seems like a bit of overkill. One of the elegant features of PyLit was that it was so simple, it didn't even have a setup.py. However, there may be a community of staunchly Python2 literate programming advocates. 

Introduction to Programming: iBook Edition for Python 3.2

That was challenging.

I rewrote almost all of my Introduction to Programming book into an iBook. Trimmed it down. Refocused it. Changed from Python 2.6 to 3.2. A complete refactoring from which almost nothing of the original book survives except the goals.

Look for it October 1st in the iTunes bookstore. Programming for Absolute Beginners: Building Skills in Programming.

[My intent is to have several Building Skills titles. We'll see how far I get.]

The rewrite involved three substantial changes.

1. I removed all of the duplicative reference material. The Python library reference is (now)  utstandingly good. When I started using Python over ten years ago, it was not very good, and I started writing a Python reference of my own merely to organize the documentation. The books grew from there; the reference aspect is now useless.
2. I dropped almost all Python 2 references. There's a little bit of Python 2 history, but that's it. It's time to move forward, now that most of the major packages seem to have made the switch. 
3. I changed the focus from processing to data.

Processing vs. Data

When looking at a multi-faceted language like Python, it's difficult to know what's the most gentle introduction to software development.

Historically, the procedural, imperative style of programming appears the most appealing. The roots of Python come from procedural programming. It reaches back to Pascal (and even Algol 60) by elegantly restating the core principles of those languages with an easier-to-read syntax.

Indeed, if you read classic foundational CACM articles where essential algorithms were first formally described, they used a neatly typeset variant on Algol that (for the early years of my career) was the gold standard in how code should look. Python follows this tradition nicely.

But.

That doesn't mean that procedural programming is really the absolutely best way to introduce the language.

Data First

I think that it may be possible to introduce the language with a focus on data objects first and the procedural/imperative statements as a secondary consideration.

When it comes to anything beyond trivial Rate-Time-Distance calculations, the data structure matters more than almost any other aspect of the software. The objects, their relationships, their operations and their attributes are core to the problem. The presentation, user actions and persistent representation are secondary considerations after the structure of the data.

It seems like the data structures should "drive" the presentation. The outline of the book should be introductions of each of the important and visible builtin data structures. Additionally, the library extensions that are most often used can be introduced, also.

Definitional features (def, return, yield, class, and the ideas of module and package) are central, but a step behind the builtin data structures.

Procedural features (if, for, while, break, continue, with, etc.) are clearly second-class; they exist only to support access to the data structures. A for statement, makes a "for all" assertion about a data structure. A for with a break (or a while) makes a "there exists" assertion about a data structure. The data is central. The imperative statements are secondary.

Other features (global, nonlocal, del, raise, try, etc.) are tertiary, and exist to create more elegant programs that don't annoy the other developers as much. 

This also means that generator expressions and comprehensions are first-class, front-and-center features of the language. This parallels the approach in the NLTK Book, which puts the focus on generator expressions as a way to clearly state the processing.

Other Forms

Currently, I only have the iBook available.

The iBook Author application can (and does) produce a PDF. I think I may offer that separately through www.lulu.com. 

iWeb File Extract and XML Iterators

Once upon a time, Apple offered iBlog. Then they switched to iWeb. Then they abandoned that market entirely.

That leaves some of us with content in iBlog as well as iWeb. Content we'd like to work with without doing extensive cutting and pasting. Or downloading from a web server. After all, the files are on our computer.

The iWeb files are essentially XML, making them relatively easy to work with. We can reduce the huge, and hugely complex iWeb XML to a simple iterator and use a simple for statement to extract the content.

[Historical note. I wrote a Python script to convert iBlog to RST. It worked reasonably well, all things considered. This is not the first time I've tried to preserve old content from obsolete tools. Sigh.]

Some tools (like SandVox) have a "extract iWeb content" mode. But that's not what we want. We don't want to convert from iWeb to another blog. We want to convert from iWeb to CVS or some other more useful format so we can do some interesting processing, not simple web presentation.

This is a note on how to read iWeb files to get at the content. And further, how to get at XML content in the form of a simple iterator.

Opening The Package

Here's how to overview the package.

    path="~/Documents/iWeb/Domain"
    path_full= os.path.expanduser(path+".sites2")
    for filename in os.listdir(path_full):
        name, ext = os.path.splitext( filename )
        if ext.lower() in ( ".jpg", ".png", ".mov", ".m4v", ".tiff", ".gif", ".m4a", ".mpg", ".pdf" ): continue
        print( filename )

This will reveal the files; we only really care about the "index.xml.gz" file since that has the bulk of the content.

    with closing( gzip.GzipFile( os.path.join(path_full,"index.xml.gz") ) ) as index:
        index_doc= xml.parse( index )
        index_root= index_doc.getroot()

This gets us the XML version of the blog.

Finding the Pages

We can use the following to thread through the XML. We're looking for a particular "Domain", a "Site" and a particular blog page within that site. The rest of the blog is mostly text. This portion of the blog is more structured.

For some reason, the domain is "Untitled". The site is "Cruising", and the blog page is "Travel 2012-2013". We insert these target names into XPath search strings to locate the relevant content.

search= '{{http://developer.apple.com/namespaces/bl}}domain[@{{http://developer.apple.com/namespaces/sf}}name="{0}"]'.format(domain_name)
domain= index_root.find( search )
mdu_uuid_tag= domain.find('{http://developer.apple.com/namespaces/bl}metadata/{http://developer.apple.com/namespaces/bl}MDUUID')
mdu_uuid_value= mdu_uuid_tag.find('{http://developer.apple.com/namespaces/bl}string').get('{http://developer.apple.com/namespaces/sfa}string')
domain_filename= "domain-{0}".format( mdu_uuid_value )

search= './/{{http://developer.apple.com/namespaces/bl}}site[@{{http://developer.apple.com/namespaces/sf}}name="{0}"]'.format(site_name)
cruising= domain.find(search)
mdu_uuid_tag= cruising.find('{http://developer.apple.com/namespaces/bl}metadata/{http://developer.apple.com/namespaces/bl}MDUUID')
mdu_uuid_value= mdu_uuid_tag.find('{http://developer.apple.com/namespaces/bl}string').get('{http://developer.apple.com/namespaces/sfa}string')
site_filename= "site-{0}".format(mdu_uuid_value)

search= '{{http://developer.apple.com/namespaces/bl}}site-blog[@{{http://developer.apple.com/namespaces/sf}}name="{0}"]'.format(site_blog_name)
site_nodes= cruising.find('{http://developer.apple.com/namespaces/bl}site-nodes')
travel= site_nodes.find(search)
mdu_uuid_tag= travel.find('{http://developer.apple.com/namespaces/bl}metadata/{http://developer.apple.com/namespaces/bl}MDUUID')
mdu_uuid_value= mdu_uuid_tag.find('{http://developer.apple.com/namespaces/bl}string').get('{http://developer.apple.com/namespaces/sfa}string')
site_blog_filename= "site-blog-{0}".format(mdu_uuid_value)

This will allow us to iterate through the blog entries, called "pages". Each page, it turns out, is stored in a separate XML file with the page details and styles. A lot of styles. We have to assemble the path from the base path, the domain, site,  site-blog and site-page names. We'll find an ".xml.gz" file that has the individual blog post.

    for site_page in travel.findall('{http://developer.apple.com/namespaces/bl}series/{http://developer.apple.com/namespaces/bl}site-page'):
        mdu_uuid_tag= site_page.find('{http://developer.apple.com/namespaces/bl}metadata/{http://developer.apple.com/namespaces/bl}MDUUID')
        mdu_uuid_value= mdu_uuid_tag.find('{http://developer.apple.com/namespaces/bl}string').get('{http://developer.apple.com/namespaces/sfa}string')
        site_page_filename= "site-page-{0}".format(mdu_uuid_value)

        blog_path= os.path.join(path_full, domain_filename, site_filename, site_blog_filename, site_page_filename )
        with closing( gzip.GzipFile( os.path.join(blog_path,site_page_filename+".xml.gz") ) ) as child:
            child_doc= xml.parse( child )
            child_root= child_doc.getroot()
        main_layer= child_root.find( '{http://developer.apple.com/namespaces/bl}site-page/{http://developer.apple.com/namespaces/bl}drawables/{http://developer.apple.com/namespaces/bl}main-layer' )

Once we have access to the page XML document, we can extract the content. At this point, we could define a function which simply yielded the individual site_page tags.

Summary Iterable

The most useful form for the pages is an iterable that yields the date, title and content text. In this case, we're not going to preserve the internal markup, we're just going to extract the text in bulk.

        content_map = {}
        for ds in main_layer.findall( '{http://developer.apple.com/namespaces/sf}drawable-shape' ):
            style_name= ds.get('{http://developer.apple.com/namespaces/sf}name')
            if style_name is None:
                #xml.dump( ds ) # Never has any content.
                continue
            for tb in ds.findall('{http://developer.apple.com/namespaces/sf}text/{http://developer.apple.com/namespaces/sf}text-storage/{http://developer.apple.com/namespaces/sf}text-body' ):
                # Simply extract text. Markup is lost.
                content_map[style_name] = tb.itertext()
        yield content_map

This works because the text has no useful semantic markup. It's essentially HTML formatting full of span and div tags.

Note that this could be a separate generator function, or it could be merged into the loop that finds the site-page tags. It's unlikely we'd ever have another source of site-page tags. But, it's very like that we'd have another function for extracting the text, date and title from a site-page tag. Therefore, we should package this as a separate generator function.  We didn't, however. It's just a big old function named postings_iter().

There are three relevant style names. We're not sure why these are used, but they're completely consistent indicators of the content.

• "generic-datefield-attributes (from archive)"
• "generic-title-attributes (from archive)"
• "generic-body-attributes (from archive)"

These becomes keys of the content_map mapping. The values are iterators over the text.

Processing The Text

Here's an iterator that makes use of the postings_iter() function shown above.

def flatten_posting_iter( postings=postings_iter(path="~/Documents/iWeb/Domain") ):
    """Minor cleanup to the postings to parse the date and flatten out the title."""
    for content_map in postings:
        date_text= " ".join( content_map['generic-datefield-attributes (from archive)'] )
        date= datetime.datetime.strptime( date_text, "%A, %B %d, %Y" ).date()
        title= " ".join( content_map['generic-title-attributes (from archive)'] )
        body= content_map['generic-body-attributes (from archive)']
        yield date, title, body

This will parse the dates, compress the title to remove internal markup, but otherwise leave the content untouched. 

Now we can use the following kind of loop to examine each posting.

    flat_postings=flatten_posting_iter(postings_iter(path="~/Documents/iWeb/Domain"))
    for date, title, text_iter in sorted(flat_postings):
        for text in text_iter:
           # examine the text for important content.

We've sorted the posting into date order. Now we can process the text elements to look for the relevant content.

In this case, we're looking for Lat/Lon coordinates, which have rather complex (but easy to spot) regular expressions. So the "examine" part is a series of RE matches to collect the data points we're looking for.

We'll leave off those application-specific details. We'll leave it at the following outline of the processing.

def fact_iter( flat_postings=flatten_posting_iter(postings_iter(path="~/Documents/iWeb/Domain")) ):
    for date, title, text_iter in sorted(flat_postings):
        fact= Fact()
        for text in text_iter:
           # examine the text for important content, set attributes of fact
           if fact.complete(): 
               yield fact
               fact= Fact()

This iterates through focused data structures that include the requested lat/lon points.

Final Application

The final application function that uses all of these iterators has the following kind of structure.

source= flat_postings=flatten_posting_iter(postings_iter(path="~/Documents/iWeb/Domain"))
with open('target.csv', 'w', newlines='') as target:
    wtr= csv.DictWriter( target, Fact.heading )
    wtr.writeheader()
    for fact in fact_iter( source ):
        wtr.writerow( fact.as_dict() )

We're simply iterating through the facts and writing them to a CSV file.

We can also simplify the last bit to this.

wtr.writerows( f.as_dict() for f in fact_iter( source ) )

The iWeb XML structure, while bulky and complex, can easily be reduced to a simple iterator. That's why I love Python.