Brain-Damaged Data

We process a fair amount of externally-prepared datasets.  40,000 rows of econometric data that we purchased from a third-party.  Mostly, the data is in a usable format: .CSV or .XSLX.

Once in a while, we get CSV with | (pipe).  A few times, we got fixed-format COBOL-style records.

Recently, we got a CSV-with-pipe that included 2 records with embedded \n sequences in the middle of a CSV row of data.  Really.

Painful Elimination

There are two ways to "eliminate" this problem. 

• Subclass our input processing to handle this special CSV-with-pipe case.
• Actually read and parse the source file creating a clean intermediate file that we can simply process with an existing CSV-with-pipe configuration.

I elected to do the first.  The second is (to my mind) an auditing nightmare because we touched the file.  We have to prove that we didn't disturb any other fields.  While not impossible, it becomes a very strange special case for this one-and-only file.

CSV Simplicity

The CSV module's epic simplicity makes it easy to work around this kind of goofy data.  Our subclass for this case had the following extra foolishness put in

def make_reader( self ):
        def filter_damage( aFile ):
            file_iter= iter(aFile)
            for row in file_iter:
                if row.rfind('"') >= len(row)-3:
                    logger.error( "Damaged Line: %r", row )
                    rest= next(file_iter)
                    line= row[:row.rfind('"')] + rest[3:]
                    logger.warning( "Repaired Line: %r", line )
                    yield line
                else:
                    yield row
        tweaked_file= filter_damage( self.sourceFile )
        return csv.reader( tweaked_file, delimiter='|', doublequote=False, escapechar='"' )

That's it.  Since the Python CSV reader merely wants an iterator over lines, we can (with a simple generator function) provide the necessary "iterator-over-lines". 

Delightful.

Apology

The murky-looking row.rfind('"') >= len(row)-3 condition is one of those consequences of trying to find just a few irregular line endings in an otherwise regular file.  For CSV processing, files often have to be opened in "rb" mode because they originate (or will be used with) MS-Excel.  This makes the damaged line-ending either '"\n' or maybe '"\r\n'.  Rather than spend too much time negotiating with Python's universal newline and "rb" mode, it's slightly easier to look for a '"' near the end. 

We're hoping this is a one-time-only subclass that we can safely ignore in the future.  If hope is dashed, it's a distinct subclass, so it's easily reused and didn't break anything else.

One of Those Things

Check out this question on Stack Overflow: "Python: replace a string by a float in txt file".

The question is confusing, but it appears to be a longish and confused description of simple formatting or template substitution.  It's hard to be sure, but it sounds like one of Those Things™ (TT).

Most of Those Things (TT) are standard problems with standard solutions.  Until you've seen a lot TT's, it seems like your problem is unique and special.  It's hard to see TT's for what they are.

In this case, the problem appears to be solved by Python's string.Template class with minor modifications.  The documentation for customizing string.Template isn't clear, so here's an example.


from string import Template
class MyTemplate( Template ):
    delimiter= '@'
    pattern= r"@(?P<escaped>@)|@(?P<named>[_a-z][_a-z0-9]*)@|@(?P<braced>[_a-z][_a-z0-9]*)@|@(?P<invalid>)"


That appears to be the standard solution to the standard problem.  Define a new delimiter ('@') and some slightly different delimiter parsing rules and away you go.

This can be used as follows to replace any '@x@' variables in any template file.  What's important is that very little actual code is needed, since it's one of Those Things that's already been solved.

with open( 'a.txt', 'r' ) as source:
    t = MyTemplate(source.read())
    result= t.substitute( x=15 )
    print result


Spam Email Footers

I don't want the spamilicious email.  I'm trying to actually unsubscribe.

The footer says "If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of any information contained in or attached to this communication is strictly prohibited. If you have received this message in error, please notify the sender immediately and delete the material from any computer."

I don't feel like the intended recipient because it's just irrelevant junk.  Perhaps you should not have disseminated, distributed, copied or sent me this.  Wouldn't that have been simpler? Keep it to yourself?

I also think I've received the message in error.  Since I don't want the damn thing. And that means that I have to delete it?  Why can't you stop sending it?  Wouldn't that be simpler for both of us?

757 Python User's Group Meetup

Wednesday night.  At 757 Labs.  Be there.

Here's the details on meetup.com

Lacking any other agenda, I'll do some more presentation on the supreme coolness of Django.

I almost wet myself

Someone sent me this: "“Building Skills in Python” – Steven F. Lott".

I had a vague idea that this book would get some traction.  This response was surprising.  I guess I should get to work on the upgrades.  And focus on the "no-nonsense" comment.

Security Vulnerabilities

Just saw this for the first time today:  http://cwe.mitre.org/top25/

I'd always relied on this: https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project

Both are really good lists of security vulnerabilities.

I once had to listen to a DBA tell me that "we don't know what we don't know" as a way of saying that there was no way to be sure that a web app was "secure".  That comment lead the project manager to go  through the classic "risk exposure" exercise (and hours of discussion) to determine that security mattered.  We defined the risks, the costs and the probability of occurrence so that we could document all kinds of potential exposures or something.

Instead of hand-wringing, these kinds of simple lists of the common vulnerabilities provides actionable steps for design, code, test and audit of operations.  Further, they guide selection, configuration and operation of web server technology to assure that the vulnerabilities are addressed.

Implementing the Unsubscribe User Story

I've been unsubscribing from some junk email recently.

The user story is simple: As a not-very-interested person, I want to get off your dumb-ass mailing list so that I don't have to flag your crap as spam any more.

The implementations vary from good to evil.  Here's what I've found.

The best sites have an unsubscribe link that simply presents the facts -- you are unsubscribed.  I almost feel like re-subscribing to a site that handles this use case so well.

The first level of crap is a site which forces me to click an OK or Unsubscribe button to confirm that I really want to unsubscribe and wasn't clicking the tiny little links at the end of the message randomly.

The deeper level of "marketing" crap is a form that allows me to "configure my subscription settings".  This is done by some marketing genius who wanted to "offer additional value" rather than simply do what I asked.  This is a hateful (but not yet evil) practice.  I don't want to "configure" my settings.  I want out.

The third-from-worst is a form in which I must enter my email address.  What?  I have several email aliases that redirect to a common mailbox.  I have to -- what? -- guess which of the aliases was used?  This is pernicious because I can make a spelling mistake and they can continue to send me dunning email.  This fill-in-the-blanks unsubscribe is simply evil because it gives them plausible deniability when the continue to send me email.  It's now my fault that I didn't spell my name correctly.

The next-to-worst is a "mailto:" link that jumps into my emailer.  I have to -- what? -- fill in the magic word "Complete" somewhere?  You're kidding, right?  This is so 1980's-vintage listserv that I'm hoping these companies can be sued because they failed to actually unsubscribe folks.  Again, this gives the spammer a legitimate excuse because I failed to do the arcane step properly.

The worst is no link at all.  Just instructions explaining that an email must be send with the magic word "Complete" or "Unsubscribe" in the subject or body.  Because I use aliases, this will probably not unsubscribe anything useful, but will only unsubscribe my outbound email address.  This is the worst kind of evil.  In a way, it meets the user story.  But only in a very, very oblique way.