You will learn a lot.
You may learn something from my presentations.
Tuesday, November 8, 2016
PyData DC 2016 -- YouTube Channel
All The Talks: https://www.youtube.com/playlist?list=PLGVZCDnMOq0qLoYpkeySVtfdbQg1A_GiB
Tuesday, November 1, 2016
Handling Irregular File Formats
This is a common issue. We have a file which was printed for human consumption. Consequently, it has many different kinds of lines.
These are the two kinds of lines of interest:
900296268 4/9/16 Mobility, Data Mining and Privacy Expired
900295204 4/1/16 Pro .NET Best Practices
Expired
The first is a single physical line. It has four data elements. The second is two physical lines. The first has three data elements.
There are a number of other noise lines in the file which must be filtered out.
The first "solution" pitched to me could be summarized with this:
Move "Expired" on a line by itself to the previous line
That was part of the email subject line. The body of the email was some whining about regular expressions. Which I mostly ignored. Multiline regular expressions are their own kind of challenge.
We (should) all know this: https://blog.codinghorror.com/regular-expressions-now-you-have-two-problems/
Let's do this without regular expressions. There are two things we need to know. One is buffering, and the other is the best way to split each line. It turns out that there are spaces as well as tabs, and can can, by splitting on tabs, make a lot of progress.
Instead of the good approach, I'll pick the other approach that doesn't involve splitting on tabs.
Here's the simulated file, with data lightly redacted.
sample_text = '''
"Your eBooks"
Show 200
Page: 1
Order # Date Title Formats Status Download
-------
xxx315605 9/30/16 R for Cloud Computing Available
xxx304790 6/21/16 Java XML and JSON Available
xxx304790 6/21/16 Accelerated DOM Scripting with Ajax, APIs, and Libraries Available
xxx291633 2/28/16 Practical Google Analytics and Google Tag Manager for Developers
Expired
'''
It's not perfectly obvious (because of line wrapping) but there are three examples of the "all-complete-in-one-line" records. There's one example of the "two-lines" record.
Rather than mess with the file, we'll build a file-like object with our sample data.
import io
file_like_object = io.StringIO(sample_text)
I like this because it lets me write proper unit test cases.
The file has four kinds of lines:
We'll create some decision rules for the two obvious kinds of file lines: complete records and trailers. We can deduce the headers based on a simple adjacency rule: they precede a trailer. The fourth kind of lines are those which are possible headers but are not immediately prior to a trailer.
def complete(words):
return len(words) > 3 and words[-1] in ('Available', 'Expired')
def trailer(words):
return len(words) == 1 and words[0] in ('Available', 'Expired')
We can spot these two kinds of lines easily. The other kinds require a Buffered Generator.
def emit_clean(source):
header = None
for line in (line.strip() for line in source):
words = [w.strip() for w in line.split()]
if len(words) == 0: continue
if complete(words):
yield(line)
header = None
elif trailer(words) and header:
yield(header + '\t\t' + line)
header = None
else:
# Possible header
# print('??', line)
header = line
The Buffered Generator is a way to implement a "look ahead one item" (LA1) algorithm. We do this by buffering rows. When we get to the next row we can use the buffered row and the current row to implement the look-ahead logic.
The actual implementation uses a look-behind buffer, header.
The (line.strip() for line in source) generator expression strips away leading and trailing spaces. This gets rid of the newline characters at the end of each input line.
The default behavior of split() is to split on whitespace. In this case, it will create a number of words for complete records or header records, and a single word for a trailer record. If we had split on tab characters, some of this logic would be simplified.
That's left as an exercise for the reader.
If the len(words) is zero, the line is blank.
If the line matches the complete() function, we can yield it as one of the iterable results of the generator function. We also clear out the look-behind buffer, header.
If the line is a trailer and we have a buffered look-behind line, this is the two-physical-line case. We can assemble a complete record and emit it.
Otherwise, we don't know what the line is. It's a possible header line, so we'll save it for later examination.
This algorithm involves no regular expressions.
An alternative would use three regular expressions to match the three kinds of lines.
import re
all_one_pat = re.compile("(.*)\t(.*)\t(.*)\t\t((?:Available)|(?:Expired))")
header_pat = re.compile("(.*)\t(.*)\t(.*)")
trailer_pat = re.compile("((?:Available)|(?:Expired))")
The essential processing involves seeing which of the regular expressions match the line at hand. If it's all-in-one, this is good. We can yield the groups of meaningful data. If it's a header, we can save the groups. If it's a trailer, we can combine header and trailer groups and yield the composite.
These are the two kinds of lines of interest:
900296268 4/9/16 Mobility, Data Mining and Privacy Expired
900295204 4/1/16 Pro .NET Best Practices
Expired
The first is a single physical line. It has four data elements. The second is two physical lines. The first has three data elements.
There are a number of other noise lines in the file which must be filtered out.
The first "solution" pitched to me could be summarized with this:
Move "Expired" on a line by itself to the previous line
That was part of the email subject line. The body of the email was some whining about regular expressions. Which I mostly ignored. Multiline regular expressions are their own kind of challenge.
We (should) all know this: https://blog.codinghorror.com/regular-expressions-now-you-have-two-problems/
Let's do this without regular expressions. There are two things we need to know. One is buffering, and the other is the best way to split each line. It turns out that there are spaces as well as tabs, and can can, by splitting on tabs, make a lot of progress.
Instead of the good approach, I'll pick the other approach that doesn't involve splitting on tabs.
Here's the simulated file, with data lightly redacted.
sample_text = '''
"Your eBooks"
Show 200
Page: 1
Order # Date Title Formats Status Download
-------
xxx315605 9/30/16 R for Cloud Computing Available
xxx304790 6/21/16 Java XML and JSON Available
xxx304790 6/21/16 Accelerated DOM Scripting with Ajax, APIs, and Libraries Available
xxx291633 2/28/16 Practical Google Analytics and Google Tag Manager for Developers
Expired
'''
It's not perfectly obvious (because of line wrapping) but there are three examples of the "all-complete-in-one-line" records. There's one example of the "two-lines" record.
Rather than mess with the file, we'll build a file-like object with our sample data.
import io
file_like_object = io.StringIO(sample_text)
I like this because it lets me write proper unit test cases.
The file has four kinds of lines:
- Complete Records
- Record Headers (without Available/Expired)
- Record Trailers (only Available/Expired)
- Noise
We'll create some decision rules for the two obvious kinds of file lines: complete records and trailers. We can deduce the headers based on a simple adjacency rule: they precede a trailer. The fourth kind of lines are those which are possible headers but are not immediately prior to a trailer.
def complete(words):
return len(words) > 3 and words[-1] in ('Available', 'Expired')
def trailer(words):
return len(words) == 1 and words[0] in ('Available', 'Expired')
We can spot these two kinds of lines easily. The other kinds require a Buffered Generator.
def emit_clean(source):
header = None
for line in (line.strip() for line in source):
words = [w.strip() for w in line.split()]
if len(words) == 0: continue
if complete(words):
yield(line)
header = None
elif trailer(words) and header:
yield(header + '\t\t' + line)
header = None
else:
# Possible header
# print('??', line)
header = line
The Buffered Generator is a way to implement a "look ahead one item" (LA1) algorithm. We do this by buffering rows. When we get to the next row we can use the buffered row and the current row to implement the look-ahead logic.
The actual implementation uses a look-behind buffer, header.
The (line.strip() for line in source) generator expression strips away leading and trailing spaces. This gets rid of the newline characters at the end of each input line.
The default behavior of split() is to split on whitespace. In this case, it will create a number of words for complete records or header records, and a single word for a trailer record. If we had split on tab characters, some of this logic would be simplified.
That's left as an exercise for the reader.
If the len(words) is zero, the line is blank.
If the line matches the complete() function, we can yield it as one of the iterable results of the generator function. We also clear out the look-behind buffer, header.
If the line is a trailer and we have a buffered look-behind line, this is the two-physical-line case. We can assemble a complete record and emit it.
Otherwise, we don't know what the line is. It's a possible header line, so we'll save it for later examination.
This algorithm involves no regular expressions.
With Regular Expressions
An alternative would use three regular expressions to match the three kinds of lines.
import re
all_one_pat = re.compile("(.*)\t(.*)\t(.*)\t\t((?:Available)|(?:Expired))")
header_pat = re.compile("(.*)\t(.*)\t(.*)")
trailer_pat = re.compile("((?:Available)|(?:Expired))")
This has the advantage that we can then use the groups() method of each successful match to emit useful data instead of text which needs subsequent parsing. This leads to a slightly more robust process.
def emit_clean2(source):
header = None
for line in (line.strip() for line in source):
if len(line) == 0: continue
all_one_match = all_one_pat.match(line)
header_match = header_pat.match(line)
trailer_match = trailer_pat.match(line)
if all_one_match:
yield(all_one_match.groups())
header = None
elif header_match and not header:
header = header_match.groups()
elif trailer_match and header:
yield header + trailer_match.groups()
header = None
else:
pass # noise
This has the advantage of explicitly rejecting noise lines instead of treating each noise line as a possible header.
Handling Irregular File Formats
This is a common issue. We have a file which was printed for human consumption. Consequently, it has many different kinds of lines.
These are the two kinds of lines of interest:
900296268 4/9/16 Mobility, Data Mining and Privacy Expired
900295204 4/1/16 Pro .NET Best Practices
Expired
The first is a single physical line. It has four data elements. The second is two physical lines. The first has three data elements.
There are a number of other noise lines in the file which must be filtered out.
The first "solution" pitched to me could be summarized with this:
Move "Expired" on a line by itself to the previous line
That was part of the email subject line. The body of the email was some whining about regular expressions. Which I mostly ignored. Multiline regular expressions are their own kind of challenge.
We (should) all know this: https://blog.codinghorror.com/regular-expressions-now-you-have-two-problems/
Let's do this without regular expressions. There are two things we need to know. One is buffering, and the other is the best way to split each line. It turns out that there are spaces as well as tabs, and can can, by splitting on tabs, make a lot of progress.
Instead of the good approach, I'll pick the other approach that doesn't involve splitting on tabs.
Here's the simulated file, with data lightly redacted.
sample_text = '''
"Your eBooks"
Show 200
Page: 1
Order # Date Title Formats Status Download
-------
xxx315605 9/30/16 R for Cloud Computing Available
xxx304790 6/21/16 Java XML and JSON Available
xxx304790 6/21/16 Accelerated DOM Scripting with Ajax, APIs, and Libraries Available
xxx291633 2/28/16 Practical Google Analytics and Google Tag Manager for Developers
Expired
'''
It's not perfectly obvious (because of line wrapping) but there are three examples of the "all-complete-in-one-line" records. There's one example of the "two-lines" record.
Rather than mess with the file, we'll build a file-like object with our sample data.
import io
file_like_object = io.StringIO(sample_text)
I like this because it lets me write proper unit test cases.
The file has four kinds of lines:
We'll create some decision rules for the two obvious kinds of file lines: complete records and trailers. We can deduce the headers based on a simple adjacency rule: they precede a trailer. The fourth kind of lines are those which are possible headers but are not immediately prior to a trailer.
def complete(words):
return len(words) > 3 and words[-1] in ('Available', 'Expired')
def trailer(words):
return len(words) == 1 and words[0] in ('Available', 'Expired')
We can spot these two kinds of lines easily. The other kinds require a Buffered Generator.
def emit_clean(source):
header = None
for line in (line.strip() for line in source):
words = [w.strip() for w in line.split()]
if len(words) == 0: continue
if complete(words):
yield(line)
header = None
elif trailer(words) and header:
yield(header + '\t\t' + line)
header = None
else:
# Possible header
# print('??', line)
header = line
The Buffered Generator is a way to implement a "look ahead one item" (LA1) algorithm. We do this by buffering rows. When we get to the next row we can use the buffered row and the current row to implement the look-ahead logic.
The actual implementation uses a look-behind buffer, header.
The (line.strip() for line in source) generator expression strips away leading and trailing spaces. This gets rid of the newline characters at the end of each input line.
The default behavior of split() is to split on whitespace. In this case, it will create a number of words for complete records or header records, and a single word for a trailer record. If we had split on tab characters, some of this logic would be simplified.
That's left as an exercise for the reader.
If the len(words) is zero, the line is blank.
If the line matches the complete() function, we can yield it as one of the iterable results of the generator function. We also clear out the look-behind buffer, header.
If the line is a trailer and we have a buffered look-behind line, this is the two-physical-line case. We can assemble a complete record and emit it.
Otherwise, we don't know what the line is. It's a possible header line, so we'll save it for later examination.
This algorithm involves no regular expressions.
An alternative would use three regular expressions to match the three kinds of lines.
import re
all_one_pat = re.compile("(.*)\t(.*)\t(.*)\t\t((?:Available)|(?:Expired))")
header_pat = re.compile("(.*)\t(.*)\t(.*)")
trailer_pat = re.compile("((?:Available)|(?:Expired))")
The essential processing involves seeing which of the regular expressions match the line at hand. If it's all-in-one, this is good. We can yield the groups of meaningful data. If it's a header, we can save the groups. If it's a trailer, we can combine header and trailer groups and yield the composite.
These are the two kinds of lines of interest:
900296268 4/9/16 Mobility, Data Mining and Privacy Expired
900295204 4/1/16 Pro .NET Best Practices
Expired
The first is a single physical line. It has four data elements. The second is two physical lines. The first has three data elements.
There are a number of other noise lines in the file which must be filtered out.
The first "solution" pitched to me could be summarized with this:
Move "Expired" on a line by itself to the previous line
That was part of the email subject line. The body of the email was some whining about regular expressions. Which I mostly ignored. Multiline regular expressions are their own kind of challenge.
We (should) all know this: https://blog.codinghorror.com/regular-expressions-now-you-have-two-problems/
Let's do this without regular expressions. There are two things we need to know. One is buffering, and the other is the best way to split each line. It turns out that there are spaces as well as tabs, and can can, by splitting on tabs, make a lot of progress.
Instead of the good approach, I'll pick the other approach that doesn't involve splitting on tabs.
Here's the simulated file, with data lightly redacted.
sample_text = '''
"Your eBooks"
Show 200
Page: 1
Order # Date Title Formats Status Download
-------
xxx315605 9/30/16 R for Cloud Computing Available
xxx304790 6/21/16 Java XML and JSON Available
xxx304790 6/21/16 Accelerated DOM Scripting with Ajax, APIs, and Libraries Available
xxx291633 2/28/16 Practical Google Analytics and Google Tag Manager for Developers
Expired
'''
It's not perfectly obvious (because of line wrapping) but there are three examples of the "all-complete-in-one-line" records. There's one example of the "two-lines" record.
Rather than mess with the file, we'll build a file-like object with our sample data.
import io
file_like_object = io.StringIO(sample_text)
I like this because it lets me write proper unit test cases.
The file has four kinds of lines:
- Complete Records
- Record Headers (without Available/Expired)
- Record Trailers (only Available/Expired)
- Noise
We'll create some decision rules for the two obvious kinds of file lines: complete records and trailers. We can deduce the headers based on a simple adjacency rule: they precede a trailer. The fourth kind of lines are those which are possible headers but are not immediately prior to a trailer.
def complete(words):
return len(words) > 3 and words[-1] in ('Available', 'Expired')
def trailer(words):
return len(words) == 1 and words[0] in ('Available', 'Expired')
We can spot these two kinds of lines easily. The other kinds require a Buffered Generator.
def emit_clean(source):
header = None
for line in (line.strip() for line in source):
words = [w.strip() for w in line.split()]
if len(words) == 0: continue
if complete(words):
yield(line)
header = None
elif trailer(words) and header:
yield(header + '\t\t' + line)
header = None
else:
# Possible header
# print('??', line)
header = line
The Buffered Generator is a way to implement a "look ahead one item" (LA1) algorithm. We do this by buffering rows. When we get to the next row we can use the buffered row and the current row to implement the look-ahead logic.
The actual implementation uses a look-behind buffer, header.
The (line.strip() for line in source) generator expression strips away leading and trailing spaces. This gets rid of the newline characters at the end of each input line.
The default behavior of split() is to split on whitespace. In this case, it will create a number of words for complete records or header records, and a single word for a trailer record. If we had split on tab characters, some of this logic would be simplified.
That's left as an exercise for the reader.
If the len(words) is zero, the line is blank.
If the line matches the complete() function, we can yield it as one of the iterable results of the generator function. We also clear out the look-behind buffer, header.
If the line is a trailer and we have a buffered look-behind line, this is the two-physical-line case. We can assemble a complete record and emit it.
Otherwise, we don't know what the line is. It's a possible header line, so we'll save it for later examination.
This algorithm involves no regular expressions.
With Regular Expressions
An alternative would use three regular expressions to match the three kinds of lines.
import re
all_one_pat = re.compile("(.*)\t(.*)\t(.*)\t\t((?:Available)|(?:Expired))")
header_pat = re.compile("(.*)\t(.*)\t(.*)")
trailer_pat = re.compile("((?:Available)|(?:Expired))")
This has the advantage that we can then use the groups() method of each successful match to emit useful data instead of text which needs subsequent parsing. This leads to a slightly more robust process.
def emit_clean2(source):
header = None
for line in (line.strip() for line in source):
if len(line) == 0: continue
all_one_match = all_one_pat.match(line)
header_match = header_pat.match(line)
trailer_match = trailer_pat.match(line)
if all_one_match:
yield(all_one_match.groups())
header = None
elif header_match and not header:
header = header_match.groups()
elif trailer_match and header:
yield header + trailer_match.groups()
header = None
else:
pass # noise
This has the advantage of explicitly rejecting noise lines instead of treating each noise line as a possible header.
Tuesday, October 25, 2016
Speakers advice
First. Read this: http://webapplog.com/10-conf-donts/
Some additional thoughts on the don't list.
Some additional thoughts on the don't list.
- Avoid reading to your audience unless you are a poet, journalist, judge or politician. Poets and journalists are paid to write well and read there words well. Judges and politicians are paid to be ultra precise.
- Avoid Type and Talk unless it is a Code Dojo presentation where the typing is essential. I've seen too many bad type-and-talk where the lack of organization made it nearly impossible to figure out what was going on.
- Avoid GIFs and clever graphics.
- Avoid insulting people. Don't alienate your audience. If you can't be completely 100% inclusive of every single human being in the room, don't speak in public.
- Avoid sitting if you are able to stand. If you must sit, please try to sit where folks can see you. This can't always work. Someone able to stand who chooses to sit is doing themselves a disservice. A singer or vocal coach will tell you that your standing posture helps you breathe properly and project properly. If you are able to stand, please stand.
- Avoid nervous behaviors. Avoid drawing attention to yourself, and draw attention to your material Fear (or nervousness) is hard to avoid. It's important to focus on the audience and their curiosity about your talk. They were intrigued by the title. They want to hear you..
- Avoid apologies. Apologize if you offend someone, of course. But don't "pre-apologize" for some irrelevant aspect of your presentation. Your audience came for the content, not for apologies.
- Avoid too much sales pitch. I've sat through too many product demos that had a half-hour sales pitch that left only a half-hour for the actual useful information. This has happened even when I told vendors -- explicitly -- not to provide any sales information during the product demo.
- Avoid too much personal background. A complete recitation of your CV isn't interesting and brushes up against an Argument from Authority fallacy.
- Avoid dressing badly.
A list of things to do.
- Speak with passion about your topic. Your slides are your road-map through the agenda. A few key points and reminders are all you should have.
- Speak to the people listening. Canned code examples are good, if they emphasize your point. Copy and paste into an IDE if you are demonstrating the IDE.
- Focus on the material, not other irrelevant cleverness.
- Focus on the audience as people interested in your topic.
- To project your voice -- and your presence -- you need to be visible. Stand if you can. Try to be as visible as possible.
- Focus on your audience's need to hear your material. It's not about you, it's about your content.
- Focus on the good, useful, informative information you're providing.
- Present outstanding content first. Sales are merely a hoped-for consequence of a good presentation.
- Your content should stand on it's own. You only need a brief summary of your qualifications.
- Project your presence. Dress so that you can be seen without being distracting.
Tuesday, October 18, 2016
Optimizing complex generator expressions [Updated]
See this: https://twitter.com/jakevdp/status/786920174595158018
The core expression is similar to this
y = (f(x) for x in L if f(x) is not None)
There are a lot of variations on the filter. The point is that the function appears twice in the above expression.
We have a number of alternatives.
My preference is two steps, even though I don't really have a good reason for this.
y1 = (f(x) for x in L)
y2 = (f for f in y1 if f)
The thread leads to this path: https://twitter.com/TomAugspurger/status/786922167522828289 and the idea of "Let Bindings." We could extend the language slightly to bind a variable within the confines of the generator expression.
The core expression is similar to this
y = (f(x) for x in L if f(x) is not None)
There are a lot of variations on the filter. The point is that the function appears twice in the above expression.
We have a number of alternatives.
- y = filter(None, f(x) for x in L)
- y = filter(None, map(f, L))
- y = (x for x in map(f, L) if x)
- y = (x for x in (f(y) for y in L) if x is not None)
- y = (val for x in L for val in (f(x),) if val is not None)
My preference is two steps, even though I don't really have a good reason for this.
y2 = (f for f in y1 if f)
Like this:
y = (f(x) as val for x in L if val is not None)
The as clause binds the f(x) to val so that it can be used in the if clause.
Summary: Interesting.
Friday, October 7, 2016
Tuesday, October 4, 2016
Alternatives to PowerPoint (or Keynote) for Presentations
This: https://opensource.com/business/16/9/alternatives-powerpoint
Missing from the list? The S5-based slide-show tools that are part of docutils.
The only issue with S5 is that you need to carefully review each and every page to be sure you material fits. There's no autosizing of the fonts, or other trickery to pack too much trash onto a slide.
TIL that Lync/Skype doesn't politely handle Keynote on a Mac. You must mirror displays because Lync can only share one of your displays, and it's not the one Keynote chooses to display on.
I often use Keynote because it's expedient. I sometimes use S5 to show off an entirely open-source toolchain.
Missing from the list? The S5-based slide-show tools that are part of docutils.
The only issue with S5 is that you need to carefully review each and every page to be sure you material fits. There's no autosizing of the fonts, or other trickery to pack too much trash onto a slide.
TIL that Lync/Skype doesn't politely handle Keynote on a Mac. You must mirror displays because Lync can only share one of your displays, and it's not the one Keynote chooses to display on.
I often use Keynote because it's expedient. I sometimes use S5 to show off an entirely open-source toolchain.
Subscribe to:
Posts (Atom)