Another shiny new MacBook pro

See https://slott-softwarearchitect.blogspot.com/2014/03/shiny-new-macbook-pro.html

At the time (2014), the 8Gb machine was way more than adequate for all my needs as a writer.

Enter bloat.

Mac OS Catalina has essentially filled this machine to the breaking point. Six short years is the lifespan. Things (generally) work, but it now crashes frequently. Sometimes, streaming TV won't play properly. I've tried a large number of remedies (reboot WiFi, reboot computer, reset Bluetooth) and it glitches too offten to be comfortable.

(Rumors suggest the crashes seem to be associated with going to sleep. The machine crashes when it's idle. I come back to it and find it has restarted, and needs to restart my apps. It's not horrible. But it's an indication of a deeper problem. And it's time.)

It works. But. I've spent too many years waiting for slow computers and slow networks. An hour a day (cumulative) for 300 days a year for 40 years means I've spent 1.3 years of my life waiting for a computer to do something.

I’m reluctantly replacing my kind-of-working "Late 2013" vintage machine with a new 13” MacBook Pro. At least 16Gb RAM. At least a terabyte of storage. Hopefully, things will not be "glitchy" and I won't have constant crashes.

I’ve gotten used to having an 27" Thunderbolt Display, and a USB Querkywriter keyboard, and two USB disks doing backups. That's a lot of stuff plugged in all the time. Also. I really need a slot for SD cards (the boat uses micro SD cards, as does the old GoPro camera.). So. A fancy USB-C hub will be essential, I think.

The question is 2 ports (power and hub) or 4 ports (power, hub, and two other things)?  I suspect I can live with 2 ports.  4 ports ships immediately.

I have several use cases:

• Writing Python books (e.g. Modern Python Cookbook 2e, https://www.packtpub.com/product/modern-python-cookbook-second-edition/9781800207455)
• Writing fiction. (Still editing second drafts, don’t hold your breath)
• Creating a MicroPython (or Arduino) device to track my boat’s position and act as an alarm if the anchor drags
• Boating (in a vague, general way)

Writing books actually requires some computing power. But. Not *too* much power. The general reader doesn’t always have a huge computer. If my examples require more computing power than my readers have access to, that’s a problem. The advantage of having a smallish computer is I’m not overstepping what’s available to my readers. This is a handy way to take a tax deduction to pay for this extravagance.

Writing fiction requires a small machine. Scrivener works on an iPad Pro. I’m good with almost anything. Even an iPhone can be used for writing and editing fiction. It’s hard, of course, with a tiny screen. But not impossible. And. I'm trying to learn the craft, so tools aren't as important as understanding character arc.

Creating MicroPython-based devices is a bit confusing right now. A lot of the development environments depend on a reliable USB connectivity to the Arduino or Circuit Playground Express board. I worry about the (potential) complexity of introducing a USB hub into the mix.  I suspect I only need to replace some of my USB cables; the Arduino boards all seem to use a bulky USB type B. The CPX use USB type Micro B. (I thinks one can be replaced with a USB C to USB B “printer cable”, the other is a USB microB to C adapter. Or, maybe a USB C to USB A adapter can be used with my vast collection of legacy cables. Don't know.)

Boating involves connecting external devices like the GPS antenna to the laptop and tracking position or planning routes. This is a Bluetooth thing, generally. 

It does require considerable power for the laptop; the 60W power brick becomes a constraint. The boat have an inverter and can handle the load gracefully. A computer is a dedicated 5A draw, though; twice what the fridge pulls (and the fridge runs infrequently.) We have 225Ah available. The computer could be as bad as 120Ah if it was left on for 24 hours during an overnight passage.

The good news is that the use cases are more-or-less exclusive. The boating use case is rare. We have more thrifty navigation systems permanently installed on the boat. Many folks are using CPX and Arduino’s with MacBook Pro’s, so I shouldn’t worry too much, just buy new cables.

The best part?

Since I use Time Machine, the new machine recovers from the Time Machine backups. It has to be left to run overnight, but. Boom. Done.

(On the to-do list -- encrypt the backup volumes. Ugh. But. Necessary.)

Python under XCode?!? Cool. Sort of.

	Dan Bader (@dbader_org)
5/5/17, 10:47 PM "Running Python in Xcode: Step by Step" ericasadun.com/2016/12/04/run…

Thanks for the link. There's a fair amount of "this doesn't seem to be what Xcode was designed to do." But it does seem to work.

I'll stick with Komodo Edit.

Audio Synth [Updated]

I learned about synthesizers in the '70's using a Moog analog device. Epic coolness.

Nowadays, everything is digital. We use wave tables and (relatively) simple additive synth techniques.

I made the mistake of reading about Arduino wave table synthesis:

http://learning.codasign.com/index.php?title=Wavetable_Synthesis

http://makezine.com/projects/make-35/advanced-arduino-sound-synthesis/

http://playground.arduino.cc/Main/ArduinoSynth

The idea of an Arduino alarm that uses a chime instead of a harsh buzz is exciting. The tough part about this is building the wave tables.

What a perfect place to use Python: we can build wave tables that can be pushed down to the Arduino. And test them in the Python world to adjust the frequency spectrum and the complex envelope issues around the various partials.

See http://computermusicresource.com/Simple.bell.tutorial.html

Except.

Python3.4 doesn't have PyAudio support.

Yet.

Sigh. Before I can work with Arduino wave tables, I'll have to start by figuring out how to build PyAudio for Python 3.4 on Mac OS X.

Look here: http://people.csail.mit.edu/hubert/git/pyaudio.git for the code.

Look here for the secret to building this on Mac OS X: https://stackoverflow.com/questions/2893193/building-portaudio-and-pyaudio-on-mac-running-snow-leopard-arch-issues/2906040#2906040.

Summary.

1. Get pyaudio source.
2. Inside pyaudio create a portaudio-v19. Get the portaudio source and put it here.
3. Inside pyaudio/pyaudio, do ./config; make and sudo make install
4. Inside pyaudio, do python3.4 setup.py install --static-link

On Pre-built Binaries for Python Packages

Or.

Why I Hate Windows.

For Mac OS X, you download XCode (for free) and you can build anything. For Linux, you use some kind of yum or rpm installer for the developer tools, and you can build anything.

For Windows...

Pre-built binaries. 😂

And a hope that the version numbers all match up properly. 😖

In many cases, you can use http://www.mingw.org or http://www.cygwin.com. Many projects can work well with one or both of these compilers.

In some cases, however, you have for fork over $$$ for Microsoft Visual Studio to download and build a Python module with a C extension.

The problem is a show-stopper for many n00bz. They are lead to believe that pip does everything. And it does -- for Mac OS X and Linux; for Windows, however, it does almost everything. And it's not obvious to the n00bz what the problem is when pip barfs because there's no suitable C compiler.

"Replace that junk Windows PC" is not an appropriate response. Although I often suggest it as the first solution when things won't install. 😡

Often Anaconda is the solution. It includes MinGW and you can (for a fee) buy their bundle of database drivers. The install for Anaconda is breathtakingly simple, removing a great deal of the potential complexity of assembling a tech stack for Python.

In other cases, we have to do some hand-holding to show how to find a pre-built binary for Windows.

Shiny New MacBook Pro

Wow. Just Wow. An almost seamless technology change. Almost.

The old MacBook Pro (dual core 4Gb RAM) was struggling to keep up. Struggling. It had been dropped once, so there was a ding in the corner. The trackpad "click" wasn't reliably clicking. It was shaky.

Nothing that couldn't be cured by a new Bluetooth keyboard and/or mouse. Awkward, but cheap.

Instead, I opted for a new quad-core 8Gb MacBook Pro.

Hence the Wow.

Here's how the upgrade worked.

I logged in once in the Apple Store to create an "Administrator" account. That's Not Me, but it allowed me to configure and register the machine.

Go Home.

1. Finish the last Time Machine backup of the old machine.
2. Move the Time Machine device to the new machine.
3. Use the Migration Assistant to recover everything from the old machine. There was 300+ Gb of stuff, so it took a few hours. Completely hands-off. Completely successful the first time.

Turn on WiFi (it's not always on for me, the story is complicated; it involves going to a coffee shop.)

Almost everything is perfectly normal and usable on the new machine.

1Password wanted me to login to the App Store to be sure the licenses were all up-to-snuff.

DropBox wanted me to login again to their server.

GPSNavX needs a license key. Their keys are delightfully short, but apparently encode a date or something and can't be reused easily.

Python3.3 was -- of course -- a non-starter. Not surprising, really, since it's not an "app" that can be moved neatly by Mac OS X Migration Assistant.

The Python download and install was painless. The ActiveState ActiveTcl is also important because I do use tinter and IDLE. The Python page is very explicit about the correct release of ActiveTcl for Mac OS X. And I still did it wrong the first time.

while the ActiveState web site refers to 8.5.15.0, the installer dmg link has been updated to download ActiveTcl 8.5.15.1.

Today's job, then, is to put setuptools (easy_install) and pip onto this Mac and begin the process of figuring out what's missing that I really use. I install a fair amount of stuff experimentally; stuff I don't really want or need.  And I always install it "for real" in Python's site-packages because I'm too lazy to simply download the Git repository and update the PYTHONPATH manually.

We're talking about docutils, Sphinx, Django, Jinja2, and SQLAlchemy. To get started. PyYAML and PIL are probably required, but I'll wait until I need them.

Apple's Feckless Download Protocol

For those of us who live mostly off the grid, with inconsistent WiFi through hosts that flake out frequently, Apple's AppStore download protocol is absolutely infuriating. It's the most brittle damn protocol I've ever seen.

When there's any disruption, it simply discards the data it has and starts again.

How stupid. How blatantly and infuriatingly stupid.

If I pause a download, it will resume. If it breaks, it will not resume. WTF?

For some things, I can use BitTorrent, which tolerates noisy links. But for proper AppStore Apps, their protocol is the pits.

Anyone know anyone at Apple who's able to work on a solution to this?

Almost a good idea

Appleworks (formerly Clarisworks) is software that's been dead meat since 2007.

See http://en.wikipedia.org/wiki/AppleWorks#End_of_Appleworks

Which is fine unless you have an old computer with old applications that still works. For example, a 2002-vintage iMac G4 http://www.imachistory.com/2002/ still works. Slowly.

When someone jumps 11 years to a new iMac, they find that their 2002 iMac with 2007 apps has files which are essentially unreadable by modern applications.

How can someone jump a decade and preserve their content?

1.  iWork Pages is cheap. Really. $19.99.  I could have used it to convert their files to their new iMac and then told them to ignore the app. Pages can be hard to learn. For someone jumping from 2007-vintage apps, it's probably too much. However, they can use TextEdit once the files are converted to RTF format.
2.  iWork for iCloud may be a better idea. But they have to wait a while for it to come out. And they want their files now.
3.  Try to write a data extractor.

Here are some places to start.

• https://github.com/teacurran/appleworks-parser
• http://fossies.org/linux/misc/abiword-2.9.4.tar.gz/dox/ie__imp__ClarisWorks_8cpp_source.html This appears to have a known bug in chaining through the ETBL resources.
• https://github.com/joshenders/appleworks_format This project is more notes and examples than useful code.

Documentation on the Appleworks file format does not seem to exist. It's a very weird void, utterly bereft of information.

In the long run $19.99 for a throw-away copy of Pages is probably the smartest solution.

However, if you're perhaps deranged, you can track down the content through a simple brute-force analysis of the file. This is Python3 code to scrape the content out of a .CWK file.

import argparse
import struct
import sys
import os
from io import open

class CWK:
    """Analyzes a .CWK file; must be given a file opened in "rb" mode.
    """
    DSET = b"DSET"
    BOBO = b"BOBO"
    ETBL = b"ETBL"

    def __init__( self, open_file ):
        self.the_file= open_file
        self.data= open_file.read()

    def header( self ):
        self.version= self.data[0:4]
        #print( self.version[:3] )
        bobo= self.data[4:8]
        assert bobo == self.BOBO
        version_prev= self.data[8:12]
        #print( version_prev[:3] )
        return self.version

    def margins( self ):
        self.height_page= struct.unpack( ">h", self.data[30:32] )
        self.width_page= struct.unpack( ">h", self.data[32:34] )
        self.margin_1= struct.unpack( ">h", self.data[34:36] )
        self.margin_2= struct.unpack( ">h", self.data[36:38] )
        self.margin_3= struct.unpack( ">h", self.data[38:40] )
        self.margin_4= struct.unpack( ">h", self.data[40:42] )
        self.margin_5= struct.unpack( ">h", self.data[42:44] )
        self.margin_6= struct.unpack( ">h", self.data[44:46] )
        self.height_page_inner= struct.unpack( ">h", self.data[46:48] )
        self.width_page_inner= struct.unpack( ">h", self.data[48:50] )

    def dset_iter( self ):
        """First DSET appears to have content.

        This DSET parsing may not be completely correct. 

        But it finds the first DSET, which includes all
        of the content except for headers and footers.

        It seems wrong to simply search for DSET; some part of the
        resource directory should point to this or provide an offset to it.
        """
        for i in range(len(self.data)-4):
            if self.data[i:i+4] == self.DSET:
                    #print( "DSET", i, hex(i) )
                    pos= i+4
                    for b in range(5): # Really? Always 5?
                        size, count= struct.unpack( ">Ih", self.data[pos:pos+6] )
                        pos += size+4
                    #print( self.data[i:pos] )
                    yield pos
    def content_iter( self, position ):
        """A given DSET may have multiple contiguous blocks of text."""
        done= False
        while not done:
            size= struct.unpack( ">I", self.data[position:position+4] )[0]
            content= self.data[position+4:position+4+size].decode("MacRoman")
            #print( "ENDING", repr(self.data[position+4+size-1]) )
            if self.data[position+4+size-1] == 0:
                yield content[:-1]
                done= True
                break
            else:
                yield content
                position += size+4

The function invoked from the command line is this.

def convert( *file_list ):
    for f in file_list:
        base, ext = os.path.splitext( f )
        new_file= base+".txt"
        print( '"Converting {0} to {1}"'.format(f,new_file) )
        with open(f,'rb') as source:
            cwk= CWK( source )
            cwk.header()
            with open(new_file,'w',encoding="MacRoman") as target:
                position = next( cwk.dset_iter() )
                for content in cwk.content_iter(position):
                    # print( content.encode("ASCII",errors="backslashreplace") )
                    target.write( content )
        atime, mtime = os.path.getatime(f), os.path.getmtime(f)
        os.utime( new_file, (atime,mtime) )

This is brute-force. But. It seemed to work. Buying Pages would have been less work and probably produced better results.

This does have the advantage of producing files with the original date stamps.  Other than that, it seems an exercise in futility because there's so little documentation.

What's potentially cool about this is the sane way that Python3 handles bytes as input. Particularly pleasant is the way we can transform the file-system sequence of bytes into proper Python strings with a very simple bytes.decode().

couchdb on Mac OS X

I've started to work with couchdb.

I've blogged before about the problems of SQL schema in Escaping the Relational Schema Trap.

A SQL schema -- for many applications -- is too confining.  It creates cost with relatively little value.  Once upon a time (when disks where expensive and computers were slow) it was essential.

The funny part about using couchdb is the build process.

In the couchdb Wiki, they have a page for Mac OS X installation.  The relevant part is the following line of shell script.

brew install couchdb

That's it?

Yes.  If--and only if--you follow the directions.

If you don't follow the directions, however, it can take all day.  Here are the steps.

1. Install Apple's Developer Tools.  I have Mac OS X 10.7.  XCode 4.3 for Lion.
2. Launch XTools and install the command-line utilities.  This is important because it includes things like make.
3. Remove fink or MacPorts if you happened to have used them for anything.  For fink, you'll need to clean it out of your ~/.profile or ~/.bash_profile and rename the /sw directory.
4. Install Homebrew.   Use the one-line ruby script from the Installation page of the Homebrew wiki.  This: /usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)" I tried several wrong ways before doing it the right way.
5. Install couchdb using homebrew.  It takes a while.

There are numerous things which may go wrong.

A missing lib/crt.1.10.6.o, for example. This is just an out-of-date Xcode. It took few hours of failed experiments to (a) realize it and (b) get the right one.  There are several proposed solutions around the web.  Most of them are clever, but ineffective.  Just get the right Xcode.

A failure to build Erlang. This was just an improperly installed version of Homebrew.  There were a lot of message.  A lot.  I messed around with a lot of things until I finally crashed brew doctor. I deleted and reinstalled Homebrew and everything built.  First try.

Where is Python Used? (Update)

This is a fair-to-partly silly question that shows up on places like StackOverflow once in a while.

Python is used widely and pretty heavily.

It's a built-in feature to many operating systems in common use. The exception, of course, is Windows.

I just found out -- the hard way -- that Python 2.6 is an integral part of Apple's iLife suite of products.

Important safety tip for Mac OS X users. System.Libary.Frameworks should not be touched.

Also, it helps to get used to the idea of typing python3 on the command-line. Further, it helps to skip Python 3.1 and go straight to Python 3.2.

Python 3.2 has argparse and the new dictionary-based configuration of logging.

BBEdit Configuration

After installing Python 2.6 in Mac OS X, I had problems with BBEdit not finding the right version of Python. It kept running an old 2.5 version.

I finally tracked down the BBEdit documentation, http://pine.barebones.com/manual/BBEdit_9_User_Manual.pdf.

Found this: "BBEdit expects to find Python in /usr/bin, /usr/local/bin, or /sw/bin. If you have installed Python elsewhere, you must create a symbolic link in /usr/local/bin pointing to your copy of Python in order to use pydoc and the Python debugger."

Checked in /usr/bin and found an old Python there. I think Fink did that. Removed it and BBEdit is much happier. As is Komodo Edit.

Macintosh Support

A handy CNET review site: http://reviews.cnet.com/macfixit/