Thursday, January 12, 2012

Multiprocessing and Shared Objects [Revised]

Read this: Shared Counter with Python Multiprocessing.

Brilliant.  Thank you for this.

Too many of the questions on StackOverflow that include multi-threading are better approached as multi-processing.  In Linux, there are times when all threads of a single process are stopped while the process (as a whole) waits for system services to complete.  It's a consequence of the way select and poll work.  An example of the kind of sophisticated design required to avoid this can be found here.  Most I/O-intensive applications should be done via multi processing, not multi threading.

And.  The kind of shared objects that multi threading allows are often rare and require locks.

So, simplify your life.  When you hear about "threads", replace the word with "processes" and move on.  The implementation will be much nicer.

The standard gripe is that process creation is so expensive, and thread creation is relatively cheap.  All true.  That's why folks use process pools: to amortize the creation cost over a long period of operation.