StackOverflow Meetup

See http://www.meetup.com/stackoverflow/Hampton-VA/ for information on next week's Stack Exchange meetup.

For other events near you, see the Meetup page.

I'll be wearing my "official" Stack Overflow shirt. I'll try to grab a seat by the door, also, to be easy to find.

#SOMeetup

Code Deletion

A joyous milestone today. Removed much of our pre-Piston RESTful web services code.

We started with the Django-REST Interface. While nice, it imposed a number of restrictions that were onerous. In particular, we have a lot of non-model responses. They're model-like data that we serialize to be compatible with Django, but without actually being first-class Django Model objects.

In order to provide a generic, but detailed "status" message, we actually defined a Model that we never instantiated in the database. We'd build (but not save) instances, just to make it easy to serialize them.

What a hack.

To further complicate things, I failed to really understand the way that the alternate user authentication sources worked, and how much of the Django authentication process was better handled through middleware. Failing to fully understand that, I wrote too much code. We tinkered with incoming requests to extract HTTP Authorization headers. We tinkered to handle Amazon-style key/signature values in the GET or POST. And we tinkered to handle OpenAM authentication cookies.

Too much code.

And it gets worse. I tried to use urllib2 for a wide variety of RESTful requests. This means more than GET and POST. That was a mistake. httplib works out a little better for doing RESTful web services requests. If you don't have a lot of complex proxy server handling. And if you don't have a lot of complex authentication.

In our case, the urllib2 was handling the 401 retries, cookies and also had some extra handler code to treat a 201 Created response as a non-error (by default, urrlib2 gagged on 201 Created). Also, urllib2 appears to be lazy and doesn't send everything or close the sockets in the event of a problem. This makes unit testing just a bit more complex than necessary. Also, urllib2 required a couple of monkey patches to let us use PUT and DELETE without problems.

Needless Complexity.

It turns out that handling a 401 retry in httplib isn't really all that difficult. That ended the use case for urllib2.

What's nice is

1. Being able to unmake some bad decisions.
2. Rerunning the entire unit test suite to ferret out the remaining concealed dependencies.
3. Removing hack-arounds, volume and complexity.

We still have a lot of work to make full use of Piston. That will lead to removing yet more code. It will, however, also change the API's slightly because the ".../xml/..." URL's will have a different format and we'll introduce ".../django/..." URL's which will have the current format.

Where is Python Used? (Update)

This is a fair-to-partly silly question that shows up on places like StackOverflow once in a while.

Python is used widely and pretty heavily.

It's a built-in feature to many operating systems in common use. The exception, of course, is Windows.

I just found out -- the hard way -- that Python 2.6 is an integral part of Apple's iLife suite of products.

Important safety tip for Mac OS X users. System.Libary.Frameworks should not be touched.

Also, it helps to get used to the idea of typing python3 on the command-line. Further, it helps to skip Python 3.1 and go straight to Python 3.2.

Python 3.2 has argparse and the new dictionary-based configuration of logging.

XBox Live -- Can't Unsubscribe

Here's a lack of a use case for you.

Someone -- fraudulently -- used my email address to subscribe to XBox live. I cannot remedy this. Apparently, neither can Microsoft.

I get spam from XBox. I change my passwords all over the place.

I go to the XBox live web site to cancel this fraudulent account. I can't. There's no place to do that. I cannot cancel the account because it can only be done through the XBox console. Except -- of course -- in the case of fraud, the email user doesn't have a console.

So I call the help desk. "Please remove my email from this account that fraudulently uses it." They can't. Absolutely can't. All I can do is route xbox.com email into the spam folder. That's it.

Nice help desk agent. Doing the best she can. But, she cannot find the email address and disconnect me from spam or XBox or XBox live. Someone at the console needs to do that.

How do we contact the person at the XBox Console? Can't send them email -- it goes to me!

Somehow, someone at Microsoft has to call "beezyNdetroit" on the phone (I guess) and break the bad news to them that they're fraudulently using one of my email addresses for their XBox spam.

To Robert Fulton, Regarding the "Steam Boat"

"What sir, would you make a ship sail against the wind and currents by lighting a bonfire under her deck? I pray you excuse me. I have no time to listen to such nonsense."

-- Napoleon Bonaparte

There's no authoritative source for this quote. Since Fulton was commissioned to build a submarine and did build a steam-powered boat in France, it's unlikely for this quote to be actually true.

A great list of related quotes: Famous Authoritative Pronouncements.

Improving the "Velocity" of IT

Check this out: "IT Is Too Darn Slow".

This article is packed with helpful advice on how to improve "velocity" and the pace of innovation.

Good quotes: "Once IT decides to focus on speed, two obstacles get in the way: security and governance." This is important. Manage security without it becoming an impediment.

One of the most important things not mentioned here is the idea that internal security must be a simple, cheap commodity. A identity manager and SSO framework needs to be standard, available, and well-understood. Each project shouldn't involve head-scratching and deep thinking about security. Like an OS and a file system, security infrastructure should be a given.

Another good quote: "IT can't set priorities for 10 projects spread over the next two years because, once projects one, two, and three are done, that will change what would have been four, five, and six."

What isn't provided is the a way to handle the accounting practices that always seem to take IT hostage. The idea of "capital" vs. "expense" can serve as a weird artificial boundary on innovative projects. Lots of innovation gets chopped off when the "capital" budget is spent, and we switch over to "expense" where we can't invent anything new. It's the same people. Yet, the money is "different".

DIY and the Dumb Rules of Prevention

Check out this little item in eWeek. "Transforming DIY Projects from the Painful to the Productive".

As an outside consultant in a large number of organizations, I've seen a lot of DIY projects—what we used to call "end user computing". Indeed, I've even been hired by the user organizations because IT was ineffective.

But IT gets in it's own way. There are some Dumb Rules of IT that creates an environment in which software development and project management are nightmares over burdensome inefficiency. Why? How?

How is easy. IT has too much process. I sat with angry users at meetings where they begged me to influence the CIO to stop making everything into a large-scale, heavy-weight, closely-monitored "project". All that the users wanted were simple conversations on how best to interface with existing applications and frameworks.

They didn't want a "project".

Root Cause

Why does IT anger the users with too much process?

That's sometimes hard to see when we're an insider. From end-users, I learned that IT's response to the inherent complexity of software development is to treat the act of creating software solutions as if the development work were—itself—just more software. Software executed by people and organizations. They write process they way they write code.

Here's Dumb Rule 1 of success in IT: "When in doubt, define a process."

And that rule is often just wrong. Users hate it.

Process is wrong? For software development, yes, too much process is a bad thing. Why do you think we had to invent Agile methods? The way we create complex software is not helped in any way by having a large-scale, heavy-weight process defined. Adding process steps to the already complex task of creating software merely slows the work down.

All processes aren't bad. Processes to create software are demonstrably unhelpful.

Suggestion One: Lighten UP

"The first step is to bring business developers out of the shadows and into the limelight."

This is important, and difficult. IT is so enamored of large processes that simply embracing non-IT developers means wrapping them in processes they don't need and can't comply with.

While good advice, this isn't the real first step.

The real first step is stop weighing down everything with overly-defined, heavy-weight process definitions. Embrace Agile development where there are sprints and releases and leave it at that.

DIY folks who are not in IT can understand sprint and release, since that's how they work anyway. IT needs to help them get started and help them release. The rest has to be hands-off. Inside corporate IT and outside the corporate IT enclave.

Suggestion Two: Open The Door

"Next, it’s important for IT to take the lead in providing tools and guidelines that make it easy for business developers to start off on the right foot."

While also good advice, it's almost impossible to actually do.

Why? IT folks who have drunk the Proprietary (i.e., Microsoft) Kool Aid laced with contractual terms and conditions won't let end-users have expensive Visual Studio toolkits. And the true believers in Microsoft hegemony can't let users have non-Microsoft tools. All the large IT organizations that are all-singing, all-dancing, all-Microsoft can't empower end users very well at all.

They offer up Sharepoint and MS-Access and then shut down all further access to tools.

Here's Dumb Rule 2 of success in IT: "Measure Success By Licensing Fees".

Visual Studio works because its expensive. Right? Oracle is "enterprise" scale because it's expensive. SQLite is unacceptable because there's no support, right?

To provide tools and guidelines, Corporate IT has to either (1) stop paying so much for tools or (2) budget for user departments to have access to real tools or both.

If we don't allow the DIY folks a decent tool set they'll still create software using crappy tools. They're remarkably capable of building vast shadow systems in MS-Access. Seven interlocked Access databases that all magically work together on a single desktop is the kind of thing they'll do if we don't help them do something better.

Suggestion Three: Stop Driving and Start Guiding

DIY folks want first class access to servers, data and tools.

"IT can take the lead in providing guidelines and even code snippets or basic scripts that work with platforms that are commonly used to create DIY projects in your organization."

"Provide small doses of advice on how to design applications for scale and security that are led by your professional-development staff. "

This can't happen, of course. The reason is that we can't "trust" the end users. If we give them access to the data, they'll make a total mess of it. We know they'll make a mess of it because we have columns in the database where the users have stopped entering the data we carefully defined in the data dictionary and started entering other data.

Why, oh why do end users corrupt the columns in the database? We tear at our hair and rend our clothes asking this. Everyone asks this. It's the great rhetorical argument against allowing users into the hallowed halls of IT.

No one tries to answer it, of course. It's a rhetorical question. The answer is embarrassing.

Users corrupt and misuse the database because a small change to add a column is a 9-month delay preceded by endless useless meetings, endless useless IT process. While waiting for IT to make a simple change, it's easier to simply enter the data into another unused field.

IT creates they're own nightmare by being unable to adapt to business change.

Here's Dumb Rule 3 of success in IT: "Control The Data".

Rather than adapt to business changes, wrap them in process as a way to say "no". Have elaborate processes. Complex budget negotiations. Project prioritization meetings.

Yes, the resources are finite. More process, however, means fewer resources devoted to solving business problems.

Code that does updates needs a lot of code review and testing no matter who wrote it. If end-users have a DIY project, IT should be deeply involved in the QA process. End users don't particularly like the kind of rigorous, expensive, sophisticated testing that IT imposes on itself. That doesn't remove the need to engage the DIY folks in some kind of test plan.

Making Progress

IT needs to offer a menu of services including "secure, reliable and available data and processing resources".

IT needs to have a simple, easy-to-understand quality threshold for software.

Programming, software purchase, integration installation and configuration and the like can be done by IT or by anyone else who meets the simple quality threshold.

A test plan. Evidence that the tests are passed. Compatibility with some standards.

It should be very, very simple so that DIY can be made to work efficiently. Since it's going to happen anyway, it's better to do it well than to fight against it with Dumb Rules.