CI/CD DevOps and Python

See https://www.slideshare.net/ITRevolution/does-sfo-2016-topo-pal-devops-at-capital-one for the 16 gates that separate a good idea from secure, productive use of software. While a lot of DevOps folks like the idea, when it comes to implementing it for Python apps, they get confused.

The confusion seems to stem from Python's lack of a proper "build" step in the CI/CD pipeline. I've had the "everything involves a build" argument and the "well setup.py is analogous to a build" arguments. I have to acquiesce to these positions as part of making progress. In this case, reasoning by analogy can be misleading.

I want to focus on the two gates that are part of the code itself, separate from the rest of the pipeline.

• Static Analysis 
• >80% Code Coverage (which implies Unit Tests)

Unit Testing

My preference is to run the unit test suite first and get that out of the way. If the unit test suite fails, or fails to cover 80% of the code, any other considerations are moot.

I like Git triggers based on Pull Requests (PR's) and Merge to Master for checking these two conditions. I like the idea that a PR can't be discussed until unit tests pass. They can also be part of whatever other pipeline is going on, but I like them to be done early and often.

(I worked on a sprint team where the PR unit test wasn't trusted by one of the devs: he'd carefully check out the branch and rerun the unit tests. His comments were good, so the extra effort paid off. I guess.)

After flirting with a lot of frameworks, I'm happiest with py.test. I like the py.test-coverage plug-in and the py.test-BDD plug-in.

Yes. We have acceptance tests for our features written in Gherkin. And we have pytest fixtures that are used by pytest-bdd to process the scenarios in the Gherkin feature files. It actually works out nicely because we have a cucumber.json file that makes everyone happy that we've run an acceptance test suite along with our unit test suite.

What's important is the coverage report is painless and automatic.

And it's compatible with the Ruby-based cucumber tool without involving any actual ruby.

For integration testing, we use Behave. This is a bit more cumbersome than pytest-bdd, but it's appropriate for the bigger-picture testing where we have a docker cluster and have to see a number of "Then" steps to confirm operations spread across a suite of microservices.

The goofy question that often leads to endless confusion is the relationship between unit testing and "build." The setup.py setup definition includes a `tests_require` parameter. This *should be* all that's needed to do `python setup.py test`, which *should be* all that's involved in testing.

Is it a "build"? No. But. You can tell the DevOps folks it's a build if it makes them happy.

Static Analysis

There are several kinds of static analysis. Folks who work in Java are used to having Sonar analysis performed. This is above and beyond the static analysis already performed by the compiler. It seems excessive to me, but folks deploying Java seem to like it.

For Python, there are two important static analysis tools. And this is another source of profound confusion for DevOps folks new to Python.

• pylint (I prefer this over pyflakes and pep8. See About style guide of python and linter tool for a long list of tools.)
• mypy

I like to extract the last line of the pylint output and use that numeric score as the "bottom line" on static analysis part 1. While the default setting is 9.5, that can be a challenge, and we prefer 9.0 as well as some local pylintrc modifications to modify some checkers (e.g., set line length to 120.)

For mypy, it's a little bit more complex. We're still fumbling around here.

Ideally, the type hints are all clean and mypy has nothing to say. We can, of course, fix any errors by claiming everything uses Any and returns Any and every assignment statement sets an Any value. But that's so wrong.

There are (still) modules which require typeshed stub definitions. Ideally, we'd provide these. This would be better than using Any as a hack-around. While good, it's a lot of work.

For now, I think it's sensible to have two "pass" rules for mypy: clean or typeshed error. If mypy is silent, that's perfect. If mypy can't find stubs in typeshed, we can let this go for now and log an issue from the CI/CD pipeline to note the presence of technical debt.

In the best of all worlds, we'd fork the package, fix the type hints, and put in a PR. That's a lot better than using typeshed to work around the lack of hints. 

And, of course, there's the "build" question. For mypy to work, the dependencies (or their typeshed stubs) must be present. We wind up doing a `python setup.py install` to build out the requirements. Is this a "build"? Maybe. You can tell the DevOps folks it's a build if it makes them happy.

If you want idempotent server (or container) builds, you'll need to be sure that you pin specific versions. It can help to break this into two parts:

• a requirements.txt with specific versions 
• a generic version-free high-level list in setup.py

The reason for this separation is to make it easy to do a `pip install` or `conda create` from the detailed requirements. Once that's out of the way, the `python setup.py` will run very quickly. If you're working with Docker containers, the `pip install` (or `conda create`) can be part of the Dockerfile, and then tests or static analysis can be run separately, after the initial wave of installations.

Needlessly Redundant Overcommunication and DevOps

At the "day job" I use a Windows laptop. It was essential for a project I might have started, but didn't. So now I'm stuck with it until the budgetary gods deem that it's been paid for and I can request something more useful.  Mostly, however, Windows is fine. It doesn't behave too badly and most of the awful "features" are concealed by Python's libraries.

This is context for a strange interaction today. It seems to exemplify DevOps and the cruddy laptop problem.

The goofy Microsoft Office Communicator -- the one that's so often used instead of a good chat program like Slack or HipChat -- pinged.  The message went something like this.

"I sent you an email just now. Can you read it and reply?"

I was stunned. Too stunned to save the text.  This is either someone being aggressive almost to a point that hints at rudeness, or someone vague on how email works. Let's assume the second option. I can only reply, "I agree with you, that is how email works."

The email was a kind of vague question about server provisioning.  It was something along the lines of

"Do we provision our own server with Ansible or Chef? Or is there a team to provision servers for us? ..."

It went on to describe details of a fantasy world where someone would write Chef scripts for them.  The rest of the email mostly ignored the first question entirely.

The Real Question

If you're familiar with DevOps as a concept, then server provisioning is -- like most problems -- something that the developers need to solve. Technical Support folks may provide tools (Ansible, for example) to help build the server, but there aren't a room full of support people waiting for your story ("make me a server") to appear on their Kanban board.

Indeed, there was never the kind of support implied in the email, even in non-DevOps organizations. In a "traditional"  Dev-vs.-Ops organization, the folks that built servers were (a) overbooked, (b) uninterested in the details of our particular problem, or (c) only grudgingly let us use an existing server that doesn't quite fit our requirements. They rarely built servers for us.

Reason A, of course, is business as usual. Unless we're the Hippo (Highest Paid Person in the Organization,) there's always some other project that's somehow more important than whatever foolishness we're engaged in. How many times have we been told that "The STARS Project is tying up all our resources. It will be 90 days before..."? Gotcha. The bad part about this situation is when the person paying the bills says to me "You need to make them respond." How -- precisely -- do you propose that I change the internal reward system of the ops people?

We could label this as a passive-aggressive approach. They're waiting for us establish a schedule so that they can shoot it down. Or maybe that's reading way too much into the situation. Maybe they're really just overbooked.

Regarding reason B. Years ago, I had a hilarious interaction where we sent a stream of emails explaining our server requirements. The emails were not exactly ignored. But. When we asked about the status of our servers, the person responsible for the team brought a yellow pad and wrote down the requirements. I read the email to them. Without a trace of embarrassment, they wrote down what I was reading from an email.  (It was long enough ago, that we didn't have laptops, and I had a hard-copy of the email. They refused the hard-copy. I had to read it. Really.)

Were they clueless about how email works? Or. Was this a kind of passive-aggressive approach to architecture where our input was discounted to zero because it didn't count until they wrote it on a their yellow pad? The behavior was bizarre.

Something similar happened with another organization. We made server recommendations. They didn't like the server recommendations. Not because the recommendations seemed wrong, but because we didn't have a formal sciency-seeming methodology for fantasizing about servers that were required to support the fantasy software which hadn't been written yet. They felt it necessary to complain. And when we talked with hardware vendors, they felt it necessary to customize the cheap commodity servers.

[It got weirder. They were convinced that a server farm needed to be designed from the bottom up.  I endured a lecture on how a properly sciency-seeming methodology started by deciding on L1 and L2 cache sizing, bus timing, and worked through memory allocation and then I slowly grew to see that they had no clue what they were talking about when buying commodity servers by the rack-full for software that doesn't exist yet.]

We all know about reason C. The reason for DevOps is to avoid being stuffed into a kind of random server where there are upgrades that we all have to agree on. Or -- worse -- a server that can't be upgraded because no one will agree. A single app team vetos all changes.

"We can't install Anaconda 3 because we know that Python 3 is incompatible with Python 2"...

What?

I stopped understanding at that point. It seemed like the rest of the answer amounted to "having the second Anaconda on a separate path could lead to problems. It can't be proven that no problems will arise, so we'll assume that -- somehow -- PATH settings will get altered randomly and a Python 2 job will crash because it accidentally had the wrong PATH and accidentally ran with Python 3."

It was impossible to explain that this is a non-problem. Their response was "But we can't be sure." That's the last resort of someone who refuses to change. And it's the final answer. Even if you do a proof-of-concept, they'll find reasons to doubt the POC's results because they can't be sure the POC mirrors production.

The Real Answer

The answer to the original Ping and the Email was "You're going to do this yourself."  I included links to four or five corporate missives on Chef, Ansible, DevOps, and how to fill in the form for a cloud server.

I have my doubts -- though -- that this would be seen as helpful.

They may not be happy because they don't get to use Communicator and Email and someone else's Kanban board to get this done. They don't get to ask someone else what they're doing and why they're not getting it done on time. They don't get to second-guess their technical decisions. They actually have to do it. And that may not work out well.

The truly passive-aggressive don't seem to do things by themselves. It appears to me that they spend a lot of time looking for reasons to stall. Either they need to get more information or get organized or they need to have some kind more official "permission" to proceed. Lacking any further information, I chalk it up to them only feeling successful when they've found the flaws in what someone else did.

It's challenging sometimes to make it clear that a rambling email asking for someone else to help is going nowhere. A Communicator ping followed by an email isn't actually getting anything done. It's essentially stalling, waiting for more information, getting organized, or waiting for permission. Overcommunication can become a stalling tactic or maybe a way to avoid responsibility.

I'm stuck with a cruddy laptop because the budget gods have laid down some laws that don't make a lick of technical sense. I think that the short-sighted "use it until it physically wears out" might be more costly than "find the right tool, we'll recycle the old one appropriately." In the same way, the shared server world view is clearly costly.  We shouldn't share a server "because it's there."

The move to DevOps allows us to build a server rather than discuss building a server.

I want a DevOps parallel for my developer workstation. I don't want permission or authorization. I don't want to overcommunicate with the budget gods. I want a workstation unencumbered by permission-seeking.

More Thoughts on the friction of DevOps

Read this: How 'DevOps' is Killing the Developer

My pull-out quote:

This is why we see so many developers that can't pass FizzBuzz: they never really had to write any code.

I agree: It appears that DevOps may be more symptom than solution.

I have one tiny objection to any otherwise excellent series of points: I don't like the totem pole analogy.

I prefer a supply-chain:

• Release Engineers respond to user needs.
• Quality Engineers respond to the Release Engineers' needs for assurance that something is fit for use.
• Developers respond to Release Engineers by providing software.
• Similarly, procurement folks may purchase or lease or download and pay royalties for software. 

I think of it like this:

Developer ⇒ QE ⇒ RE ⇒ Users

No top-to-bottom. More a sequence of more-or-less peers.

I still agree with the central tenet: a developer is able to march the software from concept to user. We don't really expect QE or RE to create software. We might expect some skill sharing between QE and RE.

Many years ago, I posted this: IT’s Drive to Self-Destruction, which is random and whiny but related to this point about DevOps. The idea is that key developers create competitive advantage. Release Engineers put it in the hands of users. Both are important. Without creation there's no deployment. Without deployment, creators can be diverted to deployment, so deployment can still go forward, but it will be slower.

The key point is this:

If a developer is spending time with DevOps (and TechOps) trying to get stuff deployed, who's developing the Next Big Thing? 

It appears that DevOps may be more symptom than solution

It appears that DevOps may be a symptom of a bigger problem. The bigger problem? Java.

Java development -- with a giant framework like Spring -- seems to accrete layers and layers of stuff. And more stuff.  And bonus stuff on top the the excess stuff.

The in-house framework that's used on top of the Spring Framework that's used to isolate us from WebLogic that used to isolate us from REST seems -- well -- heavy. Very heavy.

And the code cannot be built without a heavy investment in learning Maven. It can't be practically deployed without Hudson, Nexus, and Subversion. And Sonar and Hamcrest and mountains of stuff that's implicitly brought in by the mountain of stuff already listed in the Maven pom.xml files.

The deployment from Nexus artifacts to a WebLogic server also involves uDeploy. Because the whole thing is AWS-based, this kind of overhead seems unavoidable.

Bunches of web-based tools to manage the bunch of server-based tools to build and deploy.

Let me emphasize this: bunches of tools.

Architecturally, we're focused on building "micro services". Consequently, an API takes about a sprint to build. Sometimes a single developer can do the whole CRUD spectrum in a sprint for something relatively simple. That's five API's by our count: GET one, GET many, POST, PUT and DELETE: each operation counts as a separate API.

Then we're into CI/CD overheads. It's a full sprint of flailing around with deployment onto a dev servers to get something testable and get back test results so we can fix problems. A great deal of time spent making sure that all the right versions of the right artifacts are properly linked. Doesn't work? Oh. Stuff was updated: fix your pom's.

It's another sprint after that flailing around with the CI/CD folks to get onto official QA servers. Not building in Husdon? Wrong Nexus setup in Maven: go back to square one. Not deployable to WebLogic? Spring Beans that aren't relevant when doing unit testing are not being built by WebLogic on the QA server because the .XML configuration or the annotations or something is wrong.

What's important here is that ⅔ of the duration is related to the simple complexity of Java.

The DevOps folks are trying really hard to mitigate that complexity. And to an extent, they're sort of successful.

But. Let's take a step back.

1. We have hellish complexity in our gigantic, layered software toolset.
2. We've thrown complicated tools at the hellish complexity, creating -- well -- more complexity.

This doesn't seem right. More complexity to solve the problems of complexity just don't seem right.

My summary is this: the fact that DevOps even exists seems like an indictment of the awful complexity of the toolset. It feels like DevOps is a symptom and Java is the problem.