Archive for the ‘Software Engineering’ Category

Publishing packages with npm and CircleCI

A common workflow

In recent years, I’ve pushed more and more for common, automated, deployment processes. In practice, this has usually meant:

  • Code is managed with Git, and tags are used for releases
  • Tags (and hence releases) are created via GitHub
  • Creating a tag executes everything in the CI pipeline + a few more tasks for the deployments

The result is that all deployments go through the same process (no deploy scripts run on personal machines), in the same environment (the CI container). It eliminates discrepancies in how things are deployed, avoids workflow differences and failures due to environment variance, and flattens the learning curve (developers only need to learn about Git tags).

Here I’ll present how I’ve been approaching this when it comes to publishing npm packages, with deployment tasks handled via CircleCI. The flow will look something like this:

Setting up CircleCI

First things first, we need the CircleCI pipeline to trigger when a tag is created. At the bottom of your circle.yml file, add filter for “deployment.”

version: 2 jobs: build: docker: - image: circleci/node:10.0.0 working_directory: ~/repo steps: - checkout # # Other stuff (run npm install, execute tests, etc.) # ... deployment: trigger_tag: tag: /.*/

Authenticating with the npm registry

Create an npm token and expose it as an environment variable in CircleCI (in this case, I’ve named it NPM_TOKEN). Then, add a step to authenticate with the npm registry in your circle.yml:

version: 2 jobs: build: docker: - image: circleci/node:10.0.0 working_directory: ~/repo steps: - checkout # # Other stuff (run npm install, execute tests, etc.) # ... - run: name: Authenticate with registry command: echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" > ~/repo/.npmrc deployment: trigger_tag: tag: /.*/

Versioning

Things get a little weird when it comes to versioning. npm expects a version declared in the project’s package.json file. However, this goes against managing releases (and thus versioning) with Git tags. I see two potential solutions here:

  • Manage versions with both Git and npm, with the npm package version mirroring the tag. This would mean updating the version in package.json first, then creating the Git tag.
  • Only update/set the version in package.json within the pipeline, and set it to the version indicated by the Git tag.

I like the latter solution, as forgetting to update the version number in package.json is an annoyance that pops up frequently for me. Also, dealing with version numbers in 2 places, across 2 systems, is an unnecessary bit of complexity and cognitive load. There is one oddity however, you still need a version number in package.json when developing and using the npm tool, as npm requires it and will complain if it’s not there or in an invalid format. I tend to set it to “0.0.0”, indicating a development version; e.g.

{ "name": "paper-plane", "version": "0.0.0", // ... }

In the pipeline, we’ll reference the CIRCLE_TAG environment variable to get the Git tag and use to correctly set the version in package.json. Based on semantic versioning conventions, we expect the tag to have the format “vX.Y.Z”, so we’ll need to strip away the “v” and then we’ll use “X.Y.Z” for the version in package.json. We can use npm version to set the version number:

npm --no-git-tag-version version ${CIRCLE_TAG:1}

Note the –no-git-tag-version flag. This is necessary as the default behavior of npm version is to commit the tag to the git repo.

Publishing

Publishing is simply done via npm publish. Pulling together the CIRCLE_TAG check, applying the version, and publishing into a deploy step, we get something like this:

version: 2 jobs: build: docker: - image: circleci/node:10.0.0 working_directory: ~/repo steps: - checkout # # Other stuff (run npm install, execute tests, etc.) # ... - run: name: Authenticate with registry command: echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" > ~/repo/.npmrc - deploy: name: Updating version num and publishing command: | if [[ "${CIRCLE_TAG}" =~ v[0-9]+(\.[0-9]+)* ]]; then npm --no-git-tag-version version ${CIRCLE_TAG:1} npm publish fi deployment: trigger_tag: tag: /.*/

… and we’re done 🚀!

For further reference, this circle.yml uses the steps presented above.

Null

One of my favorite videos is Null Island from Minute Earth. I frequently link to it when I get into a discussion about whether null is an acceptable value for a certain use case.

What I really like is the definition around null and the focus of null having a concrete definition, that is: “we don’t know”. When used in this way, we have a clear understanding of what null is and the context in which it’s used (whether some field in a relational table, JSON object, value object, etc.) inherits this definition (i.e. “it’s either this value or we don’t know”), yielding something that’s fairly easy to reason about.

When nulls are ill-defined or have multiple definitions, complexity and confusion grow. Null is not:

  • Zero
  • Empty set
  • Empty string
  • Invalid value
  • A flag value for an error

Equating null to any of the above means that if you come across a null, you need to dig deeper into your code or database to figure out what that null actually means.

The flip side of this is avoiding nulls altogether, and there are really 2 cases here:

  • There is no need for null (i.e. we do know what the value is, in every use case)
  • Architect the system such that a null isn’t surfaced

In the first case, null doesn’t fit the use case, so there’s no need for it. When possible, this is ideal, and you avoid the necessity for null checks.

For the second case, architecting this way always seems to involve adding more complexity, to the point where it’s questionable if there’s a net benefit.

The road never built

On reliability, Robert Glass notes the following in Frequently Forgotten Fundamental Facts about Software Engineering:

Roughly 35 percent of software defects emerge from missing logic paths, and another 40 percent are from the execution of a unique combination of logic paths. They will not be caught by 100-percent coverage (100-percent coverage can, therefore, potentially detect only about 25 percent of the errors!).

John Cook dubs these “sins of omission”:

If these figures are correct, three out of four software bugs are sins of omission, errors due to things left undone. These are bugs due to contingencies the developers did not think to handle.

I can’t validate the statistics, but they do ring true to my experiences as well; simply forgetting to handle an edge case, or even a not-so-edge case, is something I’ve done more often than I’d like. I’ve introduced my fair share of “true bugs”, code paths that fails to do what’s intended, but with far less frequency.

The statistics also hint at the limitations of unit testing, as you can can’t test a logic path that doesn’t exist. While you can push for (and maybe even attain) 100% code coverage, it’s a fuzzy metric and by no means guarantees error-free functionality.

Code Coverage