/usr/local/bin/polkadot: line 1: Not: command not found

Or: A lesson in paying attention

What do you do when you see an error like that? If you are like me, you do not read the full message. You instead rush to create hypotheses and test them.

If you are not like me, you might have picked up on the curious fact that the first line of code of the polkadot command is the word Not.

If you are like me, you skim the error message, not notice the Not:, and assume that the computer does not have the polkadot binary. Why do you skim? Because you are eternally, anxiously, irretrievably, and intractably in a hurry. And this hurry leads to a dead end, which leads to frustration. And this hurry leads to a couple more dead ends.

Thank God for wiser people than myself, like my colleague Bruno, who retrieved the container with the polkadot binary from our repository and ran it locally to look around:

$ docker run -ti eu.gcr.io/$COMPANY/polkadot:v0.9.17-5e15a6a -- bash

Meanwhile, I found that the Dockerfile of our polkadot container assumes that every release conforms to a standard version notation: v0.9.17 in this case. Well, they misnamed this release. It was v0.9.17-rc4. Go to the release page and hover over the polkadot binary to see for yourself.

Cool catch, huh? But what caused the strange error message?

Again, Bruno was wiser than me and had the courage to look inside the polkadot binary, something which I considered to be scary and not necessary.

$ cat polkadot
Not Found

Nice! So the container tried to execute the binary but because Not is not a command present in the container, it returned the puzzling error.

But why does polkadot have Not Found as its code?

Unfortunately, this was the point where both of us got stuck. The output made it worse. Due to its simplicity, it made the problem more mysterious. Two words?! What do they mean? Why is "Found" capitalized? This caused a bit of exasperation. With time ticking, it culminated into a feeling of desperation. No light at the end of the tunnel. You revert to testing hypotheses that you feel to be dead ends, you start to form wild conspiracy theories, you degenerate. You compare working versions of the same container with this one, you read that error message five more times, you start to procrastinate.

Thankfully, time does funny things to our brains, if you let it. Just take a break. You will be different after. Maybe better.

Your intuition pulls you to look at the Dockerfile again.

VERSION=$(git -C $BUILD_DIR/polkadot describe --tags |awk -F '-' '{print$1}’)
... stuff ...
curl -O -L "https://github.com/paritytech/polkadot/releases/download/${VERSION}/polkadot"

The way the Dockerfile builds the version is alright, save for cases where the version is not what you expect. In these cases, the curl command does something you did not take into consideration. From the curl man page:

-O, --remote-name
Write output to a local file named like the remote file we get. (Only the file part of the remote file is used, the path is cut off.)

Maybe because curl did not find the ${VERSION} in question, it wrote the error message that would otherwise be written to stdout to the polkadot file.

At this point you feel that you are close. You swiftly craft a modified curl command and run it locally to test your hypothesis. Something like this, maybe?

curl -O "https://github.com/paritytech/polkadot/releases/download/v0.9.17/polkadot"

Yeah, that should do the trick.

The error message?

Not Found

Do you feel that wave of relief, breaking into your being, filling you with what feels like one of the holiest states of existence? Even better, add joy to it, since you discovered something that you now can share with your colleague. Even better! Add laughter, since you both independently discovered the same thing at the same time! Being an engineer makes you feel alive, no matter if the emotions are positive or negative. Let us cherish that, shall we?

Great! Mystery solved! No time to take a beat and celebrate this small victory. Remember, you are in a hurry.

So, how do you solve this problem? Do you spend several hours testing an improved Dockerfile for this particular edge case? No, you hardcode the desired version for this run, rebuild the container image, and go about your merry way.

But! You do not forget! If this happens again, you might consider improving the Dockerfile (and all other Dockerfiles).

And this story will be the memory, so that we can never forget.