Magical thinking in Software Development


I guess we all Python developers heard this kind of argument from time to time:

Python is slower than C++/Java/C# because is not compiled.

Other than the usual “blame the others” when working with other companies (usually big corporations than thinks than using anything except C# or Java is laughable), you can also see a lot of comments in technical blogs or places like Hacker News or Reddit with similar, simplistic arguments. You can recognise them on the usual rants about how technology X is The Worst Thing That Ever Happened™ and Should Never Be Used™

That’s a form of Software Development Magical Thinking. This can be really harmful for software development, specially when the opposite, positive form is used. Let me define Software Development Magical Thinking in this context:

Software Development Magical Thinking noun Assuming that a technology will magically avoid a complex problem just by itself.

Probably that will become clearer after a couple of examples:

Java is a static type language and it is safer than dynamic type languages like Ruby.

We program in C++ so our code is very fast.

MongoDB / NodeJS / Riak is web-scale.

Please note that those are not completely, utterly wrong statements. C++ can be very fast. Static typed languages can avoid some bugs related with input parameters type. But there is no guarantee that creating a system in C++ is going to act like a magic wand against slow code. Or that Erlang will avoid having a single point of failure. And you’ll get as sick of bugs and security issues both on static type language and dynamic type languages. *

Those are all complex problems that need careful design and possibly measurements to deal with them. Deep analysis of the problem, which usually is more complicated that looks on the first place. Or even worst, the problem is not as bad as it looked and the designed system is more complex that it should, trying to catch a problem that never arises. Not to exclude having previous experience to avoid subtle errors.

Let me say it again. There are problems that are HARD. In software systems they are confronted almost daily. And no single thing will make you forget them. Even if you use a very good tool for what you’re doing (like Erlang for concurrency), which usually implies paying a price (in development time, etc), doesn’t replace vigilance and issues could eventually appear. Unfortunately, making software is tough.

The problem with Software Development Magical Thinking is that it is very easy and it is also very natural. Seductive. We know that “general Magical Thinking”, simple solutions to very complex problems, is quite common. Hey, a lot of times, it even seems to work, because the Feared Problem will only present after certain size that is never attained, or after the designer leave the company and left a latent problem behind. Most of the time, making a totally informed decision is unrealistic, or simply not possible, and some risks must be taken.

But as software developers we should know that things are not that easy, even if we have to compromise. Each bug that takes time methodically eliminating causes. Every measurement that makes you wonder what is the best metric to reflect a value. Every time you realise that there was a back-of-the-envelope calculation that shows something that will have an impact on some design aspects. Those are all reminders that should makes us think that there are no silver bullets and we shouldn’t take lightly all those difficult problems.

Make decisions. Design systems. Choose a tool over others. Take risks. But don’t be delusional and careless. Be conscious that software can bite you back. Be vigilant. Be skeptic. Avoid Magical Thinking.

PD: And please, don’t say “Python is slow”. Just don’t. It is not for most of the jobs. It is not going to make you win a discussion unless you carefully measure and proof it. And, perhaps most importantly, raises my urge to kill.

* No, I am not going to comment anything the Mythical Web Scale property.

EDIT: Wow, it has been submitted to Hacker News here. Just in case any one whats to add to the discussion there.