The state of programming

Recently, the company I worked for was acquired by another company. Suddenly, I had to face the unpleasant necessity to sell myself on the job market again. Last time I did it five years ago and, since then, the programming landscape has changed. The most advanced and growing companies now talk about machine learning and big data.

Luckily for me, I did some of it last few years. We built models that predicted user behavior based on historical data. The basic approach is pretty straightforward. You collect user data (say, each user is presented by 30 parameters, like name, age, address, education level, etc) and you mark each user by a flag, whether s/he did the action of interest: bought certain product (true/false), clicked certain advertisement (true/false), spent more than 5 seconds reading the advertisement (true/false), and similar. Then you feed these data into a machine learning algorithm and generate a model, which gives you a score for each user (the probability that the user will do this or that action).

Then you test the model on another set of historical data, tune the model if needed and then use it to predict new user behavior. I was surprised to see how much the sales revenue grows with such a simple technique.

That is what encompasses basic machine learning functionality. There is a lot of mathematics around it. And you have to understand your data too: do some data depend on each other? how reliable are the data? which combination of data identify a user uniquely? are there duplicates? which algorithms better for the kind of prediction you need? And you have to know the limitations and parameters of the algorithms.

Then pure programming challenges come in: how to process a big amount of data quickly enough? You might not be able to get away with one computer, even a very powerful one. And such companies like Google and Amazon use many thousands of computers to do that.

The adjective “big” has spread around like a ground fire during a dry season in the prairie. The last version of Java includes the language constructs that support parallel computing and asynchronous processing. New — non-blocking — servers were created that took on themselves multi-threading, but you, as a programmer, have to structure your data and algorithm in a way that they can be used in parallel. “Reactive”, “dynamic”, and “functional” become standard adjectives of the word “programming”.

The traditional sequential programming becomes a history. Now the huge amount of data are processed in parallel by a huge number of independent agents (microservices). The results are picked up by a swarm of other agents (microservices) and so on until the final result is reached.

The most exciting aspect of this new brave world for me was the discovery that my human intuition could not compete with machine learning algorithms. I used to think that computers just can do faster the job a human can do, and that programmer can predict the result in principle, if not with high precision. What I find more and more is that my intuition does not give me even a hint in the direction that later is identified as the most fruitful by an algorithm. I even do not talk here about deep learning, when a computer decides which data parameters to use and which not. Even the basic machine learning algorithm can find trends in the data you never even imagined existed.

My friends, I admit my total surrender to the power of artificial intelligence. It is not even a question of my goodwill. It is just simple and straightforward fact that faces everybody who works in this area. The race is over. Humans can rule or have an illusion they do, but we cannot compete with machines anymore. So far, we do not allow machines limited freedom in acting based on the results of the data analysis, but let us see what will happen in just a few years from now.

Then I thought about the latest trends in physics. The science used to be about the nature of things around us. Physicists talked about the structure of the material world, about forces, causes and effects. Then we were forced to admit that we did not know what was actually “out there.” All we can do is to predict the outcome, using the formulas (called “laws”). Given the input, we can compute the results, but we do not know why everything happens the way it happens. Even when we use the notion of “God”, we still cannot explain why s/he decided to do it this or that way. Poor schoolchildren! I remember how we struggled to explain physical processes, while all we actually were required to do was to memorize words and sentences that were approved in lieu of explanation.

In the course of the last decade this understanding, once a heretical and dissenting, became public knowledge. Many physicists accepted the notion of a “computational universe”: physics describes the behavior of the world around us not as the system of “cause-effect”, but as an interplay of various states, whose nature remains unknown and unknowable to us on principle.

Computing becomes not just a tool but the nature of the world around and inside us. Well, it might be the result of the human tendency to explain things by analogy. We think we know the latest technology and have an impression that it can be the key to every lock. Just barely 100 years ago, we thought we understood the universe by comparing it to the mechanical clockwork, remember?

--

--

--

Born in Moscow, lived in Crimea, now lives in the US. Used to be physicist and rock climber, now programmer and writer.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Meet Anja-Your 24/7 social media manager.

Kardome Joins Intel® Partner Alliance to Deliver Breakthrough Voice Technology Solutions

3 Reasons Why Marketers Are Getting Excited About AI

Bias in algorithms

IoB XAI and Behavioural Data

The Effect DAO: The Guardians of the Galaxy Pool

AI Economist sums up Communism

The Sequence Scope: Amazon Big ML Week

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Nick Samoylov

Nick Samoylov

Born in Moscow, lived in Crimea, now lives in the US. Used to be physicist and rock climber, now programmer and writer.

More from Medium

Does your autonomous vehicle know when to stop?

AlphaFold: A simple explainer — Part 2

Some far-fetched thoughts on “inverting” algorithms

Things That Make You Love And Hate Quantum Computer.