Current risks of AI – Part 1

AI is here to stay. It has been here for a long time. I have, personally, been using it since 2018. It can bring short and long term gains. But it is also not without its risks and we must be aware of them to safely and responsibly use them.

This is Part 0 of a, as of now undetermined, number of future articles about the current risks of AI. In this Part I wanted to start off by talking about the Risk that is most often brought up: AI is going to become so smart it will take over everything.

RSI and the Terminator

There is a lot of talk about the potential for AI to "go rogue" or become so good it will replace or otherwise make humans redundant. This is usually coupled with a fear of AI endangering life at a massive scale such as what we see in the Terminator Movies.

These fears tend to sit atop the premise that, at some point in the future, AI will start "speeding up" and we will lose control of it. Or it will become so smart that not even the smartest humans can keep it under control. The more "technically accepted" version of this happens through something called Recursive Self Improvement (RSI).

The idea of RSI is actually quite simple. Take a smart AI (Billy-1), ask it to build an even smarter AI (Billy-2) that is maybe just a little bit smarter. This is called self-improvement, since Billy is building a newer better version of itself. Repeat this process ad infinitum, at some point Billy-9000 will be so smart not even the AI experts can keep up. That is what also makes it Recursive: Billy-1 builds Billy-2, then Billy-2 builds Billy-3 and on and on continuously. Thus, Recursive Self Improvement.

Who knows, maybe Billy-1 makes Billy-2 not just a little bit smarter but actually 20% smarter and every new Billy keeps that up. For the mathematically inclined among you this roughly means that by the time we reach Billy-9000 it will be 4.28 × 10⁷¹² times smarter than Billy-1. That is a really large number which goes above what any intuition we have can handle, it is essentially a 4 with 712 zeroes after it. For reference the US National Debt is around 40 Trillion Dollars (at the time of writing), so a 4 with 13 zeroes after it.

Even if we say the rate of improvement is 2% at each iteration, Billy-9000 will still be around 2.52 × 10⁷⁷ times smarter than Billy 1. Boom! Singularity Time!

We don't even know how fast it can go. A model that is that smart, such as Billy-5 or Billy-200 or Billy-1000 can probably come up with a way to train the following one even faster right? So even if Billy-1 → Billy-2 is a 2 year affair, maybe Billy-4000 → Billy-4001 takes seconds or even less. If that is the case then Billy 4000 → 9000 can happen in a matter of hours.

If you have used Chat-GPT, Claude, Gemini or any of the other models you have experienced their limitations. But the thing about numbers this big is that, even if the starting point is a model that is not that smart, the end product is something insanely smart. Who can even control a system that smart?

Scary right?

Well, yes. But, things aren't that simple.

Pouring some water on the fire

For starters, RSI is not a proven concept. It is not happening. It is not something that we can even prove will happen. Is it possible that a future AI model will one day learn how to improve itself? Yes, but we are cannot be sure. It is also possible that RSI will always remain just over the horizon no matter how "smart" the starting point is.

Another problematic point is determining how "smart" a model is. Intelligence is not just one single value like height or weight. We don't really have a good way of measuring how "smart" a person is because we also don't have a good working definition of what makes a person smarter than another.

You can probably think of at least one area where you are much smarter than your parents, your partner or a close friend. You can also probably think about several areas where these people are much better at you. Who is "smarter" then? You because you can code a really cool Python tool or your friend because he can do most of his own car maintenance?

If you then use the same tests we use to measure intelligence in humans, and have AI test on them you can run into these problems as well as new ones. You can IQ test Billy-1 yes. But if Billy-1 is an LLM trained on all of humanity's collective knowledge, then isn't the IQ test itself inside the training dataset? If Billy-1 scores an IQ of 80 and Billy-2 scores an IQ of 90, is Billy-2 smarter or just better at remembering the training for that specific data point of IQ tests?¹

I would also invite you to read Goodhart's work on metrics, when it comes to the problems that can be caused when you focus too much on optimizing for any given measure (or set of measures). As he himself put it: "When a measure becomes a target, it ceases to be a good measure." Thus, if you focus on making every next model "smarter" as measured by some specific benchmark² perhaps you end up with a model that is just better at that specific benchmark.

But, for the sake of argument, let us assume we have a perfect way of measuring this and that we do determine the models are, across the board, becoming better. We can see Billy-1 building Billy-2 and so on and when we test them they are so much better. That still doesn't tell us if we can reach Billy-9000. What if there is a ceiling on how smart a model can be? That is also not a solved question.

RSI is not proven, we don't really have an accurate and effective way of measuring how much better the models are becoming and there is no guarantee that a ceiling doesn't exist.

What the following Parts will (and will not) be

Can RSI happen? Maybe.

Will RSI happen? I honestly do not know.

What I do know is that there is, to my knowledge, no sign that RSI is currently happening. What I also know is that there are risks with using AI that need to be addressed right now. Because it is not just RSI that can cause serious problems. That is what this series of articles will be about.

AI is here, AI is a powerful tool but we need to be aware of the problems it can cause. If we focus solely on RSI and Terminator-type scenarios (which are, of course, scary) we are forgetting the much less exciting and yet more insidious issues that are already here.

We need to be watchful of what we are building in terms of AI. Security and Alignment research is important. But we also have other problems to fix. The average business cannot solve RSI, but it can maybe mitigate these other problems and risks.

The exact list of follow-up topics is a work in progress and subject to change according to developments in the field, feedback and shifting priorities from Axveco. But, tentatively, we will be talking about the following:

• Part 1: The loss of Junior developers/engineers
• Part 2: Cognitive debt and Cognitive Offloading
• Part 3: Vendor Lock-in
• Part 4: Economical Viability

1. (I am using IQ tests here as an example. They have their own problems which I will not get into here but which I invite you to look up.) ↩
2. A benchmark is a standard that you can use to compare the performance of your system against that of others. A good example of benchmarking would be: "My AI system can detect 20% more cats than yours, I benchmarked it against the COCO database which contains images of cats and dogs (among other things) annotated in a standardized way.". ↩