The One with reinforcement learning

“ Hello guys it’s Estelle ! ” , Oh … My … God …, I need to stop … please help :(

I know it’s been a while since my last article, so let me make up for it with this one. I assume that you are a little bit familiar with machine learning stuff, if not let me know so I can make an article about that, but for now, let’s talk about reinforcement learning.

First let’s present what are the model of the reinforcement learning approach :

Source : A Brief Survey of Deep Reinforcement Learning, 2017

We have an agent ( or multiple ) in a state who choses an action to do from a set of actions based on a certain policy ( could be fixed like taking always the action that maximizes the next reward ), this action is going to change ( or not ) the environment, and the agent receives a reward signal ( the values of the reward could be continuous/discrete, positive/negative, it depends on the case where you are using RL ), and the agent changes its state to a new state. That’s basically the main idea of reinforcement learning, now let’s see where reinforcement learning is located among other fields :

Source : David Silver’s Reinforcement learning course, Introduction to RL

A little scary right ? How can you study a thing that needs knowledge in all these fields ? Well it’s all right, because you do not have to be an expert to understand reinforcement learning, but the first step is to be curious and try to understand. So, how’s RL different from other machine learning paradigms such as supervised learning ? 

1. The first difference is that there’s no supervisor : the agent judges the choices he’s making based only on a reward signal ( also called trial-and-error paradigm )
2. The feedback is delayed, not instantaneous, which means that the impact of a choice that the agent make could be after so many steps and decide whether it was or not a good move.
3. Time really matters, in other words, it’s a sequential process of decision making which makes it a dynamic system where data is not i.i.d. ( independent and identically distributed ) like in supervised or unsupervised learning.
4. Agent’s actions affect the subsequent data it receives, imagine a red light, where to cars are waiting, when the green light comes, they could pick the same road, which leads us to the same data distribution in the case of RL, but they can choose different paths, which means observing different things, and receiving different rewards in RL.

We can compare RL vs SL like this :

Source : CIFAR Reinforcement Learning Summer School (RLSS) 2017, Montréal

I don’t want to fill you with a lot of information, so that’s all for this article. I will leave some examples of what’s done with reinforcement learning :

 

Google’s AI AlphaGo Is Beating Humanity At Its Own Games

 

DeepMind AlphaGo Zero Explained

 

DeepMind Reinforcement Learning

If you have any questions or comments, please feel free to let me know, thank you !

PS : you can find my last article here

The "local search" problem - part 1

This week, since we are 2 working on this project, we had to start thinking in a deep and formal way to optimize the already existing metaheuristic ... Oh I forgot, I didn't explain how we got here, so I'm
going to start by first introducing our project.

Supervised classification is a very important task in data mining and a part of the machine learning techniques, it is affecting objects into groups that have the same characteristics based on a set of features or attributes. To simplify things, imagine your data as a table that has a number of columns same as the number of features, let’s call it X, and in addition, you have a number of lines equals the number of your clients, called Y ( when talking about a database in a company for example ), now you’d have a table with a size of ( X x Y ) element at a moment t1.

Now imagine you train your build a model based on this table ( X, Y , t1 ) to try and predict certain future data, but you have a new client who just got into your database, so you need to update your model, now you build a new model based on ( X , Y+1 , t2 ), but hey, 4 more clients came in just moments after you build your model, it becomes (X,Y+5, t3), and while your database grows, the cost of training a new model each time becomes higher after each new model, you should know that the cost to calculate the determinant of a 25x25 matrix is too high, even for a computer, then imagine a 1 00 x 1 000 000 or even more, and this is just one time, and that's what we call the curse of dimensionality.

So people who do research said, since we have no control over new instances of data ( the lines ), let's try and reduce the number of attributes ( the columns ), but the problem with this approach is that ...

Well that would be something for the next article ( a way for me to commit to writing this time since I have that OCD for finishing things I start ).

PS : I wanted to say that, anyone should be proud of what he is/do now, because it is a part of what you will be tomorrow, and for me, this kind of stuff is my mindset now, and after some years, even I won't have the same ideas, that doesn't contradict the fact that I was this way at a certain moment, and it will always be a part of who I am.

Why a "blog" & why now ?

If you ask me why I'm doing this, I'd be like :

The blog concept is now "deprecated", so why would I try to have mine now ?
You can say that I am an old school guy, who's only objective is to change the world bla bla bla ...
Nah more seriously, when you write in your personal blog, it's like writing a public diary where you have no pressure, and you feel that you can write whatever you want, and that's what I will be trying to do.
The main purpose of this blog is an initiative to share the knowledge I acquired during my 5 years as a computer engineering student at Ecole nationale Supérieure d'Informatique - ESI, Algiers, and will be acquired in the future. I felt the need to do this because I will graduate this year and I am working on using reinforcement learning to boost a metaheuristic called Bee Swarm optimization, for the feature selection problem, and to be honest, I've spend some time to figure things out ( when I wrote this, I had 2 months and a half working on the project ), because when you read articles, you have so many points of view and each researcher or group of researchers has his own perspective, which means that, your contribution could be to do the abstraction of the studies, and make a sort of survey ( a concept that exists, but still, you can't understand the fundamentals just by reading surveys, you need to do you own ).

I actually tried to open my personal blog once, wrote 2-3 articles then stopped, I hope this time it will be different, because I think that this time, I have a solid content to share.

And now ...