3.1 Markov Decision Process

The Markov Decision Process (MDP) is an ideal mathematical form that is able to elaborately express reinforcement learning problems theoretically ^[1]. In MDPs, learners are called agents, and everything other than agents that interact with the learner is classed as the environment. The agent and the environment interact in every successive discrete time step. At every time step, the agent receives a representation of the state of the environment $s_{t}\in S$, where $S$ is the state space. The agent then selects an action $a_{t}\in A$ based on $s_{t}$, where $A~ $ is the action space.

In the next time step the agent receives a reward function $r_{t+1}$ and the next state of the environment $s_{t+1}\in S$. At this time, the learner's goal is to maximize the gain $G_{t}$, which is the sum of the attenuated rewards:

where $\gamma \in \left[0,1\right]$ is the discount rate, which gives the present value of future compensation. The action value function evaluates the agent's action in a given state:

where $\pi $ is called the policy and the probability of choosing each possible action from that state. There is one policy that is better than or as good as all other policies, which is called the optimal policy $\pi _{*}$ and satisfies the following:

3.2 DQN

One of the main factors in the success of DQN is experience replay. First, the agent's experience $e_{t}\doteq \left(s_{t},a_{t},r_{t+1},s_{t+1}\right)$ is stored in the experience replay memory $D=\left\{e_{1},e_{2},\cdots ,e_{N}\right\}$ at every time step, where N gives the size of the experience replay memory. At a later point, during learning, experiences $e\sim D$ are uniformly randomly sampled from the memory, the number of experiences sampled is a predetermined parameter, and these experiences form a mini-batch that is entered as an input of the agent.

Due to experiential replay, the correlations between input data, which cause instability when reinforcement learning uses neural networks, are significantly reduced ^[7]. A second factor in the success of DQN is the use of a separate target network. The target network has the same structure as the network in which learning happens but has independent $\theta ^{-}$ parameters, which are used to generate the target value $y$ used during learning:

where $s'$ is the next state stored by $e=\left(s,a,r,s'\right)\sim D$, $a'$ is the action selected by the target network at $s'$, $i$ indicates how many times the update of the learning network parameters is repeated, and $\gamma $ is the discount rate. Due to the use of a target network, the parameters of the learning network in the i-th iteration are updated to minimize the loss function $L_{i}\left(\theta _{i}\right)$

Fig. 1. Robot manipulator and work environment.

3.3 Double DQN

Double DQN is a deep reinforcement learning algorithm that combines DQN with Double Q-learning. It is intended to reduce overestimation of Q-learning's action value function. This approach estimates the action value function more accurately and shows better performance than DQN in some Atari games ^[4].

This action value function overestimation is a problem caused by the maximization operation for the target value that is carried out while updating the current action value function. This problem causes significant deviations in the estimated action value function. Double DQN uses a target value $y$:

As shown in (6), the process of selecting an action and the process of evaluating the action are distributed to the learning and target networks to prevent overestimation of the action value function.

3.4 Dueling DQN

Dueling DQN uses a dueling architecture that has two streams for the state value function and advantage function. It combines these to estimate the action value function ^[5]. The new action value function $q$ is then:

The dueling architecture has the following advantages. First, as the action value function is updated, the state value function is also updated. Therefore, other action value functions that have not been selected by the agent may also be updated frequently. Second, when updating the action value function, it is less affected by the difference in size between the target value and the current action value function, making the learning process more robust.

3.5 Prioritized Experience Replay

The prioritized experience replay method prioritizes and samples the data in the experience replay memory that will be used for learning according to the magnitude of the temporal difference error. This makes learning more efficient and effective compared to random uniform sampling ^[6]. The priority of the samples to be stored in the experience replay memory is based on the magnitude of their temporal difference error (that is, how "surprising" the sample is). However, if only samples with high priority are greedily selected, and samples with low temporal difference errors are never selected, only a subset of the experience replay memory will ever be used. To solve this problem, stochastic sampling is used. The probability that the i-th sample in the memory is selected, $P\left(i\right)$, is given by:

where $p_{i}$ is the priority of the i-th sample, and the constant $\alpha ~ $determines how much effect priority has on a sample’s chance of being selected.

4.1 Simulation Environment

The robot manipulator and environment used in our simulation were created using the realistic robot simulator Webots ^[12]. Webots is a professional robot simulation software package that provides a fast prototyping environment that allows users to create 3D virtual worlds by defining physical properties such as the mass, type of joint, coefficient of friction, and so on. In addition, various sensors and actuator devices such as distance sensors, drive wheels, cameras, motors, and touch sensors can be modeled. Webots is a more realistic simulator than other simulation tools available. Webots not only allows you to simulate the physical properties of the environment, but also provides realistic photo-like camera images that reduce the gap between the real world and the simulated environment. Due to these advantages, Webots is a great simulation tool for reinforcement learning researchers. In addition, a useful framework for developers called Deepbots has also been created for this software ^[13].

Fig. 1 shows the robot manipulator, which is a three-axis robot that has two linear motors to operate the gripper. In the work environment, both objects to be moved and obstacles are on a table. Our goal is to bring the target object to the target point while avoiding obstacles in the shortest time. A camera to monitor the work environment hangs from the ceiling.

4.2 Proposed Method

The camera in Webots supports object detection, recognizes target objects, and allows us to track their positions using images. Based on information obtained from images captured by the camera, a new representation is created, as shown in Fig. 2. Fig. 2 shows the position of the target object, the position of the obstacle, and the destination expressed as a grid world composed of square boxes. The grid world has dimensions of 12x12 and is created as an 84x84 image. This grid world image becomes the input image to the training network of the deep reinforcement learning algorithm, which will search for the optimal path to complete the pick and place tasks.

As shown in Fig. 3, the robot simulator extracts object information from the input image of the camera. Then, a new grid world image is created and delivered to the deep reinforcement learning agent. After receiving the image, the agent selects the next action and transmits it to the robot simulator. The robot simulator calculates the position where the end effector should move based on that action and then moves each joint through inverse kinematics. Next, a new image is then recorded by the camera after the robot has moved, and the information obtained from it is used to create the new grid world image. This process repeats from the moment the object is picked up to the moment it is put down on the target point.

Fig. 2. How work environment is represented by grid world.

Fig. 3. System architecture.

Fig. 4. Training results for simple environment.

In reinforcement learning, there are various episodes, and learning takes place from the experiences that are created within them. It takes a considerable amount of time to acquire sufficient experience for a robot to learn a skill well while in the simulation tool. To tackle this problem, we created training data using OpenAI Gym frame, a toolkit for developing and comparing reinforcement learning algorithms ^[14]. OpenAI Gym frame allows you to test your own reinforcement learning algorithms using various internal game environments. Furthermore, if you use OpenAI Gym frame, you can create your own game environment where your algorithms can learn how to complete tasks. Since we represented the work environment using a grid world image, we were able to create virtual training data and significantly cut the time required for training.

5.1 Training of Deep Reinforcement Learning Algorithms

We let the deep reinforcement learning algorithm learn two environments. In the first environment, the starting point and destination are randomly generated, and there are no obstacles in the way. The scalar reward function for this is as follows.

The robot has a maximum of 500 time steps per episode, and the worst total reward per episode is -50. The goal of the robot is to reach the destination in the shortest time. The training was conducted over a total of 50,000 episodes, where two deep reinforcement learning algorithms were trained: DQN and D3QN with PER. The results achieved by these two algorithms in this task are shown in Fig. 4.

The vertical axis represents the total reward per episode, and the horizontal axis represents the number of episodes. For each episode, the total reward from the most recent 1000 episodes was averaged and used as the output. In a simple environment with no obstacles, both algorithms successfully learned how to compete the task well. By episode 50,000, the average reward was about -0.65, which means that the destination was reached in 6 to 7 time steps on average. The overall average and highest average rewards of D3QN with PER were slightly higher than those of DQN.

The second environment is a more complex and difficult environment than the first one. In the second environment, three obstacles appear stochastically with random positions and sizes. The goal of the robot is to pick up the target and move it to the destination in the shortest time possible (measured in steps) while avoiding obstacles. If the robot collides with an obstacle, it returns to its original position. This is more difficult than the first environment because the agent needs to know where the obstacles are in the work environment and to find a path to avoid them. The reward function and the learning process are the same as in the first environment. The results achieved by the two algorithms in this task are shown in Fig. 5.

Fig. 5. Training results for complex environment: (a) Comparison of training results of DQN and D3QN with PER during 10,000 episodes; (b) Training result of D3QN with PER during 50,000 episodes.

Fig. 6. Optimal path when there is no obstacle in the workspace.

In Fig. 5, the difference between the two algorithms can be seen clearly. As the learning progressed, DQN was not able to find the destination and could only avoid obstacles. Eventually, we decided that DQN would not be able to complete the task and ended training in the 20,000th episode. In contrast, D3QN with PER successfully learned how to complete the task. It can be seen that the average reward is about -1.2 by the 50,000th episode, which indicates that the robot was able to reach the destination by moving 12 times on average. To teach the 3-axis robot manipulator, the weights found by the deep reinforcement learning algorithm model in Gym Frame were used. Table 1 shows the hyperparameters for Double Dueling DQN with PER used for training.

5.2 Optimal Path Search

Figs. 6-8 show the optimal path search results of the 3-axis robot manipulator that was taught using the stored weights of the trained model. These were passed to the deep reinforcement learning algorithm model, which was then able to interact with the working environment implemented in the simulation. The first test was to ensure that the optimal path could be found when there were no obstacles. Fig. 6 shows that the robot manipulator is able to find the fastest straight path when it picks up and moves the object to the target.

Fig. 7. Optimal path when there are static obstacles in the workspace.

Fig. 8. Optimal path when there are dynamic obstacles in the workspace.

The second test was to check that our model could find the optimal path when there were static obstacles. There are two obstacles in the work environment. The tests confirmed that the robot manipulator is able to avoid obstacles and reach the destination. In Fig. 7, the robot follows the fastest path while avoiding collisions with the obstacles.

The final test was to ensure that our model could find the optimal path when there were moving obstacles. One obstacle moves and blocks the simple path to the target while the robot manipulator performs a pick and place operation. As shown in Fig. 8, the robot manipulator moved along the path with the greatest reward while making corrections to the optimal path in real time to avoid the moving obstacle.