Performance Improvement of a Virtual Network Embedding Algorithm based on Temporal-difference
Learning by Resource-Constraint-Aware Candidate Solution Selection
FukushimaYukinobu1
SagawaYuta2
TarutaniYuya3
YokohiraTokumi3
-
(Faculty of Natural Science and Technology, Okayama University / Okayama city, Okayama,
Japan
fukusima@okayama-u.ac.jp
)
-
(Graduate school of Natural Science and Technology, Okayama University / Okayama City,
Okayama, Japan)
-
(Faculty of Interdisciplinary Science and Engineering in Health Systems, Okayama University
/ Okayama city, Okayama, Japan {y-tarutn, yokohira}@okayama-u.ac.jp )
Copyright © The Institute of Electronics and Information Engineers(IEIE)
Keywords
Network virtualization, Virtual network embedding, VNE-TD, Resource constraints
1. Introduction
The development of virtualization technology has increased the interest in network
virtualization. In network virtualization, multiple virtual networks can be constructed
and operated on a physical network by using node resources (e.g., CPU, memory, and
storage) and link resources (e.g., bandwidth) [1,2]. Network virtualization allows users to use virtual networks on demand in a pay-as-you-go
manner, realizing NaaS (Network as a Service). In addition, network virtualization
enables efficient use of physical network resources and early deployment of new network
services.
In network virtualization, it is essential to determine to which physical node and
physical route in a physical network every virtual node and virtual link in a virtual
network should be mapped in order to maximize the revenue of a virtual network provider,
who constructs and provides virtual networks. In determining the mapping, resource
constraints must be satisfied. The resource constraints mean that (1) each virtual
node can only be mapped to a physical node with sufficient remaining node resources
(node resource constraint), and (2) each virtual link can only be mapped to a physical
route with sufficient remaining link resources on all the physical links along the
route (link resource constraint). This problem is called the virtual network embedding
(VNE) problem [3-12].
For this problem, a VNE algorithm (VNE-TD) based on temporal difference (TD) learning
[13], which is a type of reinforcement learning, has been proposed [12]. VNE-TD defines the remaining amounts of node/link resources in the physical network
as the state of the physical network and introduces a state value function for estimating
the value (i.e., the expected total revenue obtained by the virtual network provider
in the future) of the state of the physical network. The procedure for constructing
a virtual network in VNE-TD is as follows. First, VNE-TD solves a node mapping problem,
where we determine to which physical node each virtual node is mapped, using a conventional
VNE algorithm (e.g., GRC-VNE [11]), and generates multiple candidate solutions. Then, from the candidate solutions,
VNE-TD selects the solution that leads to the state of the physical network with the
highest value after mapping the virtual network.
VNE-TD, however, does not consider whether the candidate solutions to the node mapping
problem satisfy the resource constraints. Therefore, when constructing a virtual network
based on the candidate solutions, embedding the virtual network may fail due to insufficient
resources, resulting in a decrease in revenue for the virtual network provider.
In this paper, we modify VNE-TD to select only those candidate solutions that satisfy
the resource constraints. We add a function to check the satisfiability of the node
and link resource constraints to VNE-TD. This function eliminates physical nodes and
links that do not have sufficient resources for mapping virtual nodes and links when
generating candidate solutions to the node mapping problem, so that these physical
nodes and links are not included in candidate solutions. The key contributions of
this paper are summarized as follows.
1. We modify VNE-TD so that it has a function to check the satisfiability of the node
resource constraint and the link resource constraint.
2. Simulation evaluations are conducted to evaluate the performance of the modified
VNE-TD. The simulation results show that the performance is improved the most when
both the node and link resources constraints are considered in VNE-TD.
The remainder of this paper is organized as follows. In Section 2, we introduce related
work on virtual network embedding. In Section 3, we describe the service model, network
model, and revenue model of a virtual network provider assumed in this paper, followed
by an explanation of the virtual network embedding problem addressed in this paper.
In Section 4, we describe VNE-TD and its problem, followed by an explanation of our
modified methods. In Section 5, we evaluate the performances of our modified methods
through simulation evaluations. Finally, in Section 6, we conclude this paper.
2. Related Work
Many approaches were proposed to solve VNE problems [3-12]. Refs. [4-6] formulated VNE problems as integer linear programming problems (ILPs). Ref. [4] tackled an ILP to minimize the embedding cost and to maximize the acceptance ratio
of VNRs. Refs. [5,6] studied an ILP to minimize the power consumption of the physical network. On the
other hand, since ILPs are NP-hard, it is difficult to find the exact solutions of
the ILPs.
In Refs. [7-12], heuristic algorithms were proposed to find solutions in shorter computation time.
The existing algorithms are classified into offline algorithms [7-9] and online algorithms [10-12].
The offline algorithms [7-9] solve a VNE problem, assuming that all virtual network requests are known beforehand.
Ref. [7] proposed a distributed VNE algorithm and a virtual network mapping protocol. Ref.
[8] proposed a heuristic VNE algorithm based on a max-min ant colony metaheuristic. Ref.
[9] proposed DPVNE, a distributed, parallel, and generic VNE framework. On the other
hand, knowing all the requests for virtual networks beforehand is impractical.
The online algorithms [10-12] solve a VNE problem for each virtual network request in an online manner without
prior knowledge about future requests for virtual networks. Ref. [10] proposed a heuristic VNE algorithm whose goal is to maximize the financial profit
of the virtual network provider. The algorithm first solves the node mapping problem
in a greedy manner by assigning the virtual nodes with larger node resource requests
to the physical nodes with larger remaining node resources. Then, the algorithm solves
the link mapping problem by assigning each virtual link to multiple physical routes
(i.e., multipath routing). However, this VNE algorithm is not applicable to physical
networks adopting single path routing protocols. Ref. [11] proposed a heuristic VNE algorithm called GRC-VNE. The algorithm adopts a metric
called GRC (Global Resource Capacity) to quantify the embedding potential of every
node. The algorithm first solves the node mapping problem in a greedy manner by assigning
the virtual nodes with larger GRC values to the physical nodes with larger GRC values.
The algorithm then solves the link mapping problem by using the shortest-path algorithm.
Ref. [12] proposed a heuristic VNE algorithm (VNE-TD) based on temporal difference learning.
VNE-TD generates multiple candidate solutions and selects the one with the highest
value estimated using a state value function. The simulation results show that VNE-TD
outperforms GRC-VNE. VNE-TD, however, does not consider the resource constraints when
generating the multiple candidate solutions. Consequently, VNE-TD can cause VNE failures
because of the lack of resources.
3. Virtual Network Embedding
3.1 Service Model
Network virtualization provides users with virtual networks embedded in a physical
network. The users of virtual networks are assumed to be providers of network services
that require high communication quality, such as live video streaming and online games.
The service flow in network virtualization is shown below.
A user requests a virtual network provider to construct a virtual network. This request
is called a virtual network request (VNR).
The virtual network provider attempts to construct the requested virtual network on
the physical network. If it succeeds, the virtual network provider delivers the constructed
virtual network to the user and receives a network usage charge from the user. If
it fails, the virtual network provider informs the user.
A virtual network provider receives VNRs dynamically one after another. Therefore,
the virtual network provider needs to construct and provide a virtual network in response
to each VNR in an online manner.
3.2 Network Model
The lower left of Fig. 1 presents an example of a physical network. Each vertex of the physical network represents
a physical node, and each edge represents a physical link. The numbers on the side
of a physical node and a physical link represent the amount of remaining node resources
(e.g., CPU, memory, and storage) and the amount of remaining link resources (e.g.,
bandwidth), respectively. For example, physical node B has 20 remaining node resources,
which can be provided to virtual nodes, and physical link (A, B) has 60 remaining
link resources, which can be provided to virtual links.
The upper left of Fig. 1 presents an example of a virtual network. Each vertex of the virtual network represents
a virtual node, and each edge represents a virtual link. The numbers on the side of
a virtual node and a virtual link represent the amount of requested node resources
and the amount of requested link resources, respectively. For example, virtual node
b requires five node resources on the physical node to which virtual node b is mapped,
and virtual link (a, b) requires 20 link resources on the physical route to which
virtual link (a, b) is mapped.
An example of mapping a virtual network onto a physical network is shown on the right
side of Fig. 1. In this example, virtual nodes a, b and c are mapped to physical nodes A, B and
D, respectively. The virtual link connecting virtual node a to virtual node b is mapped
to a physical route that passes through physical link (A, B), the virtual link connecting
virtual node a to virtual node c is mapped to a physical route that passes through
physical link (A, D), and the virtual link connecting virtual node b to virtual node
c is mapped to a physical route that passes through physical links (B, C) and (C,
D).
3.3 Revenue Model
Similar to Ref. [11], the objective of the virtual network embedding in this paper is to maximize the
long-term time-average revenue of the virtual network provider (Eq. (1)).
where $K_{T}=\{k\,| \,0< t_{k}< T\}$ denotes the set of identifiers of virtual network
requests that arrived by time T, $t_{k}$ denotes the arrival time of the $k$th arriving
virtual network request (VNR $k$), and $Rvn\left(k\right)$ denotes the revenue obtained
by embedding VNR k. $Rvn\left(k\right)$ is calculated as the weighted sum of the amount
of node and link resources used by VNR $k$ (Eq. (2))
where $V^{k}$ is the set of virtual nodes in VNR $k$, $E^{k}$ is the set of virtual
links in VNR $k$, $NR\left(v\right)$ is the amount of node resources requested by
virtual node $v$, $LR\left(e\right)$ is the amount of link resources requested by
virtual link $e$, $\eta $ is the unit price of node resource, and $\beta $ is the
unit price of the link resource.
3.4 Virtual Network Embedding Problem
The virtual network embedding problem tackled in this paper is described as follows.
● Input:
A newly arriving VNR
● Output:
Whether the VNR can be embedded in the physical network. (If the VNR can be embedded,
it is provided to the user.)
● Objective function:
To maximize the total revenue of the virtual network provider (the numerator of Eq.
(1))
● Constraints:
1. Node resource constraint:
Each virtual node can only be mapped to a physical node that has sufficient remaining
node resources.
2. Link resource constraint:
Each virtual link can only be mapped to a physical route that has sufficient remaining
link resources on all the physical links along the route.
4. Modification of the Conventional Virtual Network Embedding Algorithm
4.1 VNE-TD (Virtual Network Embedding Algorithm based on Temporal-difference Learning)
VNE-TD, proposed in Ref. [12], is a virtual network embedding algorithm using TD learning. VNE-TD uses the state
value function $V$ to estimate the expected value of the cumulative revenue obtained
by the virtual network provider in the future and constructs a virtual network so
that the expected value is as high as possible.
Fig. 2 presents the interaction between the agent and environment in VNE-TD. In VNE-TD,
the agent corresponds to the virtual network provider, and the environment corresponds
to the physical network and VNR, respectively. The state of the environment, the agent's
action, and the reward that the agent gains from the environment are defined as follows.
The state of the environment corresponds to the remaining resources in the physical
network. The state is represented by a vector whose elements are the amounts of remaining
resources of all physical nodes and links (normalized based on the maximum amounts
of remaining resources). The agent's action corresponds to how the current VNR is
embedded in the physical network (mapping solution). The reward corresponds to the
revenue obtained from the current VNR (Eq. (2)).
Fig. 2. Interaction between the agent and the environment in VNE-TD.
VNE-TD uses the state value function $V$ to estimate the expected value of the cumulative
revenue $V(s_{k+1})$ obtained by the virtual network provider in the future at state
$s_{k+1}$ of the physical network, which corresponds to the state after mapping the
current VNR k. The state value function $V$ is approximated using a neural network
in order to appropriately estimate the values for states that have never been experienced.
The estimation accuracy of the state value function $V$ is improved by learning based
on the experiences ($s_{k},\,\,r_{k},\,\,s_{k+1}$) obtained in virtual network embeddings
in the past. The stochastic gradient descent (SGD) method is used to train the state
value function $V$. The SGD method updates the parameter vector $\vec{p}_{k}$ of the
neural network using the experiences ($s_{k},\,\,r_{k},\,\,s_{k+1}$) in the past,
which are stored in an experience buffer, based on Eq. (3) so that the TD error (the difference between $r_{k+1}+\gamma V\left(s_{k+1}\right)$
and $V\left(s_{k}\right)$) is minimized.
where $\alpha $ is the learning rate, $\gamma $ is the discount rate of the reward,
and $\nabla _{{\vec{p}_{k}}}V\left(s_{k}\right)$ is the gradient of $V\left(s_{k}\right)$
with respect to $\vec{p}_{k}$.
Fig. 3 presents the algorithm of VNE-TD. In VNE-TD, GC-GRC algorithm (Fig. 4) is first used to generate $L$ candidate solutions for the node mapping problem in
order to decrease the calculation time of a mapping solution (line 2 in Fig. 1). In GC-GRC, a GRC value [12] is calculated for each physical node in the physical network before performing node
mapping. The GRC value of a physical node represents its embedding potential and is
calculated based on the amount of remaining node resources of the physical node and
the amounts of remaining link resources of the physical links connected to the physical
node. VNE-TD then selects the candidate solution that leads to the state of the physical
network with the highest expected value of the cumulative revenue of the physical
network after mapping the current VNR (lines 5 to 24 in Fig. 3). For the link mapping problem, the shortest path method is used to determine the
physical route to be assigned to each virtual link. When VNE-TD finishes embedding
the current VNR, it stores the experience obtained this time to the experience buffer
(line 25). Finally, it trains the state value function $V$ based on past experiences
(line 26).
Fig. 3. Algorithm of VNE-TD.
Fig. 4. Algorithm of GC-GRC.
4.2 Modification of VNE-TD
VNE-TD does not consider whether or not the candidate solutions to the node mapping
problem satisfy both the node resource constraint and the link resource constraint
when selecting the candidate solutions. Therefore, when constructing a virtual network
based on the candidate solutions, the embedding of the virtual network may fail because
of insufficient node or link resources.
Fig. 5 gives examples of failures of VNE due to non-consideration of the node/link resource
constraints in VNE-TD. In these examples, a virtual network consisting of three nodes
and three links is embedded in a physical network consisting of five nodes and six
links. In candidate solution 1 (upper right of Fig. 5), virtual nodes a, b and c are mapped to physical nodes A, B and E, respectively.
In this case, the VNE fails because of the lack of remaining node resource at physical
node B. In candidate solution 2 (lower right of Fig. 5), virtual nodes a, b and c are mapped to physical nodes E, C and D, respectively.
In this case, the node resource constraint is satisfied. For link mapping, virtual
links (a, b), (a, c) and (b, c) are mapped to the physical route passing through physical
link (E, C), the physical route passing through physical link (E, D) and the physical
route passing through physical link (C, D), respectively. In this case, the VNE fails
because of the lack of remaining link resources at physical links (E, D) and (C, D).
In this paper, in order to improve the performance of VNE-TD, we modify VNE-TD so
that it has a function to check the satisfiability of the node resource constraint
and the link resource constraint.
In our modified VNE-TD, Resource-Constraint-Aware GC-GRC (Fig. 6) is used instead of GC-GRC (Fig. 4) when finding candidate solutions for node mapping in line 2 of Fig. 3. As a physical node to which virtual node $j$ is mapped, the Resource-Constraint-Aware
GC-GRC selects only those physical nodes that can satisfy both the node resource constraint
and the link resource constraint (lines 8 to 17 in Fig. 6) instead of all physical nodes. The function isRscCstSatisfied (Fig. 7) checks whether the node and link resource constraints are satisfied when virtual
node $j$ is mapped to physical node $l$. The function isRscCstSatisfied first checks
whether the node resource constraint is satisfied in lines 1 to 3. Here, $NR^{v}\left(j\right)$
is the amount of node resource requested by virtual node $j$ and $NR^{p}\left(l\right)$
is the amount of remaining node resource of physical node $l$. Then, the function
isRscCstSatisfied checks whether the link resource constraint is satisfied for each
of virtual links that are set up between virtual node $j$ and the virtual nodes that
have already been mapped to physical nodes in lines 4 to 11. Here, $LR^{v}\left(j,m\right)$
is the amount of link resource requested by virtual link $\left(j,m\right)$.
Fig. 5. Examples of failures of VNE due to non-consideration of resource constraints in VNE-TD.
Fig. 6. Algorithm of Resource-Constraint-Aware GC-GRC.
Fig. 7. Algorithm of function isRscCstSatisfied().
5. Performance Evaluation
5.1 Parameter Settings
We evaluate the performances of our modified methods by computer simulation. We use
the following three modified methods: 1) NLRC-VNE-TD (Node and Link Resource Constraints
aware VNE-TD): Both the node and link resource constraints are considered and the
function isRscCstSatisfied (Fig. 7) is used as is, 2) NRC-VNE-TD (Node Resource Constraint aware VNE-TD): only the node
resource constraint is considered and the link resource constraint is not considered
(i.e., lines 4 to 11 in the function isRscCstSatisfied in Fig. 7 are commented out), 3) LRC-VNE-TD (Link Resource Constraint aware VNE-TD): only the
link resource constraint is considered and the node resource constraint is not considered
(i.e., lines 1 to 3 in the function isRscCstSatisfied in Fig. 7 are commented out). VNE-TD is used for comparison.
Table 1 lists the parameter settings for the simulations. We developed a VNE simulator using
OpenAI Gym [14]. We implemented the neural network representing the state value function $V$ using
Keras [15]. We used the Waxman model as the topology generation model for the physical network.
We used the Erdos-Renyi model as the topology generation model for virtual networks.
In the simulations, VNRs arrive dynamically, and their arrival intervals follow an
exponential distribution with an average of one second. The holding times of the constructed
virtual networks follow an exponential distribution with an average of 70 seconds.
The total number of VNRs is set to 10000.
In order to evaluate the effect of the ratio of the amount of node resources to the
amount of link resources on the performances of the VNE methods, we set the amount
of node resource provided by a physical node as follows: Case 1) uniformly distributed
over the integers from 10 to 100, Case 2) uniformly distributed over the integers
from 5 to 50 and Case 3) uniformly distributed over the integers from 40 to 400.
We use blocking ratio of VNRs and revenue of virtual network provider as performance
metrics. The blocking ratio of VNRs is the ratio of the number of VNRs that failed
to be embedded in the physical network to the number of all the VNRs. The revenue
of a virtual network provider is the revenue per second earned by the virtual network
provider.
Table 1. Parameter Settings.
|
Parameters
|
Values
|
|
Number of physical nodes
|
500
|
|
Number of physical links
|
1000
|
|
Node resource provided by a physical node
|
Case 1: [10:100]
Case 2: [5:50]
Case 3: [40:400]
|
|
Link resource provided by a physical link
|
[40:400]
|
|
Number of virtual nodes
|
Case 1: [2:50]
Case 2: [2:40]
Case 3: [2:60]
|
|
Probability to connect a pair of virtual nodes with a virtual link in the Erdos-Renyi
model
|
0.2
|
|
Node resource requested by a virtual node
|
[1:10]
|
|
Link resource requested by a virtual link
|
[1:10]
|
|
Inter-arrival time of VNR
|
Exponential distribution with an average of 1 [s]
|
|
Holding time of VNR
|
Exponential distribution with an average of 70 [s]
|
|
Number of candidate solutions
|
40
|
|
Number of hidden layers in the neural network
|
2
|
|
Number of neurons in a hidden layer
|
300
|
|
Activation function
|
ReLU
|
|
Experience buffer size
|
1000
|
|
Batch size
|
50
|
|
Learning rate
|
0.01
|
5.2 Evaluation Results
Fig. 8 presents the blocking ratio of every method in Case 1. The horizontal axis represents
the elapsed simulation time. Compared to VNE-TD, NLRC-VNE-TD reduces the blocking
ratio by approximately 57%, suggesting that the blocking ratio of VNE-TD can be improved
by considering the node and link resource constraints. Compared to VNE-TD, NRC-VNE-TD
and LRC-VNE-TD reduce the blocking ratio by approximately 4% and 17%, respectively,
suggesting that considering only one resource constraint can also improve the blocking
ratio of VNE-TD.
Fig. 9 presents the blocking ratio of every method in Case 2. Compared to VNE-TD, NLRC-VNE-TD
and NRC-VNE-TD reduce the blocking ratio by approximately 80% and 41%, respectively.
On the other hand, LRC-VNE-TD shows a comparable blocking ratio compared to VNE-TD.
This is explained as follows. In Case 2, the amount of node resources is relatively
small, and the leading cause of blocking in VNE is the lack of node resources. LRC-VNE-TD
does not consider the node resource constraint, and consequently it cannot avoid the
blocking in VNE because of the lack of node resources.
Fig. 10 shows the blocking ratio of every method in Case 3. Compared to VNE-TD, NLRC-VNE-TD
and LRC-VNE-TD reduce the blocking ratio by approximately 44% and 41%, respectively.
On the other hand, NRC-VNE-TD shows a comparable blocking ratio compared to VNE-TD.
This is because the amount of link resources is relatively small in Case 3, and NRC-VNE-TD
does not consider the link resource constraint. Consequently, it cannot avoid the
blocking in VNE because of the lack of link resources.
Figs. 11-13 present the revenue of virtual network provider for every method in Cases 1, 2
and 3, respectively. In Fig. 11, compared to VNE-TD, NLRC-VNE-TD and NRC-VNE-TD achieve approximately 1.57 and 1.07
times higher revenues, respectively. On the other hand, LRC-VNE-TD shows similar revenue
to VNE-TD. This situation can be explained through the fact that LRC-VNE-TD successfully
embeds more VNRs than VNE-TD, but the resource requirement per VNR (i.e., the revenue
from a single VNR) of LRC-VNE-TD is smaller than that of VNE-TD. In Fig. 12, similar to Fig. 9, NLRC-VNE-TD and NRC-VNE-TD achieve approximately 2.84 and 2.09 times higher revenue
than VNE-TD, respectively while LRC-VNE-TD shows comparable revenue compared to VNE-TD.
In Fig. 13, similar to Fig. 10, NLRC-VNE-TD and LRC-VNE-TD achieve approximately 1.47 and 1.43 times higher revenues
than VNE-TD, respectively whereas NRC-VNE-TD shows comparable revenue compared to
VNE-TD.
These results show that the blocking ratio and the revenue of the virtual network
provider are improved the most when both the node and link resources constraints are
considered in VNE-TD.
Fig. 8. Blocking ratio in Case 1.
Fig. 9. Blocking ratio in Case 2.
Fig. 10. Blocking ratio in Case 3.
Fig. 11. Revenue per second in Case 1.
Fig. 12. Revenue per second in Case 2.
Fig. 13. Revenue per second in Case 3.
6. Conclusion
In this paper, we have modified VNE-TD to select only those candidate solutions that
satisfy the node and link resource constraints. The simulation results confirms that
the blocking ratio of virtual network requests and the revenue of virtual network
provider are improved the most when both the node and link resources constraints are
considered in VNE-TD.
One of our future works is to evaluate the scalability of VNE-TD.
ACKNOWLEDGMENTS
This work was supported by JSPS KAKENHI Grant Number JP23K11065.
REFERENCES
K. Tutschku, T. Zinner, A. Nakao and P. Tran-Gia, ``Network Virtualization: Implementation
Steps towards the Future Internet,'' in Proc. of Workshop on Overlay and Network Virtualization,
May 2009.
N. M. M. K. Chowdhury, ``Network Virtualization: State of the Art and Research Challenges,''
IEEE Communications Magazine, vol. 47, no. 7, pp. 20-26, Jul. 2009.
A. Fischer, J. F. Botero, M. T. Beck, H. Meer, and X. Hesselback, ``Virtual Network
Embedding: A Survey,'' IEEE Communications surveys & Tutorial, vol. 15, iss., 4, pp.
1888-1906, Feb. 2013.
I. Houidi, W. Louati, W. B. Ameur, and D. Zeghlache, ``Virtual network provisioning
across multiple substrate networks,'' Computer Networks, vol. 55, no. 4, pp. 1011-1023,
Mar. 2011.
J. F. Botero, X. Hesselbach, M. Duelli, D. Schlosser, and H. D. Meer, ``Energy Efficient
Virtual Network Embedding,'' IEEE Communication Letters, vol. 16, no. 5, May 2012.
A. Fischer, M. T. Beck, and H. D. Meer, ``An approach to Energy-efficient Virtual
Network Embeddings,'' in Proc. of IM, pp. 1142-1147, May 2013.
I. Houidi, W. Louati, and D. Zeghlache, ``A distributed virtual network mapping algorithm,''
in Proc. of ICC, pp. 5634-5640, May 2008.
I. Fajjari, N. Aitsaadi, G. Pujolle, and H. Zimmermann, ``VNE-AC: virtual network
embedding algorithm based on ant colony metaheuristic,'' in Proc. of ICC, pp. 1-6,
June 2011.
M. T. Beck, A. Fischer, H. D. Meer, J. F. Boteto, and X. Hesselbach, ``A distributed,
parallel, and genetic virtual network embedding framework,'' in Proc. of ICC, pp.
3471-3475, June 2013.
M. Yu, Y. Yi, J. Rexford, and M. Chiang, ``Rethinking virtual network embedding: substrate
support for path splitting and migration,'' ACM SIGCOMM Computer Communication Review,
vol. 38, iss. 2, Apr. 2008.
L. Gong, Y. Wen, Z. Zhu, and T. Lee, ``Toward profit-seeking virtual network embedding
algorithm via global resource capacity,'' in Proc. of INFOCOM, pp. 1-9, Apr. 2014.
S. Wang, J. Bi, J. Wu, A. V. Vasilakos, and Q. Fan, ``VNE-TD: A virtual network embedding
algorithm based on temporal-difference learning,'' Computer Networks, vol, 161, pp.
251-263, Oct. 2019.
R. S. Sutton and A. G. Barto, ``Reinforcement Learning: An Introduction,'' MIT Press,
2018,
OpenAI Gym,
Keras,
Author
Yukinobu FUKUSHIMA
Yukinobu FUKUSHIMA received the B.E., M.E. and Ph.D. degrees from Osaka University,
Japan, in 2001, 2003 and 2006, respectively. He is currently an associate professor
of the Faculty of Natural Science and Technology, Okayama University. His research
interest includes knowledge-defined networking and network virtualization. He is a
member of IEICE, IEEE and ACM.
Yuta SAGAWA
Yuta SAGAWA received an B.E. and M.E. degree in Engineering from Okayama University
in 2018 and 2021. His research interest includes virtual network embedding.
Yuya TARUTANI
Yuya TARUTANI received an B.E., M.E. and Ph.D. degree in Information Science and
Technology from Osaka University in 2010, 2012 and 2014, respectively. He was an assistant
professor in cybermedia center at Osaka University from Oct. 2014 to Nov. 2018. He
is currently an assistant professor in Graduate School of Interdisciplinary Science
and Engineering in Health Systems at Okayama University. His research interest includes
communication network, design of control method with IoT devices, and network security
in IoT network. He is a Member of IEICE and IEEE.
Tokumi YOKOHIRA
Tokumi YOKOHIRA received the B.E., M.E. and Ph.D. degrees in information and computer
sciences from Osaka University, Osaka, Japan, in 1984, 1986 and 1989, respectively.
He was an academic of Okayama University from April 1989 to March 2018. Since April
2018, he has been a professor of Graduate School of Interdisciplinary Science and
Engineering in Health Systems of the same university. His present research interests
include highly distributed cloud computing environment, design of virtual networks,
technologies to upgrade the speed of the Internet and technologies to increase the
fault tolerance of the Internet. He is a member of the IEEE Computer and Communication
Societies, the Institute of Electronics, Information and Communication Engineers Inc.
and Information Processing Society of Japan.