Author: Eitan AltmanPublisher:
ISBN:
Category :
Languages : en
Pages : 23
Book Description
Sensitivity of Constrained Markov Decision Processes
Author: Eitan AltmanConstrained Markov Decision Processes
Author: Eitan AltmanPublisher: CRC Press
ISBN: 9780849303821
Category : Mathematics
Languages : en
Pages : 260
Book Description
This book provides a unified approach for the study of constrained Markov decision processes with a finite state space and unbounded costs. Unlike the single controller case considered in many other books, the author considers a single controller with several objectives, such as minimizing delays and loss, probabilities, and maximization of throughputs. It is desirable to design a controller that minimizes one cost objective, subject to inequality constraints on other cost objectives. This framework describes dynamic decision problems arising frequently in many engineering fields. A thorough overview of these applications is presented in the introduction. The book is then divided into three sections that build upon each other. The first part explains the theory for the finite state space. The author characterizes the set of achievable expected occupation measures as well as performance vectors, and identifies simple classes of policies among which optimal policies exist. This allows the reduction of the original dynamic into a linear program. A Lagranian approach is then used to derive the dual linear program using dynamic programming techniques. In the second part, these results are extended to the infinite state space and action spaces. The author provides two frameworks: the case where costs are bounded below and the contracting framework. The third part builds upon the results of the first two parts and examines asymptotical results of the convergence of both the value and the policies in the time horizon and in the discount factor. Finally, several state truncation algorithms that enable the approximation of the solution of the original control problem via finite linear programs are given.
Markov Decision Processes with Policy Constraints
Author: John NafehPublisher:
ISBN:
Category : Decision making
Languages : en
Pages : 338
Book Description
This work is concerned with Markov Decision Processes with policy constraints. The selection of an optimum stationary policy for such processes, in the absence of policy constraints, is a problem which has received a great deal of attention, and has been satisfactorily solved. Relatively little attention has been given to the case when policy constraints are present or to the formulation of such constraints. Optimum policy sensitivity analysis is also a subject in which little has been achieved. Towards those ends, this work makes three major contributions. First, policy constraints are formulated and categorized. Secondly, a computationally efficient iterative algorithm is developed for selecting the optimum policy for completely ergodic, infinite time horizon Markov Decision Processes with policy constraints for both the risk-indifferent and risk-sensitive cases. Finally, the sensitivity of optimum policies to the policy constraints is analyzed by using the algorithm to compute the value of removing a constraint or a group of constraints. (Author).
Constrained Markov Decision Processes
Author: Eitan AltmanPublisher: Routledge
ISBN: 1351458248
Category : Mathematics
Languages : en
Pages : 256
Book Description
This book provides a unified approach for the study of constrained Markov decision processes with a finite state space and unbounded costs. Unlike the single controller case considered in many other books, the author considers a single controller with several objectives, such as minimizing delays and loss, probabilities, and maximization of throughputs. It is desirable to design a controller that minimizes one cost objective, subject to inequality constraints on other cost objectives. This framework describes dynamic decision problems arising frequently in many engineering fields. A thorough overview of these applications is presented in the introduction. The book is then divided into three sections that build upon each other.
Constrained Markov Decision Processes
Author: E. AltmanHandbook of Markov Decision Processes
Author: Eugene A. FeinbergPublisher: Springer Science & Business Media
ISBN: 1461508053
Category : Business & Economics
Languages : en
Pages : 560
Book Description
Eugene A. Feinberg Adam Shwartz This volume deals with the theory of Markov Decision Processes (MDPs) and their applications. Each chapter was written by a leading expert in the re spective area. The papers cover major research areas and methodologies, and discuss open questions and future research directions. The papers can be read independently, with the basic notation and concepts ofSection 1.2. Most chap ters should be accessible by graduate or advanced undergraduate students in fields of operations research, electrical engineering, and computer science. 1.1 AN OVERVIEW OF MARKOV DECISION PROCESSES The theory of Markov Decision Processes-also known under several other names including sequential stochastic optimization, discrete-time stochastic control, and stochastic dynamic programming-studiessequential optimization ofdiscrete time stochastic systems. The basic object is a discrete-time stochas tic system whose transition mechanism can be controlled over time. Each control policy defines the stochastic process and values of objective functions associated with this process. The goal is to select a "good" control policy. In real life, decisions that humans and computers make on all levels usually have two types ofimpacts: (i) they cost orsavetime, money, or other resources, or they bring revenues, as well as (ii) they have an impact on the future, by influencing the dynamics. In many situations, decisions with the largest immediate profit may not be good in view offuture events. MDPs model this paradigm and provide results on the structure and existence of good policies and on methods for their calculation.
Risk-sensitive Markov Decision Processes
Author: Yun ShenOptimal Control of Random Sequences in Problems with Constraints
Author: A.B. PiunovskiyPublisher: Springer Science & Business Media
ISBN: 9401155089
Category : Mathematics
Languages : en
Pages : 355
Book Description
Controlled stochastic processes with discrete time form a very interest ing and meaningful field of research which attracts widespread attention. At the same time these processes are used for solving of many applied problems in the queueing theory, in mathematical economics. in the theory of controlled technical systems, etc. . In this connection, methods of the theory of controlled processes constitute the every day instrument of many specialists working in the areas mentioned. The present book is devoted to the rather new area, that is, to the optimal control theory with functional constraints. This theory is close to the theory of multicriteria optimization. The compromise between the mathematical rigor and the big number of meaningful examples makes the book attractive for professional mathematicians and for specialists who ap ply mathematical methods in different specific problems. Besides. the book contains setting of many new interesting problems for further invf'stigatioll. The book can form the basis of special courses in the theory of controlled stochastic processes for students and post-graduates specializing in the ap plied mathematics and in the control theory of complex systf'ms. The grounding of graduating students of mathematical department is sufficient for the perfect understanding of all the material. The book con tains the extensive Appendix where the necessary knowledge ill Borel spaces and in convex analysis is collected. All the meaningful examples can be also understood by readers who are not deeply grounded in mathematics.
Examples in Markov Decision Processes
Author: A. B. PiunovskiyPublisher: World Scientific
ISBN: 1848167938
Category : Mathematics
Languages : en
Pages : 308
Book Description
This invaluable book provides approximately eighty examples illustrating the theory of controlled discrete-time Markov processes. Except for applications of the theory to real-life problems like stock exchange, queues, gambling, optimal search etc, the main attention is paid to counter-intuitive, unexpected properties of optimization problems. Such examples illustrate the importance of conditions imposed in the theorems on Markov Decision Processes. Many of the examples are based upon examples published earlier in journal articles or textbooks while several other examples are new. The aim was to collect them together in one reference book which should be considered as a complement to existing monographs on Markov decision processes. The book is self-contained and unified in presentation. The main theoretical statements and constructions are provided, and particular examples can be read independently of others. Examples in Markov Decision Processes is an essential source of reference for mathematicians and all those who apply the optimal control theory to practical purposes. When studying or using mathematical methods, the researcher must understand what can happen if some of the conditions imposed in rigorous theorems are not satisfied. Many examples confirming the importance of such conditions were published in different journal articles which are often difficult to find. This book brings together examples based upon such sources, along with several new ones. In addition, it indicates the areas where Markov decision processes can be used. Active researchers can refer to this book on applicability of mathematical methods and theorems. It is also suitable reading for graduate and research students where they will better understand the theory.
Continuous-Time Markov Decision Processes
Author: Xianping GuoPublisher: Springer Science & Business Media
ISBN: 3642025471
Category : Mathematics
Languages : en
Pages : 240
Book Description
Continuous-time Markov decision processes (MDPs), also known as controlled Markov chains, are used for modeling decision-making problems that arise in operations research (for instance, inventory, manufacturing, and queueing systems), computer science, communications engineering, control of populations (such as fisheries and epidemics), and management science, among many other fields. This volume provides a unified, systematic, self-contained presentation of recent developments on the theory and applications of continuous-time MDPs. The MDPs in this volume include most of the cases that arise in applications, because they allow unbounded transition and reward/cost rates. Much of the material appears for the first time in book form.