Class responsible for solving the gridworld problem with Q-learning, neural network used for approximation of the rewards and experience replay using for (batch) training of the neural network. In this case there is an assumption that the agent observes only part of the environment (POMPD). More...

#include <GridworldDRLExperienceReplayPOMDP.hpp>

Inheritance diagram for mic::application::GridworldDRLExperienceReplayPOMDP:

[legend]

Collaboration diagram for mic::application::GridworldDRLExperienceReplayPOMDP:

[legend]

Public Member Functions
	GridworldDRLExperienceReplayPOMDP (std::string node_name_="application")

virtual	~GridworldDRLExperienceReplayPOMDP ()

Protected Member Functions
virtual void	initialize (int argc, char *argv[])

virtual void	initializePropertyDependentVariables ()

virtual bool	performSingleStep ()

virtual void	startNewEpisode ()

virtual void	finishCurrentEpisode ()

Private Member Functions
float	computeBestValueForGivenStateAndPredictions (mic::types::Position2D player_position_, float *predictions_)

mic::types::MatrixXfPtr	getPredictedRewardsForGivenState (mic::types::Position2D player_position_)

mic::types::NESWAction	selectBestActionForGivenState (mic::types::Position2D player_position_)

std::string	streamNetworkResponseTable ()

Private Attributes
WindowCollectorChart< float > *	w_chart
	Window for displaying statistics. More...

mic::utils::DataCollectorPtr < std::string, float >	collector_ptr
	Data collector. More...

mic::environments::Gridworld	grid_env
	The gridworld environment. More...

size_t	batch_size
	Size of the batch in experience replay - set to the size of maze (width*height). More...

mic::configuration::Property < float >	step_reward

mic::configuration::Property < float >	discount_rate

mic::configuration::Property < float >	learning_rate

mic::configuration::Property < double >	epsilon

mic::configuration::Property< int >	step_limit

mic::configuration::Property < std::string >	statistics_filename
	Property: name of the file to which the statistics will be exported. More...

mic::configuration::Property < std::string >	mlnn_filename
	Property: name of the file to which the neural network will be serialized (or deserialized from). More...

mic::configuration::Property < bool >	mlnn_save
	Property: flad denoting thether the nn should be saved to a file (after every episode end). More...

mic::configuration::Property < bool >	mlnn_load
	Property: flad denoting thether the nn should be loaded from a file (at the initialization of the task). More...

BackpropagationNeuralNetwork < float >	neural_net
	Multi-layer neural network used for approximation of the Qstate rewards. More...

long long	sum_of_iterations

long long	sum_of_rewards

long long	number_of_successes

SpatialExperienceMemory	experiences

Detailed Description

Class responsible for solving the gridworld problem with Q-learning, neural network used for approximation of the rewards and experience replay using for (batch) training of the neural network. In this case there is an assumption that the agent observes only part of the environment (POMPD).

Author: tkornuta

Definition at line 50 of file GridworldDRLExperienceReplayPOMDP.hpp.

Constructor & Destructor Documentation

mic::application::GridworldDRLExperienceReplayPOMDP::GridworldDRLExperienceReplayPOMDP ( std::string node_name_ = "application" )

Default Constructor. Sets the application/node name, default values of variables etc.

Parameters

node_name_ Name of the application/node (in configuration file).

Definition at line 40 of file GridworldDRLExperienceReplayPOMDP.cpp.

References discount_rate, epsilon, learning_rate, mlnn_filename, mlnn_load, mlnn_save, statistics_filename, step_limit, and step_reward.

mic::application::GridworldDRLExperienceReplayPOMDP::~GridworldDRLExperienceReplayPOMDP ( )

virtual

Destructor.

Definition at line 67 of file GridworldDRLExperienceReplayPOMDP.cpp.

References w_chart.

Member Function Documentation

float mic::application::GridworldDRLExperienceReplayPOMDP::computeBestValueForGivenStateAndPredictions	(	mic::types::Position2D	player_position_,
		float *	predictions_
	)

private

Calculates the best value for the current state and predictions.

Parameters

player_position_	State (player position).
predictions_	Vector of predictions to be analyzed.

Returns: Value of the best possible action for given state.

Definition at line 255 of file GridworldDRLExperienceReplayPOMDP.cpp.

References grid_env, and mic::environments::Environment::isActionAllowed().

Referenced by performSingleStep().

void mic::application::GridworldDRLExperienceReplayPOMDP::finishCurrentEpisode ( )

protectedvirtual

Method called when given episode ends (goal: export collected statistics to file etc.) - abstract, to be overridden.

Definition at line 140 of file GridworldDRLExperienceReplayPOMDP.cpp.

References collector_ptr, mic::environments::Gridworld::getAgentPosition(), mic::environments::Gridworld::getStateReward(), grid_env, mlnn_filename, mlnn_save, neural_net, number_of_successes, statistics_filename, sum_of_iterations, and sum_of_rewards.

mic::types::MatrixXfPtr mic::application::GridworldDRLExperienceReplayPOMDP::getPredictedRewardsForGivenState ( mic::types::Position2D player_position_ )

private

Returns the predicted rewards for given state.

Parameters

player_position_ State (player position).

Returns: Pointer to the predicted rewards (network output matrix).

Definition at line 279 of file GridworldDRLExperienceReplayPOMDP.cpp.

References batch_size, mic::environments::Gridworld::encodeObservation(), mic::environments::Gridworld::getAgentPosition(), mic::environments::Environment::getObservationSize(), grid_env, mic::environments::Gridworld::moveAgentToPosition(), and neural_net.

Referenced by selectBestActionForGivenState().

void mic::application::GridworldDRLExperienceReplayPOMDP::initialize	(	int	argc,
		char *	argv[]
	)

protectedvirtual

Method initializes GLUT and OpenGL windows.

Parameters

argc	Number of application parameters.
argv	Array of application parameters.

Definition at line 72 of file GridworldDRLExperienceReplayPOMDP.cpp.

References collector_ptr, number_of_successes, sum_of_iterations, sum_of_rewards, and w_chart.

void mic::application::GridworldDRLExperienceReplayPOMDP::initializePropertyDependentVariables ( )

protectedvirtual

Initializes all variables that are property-dependent.

Definition at line 94 of file GridworldDRLExperienceReplayPOMDP.cpp.

References batch_size, experiences, mic::environments::Environment::getObservationHeight(), mic::environments::Environment::getObservationSize(), mic::environments::Environment::getObservationWidth(), grid_env, mic::environments::Gridworld::initializeEnvironment(), mlnn_filename, mlnn_load, and neural_net.

bool mic::application::GridworldDRLExperienceReplayPOMDP::performSingleStep ( )

protectedvirtual

Performs single step of computations.

Definition at line 352 of file GridworldDRLExperienceReplayPOMDP.cpp.

mic::types::NESWAction mic::application::GridworldDRLExperienceReplayPOMDP::selectBestActionForGivenState ( mic::types::Position2D player_position_ )

private

Finds the best action for the current state.

Parameters

player_position_ State (player position).

Returns: The best action found.

Definition at line 319 of file GridworldDRLExperienceReplayPOMDP.cpp.

References getPredictedRewardsForGivenState(), grid_env, and mic::environments::Environment::isActionAllowed().

Referenced by performSingleStep().

void mic::application::GridworldDRLExperienceReplayPOMDP::startNewEpisode ( )

protectedvirtual

Method called at the beginning of new episode (goal: to reset the statistics etc.) - abstract, to be overridden.

Definition at line 128 of file GridworldDRLExperienceReplayPOMDP.cpp.

References mic::environments::Gridworld::environmentToString(), grid_env, mic::environments::Gridworld::initializeEnvironment(), mic::environments::Gridworld::observationToString(), and streamNetworkResponseTable().

std::string mic::application::GridworldDRLExperienceReplayPOMDP::streamNetworkResponseTable ( )

private

Steams the current network response - values of actions associates with consecutive agent poses.

Returns: Ostream with description of the state-action table.

Definition at line 167 of file GridworldDRLExperienceReplayPOMDP.cpp.

References batch_size, mic::environments::Gridworld::encodeObservation(), mic::environments::Gridworld::getAgentPosition(), mic::environments::Environment::getObservationHeight(), mic::environments::Environment::getObservationSize(), mic::environments::Environment::getObservationWidth(), grid_env, mic::environments::Environment::isActionAllowed(), mic::environments::Gridworld::isStateAllowed(), mic::environments::Gridworld::isStateTerminal(), mic::environments::Gridworld::moveAgentToPosition(), and neural_net.

Referenced by performSingleStep(), and startNewEpisode().

Member Data Documentation

size_t mic::application::GridworldDRLExperienceReplayPOMDP::batch_size

private

Size of the batch in experience replay - set to the size of maze (width*height).

Definition at line 105 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by getPredictedRewardsForGivenState(), initializePropertyDependentVariables(), performSingleStep(), and streamNetworkResponseTable().

mic::utils::DataCollectorPtr<std::string, float> mic::application::GridworldDRLExperienceReplayPOMDP::collector_ptr

private

Data collector.

Definition at line 99 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by finishCurrentEpisode(), and initialize().

mic::configuration::Property<float> mic::application::GridworldDRLExperienceReplayPOMDP::discount_rate

private

Property: future discount (should be in range 0.0-1.0).

Definition at line 115 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by GridworldDRLExperienceReplayPOMDP(), and performSingleStep().

mic::configuration::Property<double> mic::application::GridworldDRLExperienceReplayPOMDP::epsilon

private

Property: variable denoting epsilon in action selection (the probability "below" which a random action will be selected). if epsilon < 0 then if will be set to 1/episode, hence change dynamically depending on the episode number.

Definition at line 126 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by GridworldDRLExperienceReplayPOMDP(), and performSingleStep().

SpatialExperienceMemory mic::application::GridworldDRLExperienceReplayPOMDP::experiences

private

Table of past experiences.

Definition at line 194 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by initializePropertyDependentVariables(), and performSingleStep().

mic::environments::Gridworld mic::application::GridworldDRLExperienceReplayPOMDP::grid_env

private

The gridworld environment.

Definition at line 102 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by computeBestValueForGivenStateAndPredictions(), finishCurrentEpisode(), getPredictedRewardsForGivenState(), initializePropertyDependentVariables(), performSingleStep(), selectBestActionForGivenState(), startNewEpisode(), and streamNetworkResponseTable().

mic::configuration::Property<float> mic::application::GridworldDRLExperienceReplayPOMDP::learning_rate

private

Property: neural network learning rate (should be in range 0.0-1.0).

Definition at line 120 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by GridworldDRLExperienceReplayPOMDP(), and performSingleStep().

mic::configuration::Property<std::string> mic::application::GridworldDRLExperienceReplayPOMDP::mlnn_filename

private

Property: name of the file to which the neural network will be serialized (or deserialized from).

Definition at line 137 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by finishCurrentEpisode(), GridworldDRLExperienceReplayPOMDP(), and initializePropertyDependentVariables().

mic::configuration::Property<bool> mic::application::GridworldDRLExperienceReplayPOMDP::mlnn_load

private

Property: flad denoting thether the nn should be loaded from a file (at the initialization of the task).

Definition at line 143 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by GridworldDRLExperienceReplayPOMDP(), and initializePropertyDependentVariables().

mic::configuration::Property<bool> mic::application::GridworldDRLExperienceReplayPOMDP::mlnn_save

private

Property: flad denoting thether the nn should be saved to a file (after every episode end).

Definition at line 140 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by finishCurrentEpisode(), and GridworldDRLExperienceReplayPOMDP().

BackpropagationNeuralNetwork<float> mic::application::GridworldDRLExperienceReplayPOMDP::neural_net

private

Multi-layer neural network used for approximation of the Qstate rewards.

Definition at line 146 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by finishCurrentEpisode(), getPredictedRewardsForGivenState(), initializePropertyDependentVariables(), performSingleStep(), and streamNetworkResponseTable().

long long mic::application::GridworldDRLExperienceReplayPOMDP::number_of_successes

private

Number of successes, i.e. how many times we reached goal till now - used in statistics.

Definition at line 189 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by finishCurrentEpisode(), and initialize().

mic::configuration::Property<std::string> mic::application::GridworldDRLExperienceReplayPOMDP::statistics_filename

private

Property: name of the file to which the statistics will be exported.

Definition at line 134 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by finishCurrentEpisode(), and GridworldDRLExperienceReplayPOMDP().

mic::configuration::Property<int> mic::application::GridworldDRLExperienceReplayPOMDP::step_limit

private

Limit of steps for episode. Setting step_limit <= 0 means that the limit should not be considered.

Definition at line 131 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by GridworldDRLExperienceReplayPOMDP(), and performSingleStep().

mic::configuration::Property<float> mic::application::GridworldDRLExperienceReplayPOMDP::step_reward

private

Property: the "expected intermediate reward", i.e. reward received by performing each step (typically negative, but can be positive as all).

Definition at line 110 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by GridworldDRLExperienceReplayPOMDP(), and performSingleStep().

long long mic::application::GridworldDRLExperienceReplayPOMDP::sum_of_iterations

private

Sum of all iterations made till now - used in statistics.

Definition at line 179 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by finishCurrentEpisode(), and initialize().

long long mic::application::GridworldDRLExperienceReplayPOMDP::sum_of_rewards

private

Sum of all rewards collected till now - used in statistics.

Definition at line 184 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by finishCurrentEpisode(), and initialize().

WindowCollectorChart<float>* mic::application::GridworldDRLExperienceReplayPOMDP::w_chart

private

Window for displaying statistics.

Definition at line 96 of file GridworldDRLExperienceReplayPOMDP.hpp.

Referenced by initialize(), and ~GridworldDRLExperienceReplayPOMDP().

The documentation for this class was generated from the following files:

src/application/GridworldDRLExperienceReplayPOMDP.hpp
src/application/GridworldDRLExperienceReplayPOMDP.cpp

Public Member Functions

Protected Member Functions

Private Member Functions

Private Attributes

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation