In this work we propose a hierarchical reinforcement learning algorithm that is able to construct planning graphs while managing to use computational resources efficiently, guided by intrinsic motivation in form of prediction error and measure of task improvement.