iQc @sddlZddlZddlZddlZddlZddlZdejfdYZdejfdYZ dd6dYZ dZ d Z d Z d Zd Zd ZdZdZdZdZdZedkreZddlZeedejZeZejejejejej eZ ddl!Z!e!j"eZ#ej$sddl%Z%e%j&eej'ej(Z#nye#j)Wne*k rej+dnXddl,Z,ddl-Z-dZ/ej0dkr'e,j1eej2ej3Z/nej0dkre eZ4dZ5iej2d6ej6d6ej7d6e5d6Z8e-j9e8Z/nbej0dkrej:dkrde_:ndd7d YZ;e;Z/nej<sd!ej0nyej< rej0dkrej=rox]e>ej3D]IZ?e,j1eej2e?Z@e#jAe@d"d#eBe?d$e#jCqWne#jAe/d"d#eBej3d$e#jCe#jDe/d"d%eBej3d$e#jCnWne*k rej+dnXd&ZEejFszej<r&ej0dkr&d'ZEqzej0dkrAd(ZEnej0dkr\d)ZEnej0dkrzd*ZEqznd+ZGejFrd,ZGnd-ZHejCrd.ZHnej<rd/ZIn e/jJZIej:dkrHd0Gej:Gd1GHHndZKxFe>d2ej:d2D].ZLeKee/e ej2eIeEeGeHeL7ZKqWej:dkrvHd3eBeKd4ej:GHHHnej0dkrej< ry`e#jDe/d"d%eBej:d5e#jCe#jAe/d"d#eBej:d5e#jCWqe*k rej+dqXqndS(8iNt GridworldcBsqeZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z RS( s Gridworld cCsFt|tgkr't|}n||_d|_d|_dS(Ngg?(ttypetmakeGridtgridt livingRewardtnoise(tselfR((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyt__init__s   cCs ||_dS(s The (negative) reward for exiting "normal" states. Note that in the R+N text, this reward is on entering a state and therefore is not clearly part of the state's future rewards. N(R(Rtreward((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pytsetLivingRewardscCs ||_dS(sG The probability of moving in an unintended direction. N(R(RR((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pytsetNoise)scCsG||jjkrdS|\}}t|j||tkrCdSdS( s Returns list of valid actions for 'state'. Note that you can request moves into walls and that "exit" states transition to the terminal state under the special action "done". texittnorthtwesttsouthteast((sexit(snorthswestssouthseast(Rt terminalStateRtint(Rtstatetxty((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pytgetPossibleActions0s  cCs|jjg}xjt|jjD]V}xMt|jjD]9}|j||dkr;||f}|j|q;q;Wq"W|S(s, Return list of all states. t#(RRtrangetwidththeighttappend(RtstatesRRR((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyt getStates?s cCsb||jjkrdS|\}}|j||}t|tksWt|tkr[|S|jS(s Get reward for state, action, nextState transition. Note that the reward depends only on the state being departed (as in the R+N book examples, which more or less use this convention). g(RRRRtfloatR(RRtactiont nextStateRRtcell((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyt getRewardLs $cCsexXt|jjD]D}x;t|jjD]'}|j||dkr,||fSq,WqWddS(NtSsGrid has no start state(RRRR(RRR((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyt getStartState\s cCs||jjkS(s= Only the TERMINAL_STATE state is *actually* a terminal state. The other "exit" states are technically non-terminals with a single action "exit" which leads to the true terminal state. This convention is to make the grids line up with the examples in the R+N textbook. (RR(RR((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyt isTerminalcsc Cs||j|krdn|j|r1gS|\}}t|j||tkswt|j||tkr|jj}|dfgSg}|j|d|r||dfp|}|j||dr|d|fp|}|j|d|r||dfp|} |j||dr@|d|fpC|} |dks^|dkr|dkr|j|d|j fn|j| d|j f|j } |j|| df|j| | dfn|dks|dkrp|dkr|j|d|j fn|j| d|j f|j } |j|| df|j| | dfn|j |}|S( s Returns list of (nextState, prob) pairs representing the states reachable from 'state' by taking 'action' along with their transition probabilities. sIllegal action!g?iR Rg@R R( RR$RRRRRt_Gridworld__isAllowedRRt_Gridworld__aggregate( RRRRRt termStatet successorst northStatet westStatet southStatet eastStatetmassLeft((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pytgetTransitionStatesAndProbsns:  :  ,,,,    cCsmtj}x$|D]\}}||c|7sN( R9R:RR?RRARBR@R>(((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyR<s      tGridcBskeZdZddZdZdZdZdZdZdZ d Z d Z d Z RS( s2 A 2-dimensional array of immutables backed by a list of lists. Data is accessed via grid[x][y] where (x,y) are cartesian coordinates with x horizontal, y vertical and the origin (0,0) in the bottom left corner. The __str__ method constructs an output that is oriented appropriately. t cCsZ||_||_gt|D]%}gt|D] }|^q2^q|_d|_dS(NtTERMINAL_STATE(RRRtdataR(RRRt initialValueRR((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyRs  ;cCs |j|S(N(RJ(Rti((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyt __getitem__scCs||j|dddd gddddgddddgg}t|S(NRHiRiR"(R(R((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyt getBookGrid(scCs\dddd gddddgddddgddddgddddgg}t|S(NRHiRR"(R(R((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyt getMazeGrid.s cCsddl}d}xtr|j}d|kr<d}nd|krQd}nd|krfd}nd |kr{d }nd |krtjd n|dkrqnPqW||}||kr|d }n|S( si Get an action from the user (rather than the agent). Used for debugging and lecture demos. iNtUpR tDownRtLeftR tRightRtqi(t graphicsUtilsRBtTruet wait_for_keystsysR (RtactionFunctionRoRtkeystactions((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyt getUserAction8s*            cCs |GHdS(N((R((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyt printStringNscCsd}d} |jdt|kr5|jn|dt|dx-tr||j} || ||j| } t| dkr|dt|dt|d|S|| } | dkrdn|j | \} }|d t| d t| d t| d t|dd t|kra|j | | | |n||| 7}| |9} qPWdt|kr|j ndS(Nig?t startEpisodesBEGINNING EPISODE: s sEPISODE s COMPLETE: RETURN WAS s!Error: Agent returned None actionsStarted in state: s Took action: s Ended in state: s Got reward: tobserveTransitiont stopEpisode( R>tdirRxR\RpR?RR^RBRARyRz(tagentt environmenttdiscounttdecisiontdisplaytmessagetpausetepisodetreturnst totalDiscountRRuRRR((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyt runEpisodePs4     &   ?cCsttj}|jddddddddd d d d |jd dddddddd dddd d|jddddddddd dddd dd|jddddddddd dddd d |jd!d"dddddd#d d$ddd d%|jd&d'dddd(dd)d d*dd+d d,|jd-d.dddd(dd/d d0dd+d d1|jd2d3dddd4dd5dd6d d7d d8|jd9d:dd;dd(dd<d d=d d>|jd?d@ddddAdd5ddBd dCd dD|jdEdFddGddHd td dI|jdJdKddGddLd td dM|jdNdOddGddPd td dQ|jdRdSddddTdtddUd dVd dW|jdXdYddGddZd td d[|jd\d]ddGd td d^|j\}}|jr=|jd_kr=d`GHd|_n|j sO|j r[t|_ n|jrpt |_ n|S(aNs-ds --discountRtstoreRRtdestR~tdefaultg?thelps%Discount on future (default %default)s-rs--livingRewardRgtmetavartRs4Reward for living for a time step (default %default)s-ns--noiseRg?tPsHow often action results in s'unintended direction (default %default)s-es --epsilontepsilong333333?tEsAChance of taking a random action in q-learning (default %default)s-ls--learningRatet learningRateg?s#TD learning rate (default %default)s-is --iterationsRtitersi tKs6Number of rounds of value iteration (default %default)s-ks --episodestepisodesis7Number of epsiodes of the MDP to run (default %default)s-gs--gridtGtstringRtBookGridseGrid to use (case sensitive; options are BookGrid, BridgeGrid, CliffGrid, MazeGrid, default %default)s-ws --windowSizetXtgridSizeisERequest a window width of X pixels *per grid cell* (default %default)s-as--agenttAR|RCsDAgent type (options are 'random', 'value' and 'q', default %default)s-ts--textt store_truet textDisplaysUse text-only ASCII displays-ps--pauseRs3Pause GUI after each time step when running the MDPs-qs--quiettquiets%Skip display of any learning episodess-ss--speedR"tspeedg?sQSpeed of animation, S > 1.0 is faster, 0.0 < S < 1.0 is slower (default %default)s-ms--manualtmanualsManually control agents-vs --valueStepss$Display each step of value iterationRns*## Disabling Agents in Manual Mode (-m) ##( toptparset OptionParsert add_optionR7Rt parse_argsRR|RBRRRRp(t optParsertoptstargs((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyt parseOptionsxsr       '   !      t__main__tgetitvalueRncCs tj|S(N(tmdpR(R((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pytstgammatalphaRtactionFnRCi t RandomAgentcBs5eZdZdZdZdZdZRS(cCstjtj|S(N(RCtchoiceRR(RR((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyt getActionscCsdS(Ng((RR((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pytgetValuescCsdS(Ng((RRR((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyt getQValuescCsdS(sDNOTE: 'random' is a special policy value; don't use it in your code.RC((RR((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyt getPolicyscCsdS(N((RRRRR((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pytupdates(R9R:RRRRR(((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyRs     sUnknown agent type: Rs VALUES AFTER s ITERATIONSsQ-VALUES AFTER cCsdS(N(RB(R((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyRscCs tj|S(N(RtdisplayNullValues(R((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyRscCstjt|dS(NsCURRENT VALUES(Rt displayValuesta(R((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyRscCstjt|dS(NsCURRENT VALUES(RRR(R((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyRscCstjt|dS(NsCURRENT Q-VALUES(RtdisplayQValuesR(R((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyRscCs t|S(N(Rw(R((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyR scCsdS(N(RB(R((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyR"scCsdS(N(RB(((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyR%scCs tjS(N(RR(((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyR'scCst|tjS(N(RvRR(R((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyR+stRUNNINGtEPISODESis"AVERAGE RETURNS FROM START STATE: gs EPISODES(((MRCRrRR}R/RtMarkovDecisionProcessRt EnvironmentR<RGRRdReRfRgRhRiRvRwRRR9Rt gridworldtgetattrRt mdpFunctionR RR RtenvttextGridworldDisplaytTextGridworldDisplayRRtgraphicsGridworldDisplaytGraphicsGridworldDisplayRRtstarttKeyboardInterruptR tvalueIterationAgentstqlearningAgentsRBRR|tValueIterationAgentR~Rt gridWorldEnvRRRt qLearnOptstQLearningAgentRRRt valueStepsRRLt tempAgentRR\RRtdisplayCallbackRtmessageCallbackt pauseCallbacktdecisionCallbackRRR(((sSC:\Users\Nicholas\Desktop\CS188X Berkeley\05 Projects\03 Reinforcement\gridworld.pyt s      &0        ( D                  !$ $               ,$ $