{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xCLcAmON-m2i",
    "colab_type": "text"
   },
   "source": [
    "# Tensor2Tensor Reinforcement Learning\n",
    "\n",
    "The `rl` package provides the ability to run model-free and model-based reinforcement learning algorithms.\n",
    "\n",
    "Currently, we support the Proximal Policy Optimization ([PPO](https://arxiv.org/abs/1707.06347)) and Simulated Policy Learning ([SimPLe](https://arxiv.org/abs/1903.00374)).\n",
    "\n",
    "Below you will find examples of PPO training using `trainer_model_free.py` and SimPLe traning using `trainer_model_based.py`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "RW7gEGp3e87G",
    "colab_type": "code",
    "colab": {},
    "cellView": "form"
   },
   "outputs": [],
   "source": [
    "#@title\n",
    "# Copyright 2018 Google LLC.\n",
    "\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "pq0BqXm4-3gJ",
    "colab_type": "code",
    "outputId": "6086719f-6268-4b61-8fa3-d251eda24c97",
    "executionInfo": {
     "status": "ok",
     "timestamp": 1.553273826475E12,
     "user_tz": -60.0,
     "elapsed": 20650.0,
     "user": {
      "displayName": "Piotr Miłoś",
      "photoUrl": "https://lh3.googleusercontent.com/-050ZBEGpNAA/AAAAAAAAAAI/AAAAAAAAk9g/r6cv_J6J5qA/s64/photo.jpg",
      "userId": "12158759908531801397"
     }
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 163.0
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[K    100% |████████████████████████████████| 1.3MB 9.4MB/s \n",
      "\u001b[K    100% |████████████████████████████████| 215kB 27.3MB/s \n",
      "\u001b[K    100% |████████████████████████████████| 143kB 29.6MB/s \n",
      "\u001b[K    100% |████████████████████████████████| 21.1MB 1.7MB/s \n",
      "\u001b[K    100% |████████████████████████████████| 409kB 24.7MB/s \n",
      "\u001b[K    100% |████████████████████████████████| 296kB 25.0MB/s \n",
      "\u001b[K    100% |████████████████████████████████| 61kB 21.5MB/s \n",
      "\u001b[?25h  Building wheel for pypng (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Building wheel for opt-einsum (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h"
     ]
    }
   ],
   "source": [
    "!pip install -q tensorflow==1.13.1\n",
    "!pip install -q tensorflow_probability==0.6.0\n",
    "!pip install -q tensor2tensor==1.13.1\n",
    "!pip install -q gym[atari]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "R7-Ni-39DGZW",
    "colab_type": "code",
    "colab": {}
   },
   "outputs": [],
   "source": [
    "# Helper function for playing videos in the colab.\n",
    "def play_video(path):\n",
    "  from IPython.core.magics.display import HTML\n",
    "  display_path = \"/nbextensions/vid.mp4\"\n",
    "  display_abs_path = \"/usr/local/share/jupyter\" + display_path\n",
    "  !rm -f $display_abs_path\n",
    "  !ffmpeg -loglevel error -i $path $display_abs_path\n",
    "  return HTML(\"\"\"\n",
    "    <video width=\"640\" height=\"480\" controls>\n",
    "      <source src=\"{}\" type=\"video/mp4\">\n",
    "    </video>\n",
    "  \"\"\".format(display_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pueuiKUmAOUT",
    "colab_type": "text"
   },
   "source": [
    "# Play using a pre-trained policy\n",
    "\n",
    "We provide pretrained policies for the following games from the Atari Learning Environment ( [ALE](https://github.com/mgbellemare/Arcade-Learning-Environment)) : alien,\n",
    "amidar,\n",
    " assault,\n",
    " asterix,\n",
    " asteroids,\n",
    " atlantis,\n",
    " bank_heist,\n",
    " battle_zone,\n",
    " beam_rider,\n",
    " bowling,\n",
    " boxing,\n",
    " breakout,\n",
    " chopper_command,\n",
    " crazy_climber,\n",
    " demon_attack,\n",
    " fishing_derby,\n",
    " freeway,\n",
    " frostbite,\n",
    " gopher,\n",
    " gravitar,\n",
    " hero,\n",
    " ice_hockey,\n",
    " jamesbond,\n",
    " kangaroo,\n",
    " krull,\n",
    " kung_fu_master,\n",
    " ms_pacman,\n",
    " name_this_game,\n",
    " pong,\n",
    " private_eye,\n",
    " qbert,\n",
    " riverraid,\n",
    " road_runner,\n",
    " seaquest,\n",
    " up_n_down,\n",
    " yars_revenge.\n",
    " \n",
    " We have 5 checkpoints for each game saved on Google Storage. Run the following command get the storage path:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "x9pKfNbDFfVh",
    "colab_type": "code",
    "outputId": "97e763cc-caaa-49c8-e532-fcbde828d1a2",
    "executionInfo": {
     "status": "ok",
     "timestamp": 1.5532741511E12,
     "user_tz": -60.0,
     "elapsed": 6162.0,
     "user": {
      "displayName": "Piotr Miłoś",
      "photoUrl": "https://lh3.googleusercontent.com/-050ZBEGpNAA/AAAAAAAAAAI/AAAAAAAAk9g/r6cv_J6J5qA/s64/photo.jpg",
      "userId": "12158759908531801397"
     }
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 147.0
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
      "For more information, please see:\n",
      "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
      "  * https://github.com/tensorflow/addons\n",
      "If you depend on functionality not listed there, please file an issue.\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'gs://tensor2tensor-checkpoints/modelrl_experiments/train_sd/143'"
      ]
     },
     "execution_count": 4,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# experiment_id is an integer from [0, 4].\n",
    "def get_run_dir(game, experiment_id):\n",
    "  from tensor2tensor.data_generators.gym_env import ATARI_GAMES_WITH_HUMAN_SCORE_NICE\n",
    "  EXPERIMENTS_PER_GAME = 5\n",
    "  run_id = ATARI_GAMES_WITH_HUMAN_SCORE_NICE.index(game) * EXPERIMENTS_PER_GAME + experiment_id + 1\n",
    "  return \"gs://tensor2tensor-checkpoints/modelrl_experiments/train_sd/{}\".format(run_id)\n",
    "\n",
    "get_run_dir('pong', 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "77fFdm-cFEOB",
    "colab_type": "text"
   },
   "source": [
    "To evaluate and generate videos for a pretrained policy on Pong:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "X-nGlbuTAQXj",
    "colab_type": "code",
    "outputId": "888968f2-f551-4a0f-9fc7-074a949362d6",
    "executionInfo": {
     "status": "ok",
     "timestamp": 1.553271580737E12,
     "user_tz": -60.0,
     "elapsed": 842128.0,
     "user": {
      "displayName": "Piotr Kozakowski",
      "photoUrl": "",
      "userId": "01014928596539690143"
     }
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 17088.0
    },
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
      "For more information, please see:\n",
      "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
      "  * https://github.com/tensorflow/addons\n",
      "If you depend on functionality not listed there, please file an issue.\n",
      "\n",
      "INFO:tensorflow:Overriding hparams in rlmb_long_stochastic_discrete with game=pong,eval_max_num_noops=8,eval_sampling_temps=[0.5]\n",
      "INFO:tensorflow:Evaluating metric mean_reward/eval/sampling_temp_0.5_max_noops_8_unclipped\n",
      "2019-03-22 16:05:45.007030: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz\n",
      "2019-03-22 16:05:45.007306: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x2697860 executing computations on platform Host. Devices:\n",
      "2019-03-22 16:05:45.007346: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>\n",
      "2019-03-22 16:05:45.105281: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2019-03-22 16:05:45.105857: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x2697440 executing computations on platform CUDA. Devices:\n",
      "2019-03-22 16:05:45.105908: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): Tesla K80, Compute Capability 3.7\n",
      "2019-03-22 16:05:45.106380: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: \n",
      "name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235\n",
      "pciBusID: 0000:00:04.0\n",
      "totalMemory: 11.17GiB freeMemory: 11.10GiB\n",
      "2019-03-22 16:05:45.106420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0\n",
      "2019-03-22 16:05:45.499212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
      "2019-03-22 16:05:45.499307: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 \n",
      "2019-03-22 16:05:45.499332: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N \n",
      "2019-03-22 16:05:45.499671: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2019-03-22 16:05:45.499741: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10754 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
      "INFO:tensorflow:Using DummyPolicyProblem for the policy.\n",
      "INFO:tensorflow:Setting T2TModel mode to 'train'\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "INFO:tensorflow:Using variable initializer: orthogonal\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/t2t_model.py:1358: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "INFO:tensorflow:Transforming feature 'input_action' with symbol_modality_6_64.bottom\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/function.py:1007: calling Graph.create_op (from tensorflow.python.framework.ops) with compute_shapes is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Shapes are always computed; don't use the compute_shapes as it has no effect.\n",
      "INFO:tensorflow:Transforming feature 'input_reward' with symbol_modality_3_64.bottom\n",
      "INFO:tensorflow:Transforming feature 'inputs' with video_modality.bottom\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_video.py:495: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "tf.py_func is deprecated in TF V2. Instead, use\n",
      "    tf.py_function, which takes a python function which manipulates tf eager\n",
      "    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to\n",
      "    an ndarray (just call tensor.numpy()) but having access to eager tensors\n",
      "    means `tf.py_function`s can use accelerators such as GPUs as well as\n",
      "    being differentiable using a gradient tape.\n",
      "    \n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_layers.py:277: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "INFO:tensorflow:Transforming feature 'target_action' with symbol_modality_6_64.targets_bottom\n",
      "INFO:tensorflow:Transforming feature 'target_policy' with identity_modality.targets_bottom\n",
      "INFO:tensorflow:Transforming feature 'target_reward' with symbol_modality_3_64.targets_bottom\n",
      "INFO:tensorflow:Transforming feature 'target_value' with identity_modality.targets_bottom\n",
      "INFO:tensorflow:Transforming feature 'targets' with video_modality.targets_bottom\n",
      "INFO:tensorflow:Building model body\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/rl.py:598: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.conv2d instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/rl.py:602: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.flatten instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/rl.py:603: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.dropout instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/rl.py:604: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.dense instead.\n",
      "INFO:tensorflow:Transforming body output with identity_modality.top\n",
      "INFO:tensorflow:Transforming body output with identity_modality.top\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_layers.py:2887: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.random.categorical instead.\n",
      "2019-03-22 16:06:00.352605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0\n",
      "2019-03-22 16:06:00.352688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
      "2019-03-22 16:06:00.352724: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 \n",
      "2019-03-22 16:06:00.352744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N \n",
      "2019-03-22 16:06:00.353037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10754 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
      "2019-03-22 16:06:00.588787: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n",
      "2019-03-22 16:06:00.647797: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n",
      "INFO:tensorflow:Restoring checkpoint gs://tensor2tensor-checkpoints/modelrl_experiments/train_sd/142/policy/model.ckpt-171992\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n",
      "2019-03-22 16:06:00.711910: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n",
      "INFO:tensorflow:Restoring parameters from gs://tensor2tensor-checkpoints/modelrl_experiments/train_sd/142/policy/model.ckpt-171992\n",
      "2019-03-22 16:06:00.793701: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n",
      "2019-03-22 16:06:00.953239: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n",
      "2019-03-22 16:06:01.086594: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n",
      "2019-03-22 16:06:01.259521: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n",
      "2019-03-22 16:06:01.322896: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n",
      "2019-03-22 16:06:03.034751: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally\n",
      "INFO:tensorflow:Step 5, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 10, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 15, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 20, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 25, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 30, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 35, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 40, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 45, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 50, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 55, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 60, mean_score: -0.015625\n",
      "INFO:tensorflow:Step 65, mean_score: -0.078125\n",
      "INFO:tensorflow:Step 70, mean_score: -0.078125\n",
      "INFO:tensorflow:Step 75, mean_score: -0.078125\n",
      "INFO:tensorflow:Step 80, mean_score: -0.078125\n",
      "INFO:tensorflow:Step 85, mean_score: -0.078125\n",
      "INFO:tensorflow:Step 90, mean_score: 0.484375\n",
      "INFO:tensorflow:Step 95, mean_score: 0.843750\n",
      "INFO:tensorflow:Step 100, mean_score: 0.828125\n",
      "INFO:tensorflow:Step 105, mean_score: 0.828125\n",
      "INFO:tensorflow:Step 110, mean_score: 0.828125\n",
      "INFO:tensorflow:Step 115, mean_score: 0.828125\n",
      "INFO:tensorflow:Step 120, mean_score: 0.828125\n",
      "INFO:tensorflow:Step 125, mean_score: 0.828125\n",
      "INFO:tensorflow:Step 130, mean_score: 0.812500\n",
      "INFO:tensorflow:Step 135, mean_score: 0.812500\n",
      "INFO:tensorflow:Step 140, mean_score: 0.812500\n",
      "INFO:tensorflow:Step 145, mean_score: 0.812500\n",
      "INFO:tensorflow:Step 150, mean_score: 0.812500\n",
      "INFO:tensorflow:Step 155, mean_score: 0.812500\n",
      "INFO:tensorflow:Step 160, mean_score: 0.812500\n",
      "INFO:tensorflow:Step 165, mean_score: 0.812500\n",
      "INFO:tensorflow:Step 170, mean_score: 0.828125\n",
      "INFO:tensorflow:Step 175, mean_score: 0.843750\n",
      "INFO:tensorflow:Step 180, mean_score: 0.843750\n",
      "INFO:tensorflow:Step 185, mean_score: 0.843750\n",
      "INFO:tensorflow:Step 190, mean_score: 1.140625\n",
      "INFO:tensorflow:Step 195, mean_score: 1.765625\n",
      "INFO:tensorflow:Step 200, mean_score: 1.765625\n",
      "INFO:tensorflow:Step 205, mean_score: 1.765625\n",
      "INFO:tensorflow:Step 210, mean_score: 1.781250\n",
      "INFO:tensorflow:Step 215, mean_score: 1.781250\n",
      "INFO:tensorflow:Step 220, mean_score: 1.765625\n",
      "INFO:tensorflow:Step 225, mean_score: 1.765625\n",
      "INFO:tensorflow:Step 230, mean_score: 1.765625\n",
      "INFO:tensorflow:Step 235, mean_score: 1.765625\n",
      "INFO:tensorflow:Step 240, mean_score: 1.765625\n",
      "INFO:tensorflow:Step 245, mean_score: 1.765625\n",
      "INFO:tensorflow:Step 250, mean_score: 1.765625\n",
      "INFO:tensorflow:Step 255, mean_score: 1.750000\n",
      "INFO:tensorflow:Step 260, mean_score: 1.750000\n",
      "INFO:tensorflow:Step 265, mean_score: 1.750000\n",
      "INFO:tensorflow:Step 270, mean_score: 2.312500\n",
      "INFO:tensorflow:Step 275, mean_score: 2.687500\n",
      "INFO:tensorflow:Step 280, mean_score: 2.703125\n",
      "INFO:tensorflow:Step 285, mean_score: 2.703125\n",
      "INFO:tensorflow:Step 290, mean_score: 2.703125\n",
      "INFO:tensorflow:Step 295, mean_score: 2.703125\n",
      "INFO:tensorflow:Step 300, mean_score: 2.703125\n",
      "INFO:tensorflow:Step 305, mean_score: 2.703125\n",
      "INFO:tensorflow:Step 310, mean_score: 2.718750\n",
      "INFO:tensorflow:Step 315, mean_score: 2.718750\n",
      "INFO:tensorflow:Step 320, mean_score: 2.718750\n",
      "INFO:tensorflow:Step 325, mean_score: 2.718750\n",
      "INFO:tensorflow:Step 330, mean_score: 2.734375\n",
      "INFO:tensorflow:Step 335, mean_score: 2.734375\n",
      "INFO:tensorflow:Step 340, mean_score: 2.734375\n",
      "INFO:tensorflow:Step 345, mean_score: 2.734375\n",
      "INFO:tensorflow:Step 350, mean_score: 2.750000\n",
      "INFO:tensorflow:Step 355, mean_score: 2.765625\n",
      "INFO:tensorflow:Step 360, mean_score: 2.765625\n",
      "INFO:tensorflow:Step 365, mean_score: 2.765625\n",
      "INFO:tensorflow:Step 370, mean_score: 3.062500\n",
      "INFO:tensorflow:Step 375, mean_score: 3.687500\n",
      "INFO:tensorflow:Step 380, mean_score: 3.687500\n",
      "INFO:tensorflow:Step 385, mean_score: 3.687500\n",
      "INFO:tensorflow:Step 390, mean_score: 3.703125\n",
      "INFO:tensorflow:Step 395, mean_score: 3.703125\n",
      "INFO:tensorflow:Step 400, mean_score: 3.703125\n",
      "INFO:tensorflow:Step 405, mean_score: 3.703125\n",
      "INFO:tensorflow:Step 410, mean_score: 3.687500\n",
      "INFO:tensorflow:Step 415, mean_score: 3.687500\n",
      "INFO:tensorflow:Step 420, mean_score: 3.687500\n",
      "INFO:tensorflow:Step 425, mean_score: 3.687500\n",
      "INFO:tensorflow:Step 430, mean_score: 3.703125\n",
      "INFO:tensorflow:Step 435, mean_score: 3.703125\n",
      "INFO:tensorflow:Step 440, mean_score: 3.703125\n",
      "INFO:tensorflow:Step 445, mean_score: 3.703125\n",
      "INFO:tensorflow:Step 450, mean_score: 4.265625\n",
      "INFO:tensorflow:Step 455, mean_score: 4.640625\n",
      "INFO:tensorflow:Step 460, mean_score: 4.656250\n",
      "INFO:tensorflow:Step 465, mean_score: 4.656250\n",
      "INFO:tensorflow:Step 470, mean_score: 4.656250\n",
      "INFO:tensorflow:Step 475, mean_score: 4.656250\n",
      "INFO:tensorflow:Step 480, mean_score: 4.656250\n",
      "INFO:tensorflow:Step 485, mean_score: 4.656250\n",
      "INFO:tensorflow:Step 490, mean_score: 4.671875\n",
      "INFO:tensorflow:Step 495, mean_score: 4.671875\n",
      "INFO:tensorflow:Step 500, mean_score: 4.671875\n",
      "INFO:tensorflow:Step 505, mean_score: 4.671875\n",
      "INFO:tensorflow:Step 510, mean_score: 4.687500\n",
      "INFO:tensorflow:Step 515, mean_score: 4.687500\n",
      "INFO:tensorflow:Step 520, mean_score: 4.703125\n",
      "INFO:tensorflow:Step 525, mean_score: 4.703125\n",
      "INFO:tensorflow:Step 530, mean_score: 4.718750\n",
      "INFO:tensorflow:Step 535, mean_score: 4.734375\n",
      "INFO:tensorflow:Step 540, mean_score: 4.734375\n",
      "INFO:tensorflow:Step 545, mean_score: 4.734375\n",
      "INFO:tensorflow:Step 550, mean_score: 5.031250\n",
      "INFO:tensorflow:Step 555, mean_score: 5.656250\n",
      "INFO:tensorflow:Step 560, mean_score: 5.656250\n",
      "INFO:tensorflow:Step 565, mean_score: 5.656250\n",
      "INFO:tensorflow:Step 570, mean_score: 5.671875\n",
      "INFO:tensorflow:Step 575, mean_score: 5.671875\n",
      "INFO:tensorflow:Step 580, mean_score: 5.671875\n",
      "INFO:tensorflow:Step 585, mean_score: 5.671875\n",
      "INFO:tensorflow:Step 590, mean_score: 5.671875\n",
      "INFO:tensorflow:Step 595, mean_score: 5.671875\n",
      "INFO:tensorflow:Step 600, mean_score: 5.671875\n",
      "INFO:tensorflow:Step 605, mean_score: 5.671875\n",
      "INFO:tensorflow:Step 610, mean_score: 5.687500\n",
      "INFO:tensorflow:Step 615, mean_score: 5.687500\n",
      "INFO:tensorflow:Step 620, mean_score: 5.703125\n",
      "INFO:tensorflow:Step 625, mean_score: 5.703125\n",
      "INFO:tensorflow:Step 630, mean_score: 6.265625\n",
      "INFO:tensorflow:Step 635, mean_score: 6.640625\n",
      "INFO:tensorflow:Step 640, mean_score: 6.656250\n",
      "INFO:tensorflow:Step 645, mean_score: 6.656250\n",
      "INFO:tensorflow:Step 650, mean_score: 6.656250\n",
      "INFO:tensorflow:Step 655, mean_score: 6.656250\n",
      "INFO:tensorflow:Step 660, mean_score: 6.656250\n",
      "INFO:tensorflow:Step 665, mean_score: 6.656250\n",
      "INFO:tensorflow:Step 670, mean_score: 6.671875\n",
      "INFO:tensorflow:Step 675, mean_score: 6.671875\n",
      "INFO:tensorflow:Step 680, mean_score: 6.671875\n",
      "INFO:tensorflow:Step 685, mean_score: 6.671875\n",
      "INFO:tensorflow:Step 690, mean_score: 6.687500\n",
      "INFO:tensorflow:Step 695, mean_score: 6.687500\n",
      "INFO:tensorflow:Step 700, mean_score: 6.703125\n",
      "INFO:tensorflow:Step 705, mean_score: 6.703125\n",
      "INFO:tensorflow:Step 710, mean_score: 6.718750\n",
      "INFO:tensorflow:Step 715, mean_score: 6.734375\n",
      "INFO:tensorflow:Step 720, mean_score: 6.734375\n",
      "INFO:tensorflow:Step 725, mean_score: 6.734375\n",
      "INFO:tensorflow:Step 730, mean_score: 7.031250\n",
      "INFO:tensorflow:Step 735, mean_score: 7.656250\n",
      "INFO:tensorflow:Step 740, mean_score: 7.656250\n",
      "INFO:tensorflow:Step 745, mean_score: 7.656250\n",
      "INFO:tensorflow:Step 750, mean_score: 7.671875\n",
      "INFO:tensorflow:Step 755, mean_score: 7.671875\n",
      "INFO:tensorflow:Step 760, mean_score: 7.671875\n",
      "INFO:tensorflow:Step 765, mean_score: 7.671875\n",
      "INFO:tensorflow:Step 770, mean_score: 7.671875\n",
      "INFO:tensorflow:Step 775, mean_score: 7.671875\n",
      "INFO:tensorflow:Step 780, mean_score: 7.671875\n",
      "INFO:tensorflow:Step 785, mean_score: 7.671875\n",
      "INFO:tensorflow:Step 790, mean_score: 7.687500\n",
      "INFO:tensorflow:Step 795, mean_score: 7.687500\n",
      "INFO:tensorflow:Step 800, mean_score: 7.703125\n",
      "INFO:tensorflow:Step 805, mean_score: 7.703125\n",
      "INFO:tensorflow:Step 810, mean_score: 8.265625\n",
      "INFO:tensorflow:Step 815, mean_score: 8.640625\n",
      "INFO:tensorflow:Step 820, mean_score: 8.656250\n",
      "INFO:tensorflow:Step 825, mean_score: 8.656250\n",
      "INFO:tensorflow:Step 830, mean_score: 8.656250\n",
      "INFO:tensorflow:Step 835, mean_score: 8.656250\n",
      "INFO:tensorflow:Step 840, mean_score: 8.656250\n",
      "INFO:tensorflow:Step 845, mean_score: 8.656250\n",
      "INFO:tensorflow:Step 850, mean_score: 8.671875\n",
      "INFO:tensorflow:Step 855, mean_score: 8.671875\n",
      "INFO:tensorflow:Step 860, mean_score: 8.671875\n",
      "INFO:tensorflow:Step 865, mean_score: 8.671875\n",
      "INFO:tensorflow:Step 870, mean_score: 8.687500\n",
      "INFO:tensorflow:Step 875, mean_score: 8.687500\n",
      "INFO:tensorflow:Step 880, mean_score: 8.703125\n",
      "INFO:tensorflow:Step 885, mean_score: 8.703125\n",
      "INFO:tensorflow:Step 890, mean_score: 8.718750\n",
      "INFO:tensorflow:Step 895, mean_score: 8.734375\n",
      "INFO:tensorflow:Step 900, mean_score: 8.734375\n",
      "INFO:tensorflow:Step 905, mean_score: 8.734375\n",
      "INFO:tensorflow:Step 910, mean_score: 9.031250\n",
      "INFO:tensorflow:Step 915, mean_score: 9.656250\n",
      "INFO:tensorflow:Step 920, mean_score: 9.656250\n",
      "INFO:tensorflow:Step 925, mean_score: 9.656250\n",
      "INFO:tensorflow:Step 930, mean_score: 9.671875\n",
      "INFO:tensorflow:Step 935, mean_score: 9.671875\n",
      "INFO:tensorflow:Step 940, mean_score: 9.671875\n",
      "INFO:tensorflow:Step 945, mean_score: 9.671875\n",
      "INFO:tensorflow:Step 950, mean_score: 9.671875\n",
      "INFO:tensorflow:Step 955, mean_score: 9.671875\n",
      "INFO:tensorflow:Step 960, mean_score: 9.671875\n",
      "INFO:tensorflow:Step 965, mean_score: 9.671875\n",
      "INFO:tensorflow:Step 970, mean_score: 9.687500\n",
      "INFO:tensorflow:Step 975, mean_score: 9.687500\n",
      "INFO:tensorflow:Step 980, mean_score: 9.703125\n",
      "INFO:tensorflow:Step 985, mean_score: 9.703125\n",
      "INFO:tensorflow:Step 990, mean_score: 10.265625\n",
      "INFO:tensorflow:Step 995, mean_score: 10.640625\n",
      "INFO:tensorflow:Step 1000, mean_score: 10.656250\n",
      "INFO:tensorflow:Step 1005, mean_score: 10.656250\n",
      "INFO:tensorflow:Step 1010, mean_score: 10.656250\n",
      "INFO:tensorflow:Step 1015, mean_score: 10.656250\n",
      "INFO:tensorflow:Step 1020, mean_score: 10.656250\n",
      "INFO:tensorflow:Step 1025, mean_score: 10.656250\n",
      "INFO:tensorflow:Step 1030, mean_score: 10.671875\n",
      "INFO:tensorflow:Step 1035, mean_score: 10.671875\n",
      "INFO:tensorflow:Step 1040, mean_score: 10.671875\n",
      "INFO:tensorflow:Step 1045, mean_score: 10.671875\n",
      "INFO:tensorflow:Step 1050, mean_score: 10.687500\n",
      "INFO:tensorflow:Step 1055, mean_score: 10.687500\n",
      "INFO:tensorflow:Step 1060, mean_score: 10.703125\n",
      "INFO:tensorflow:Step 1065, mean_score: 10.703125\n",
      "INFO:tensorflow:Step 1070, mean_score: 10.718750\n",
      "INFO:tensorflow:Step 1075, mean_score: 10.734375\n",
      "INFO:tensorflow:Step 1080, mean_score: 10.734375\n",
      "INFO:tensorflow:Step 1085, mean_score: 10.734375\n",
      "INFO:tensorflow:Step 1090, mean_score: 11.031250\n",
      "INFO:tensorflow:Step 1095, mean_score: 11.656250\n",
      "INFO:tensorflow:Step 1100, mean_score: 11.656250\n",
      "INFO:tensorflow:Step 1105, mean_score: 11.656250\n",
      "INFO:tensorflow:Step 1110, mean_score: 11.671875\n",
      "INFO:tensorflow:Step 1115, mean_score: 11.671875\n",
      "INFO:tensorflow:Step 1120, mean_score: 11.671875\n",
      "INFO:tensorflow:Step 1125, mean_score: 11.671875\n",
      "INFO:tensorflow:Step 1130, mean_score: 11.671875\n",
      "INFO:tensorflow:Step 1135, mean_score: 11.671875\n",
      "INFO:tensorflow:Step 1140, mean_score: 11.671875\n",
      "INFO:tensorflow:Step 1145, mean_score: 11.671875\n",
      "INFO:tensorflow:Step 1150, mean_score: 11.687500\n",
      "INFO:tensorflow:Step 1155, mean_score: 11.687500\n",
      "INFO:tensorflow:Step 1160, mean_score: 11.703125\n",
      "INFO:tensorflow:Step 1165, mean_score: 11.703125\n",
      "INFO:tensorflow:Step 1170, mean_score: 12.265625\n",
      "INFO:tensorflow:Step 1175, mean_score: 12.640625\n",
      "INFO:tensorflow:Step 1180, mean_score: 12.656250\n",
      "INFO:tensorflow:Step 1185, mean_score: 12.656250\n",
      "INFO:tensorflow:Step 1190, mean_score: 12.656250\n",
      "INFO:tensorflow:Step 1195, mean_score: 12.656250\n",
      "INFO:tensorflow:Step 1200, mean_score: 12.656250\n",
      "INFO:tensorflow:Step 1205, mean_score: 12.656250\n",
      "INFO:tensorflow:Step 1210, mean_score: 12.671875\n",
      "INFO:tensorflow:Step 1215, mean_score: 12.671875\n",
      "INFO:tensorflow:Step 1220, mean_score: 12.671875\n",
      "INFO:tensorflow:Step 1225, mean_score: 12.671875\n",
      "INFO:tensorflow:Step 1230, mean_score: 12.687500\n",
      "INFO:tensorflow:Step 1235, mean_score: 12.687500\n",
      "INFO:tensorflow:Step 1240, mean_score: 12.703125\n",
      "INFO:tensorflow:Step 1245, mean_score: 12.703125\n",
      "INFO:tensorflow:Step 1250, mean_score: 12.718750\n",
      "INFO:tensorflow:Step 1255, mean_score: 12.734375\n",
      "INFO:tensorflow:Step 1260, mean_score: 12.734375\n",
      "INFO:tensorflow:Step 1265, mean_score: 12.734375\n",
      "INFO:tensorflow:Step 1270, mean_score: 13.031250\n",
      "INFO:tensorflow:Step 1275, mean_score: 13.656250\n",
      "INFO:tensorflow:Step 1280, mean_score: 13.656250\n",
      "INFO:tensorflow:Step 1285, mean_score: 13.656250\n",
      "INFO:tensorflow:Step 1290, mean_score: 13.671875\n",
      "INFO:tensorflow:Step 1295, mean_score: 13.671875\n",
      "INFO:tensorflow:Step 1300, mean_score: 13.671875\n",
      "INFO:tensorflow:Step 1305, mean_score: 13.671875\n",
      "INFO:tensorflow:Step 1310, mean_score: 13.671875\n",
      "INFO:tensorflow:Step 1315, mean_score: 13.671875\n",
      "INFO:tensorflow:Step 1320, mean_score: 13.671875\n",
      "INFO:tensorflow:Step 1325, mean_score: 13.671875\n",
      "INFO:tensorflow:Step 1330, mean_score: 13.687500\n",
      "INFO:tensorflow:Step 1335, mean_score: 13.687500\n",
      "INFO:tensorflow:Step 1340, mean_score: 13.703125\n",
      "INFO:tensorflow:Step 1345, mean_score: 13.703125\n",
      "INFO:tensorflow:Step 1350, mean_score: 14.265625\n",
      "INFO:tensorflow:Step 1355, mean_score: 14.640625\n",
      "INFO:tensorflow:Step 1360, mean_score: 14.656250\n",
      "INFO:tensorflow:Step 1365, mean_score: 14.656250\n",
      "INFO:tensorflow:Step 1370, mean_score: 14.656250\n",
      "INFO:tensorflow:Step 1375, mean_score: 14.656250\n",
      "INFO:tensorflow:Step 1380, mean_score: 14.656250\n",
      "INFO:tensorflow:Step 1385, mean_score: 14.656250\n",
      "INFO:tensorflow:Step 1390, mean_score: 14.671875\n",
      "INFO:tensorflow:Step 1395, mean_score: 14.671875\n",
      "INFO:tensorflow:Step 1400, mean_score: 14.671875\n",
      "INFO:tensorflow:Step 1405, mean_score: 14.671875\n",
      "INFO:tensorflow:Step 1410, mean_score: 14.687500\n",
      "INFO:tensorflow:Step 1415, mean_score: 14.687500\n",
      "INFO:tensorflow:Step 1420, mean_score: 14.703125\n",
      "INFO:tensorflow:Step 1425, mean_score: 14.703125\n",
      "INFO:tensorflow:Step 1430, mean_score: 14.718750\n",
      "INFO:tensorflow:Step 1435, mean_score: 14.734375\n",
      "INFO:tensorflow:Step 1440, mean_score: 14.734375\n",
      "INFO:tensorflow:Step 1445, mean_score: 14.734375\n",
      "INFO:tensorflow:Step 1450, mean_score: 15.031250\n",
      "INFO:tensorflow:Step 1455, mean_score: 15.656250\n",
      "INFO:tensorflow:Step 1460, mean_score: 15.656250\n",
      "INFO:tensorflow:Step 1465, mean_score: 15.656250\n",
      "INFO:tensorflow:Step 1470, mean_score: 15.671875\n",
      "INFO:tensorflow:Step 1475, mean_score: 15.671875\n",
      "INFO:tensorflow:Step 1480, mean_score: 15.671875\n",
      "INFO:tensorflow:Step 1485, mean_score: 15.671875\n",
      "INFO:tensorflow:Step 1490, mean_score: 15.671875\n",
      "INFO:tensorflow:Step 1495, mean_score: 15.671875\n",
      "INFO:tensorflow:Step 1500, mean_score: 15.671875\n",
      "INFO:tensorflow:Step 1505, mean_score: 15.671875\n",
      "INFO:tensorflow:Step 1510, mean_score: 15.687500\n",
      "INFO:tensorflow:Step 1515, mean_score: 15.687500\n",
      "INFO:tensorflow:Step 1520, mean_score: 15.703125\n",
      "INFO:tensorflow:Step 1525, mean_score: 15.703125\n",
      "INFO:tensorflow:Step 1530, mean_score: 16.265625\n",
      "INFO:tensorflow:Step 1535, mean_score: 16.640625\n",
      "INFO:tensorflow:Step 1540, mean_score: 16.656250\n",
      "INFO:tensorflow:Step 1545, mean_score: 16.656250\n",
      "INFO:tensorflow:Step 1550, mean_score: 16.656250\n",
      "INFO:tensorflow:Step 1555, mean_score: 16.656250\n",
      "INFO:tensorflow:Step 1560, mean_score: 16.656250\n",
      "INFO:tensorflow:Step 1565, mean_score: 16.656250\n",
      "INFO:tensorflow:Step 1570, mean_score: 16.671875\n",
      "INFO:tensorflow:Step 1575, mean_score: 16.671875\n",
      "INFO:tensorflow:Step 1580, mean_score: 16.671875\n",
      "INFO:tensorflow:Step 1585, mean_score: 16.671875\n",
      "INFO:tensorflow:Step 1590, mean_score: 16.687500\n",
      "INFO:tensorflow:Step 1595, mean_score: 16.687500\n",
      "INFO:tensorflow:Step 1600, mean_score: 16.703125\n",
      "INFO:tensorflow:Step 1605, mean_score: 16.703125\n",
      "INFO:tensorflow:Step 1610, mean_score: 16.718750\n",
      "INFO:tensorflow:Step 1615, mean_score: 16.734375\n",
      "INFO:tensorflow:Step 1620, mean_score: 16.734375\n",
      "INFO:tensorflow:Step 1625, mean_score: 16.734375\n",
      "INFO:tensorflow:Step 1630, mean_score: 17.031250\n",
      "INFO:tensorflow:Step 1635, mean_score: 17.656250\n",
      "INFO:tensorflow:Step 1640, mean_score: 17.656250\n",
      "INFO:tensorflow:Step 1645, mean_score: 17.656250\n",
      "INFO:tensorflow:Step 1650, mean_score: 17.671875\n",
      "INFO:tensorflow:Step 1655, mean_score: 17.671875\n",
      "INFO:tensorflow:Step 1660, mean_score: 17.671875\n",
      "INFO:tensorflow:Step 1665, mean_score: 17.671875\n",
      "INFO:tensorflow:Step 1670, mean_score: 17.671875\n",
      "INFO:tensorflow:Step 1675, mean_score: 17.671875\n",
      "INFO:tensorflow:Step 1680, mean_score: 17.671875\n",
      "INFO:tensorflow:Step 1685, mean_score: 17.671875\n",
      "INFO:tensorflow:Step 1690, mean_score: 17.687500\n",
      "INFO:tensorflow:Step 1695, mean_score: 17.687500\n",
      "INFO:tensorflow:Step 1700, mean_score: 17.703125\n",
      "INFO:tensorflow:Step 1705, mean_score: 17.703125\n",
      "INFO:tensorflow:Step 1710, mean_score: 18.265625\n",
      "INFO:tensorflow:Step 1715, mean_score: 18.640625\n",
      "INFO:tensorflow:Step 1720, mean_score: 18.656250\n",
      "INFO:tensorflow:Step 1725, mean_score: 18.656250\n",
      "INFO:tensorflow:Step 1730, mean_score: 18.656250\n",
      "INFO:tensorflow:Step 1735, mean_score: 18.656250\n",
      "INFO:tensorflow:Step 1740, mean_score: 18.656250\n",
      "INFO:tensorflow:Step 1745, mean_score: 18.656250\n",
      "INFO:tensorflow:Step 1750, mean_score: 18.671875\n",
      "INFO:tensorflow:Step 1755, mean_score: 18.671875\n",
      "INFO:tensorflow:Step 1760, mean_score: 18.671875\n",
      "INFO:tensorflow:Step 1765, mean_score: 18.671875\n",
      "INFO:tensorflow:Step 1770, mean_score: 18.687500\n",
      "INFO:tensorflow:Step 1775, mean_score: 18.687500\n",
      "INFO:tensorflow:Step 1780, mean_score: 18.703125\n",
      "INFO:tensorflow:Step 1785, mean_score: 18.703125\n",
      "INFO:tensorflow:Step 1790, mean_score: 18.718750\n",
      "INFO:tensorflow:Step 1795, mean_score: 18.734375\n",
      "INFO:tensorflow:Step 1800, mean_score: 18.734375\n",
      "INFO:tensorflow:Step 1805, mean_score: 18.734375\n",
      "INFO:tensorflow:Step 1810, mean_score: 19.031250\n",
      "INFO:tensorflow:Step 1815, mean_score: 19.656250\n",
      "INFO:tensorflow:Step 1820, mean_score: 19.656250\n",
      "INFO:tensorflow:Step 1825, mean_score: 19.656250\n",
      "INFO:tensorflow:Step 1830, mean_score: 19.671875\n",
      "INFO:tensorflow:Step 1835, mean_score: 19.671875\n",
      "INFO:tensorflow:Step 1840, mean_score: 19.671875\n",
      "INFO:tensorflow:Step 1845, mean_score: 19.671875\n",
      "INFO:tensorflow:Step 1850, mean_score: 19.671875\n",
      "INFO:tensorflow:Step 1855, mean_score: 19.671875\n",
      "INFO:tensorflow:Step 1860, mean_score: 19.671875\n",
      "INFO:tensorflow:Step 1865, mean_score: 19.671875\n",
      "INFO:tensorflow:Step 1870, mean_score: 19.687500\n",
      "INFO:tensorflow:Step 1875, mean_score: 19.687500\n",
      "INFO:tensorflow:Step 1880, mean_score: 19.703125\n",
      "INFO:tensorflow:Step 1885, mean_score: 19.703125\n",
      "INFO:tensorflow:Step 1890, mean_score: 19.703125\n",
      "INFO:tensorflow:Step 1895, mean_score: 19.718750\n",
      "INFO:tensorflow:Step 1900, mean_score: 19.734375\n",
      "INFO:tensorflow:Step 1905, mean_score: 19.734375\n",
      "INFO:tensorflow:Step 1910, mean_score: 19.734375\n",
      "INFO:tensorflow:Step 1915, mean_score: 19.734375\n",
      "INFO:tensorflow:Step 1920, mean_score: 19.734375\n",
      "INFO:tensorflow:Step 1925, mean_score: 19.734375\n",
      "INFO:tensorflow:Step 1930, mean_score: 19.750000\n",
      "INFO:tensorflow:Step 1935, mean_score: 19.750000\n",
      "INFO:tensorflow:Step 1940, mean_score: 19.750000\n",
      "INFO:tensorflow:Step 1945, mean_score: 19.750000\n",
      "INFO:tensorflow:Step 1950, mean_score: 19.765625\n",
      "INFO:tensorflow:Step 1955, mean_score: 19.765625\n",
      "INFO:tensorflow:Step 1960, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 1965, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 1970, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 1975, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 1980, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 1985, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 1990, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 1995, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 2000, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 2005, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 2010, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 2015, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 2020, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 2025, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 2030, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 2035, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 2040, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 2045, mean_score: 19.781250\n",
      "INFO:tensorflow:Step 2050, mean_score: 19.796875\n",
      "INFO:tensorflow:Step 2055, mean_score: 19.796875\n",
      "INFO:tensorflow:Step 2060, mean_score: 19.812500\n",
      "INFO:tensorflow:Step 2065, mean_score: 19.812500\n",
      "INFO:tensorflow:Step 2070, mean_score: 19.812500\n",
      "INFO:tensorflow:Step 2075, mean_score: 19.812500\n",
      "INFO:tensorflow:Step 2080, mean_score: 19.812500\n",
      "INFO:tensorflow:Step 2085, mean_score: 19.812500\n",
      "INFO:tensorflow:Step 2090, mean_score: 19.812500\n",
      "INFO:tensorflow:Step 2095, mean_score: 19.812500\n",
      "INFO:tensorflow:Step 2100, mean_score: 19.812500\n",
      "INFO:tensorflow:Step 2105, mean_score: 19.812500\n",
      "INFO:tensorflow:Step 2110, mean_score: 19.812500\n",
      "INFO:tensorflow:Step 2115, mean_score: 19.812500\n",
      "INFO:tensorflow:Step 2120, mean_score: 19.812500\n",
      "INFO:tensorflow:Step 2125, mean_score: 19.812500\n",
      "INFO:tensorflow:Step 2130, mean_score: 19.812500\n",
      "INFO:tensorflow:Step 2135, mean_score: 19.812500\n",
      "INFO:tensorflow:Step 2140, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2145, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2150, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2155, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2160, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2165, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2170, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2175, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2180, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2185, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2190, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2195, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2200, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2205, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2210, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2215, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2220, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2225, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2230, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2235, mean_score: 19.828125\n",
      "INFO:tensorflow:Step 2240, mean_score: 19.843750\n",
      "INFO:tensorflow:Step 2245, mean_score: 19.843750\n",
      "INFO:tensorflow:Step 2250, mean_score: 19.843750\n",
      "INFO:tensorflow:Step 2255, mean_score: 19.843750\n",
      "INFO:tensorflow:Step 2260, mean_score: 19.843750\n",
      "INFO:tensorflow:Step 2265, mean_score: 19.843750\n",
      "INFO:tensorflow:Step 2270, mean_score: 19.843750\n",
      "INFO:tensorflow:Step 2275, mean_score: 19.843750\n",
      "INFO:tensorflow:Step 2280, mean_score: 19.843750\n",
      "INFO:tensorflow:Step 2285, mean_score: 19.843750\n",
      "INFO:tensorflow:Step 2290, mean_score: 19.843750\n",
      "INFO:tensorflow:Step 2295, mean_score: 19.843750\n",
      "INFO:tensorflow:Step 2300, mean_score: 19.843750\n",
      "INFO:tensorflow:Step 2305, mean_score: 19.843750\n",
      "INFO:tensorflow:Step 2310, mean_score: 19.843750\n",
      "INFO:tensorflow:Step 2315, mean_score: 19.843750\n",
      "INFO:tensorflow:Evaluating metric mean_reward/eval/sampling_temp_0.5_max_noops_0_unclipped\n",
      "2019-03-22 16:12:57.935045: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0\n",
      "2019-03-22 16:12:57.935160: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
      "2019-03-22 16:12:57.935189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 \n",
      "2019-03-22 16:12:57.935209: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N \n",
      "2019-03-22 16:12:57.935553: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10754 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
      "INFO:tensorflow:Using DummyPolicyProblem for the policy.\n",
      "INFO:tensorflow:Setting T2TModel mode to 'train'\n",
      "INFO:tensorflow:Using variable initializer: orthogonal\n",
      "INFO:tensorflow:Transforming feature 'input_action' with symbol_modality_6_64.bottom\n",
      "INFO:tensorflow:Transforming feature 'input_reward' with symbol_modality_3_64.bottom\n",
      "INFO:tensorflow:Transforming feature 'inputs' with video_modality.bottom\n",
      "INFO:tensorflow:Transforming feature 'target_action' with symbol_modality_6_64.targets_bottom\n",
      "INFO:tensorflow:Transforming feature 'target_policy' with identity_modality.targets_bottom\n",
      "INFO:tensorflow:Transforming feature 'target_reward' with symbol_modality_3_64.targets_bottom\n",
      "INFO:tensorflow:Transforming feature 'target_value' with identity_modality.targets_bottom\n",
      "INFO:tensorflow:Transforming feature 'targets' with video_modality.targets_bottom\n",
      "INFO:tensorflow:Building model body\n",
      "INFO:tensorflow:Transforming body output with identity_modality.top\n",
      "INFO:tensorflow:Transforming body output with identity_modality.top\n",
      "2019-03-22 16:13:12.260846: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0\n",
      "2019-03-22 16:13:12.260981: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
      "2019-03-22 16:13:12.261059: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 \n",
      "2019-03-22 16:13:12.261099: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N \n",
      "2019-03-22 16:13:12.261613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10754 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
      "2019-03-22 16:13:12.493082: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n",
      "INFO:tensorflow:Restoring checkpoint gs://tensor2tensor-checkpoints/modelrl_experiments/train_sd/142/policy/model.ckpt-171992\n",
      "2019-03-22 16:13:12.556955: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n",
      "INFO:tensorflow:Restoring parameters from gs://tensor2tensor-checkpoints/modelrl_experiments/train_sd/142/policy/model.ckpt-171992\n",
      "2019-03-22 16:13:12.651009: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n",
      "2019-03-22 16:13:12.715180: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n",
      "2019-03-22 16:13:12.816774: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n",
      "INFO:tensorflow:Step 5, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 10, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 15, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 20, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 25, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 30, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 35, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 40, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 45, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 50, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 55, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 60, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 65, mean_score: -0.031250\n",
      "INFO:tensorflow:Step 70, mean_score: -0.031250\n",
      "INFO:tensorflow:Step 75, mean_score: -0.031250\n",
      "INFO:tensorflow:Step 80, mean_score: -0.031250\n",
      "INFO:tensorflow:Step 85, mean_score: -0.031250\n",
      "INFO:tensorflow:Step 90, mean_score: -0.031250\n",
      "INFO:tensorflow:Step 95, mean_score: 0.937500\n",
      "INFO:tensorflow:Step 100, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 105, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 110, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 115, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 120, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 125, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 130, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 135, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 140, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 145, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 150, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 155, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 160, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 165, mean_score: 0.906250\n",
      "INFO:tensorflow:Step 170, mean_score: 0.906250\n",
      "INFO:tensorflow:Step 175, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 180, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 185, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 190, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 195, mean_score: 0.921875\n",
      "INFO:tensorflow:Step 200, mean_score: 1.890625\n",
      "INFO:tensorflow:Step 205, mean_score: 1.890625\n",
      "INFO:tensorflow:Step 210, mean_score: 1.890625\n",
      "INFO:tensorflow:Step 215, mean_score: 1.890625\n",
      "INFO:tensorflow:Step 220, mean_score: 1.890625\n",
      "INFO:tensorflow:Step 225, mean_score: 1.890625\n",
      "INFO:tensorflow:Step 230, mean_score: 1.890625\n",
      "INFO:tensorflow:Step 235, mean_score: 1.890625\n",
      "INFO:tensorflow:Step 240, mean_score: 1.890625\n",
      "INFO:tensorflow:Step 245, mean_score: 1.890625\n",
      "INFO:tensorflow:Step 250, mean_score: 1.890625\n",
      "INFO:tensorflow:Step 255, mean_score: 1.890625\n",
      "INFO:tensorflow:Step 260, mean_score: 1.890625\n",
      "INFO:tensorflow:Step 265, mean_score: 1.890625\n",
      "INFO:tensorflow:Step 270, mean_score: 1.890625\n",
      "INFO:tensorflow:Step 275, mean_score: 2.875000\n",
      "INFO:tensorflow:Step 280, mean_score: 2.890625\n",
      "INFO:tensorflow:Step 285, mean_score: 2.890625\n",
      "INFO:tensorflow:Step 290, mean_score: 2.890625\n",
      "INFO:tensorflow:Step 295, mean_score: 2.890625\n",
      "INFO:tensorflow:Step 300, mean_score: 2.890625\n",
      "INFO:tensorflow:Step 305, mean_score: 2.890625\n",
      "INFO:tensorflow:Step 310, mean_score: 2.890625\n",
      "INFO:tensorflow:Step 315, mean_score: 2.890625\n",
      "INFO:tensorflow:Step 320, mean_score: 2.890625\n",
      "INFO:tensorflow:Step 325, mean_score: 2.890625\n",
      "INFO:tensorflow:Step 330, mean_score: 2.890625\n",
      "INFO:tensorflow:Step 335, mean_score: 2.890625\n",
      "INFO:tensorflow:Step 340, mean_score: 2.890625\n",
      "INFO:tensorflow:Step 345, mean_score: 2.890625\n",
      "INFO:tensorflow:Step 350, mean_score: 2.890625\n",
      "INFO:tensorflow:Step 355, mean_score: 2.906250\n",
      "INFO:tensorflow:Step 360, mean_score: 2.906250\n",
      "INFO:tensorflow:Step 365, mean_score: 2.906250\n",
      "INFO:tensorflow:Step 370, mean_score: 2.906250\n",
      "INFO:tensorflow:Step 375, mean_score: 2.921875\n",
      "INFO:tensorflow:Step 380, mean_score: 3.890625\n",
      "INFO:tensorflow:Step 385, mean_score: 3.890625\n",
      "INFO:tensorflow:Step 390, mean_score: 3.890625\n",
      "INFO:tensorflow:Step 395, mean_score: 3.890625\n",
      "INFO:tensorflow:Step 400, mean_score: 3.890625\n",
      "INFO:tensorflow:Step 405, mean_score: 3.890625\n",
      "INFO:tensorflow:Step 410, mean_score: 3.890625\n",
      "INFO:tensorflow:Step 415, mean_score: 3.890625\n",
      "INFO:tensorflow:Step 420, mean_score: 3.890625\n",
      "INFO:tensorflow:Step 425, mean_score: 3.890625\n",
      "INFO:tensorflow:Step 430, mean_score: 3.890625\n",
      "INFO:tensorflow:Step 435, mean_score: 3.890625\n",
      "INFO:tensorflow:Step 440, mean_score: 3.890625\n",
      "INFO:tensorflow:Step 445, mean_score: 3.890625\n",
      "INFO:tensorflow:Step 450, mean_score: 3.890625\n",
      "INFO:tensorflow:Step 455, mean_score: 4.875000\n",
      "INFO:tensorflow:Step 460, mean_score: 4.890625\n",
      "INFO:tensorflow:Step 465, mean_score: 4.890625\n",
      "INFO:tensorflow:Step 470, mean_score: 4.890625\n",
      "INFO:tensorflow:Step 475, mean_score: 4.890625\n",
      "INFO:tensorflow:Step 480, mean_score: 4.890625\n",
      "INFO:tensorflow:Step 485, mean_score: 4.890625\n",
      "INFO:tensorflow:Step 490, mean_score: 4.890625\n",
      "INFO:tensorflow:Step 495, mean_score: 4.890625\n",
      "INFO:tensorflow:Step 500, mean_score: 4.890625\n",
      "INFO:tensorflow:Step 505, mean_score: 4.890625\n",
      "INFO:tensorflow:Step 510, mean_score: 4.890625\n",
      "INFO:tensorflow:Step 515, mean_score: 4.890625\n",
      "INFO:tensorflow:Step 520, mean_score: 4.890625\n",
      "INFO:tensorflow:Step 525, mean_score: 4.890625\n",
      "INFO:tensorflow:Step 530, mean_score: 4.890625\n",
      "INFO:tensorflow:Step 535, mean_score: 4.906250\n",
      "INFO:tensorflow:Step 540, mean_score: 4.906250\n",
      "INFO:tensorflow:Step 545, mean_score: 4.906250\n",
      "INFO:tensorflow:Step 550, mean_score: 4.906250\n",
      "INFO:tensorflow:Step 555, mean_score: 4.921875\n",
      "INFO:tensorflow:Step 560, mean_score: 5.890625\n",
      "INFO:tensorflow:Step 565, mean_score: 5.890625\n",
      "INFO:tensorflow:Step 570, mean_score: 5.890625\n",
      "INFO:tensorflow:Step 575, mean_score: 5.890625\n",
      "INFO:tensorflow:Step 580, mean_score: 5.890625\n",
      "INFO:tensorflow:Step 585, mean_score: 5.890625\n",
      "INFO:tensorflow:Step 590, mean_score: 5.890625\n",
      "INFO:tensorflow:Step 595, mean_score: 5.890625\n",
      "INFO:tensorflow:Step 600, mean_score: 5.890625\n",
      "INFO:tensorflow:Step 605, mean_score: 5.890625\n",
      "INFO:tensorflow:Step 610, mean_score: 5.890625\n",
      "INFO:tensorflow:Step 615, mean_score: 5.890625\n",
      "INFO:tensorflow:Step 620, mean_score: 5.890625\n",
      "INFO:tensorflow:Step 625, mean_score: 5.890625\n",
      "INFO:tensorflow:Step 630, mean_score: 5.890625\n",
      "INFO:tensorflow:Step 635, mean_score: 6.875000\n",
      "INFO:tensorflow:Step 640, mean_score: 6.890625\n",
      "INFO:tensorflow:Step 645, mean_score: 6.890625\n",
      "INFO:tensorflow:Step 650, mean_score: 6.890625\n",
      "INFO:tensorflow:Step 655, mean_score: 6.890625\n",
      "INFO:tensorflow:Step 660, mean_score: 6.890625\n",
      "INFO:tensorflow:Step 665, mean_score: 6.890625\n",
      "INFO:tensorflow:Step 670, mean_score: 6.890625\n",
      "INFO:tensorflow:Step 675, mean_score: 6.890625\n",
      "INFO:tensorflow:Step 680, mean_score: 6.890625\n",
      "INFO:tensorflow:Step 685, mean_score: 6.890625\n",
      "INFO:tensorflow:Step 690, mean_score: 6.890625\n",
      "INFO:tensorflow:Step 695, mean_score: 6.890625\n",
      "INFO:tensorflow:Step 700, mean_score: 6.890625\n",
      "INFO:tensorflow:Step 705, mean_score: 6.890625\n",
      "INFO:tensorflow:Step 710, mean_score: 6.890625\n",
      "INFO:tensorflow:Step 715, mean_score: 6.906250\n",
      "INFO:tensorflow:Step 720, mean_score: 6.906250\n",
      "INFO:tensorflow:Step 725, mean_score: 6.906250\n",
      "INFO:tensorflow:Step 730, mean_score: 6.906250\n",
      "INFO:tensorflow:Step 735, mean_score: 6.921875\n",
      "INFO:tensorflow:Step 740, mean_score: 7.890625\n",
      "INFO:tensorflow:Step 745, mean_score: 7.890625\n",
      "INFO:tensorflow:Step 750, mean_score: 7.890625\n",
      "INFO:tensorflow:Step 755, mean_score: 7.890625\n",
      "INFO:tensorflow:Step 760, mean_score: 7.890625\n",
      "INFO:tensorflow:Step 765, mean_score: 7.890625\n",
      "INFO:tensorflow:Step 770, mean_score: 7.890625\n",
      "INFO:tensorflow:Step 775, mean_score: 7.890625\n",
      "INFO:tensorflow:Step 780, mean_score: 7.890625\n",
      "INFO:tensorflow:Step 785, mean_score: 7.890625\n",
      "INFO:tensorflow:Step 790, mean_score: 7.890625\n",
      "INFO:tensorflow:Step 795, mean_score: 7.890625\n",
      "INFO:tensorflow:Step 800, mean_score: 7.890625\n",
      "INFO:tensorflow:Step 805, mean_score: 7.890625\n",
      "INFO:tensorflow:Step 810, mean_score: 7.890625\n",
      "INFO:tensorflow:Step 815, mean_score: 8.875000\n",
      "INFO:tensorflow:Step 820, mean_score: 8.890625\n",
      "INFO:tensorflow:Step 825, mean_score: 8.890625\n",
      "INFO:tensorflow:Step 830, mean_score: 8.890625\n",
      "INFO:tensorflow:Step 835, mean_score: 8.890625\n",
      "INFO:tensorflow:Step 840, mean_score: 8.890625\n",
      "INFO:tensorflow:Step 845, mean_score: 8.890625\n",
      "INFO:tensorflow:Step 850, mean_score: 8.890625\n",
      "INFO:tensorflow:Step 855, mean_score: 8.890625\n",
      "INFO:tensorflow:Step 860, mean_score: 8.890625\n",
      "INFO:tensorflow:Step 865, mean_score: 8.890625\n",
      "INFO:tensorflow:Step 870, mean_score: 8.890625\n",
      "INFO:tensorflow:Step 875, mean_score: 8.890625\n",
      "INFO:tensorflow:Step 880, mean_score: 8.890625\n",
      "INFO:tensorflow:Step 885, mean_score: 8.890625\n",
      "INFO:tensorflow:Step 890, mean_score: 8.890625\n",
      "INFO:tensorflow:Step 895, mean_score: 8.906250\n",
      "INFO:tensorflow:Step 900, mean_score: 8.906250\n",
      "INFO:tensorflow:Step 905, mean_score: 8.906250\n",
      "INFO:tensorflow:Step 910, mean_score: 8.906250\n",
      "INFO:tensorflow:Step 915, mean_score: 8.921875\n",
      "INFO:tensorflow:Step 920, mean_score: 9.890625\n",
      "INFO:tensorflow:Step 925, mean_score: 9.890625\n",
      "INFO:tensorflow:Step 930, mean_score: 9.890625\n",
      "INFO:tensorflow:Step 935, mean_score: 9.890625\n",
      "INFO:tensorflow:Step 940, mean_score: 9.890625\n",
      "INFO:tensorflow:Step 945, mean_score: 9.890625\n",
      "INFO:tensorflow:Step 950, mean_score: 9.890625\n",
      "INFO:tensorflow:Step 955, mean_score: 9.890625\n",
      "INFO:tensorflow:Step 960, mean_score: 9.890625\n",
      "INFO:tensorflow:Step 965, mean_score: 9.890625\n",
      "INFO:tensorflow:Step 970, mean_score: 9.890625\n",
      "INFO:tensorflow:Step 975, mean_score: 9.890625\n",
      "INFO:tensorflow:Step 980, mean_score: 9.890625\n",
      "INFO:tensorflow:Step 985, mean_score: 9.890625\n",
      "INFO:tensorflow:Step 990, mean_score: 9.890625\n",
      "INFO:tensorflow:Step 995, mean_score: 10.875000\n",
      "INFO:tensorflow:Step 1000, mean_score: 10.890625\n",
      "INFO:tensorflow:Step 1005, mean_score: 10.890625\n",
      "INFO:tensorflow:Step 1010, mean_score: 10.890625\n",
      "INFO:tensorflow:Step 1015, mean_score: 10.890625\n",
      "INFO:tensorflow:Step 1020, mean_score: 10.890625\n",
      "INFO:tensorflow:Step 1025, mean_score: 10.890625\n",
      "INFO:tensorflow:Step 1030, mean_score: 10.890625\n",
      "INFO:tensorflow:Step 1035, mean_score: 10.890625\n",
      "INFO:tensorflow:Step 1040, mean_score: 10.890625\n",
      "INFO:tensorflow:Step 1045, mean_score: 10.890625\n",
      "INFO:tensorflow:Step 1050, mean_score: 10.890625\n",
      "INFO:tensorflow:Step 1055, mean_score: 10.890625\n",
      "INFO:tensorflow:Step 1060, mean_score: 10.890625\n",
      "INFO:tensorflow:Step 1065, mean_score: 10.890625\n",
      "INFO:tensorflow:Step 1070, mean_score: 10.890625\n",
      "INFO:tensorflow:Step 1075, mean_score: 10.906250\n",
      "INFO:tensorflow:Step 1080, mean_score: 10.906250\n",
      "INFO:tensorflow:Step 1085, mean_score: 10.906250\n",
      "INFO:tensorflow:Step 1090, mean_score: 10.906250\n",
      "INFO:tensorflow:Step 1095, mean_score: 10.921875\n",
      "INFO:tensorflow:Step 1100, mean_score: 11.890625\n",
      "INFO:tensorflow:Step 1105, mean_score: 11.890625\n",
      "INFO:tensorflow:Step 1110, mean_score: 11.890625\n",
      "INFO:tensorflow:Step 1115, mean_score: 11.890625\n",
      "INFO:tensorflow:Step 1120, mean_score: 11.890625\n",
      "INFO:tensorflow:Step 1125, mean_score: 11.890625\n",
      "INFO:tensorflow:Step 1130, mean_score: 11.890625\n",
      "INFO:tensorflow:Step 1135, mean_score: 11.890625\n",
      "INFO:tensorflow:Step 1140, mean_score: 11.890625\n",
      "INFO:tensorflow:Step 1145, mean_score: 11.890625\n",
      "INFO:tensorflow:Step 1150, mean_score: 11.890625\n",
      "INFO:tensorflow:Step 1155, mean_score: 11.890625\n",
      "INFO:tensorflow:Step 1160, mean_score: 11.890625\n",
      "INFO:tensorflow:Step 1165, mean_score: 11.890625\n",
      "INFO:tensorflow:Step 1170, mean_score: 11.890625\n",
      "INFO:tensorflow:Step 1175, mean_score: 12.875000\n",
      "INFO:tensorflow:Step 1180, mean_score: 12.890625\n",
      "INFO:tensorflow:Step 1185, mean_score: 12.890625\n",
      "INFO:tensorflow:Step 1190, mean_score: 12.890625\n",
      "INFO:tensorflow:Step 1195, mean_score: 12.890625\n",
      "INFO:tensorflow:Step 1200, mean_score: 12.890625\n",
      "INFO:tensorflow:Step 1205, mean_score: 12.890625\n",
      "INFO:tensorflow:Step 1210, mean_score: 12.890625\n",
      "INFO:tensorflow:Step 1215, mean_score: 12.890625\n",
      "INFO:tensorflow:Step 1220, mean_score: 12.890625\n",
      "INFO:tensorflow:Step 1225, mean_score: 12.890625\n",
      "INFO:tensorflow:Step 1230, mean_score: 12.890625\n",
      "INFO:tensorflow:Step 1235, mean_score: 12.890625\n",
      "INFO:tensorflow:Step 1240, mean_score: 12.890625\n",
      "INFO:tensorflow:Step 1245, mean_score: 12.890625\n",
      "INFO:tensorflow:Step 1250, mean_score: 12.890625\n",
      "INFO:tensorflow:Step 1255, mean_score: 12.906250\n",
      "INFO:tensorflow:Step 1260, mean_score: 12.906250\n",
      "INFO:tensorflow:Step 1265, mean_score: 12.906250\n",
      "INFO:tensorflow:Step 1270, mean_score: 12.906250\n",
      "INFO:tensorflow:Step 1275, mean_score: 12.921875\n",
      "INFO:tensorflow:Step 1280, mean_score: 13.890625\n",
      "INFO:tensorflow:Step 1285, mean_score: 13.890625\n",
      "INFO:tensorflow:Step 1290, mean_score: 13.890625\n",
      "INFO:tensorflow:Step 1295, mean_score: 13.890625\n",
      "INFO:tensorflow:Step 1300, mean_score: 13.890625\n",
      "INFO:tensorflow:Step 1305, mean_score: 13.890625\n",
      "INFO:tensorflow:Step 1310, mean_score: 13.890625\n",
      "INFO:tensorflow:Step 1315, mean_score: 13.890625\n",
      "INFO:tensorflow:Step 1320, mean_score: 13.890625\n",
      "INFO:tensorflow:Step 1325, mean_score: 13.890625\n",
      "INFO:tensorflow:Step 1330, mean_score: 13.890625\n",
      "INFO:tensorflow:Step 1335, mean_score: 13.890625\n",
      "INFO:tensorflow:Step 1340, mean_score: 13.890625\n",
      "INFO:tensorflow:Step 1345, mean_score: 13.890625\n",
      "INFO:tensorflow:Step 1350, mean_score: 13.890625\n",
      "INFO:tensorflow:Step 1355, mean_score: 14.875000\n",
      "INFO:tensorflow:Step 1360, mean_score: 14.890625\n",
      "INFO:tensorflow:Step 1365, mean_score: 14.890625\n",
      "INFO:tensorflow:Step 1370, mean_score: 14.890625\n",
      "INFO:tensorflow:Step 1375, mean_score: 14.890625\n",
      "INFO:tensorflow:Step 1380, mean_score: 14.890625\n",
      "INFO:tensorflow:Step 1385, mean_score: 14.890625\n",
      "INFO:tensorflow:Step 1390, mean_score: 14.890625\n",
      "INFO:tensorflow:Step 1395, mean_score: 14.890625\n",
      "INFO:tensorflow:Step 1400, mean_score: 14.890625\n",
      "INFO:tensorflow:Step 1405, mean_score: 14.890625\n",
      "INFO:tensorflow:Step 1410, mean_score: 14.890625\n",
      "INFO:tensorflow:Step 1415, mean_score: 14.890625\n",
      "INFO:tensorflow:Step 1420, mean_score: 14.890625\n",
      "INFO:tensorflow:Step 1425, mean_score: 14.890625\n",
      "INFO:tensorflow:Step 1430, mean_score: 14.890625\n",
      "INFO:tensorflow:Step 1435, mean_score: 14.906250\n",
      "INFO:tensorflow:Step 1440, mean_score: 14.906250\n",
      "INFO:tensorflow:Step 1445, mean_score: 14.906250\n",
      "INFO:tensorflow:Step 1450, mean_score: 14.906250\n",
      "INFO:tensorflow:Step 1455, mean_score: 14.921875\n",
      "INFO:tensorflow:Step 1460, mean_score: 15.890625\n",
      "INFO:tensorflow:Step 1465, mean_score: 15.890625\n",
      "INFO:tensorflow:Step 1470, mean_score: 15.890625\n",
      "INFO:tensorflow:Step 1475, mean_score: 15.890625\n",
      "INFO:tensorflow:Step 1480, mean_score: 15.890625\n",
      "INFO:tensorflow:Step 1485, mean_score: 15.890625\n",
      "INFO:tensorflow:Step 1490, mean_score: 15.890625\n",
      "INFO:tensorflow:Step 1495, mean_score: 15.890625\n",
      "INFO:tensorflow:Step 1500, mean_score: 15.890625\n",
      "INFO:tensorflow:Step 1505, mean_score: 15.890625\n",
      "INFO:tensorflow:Step 1510, mean_score: 15.890625\n",
      "INFO:tensorflow:Step 1515, mean_score: 15.890625\n",
      "INFO:tensorflow:Step 1520, mean_score: 15.890625\n",
      "INFO:tensorflow:Step 1525, mean_score: 15.890625\n",
      "INFO:tensorflow:Step 1530, mean_score: 15.890625\n",
      "INFO:tensorflow:Step 1535, mean_score: 16.875000\n",
      "INFO:tensorflow:Step 1540, mean_score: 16.890625\n",
      "INFO:tensorflow:Step 1545, mean_score: 16.890625\n",
      "INFO:tensorflow:Step 1550, mean_score: 16.890625\n",
      "INFO:tensorflow:Step 1555, mean_score: 16.890625\n",
      "INFO:tensorflow:Step 1560, mean_score: 16.890625\n",
      "INFO:tensorflow:Step 1565, mean_score: 16.890625\n",
      "INFO:tensorflow:Step 1570, mean_score: 16.890625\n",
      "INFO:tensorflow:Step 1575, mean_score: 16.890625\n",
      "INFO:tensorflow:Step 1580, mean_score: 16.890625\n",
      "INFO:tensorflow:Step 1585, mean_score: 16.890625\n",
      "INFO:tensorflow:Step 1590, mean_score: 16.890625\n",
      "INFO:tensorflow:Step 1595, mean_score: 16.890625\n",
      "INFO:tensorflow:Step 1600, mean_score: 16.890625\n",
      "INFO:tensorflow:Step 1605, mean_score: 16.890625\n",
      "INFO:tensorflow:Step 1610, mean_score: 16.890625\n",
      "INFO:tensorflow:Step 1615, mean_score: 16.906250\n",
      "INFO:tensorflow:Step 1620, mean_score: 16.906250\n",
      "INFO:tensorflow:Step 1625, mean_score: 16.906250\n",
      "INFO:tensorflow:Step 1630, mean_score: 16.906250\n",
      "INFO:tensorflow:Step 1635, mean_score: 16.921875\n",
      "INFO:tensorflow:Step 1640, mean_score: 17.890625\n",
      "INFO:tensorflow:Step 1645, mean_score: 17.890625\n",
      "INFO:tensorflow:Step 1650, mean_score: 17.890625\n",
      "INFO:tensorflow:Step 1655, mean_score: 17.890625\n",
      "INFO:tensorflow:Step 1660, mean_score: 17.890625\n",
      "INFO:tensorflow:Step 1665, mean_score: 17.890625\n",
      "INFO:tensorflow:Step 1670, mean_score: 17.890625\n",
      "INFO:tensorflow:Step 1675, mean_score: 17.890625\n",
      "INFO:tensorflow:Step 1680, mean_score: 17.890625\n",
      "INFO:tensorflow:Step 1685, mean_score: 17.890625\n",
      "INFO:tensorflow:Step 1690, mean_score: 17.890625\n",
      "INFO:tensorflow:Step 1695, mean_score: 17.890625\n",
      "INFO:tensorflow:Step 1700, mean_score: 17.890625\n",
      "INFO:tensorflow:Step 1705, mean_score: 17.890625\n",
      "INFO:tensorflow:Step 1710, mean_score: 17.890625\n",
      "INFO:tensorflow:Step 1715, mean_score: 18.875000\n",
      "INFO:tensorflow:Step 1720, mean_score: 18.890625\n",
      "INFO:tensorflow:Step 1725, mean_score: 18.890625\n",
      "INFO:tensorflow:Step 1730, mean_score: 18.890625\n",
      "INFO:tensorflow:Step 1735, mean_score: 18.890625\n",
      "INFO:tensorflow:Step 1740, mean_score: 18.890625\n",
      "INFO:tensorflow:Step 1745, mean_score: 18.890625\n",
      "INFO:tensorflow:Step 1750, mean_score: 18.890625\n",
      "INFO:tensorflow:Step 1755, mean_score: 18.890625\n",
      "INFO:tensorflow:Step 1760, mean_score: 18.890625\n",
      "INFO:tensorflow:Step 1765, mean_score: 18.890625\n",
      "INFO:tensorflow:Step 1770, mean_score: 18.890625\n",
      "INFO:tensorflow:Step 1775, mean_score: 18.890625\n",
      "INFO:tensorflow:Step 1780, mean_score: 18.890625\n",
      "INFO:tensorflow:Step 1785, mean_score: 18.890625\n",
      "INFO:tensorflow:Step 1790, mean_score: 18.890625\n",
      "INFO:tensorflow:Step 1795, mean_score: 18.906250\n",
      "INFO:tensorflow:Step 1800, mean_score: 18.906250\n",
      "INFO:tensorflow:Step 1805, mean_score: 18.906250\n",
      "INFO:tensorflow:Step 1810, mean_score: 18.906250\n",
      "INFO:tensorflow:Step 1815, mean_score: 18.921875\n",
      "INFO:tensorflow:Step 1820, mean_score: 19.890625\n",
      "INFO:tensorflow:Step 1825, mean_score: 19.890625\n",
      "INFO:tensorflow:Step 1830, mean_score: 19.890625\n",
      "INFO:tensorflow:Step 1835, mean_score: 19.890625\n",
      "INFO:tensorflow:Step 1840, mean_score: 19.890625\n",
      "INFO:tensorflow:Step 1845, mean_score: 19.890625\n",
      "INFO:tensorflow:Step 1850, mean_score: 19.890625\n",
      "INFO:tensorflow:Step 1855, mean_score: 19.890625\n",
      "INFO:tensorflow:Step 1860, mean_score: 19.890625\n",
      "INFO:tensorflow:Step 1865, mean_score: 19.890625\n",
      "INFO:tensorflow:Step 1870, mean_score: 19.890625\n",
      "INFO:tensorflow:Step 1875, mean_score: 19.890625\n",
      "INFO:tensorflow:Step 1880, mean_score: 19.890625\n",
      "INFO:tensorflow:Step 1885, mean_score: 19.890625\n",
      "INFO:tensorflow:Step 1890, mean_score: 19.890625\n",
      "INFO:tensorflow:Step 1895, mean_score: 19.906250\n",
      "INFO:tensorflow:Step 1900, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1905, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1910, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1915, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1920, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1925, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1930, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1935, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1940, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1945, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1950, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1955, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1960, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1965, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1970, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1975, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1980, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1985, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1990, mean_score: 19.921875\n",
      "INFO:tensorflow:Step 1995, mean_score: 19.937500\n",
      "INFO:tensorflow:Step 2000, mean_score: 19.937500\n",
      "INFO:tensorflow:Step 2005, mean_score: 19.937500\n",
      "INFO:tensorflow:Step 2010, mean_score: 19.937500\n",
      "INFO:tensorflow:Step 2015, mean_score: 19.937500\n",
      "INFO:tensorflow:Step 2020, mean_score: 19.937500\n",
      "INFO:tensorflow:Step 2025, mean_score: 19.937500\n",
      "INFO:tensorflow:Step 2030, mean_score: 19.937500\n",
      "INFO:tensorflow:Step 2035, mean_score: 19.937500\n",
      "INFO:tensorflow:Step 2040, mean_score: 19.937500\n",
      "INFO:tensorflow:Step 2045, mean_score: 19.937500\n",
      "INFO:tensorflow:Step 2050, mean_score: 19.937500\n",
      "INFO:tensorflow:Step 2055, mean_score: 19.937500\n",
      "INFO:tensorflow:Step 2060, mean_score: 19.937500\n",
      "INFO:tensorflow:Step 2065, mean_score: 19.937500\n",
      "INFO:tensorflow:Step 2070, mean_score: 19.937500\n"
     ]
    }
   ],
   "source": [
    "game = 'pong'\n",
    "run_dir = get_run_dir(game, 1)\n",
    "!python -m tensor2tensor.rl.evaluator \\\n",
    "  --loop_hparams_set=rlmb_long_stochastic_discrete \\\n",
    "  --loop_hparams=game=$game,eval_max_num_noops=8,eval_sampling_temps=[0.5] \\\n",
    "  --policy_dir=$run_dir/policy \\\n",
    "  --eval_metrics_dir=pong_pretrained \\\n",
    "  --debug_video_path=pong_pretrained \\\n",
    "  --num_debug_videos=4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WKWPdwP8BW_v",
    "colab_type": "text"
   },
   "source": [
    "The above command will run a single evaluation setting to get the results fast. We usually run a grid of different settings (sampling temperatures and whether to do initial no-ops). To do that, remove `eval_max_num_noops=8,eval_sampling_temps=[0.5]` from the command. You can override the evaluation settings:\n",
    "\n",
    "```\n",
    "  --loop_hparams=game=pong,eval_max_num_noops=0,eval_sampling_temps=[0.0]\n",
    " ```\n",
    " \n",
    " The evaluator generates videos from the environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "At9LC5rxFyv2",
    "colab_type": "code",
    "outputId": "983b0e7a-2700-4e4a-d776-03c459669770",
    "executionInfo": {
     "status": "ok",
     "timestamp": 1.553253830168E12,
     "user_tz": -60.0,
     "elapsed": 4036.0,
     "user": {
      "displayName": "Piotr Kozakowski",
      "photoUrl": "",
      "userId": "01014928596539690143"
     }
    },
    "colab": {
     "resources": {
      "http://localhost:8080/nbextensions/vid.mp4": {
       "data": "[encoded data, 260204 characters]",
       "ok": true,
       "headers": [
        [
         "content-type",
         "video/mp4"
        ]
       ],
       "status": 200.0,
       "status_text": ""
      }
     },
     "base_uri": "https://localhost:8080/",
     "height": 501.0
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <video width=\"640\" height=\"480\" controls>\n",
       "      <source src=\"/nbextensions/vid.mp4\" type=\"video/mp4\">\n",
       "    </video>\n",
       "  "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 24,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "play_video('pong_pretrained/0.avi')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U-SyGcZBCmPn",
    "colab_type": "text"
   },
   "source": [
    "# Train your policy (model-free training)\n",
    "Training model-free on Pong (it takes a few hours):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "WIQazd5aCocc",
    "colab_type": "code",
    "outputId": "0a440c18-affc-4b2a-d6e1-c3cda84465bc",
    "executionInfo": {
     "status": "ok",
     "timestamp": 1.553254256733E12,
     "user_tz": -60.0,
     "elapsed": 19957.0,
     "user": {
      "displayName": "Piotr Kozakowski",
      "photoUrl": "",
      "userId": "01014928596539690143"
     }
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1516.0
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
      "For more information, please see:\n",
      "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
      "  * https://github.com/tensorflow/addons\n",
      "If you depend on functionality not listed there, please file an issue.\n",
      "\n",
      "2019-03-22 11:30:42.987149: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz\n",
      "2019-03-22 11:30:42.987392: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x30323c0 executing computations on platform Host. Devices:\n",
      "2019-03-22 11:30:42.987491: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>\n",
      "2019-03-22 11:30:43.082876: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2019-03-22 11:30:43.083442: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x3032100 executing computations on platform CUDA. Devices:\n",
      "2019-03-22 11:30:43.083493: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): Tesla K80, Compute Capability 3.7\n",
      "2019-03-22 11:30:43.083843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: \n",
      "name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235\n",
      "pciBusID: 0000:00:04.0\n",
      "totalMemory: 11.17GiB freeMemory: 11.10GiB\n",
      "2019-03-22 11:30:43.083879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0\n",
      "2019-03-22 11:30:43.475526: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
      "2019-03-22 11:30:43.475601: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 \n",
      "2019-03-22 11:30:43.475629: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N \n",
      "2019-03-22 11:30:43.476026: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2019-03-22 11:30:43.476131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10754 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/rl/envs/py_func_batch_env.py:122: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "tf.py_func is deprecated in TF V2. Instead, use\n",
      "    tf.py_function, which takes a python function which manipulates tf eager\n",
      "    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to\n",
      "    an ndarray (just call tensor.numpy()) but having access to eager tensors\n",
      "    means `tf.py_function`s can use accelerators such as GPUs as well as\n",
      "    being differentiable using a gradient tape.\n",
      "    \n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/t2t_model.py:1358: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/function.py:1007: calling Graph.create_op (from tensorflow.python.framework.ops) with compute_shapes is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Shapes are always computed; don't use the compute_shapes as it has no effect.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_layers.py:277: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/rl.py:598: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.conv2d instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/rl.py:602: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.flatten instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/rl.py:603: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.dropout instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/rl.py:604: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.dense instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_layers.py:2887: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.random.categorical instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/rl/ppo_learner.py:479: Print (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2018-08-20.\n",
      "Instructions for updating:\n",
      "Use tf.print instead of tf.Print. Note that tf.print returns a no-output operator that directly prints the output. Outside of defuns or eager mode, this operator will not be executed unless it is directly specified in session.run or used as a control dependency for other operators. This is only a concern in graph mode. Below is an example of how to ensure tf.print executes in graph mode:\n",
      "```python\n",
      "    sess = tf.Session()\n",
      "    with sess.as_default():\n",
      "        tensor = tf.range(10)\n",
      "        print_op = tf.print(tensor)\n",
      "        with tf.control_dependencies([print_op]):\n",
      "          out = tf.add(tensor, tensor)\n",
      "        sess.run(out)\n",
      "    ```\n",
      "Additionally, to use tf.print in python 2.7, users must make sure to import\n",
      "the following:\n",
      "\n",
      "  `from __future__ import print_function`\n",
      "\n",
      "2019-03-22 11:30:49.903512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0\n",
      "2019-03-22 11:30:49.903591: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
      "2019-03-22 11:30:49.903620: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 \n",
      "2019-03-22 11:30:49.903639: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N \n",
      "2019-03-22 11:30:49.903898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10754 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n",
      "2019-03-22 11:30:51.335217: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally\n",
      "mean_score: [0][0][0]\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "!python -m tensor2tensor.rl.trainer_model_free \\\n",
    "  --hparams_set=rlmf_base \\\n",
    "  --hparams=game=pong \\\n",
    "  --output_dir=mf_pong"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FbSjwVAtCvLY",
    "colab_type": "text"
   },
   "source": [
    "Hyperparameter sets are defined in `tensor2tensor/models/research/rl.py`. You can override them using the hparams flag, e.g.\n",
    "\n",
    "```\n",
    "--hparams=game=kung_fu_master,frame_stack_size=5\n",
    "```\n",
    "\n",
    "As in model-based training, the periodic evaluation runs with timestep limit of 1000. To do full evaluation after training, run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "jppi4FE5C2nB",
    "colab_type": "code",
    "outputId": "a10afb7c-edd6-4a93-eee4-e3876977e825",
    "executionInfo": {
     "status": "ok",
     "timestamp": 1.553254412202E12,
     "user_tz": -60.0,
     "elapsed": 15104.0,
     "user": {
      "displayName": "Piotr Kozakowski",
      "photoUrl": "",
      "userId": "01014928596539690143"
     }
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 4083.0
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
      "For more information, please see:\n",
      "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
      "  * https://github.com/tensorflow/addons\n",
      "If you depend on functionality not listed there, please file an issue.\n",
      "\n",
      "INFO:tensorflow:Overriding hparams in rlmf_tiny with game=pong,eval_max_num_noops=0,eval_sampling_temps=[0.5]\n",
      "INFO:tensorflow:Evaluating metric mean_reward/eval/sampling_temp_0.5_max_noops_0_unclipped\n",
      "2019-03-22 11:33:23.214052: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz\n",
      "2019-03-22 11:33:23.214294: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x2d07020 executing computations on platform Host. Devices:\n",
      "2019-03-22 11:33:23.214335: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>\n",
      "2019-03-22 11:33:23.309948: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2019-03-22 11:33:23.310546: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x2d067e0 executing computations on platform CUDA. Devices:\n",
      "2019-03-22 11:33:23.310585: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): Tesla K80, Compute Capability 3.7\n",
      "2019-03-22 11:33:23.310991: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: \n",
      "name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235\n",
      "pciBusID: 0000:00:04.0\n",
      "totalMemory: 11.17GiB freeMemory: 11.10GiB\n",
      "2019-03-22 11:33:23.311027: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0\n",
      "2019-03-22 11:33:23.707039: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
      "2019-03-22 11:33:23.707114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 \n",
      "2019-03-22 11:33:23.707139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N \n",
      "2019-03-22 11:33:23.707459: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2019-03-22 11:33:23.707523: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10754 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
      "INFO:tensorflow:Using DummyPolicyProblem for the policy.\n",
      "INFO:tensorflow:Setting T2TModel mode to 'train'\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "INFO:tensorflow:Using variable initializer: orthogonal\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/t2t_model.py:1358: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "INFO:tensorflow:Transforming feature 'input_action' with symbol_modality_6_64.bottom\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/function.py:1007: calling Graph.create_op (from tensorflow.python.framework.ops) with compute_shapes is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Shapes are always computed; don't use the compute_shapes as it has no effect.\n",
      "INFO:tensorflow:Transforming feature 'input_reward' with symbol_modality_3_64.bottom\n",
      "INFO:tensorflow:Transforming feature 'inputs' with video_modality.bottom\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_video.py:495: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "tf.py_func is deprecated in TF V2. Instead, use\n",
      "    tf.py_function, which takes a python function which manipulates tf eager\n",
      "    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to\n",
      "    an ndarray (just call tensor.numpy()) but having access to eager tensors\n",
      "    means `tf.py_function`s can use accelerators such as GPUs as well as\n",
      "    being differentiable using a gradient tape.\n",
      "    \n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_layers.py:277: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "INFO:tensorflow:Transforming feature 'target_action' with symbol_modality_6_64.targets_bottom\n",
      "INFO:tensorflow:Transforming feature 'target_policy' with identity_modality.targets_bottom\n",
      "INFO:tensorflow:Transforming feature 'target_reward' with symbol_modality_3_64.targets_bottom\n",
      "INFO:tensorflow:Transforming feature 'target_value' with identity_modality.targets_bottom\n",
      "INFO:tensorflow:Transforming feature 'targets' with video_modality.targets_bottom\n",
      "INFO:tensorflow:Building model body\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/rl.py:598: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.conv2d instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/rl.py:602: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.flatten instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/rl.py:603: dropout (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.dropout instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/rl.py:604: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.dense instead.\n",
      "INFO:tensorflow:Transforming body output with identity_modality.top\n",
      "INFO:tensorflow:Transforming body output with identity_modality.top\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_layers.py:2887: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.random.categorical instead.\n",
      "2019-03-22 11:33:24.564271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0\n",
      "2019-03-22 11:33:24.564350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
      "2019-03-22 11:33:24.564376: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 \n",
      "2019-03-22 11:33:24.564410: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N \n",
      "2019-03-22 11:33:24.564687: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10754 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
      "INFO:tensorflow:Restoring checkpoint mf_pong/model.ckpt-9\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n",
      "INFO:tensorflow:Restoring parameters from mf_pong/model.ckpt-9\n",
      "2019-03-22 11:33:24.985295: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally\n",
      "INFO:tensorflow:Step 5, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 10, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 15, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 20, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 25, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 30, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 35, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 40, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 45, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 50, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 55, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 60, mean_score: 0.000000\n",
      "INFO:tensorflow:Step 65, mean_score: -1.000000\n",
      "INFO:tensorflow:Step 70, mean_score: -1.000000\n",
      "INFO:tensorflow:Step 75, mean_score: -1.000000\n",
      "INFO:tensorflow:Step 80, mean_score: -1.000000\n",
      "INFO:tensorflow:Step 85, mean_score: -1.000000\n",
      "INFO:tensorflow:Step 90, mean_score: -1.000000\n",
      "INFO:tensorflow:Step 95, mean_score: -1.000000\n",
      "INFO:tensorflow:Step 100, mean_score: -2.000000\n",
      "INFO:tensorflow:Step 105, mean_score: -2.000000\n",
      "INFO:tensorflow:Step 110, mean_score: -2.000000\n",
      "INFO:tensorflow:Step 115, mean_score: -2.000000\n",
      "INFO:tensorflow:Step 120, mean_score: -2.000000\n",
      "INFO:tensorflow:Step 125, mean_score: -2.000000\n",
      "INFO:tensorflow:Step 130, mean_score: -2.000000\n",
      "INFO:tensorflow:Step 135, mean_score: -3.000000\n",
      "INFO:tensorflow:Step 140, mean_score: -3.000000\n",
      "INFO:tensorflow:Step 145, mean_score: -3.000000\n",
      "INFO:tensorflow:Step 150, mean_score: -3.000000\n",
      "INFO:tensorflow:Step 155, mean_score: -3.000000\n",
      "INFO:tensorflow:Step 160, mean_score: -3.000000\n",
      "INFO:tensorflow:Step 165, mean_score: -3.000000\n",
      "INFO:tensorflow:Step 170, mean_score: -4.000000\n",
      "INFO:tensorflow:Step 175, mean_score: -4.000000\n",
      "INFO:tensorflow:Step 180, mean_score: -4.000000\n",
      "INFO:tensorflow:Step 185, mean_score: -4.000000\n",
      "INFO:tensorflow:Step 190, mean_score: -4.000000\n",
      "INFO:tensorflow:Step 195, mean_score: -4.000000\n",
      "INFO:tensorflow:Step 200, mean_score: -4.000000\n",
      "INFO:tensorflow:Step 205, mean_score: -5.000000\n",
      "INFO:tensorflow:Step 210, mean_score: -5.000000\n",
      "INFO:tensorflow:Step 215, mean_score: -5.000000\n",
      "INFO:tensorflow:Step 220, mean_score: -5.000000\n",
      "INFO:tensorflow:Step 225, mean_score: -5.000000\n",
      "INFO:tensorflow:Step 230, mean_score: -5.000000\n",
      "INFO:tensorflow:Step 235, mean_score: -5.000000\n",
      "INFO:tensorflow:Step 240, mean_score: -6.000000\n",
      "INFO:tensorflow:Step 245, mean_score: -6.000000\n",
      "INFO:tensorflow:Step 250, mean_score: -6.000000\n",
      "INFO:tensorflow:Step 255, mean_score: -6.000000\n",
      "INFO:tensorflow:Step 260, mean_score: -6.000000\n",
      "INFO:tensorflow:Step 265, mean_score: -6.000000\n",
      "INFO:tensorflow:Step 270, mean_score: -6.000000\n",
      "INFO:tensorflow:Step 275, mean_score: -7.000000\n",
      "INFO:tensorflow:Step 280, mean_score: -7.000000\n",
      "INFO:tensorflow:Step 285, mean_score: -7.000000\n",
      "INFO:tensorflow:Step 290, mean_score: -7.000000\n",
      "INFO:tensorflow:Step 295, mean_score: -7.000000\n",
      "INFO:tensorflow:Step 300, mean_score: -7.000000\n",
      "INFO:tensorflow:Step 305, mean_score: -7.000000\n",
      "INFO:tensorflow:Step 310, mean_score: -8.000000\n",
      "INFO:tensorflow:Step 315, mean_score: -8.000000\n",
      "INFO:tensorflow:Step 320, mean_score: -8.000000\n",
      "INFO:tensorflow:Step 325, mean_score: -8.000000\n",
      "INFO:tensorflow:Step 330, mean_score: -8.000000\n",
      "INFO:tensorflow:Step 335, mean_score: -8.000000\n",
      "INFO:tensorflow:Step 340, mean_score: -8.000000\n",
      "INFO:tensorflow:Step 345, mean_score: -9.000000\n",
      "INFO:tensorflow:Step 350, mean_score: -9.000000\n",
      "INFO:tensorflow:Step 355, mean_score: -9.000000\n",
      "INFO:tensorflow:Step 360, mean_score: -9.000000\n",
      "INFO:tensorflow:Step 365, mean_score: -9.000000\n",
      "INFO:tensorflow:Step 370, mean_score: -9.000000\n",
      "INFO:tensorflow:Step 375, mean_score: -9.000000\n",
      "INFO:tensorflow:Step 380, mean_score: -10.000000\n",
      "INFO:tensorflow:Step 385, mean_score: -10.000000\n",
      "INFO:tensorflow:Step 390, mean_score: -10.000000\n",
      "INFO:tensorflow:Step 395, mean_score: -10.000000\n",
      "INFO:tensorflow:Step 400, mean_score: -10.000000\n",
      "INFO:tensorflow:Step 405, mean_score: -10.000000\n",
      "INFO:tensorflow:Step 410, mean_score: -10.000000\n",
      "INFO:tensorflow:Step 415, mean_score: -11.000000\n",
      "INFO:tensorflow:Step 420, mean_score: -11.000000\n",
      "INFO:tensorflow:Step 425, mean_score: -11.000000\n",
      "INFO:tensorflow:Step 430, mean_score: -11.000000\n",
      "INFO:tensorflow:Step 435, mean_score: -11.000000\n",
      "INFO:tensorflow:Step 440, mean_score: -11.000000\n",
      "INFO:tensorflow:Step 445, mean_score: -11.000000\n",
      "INFO:tensorflow:Step 450, mean_score: -12.000000\n",
      "INFO:tensorflow:Step 455, mean_score: -12.000000\n",
      "INFO:tensorflow:Step 460, mean_score: -12.000000\n",
      "INFO:tensorflow:Step 465, mean_score: -12.000000\n",
      "INFO:tensorflow:Step 470, mean_score: -12.000000\n",
      "INFO:tensorflow:Step 475, mean_score: -12.000000\n",
      "INFO:tensorflow:Step 480, mean_score: -12.000000\n",
      "INFO:tensorflow:Step 485, mean_score: -13.000000\n",
      "INFO:tensorflow:Step 490, mean_score: -13.000000\n",
      "INFO:tensorflow:Step 495, mean_score: -13.000000\n",
      "INFO:tensorflow:Step 500, mean_score: -13.000000\n",
      "INFO:tensorflow:Step 505, mean_score: -13.000000\n",
      "INFO:tensorflow:Step 510, mean_score: -13.000000\n",
      "INFO:tensorflow:Step 515, mean_score: -13.000000\n",
      "INFO:tensorflow:Step 520, mean_score: -14.000000\n",
      "INFO:tensorflow:Step 525, mean_score: -14.000000\n",
      "INFO:tensorflow:Step 530, mean_score: -14.000000\n",
      "INFO:tensorflow:Step 535, mean_score: -14.000000\n",
      "INFO:tensorflow:Step 540, mean_score: -14.000000\n",
      "INFO:tensorflow:Step 545, mean_score: -14.000000\n",
      "INFO:tensorflow:Step 550, mean_score: -14.000000\n",
      "INFO:tensorflow:Step 555, mean_score: -15.000000\n",
      "INFO:tensorflow:Step 560, mean_score: -15.000000\n",
      "INFO:tensorflow:Step 565, mean_score: -15.000000\n",
      "INFO:tensorflow:Step 570, mean_score: -15.000000\n",
      "INFO:tensorflow:Step 575, mean_score: -15.000000\n",
      "INFO:tensorflow:Step 580, mean_score: -15.000000\n",
      "INFO:tensorflow:Step 585, mean_score: -15.000000\n",
      "INFO:tensorflow:Step 590, mean_score: -16.000000\n",
      "INFO:tensorflow:Step 595, mean_score: -16.000000\n",
      "INFO:tensorflow:Step 600, mean_score: -16.000000\n",
      "INFO:tensorflow:Step 605, mean_score: -16.000000\n",
      "INFO:tensorflow:Step 610, mean_score: -16.000000\n",
      "INFO:tensorflow:Step 615, mean_score: -16.000000\n",
      "INFO:tensorflow:Step 620, mean_score: -16.000000\n",
      "INFO:tensorflow:Step 625, mean_score: -17.000000\n",
      "INFO:tensorflow:Step 630, mean_score: -17.000000\n",
      "INFO:tensorflow:Step 635, mean_score: -17.000000\n",
      "INFO:tensorflow:Step 640, mean_score: -17.000000\n",
      "INFO:tensorflow:Step 645, mean_score: -17.000000\n",
      "INFO:tensorflow:Step 650, mean_score: -17.000000\n",
      "INFO:tensorflow:Step 655, mean_score: -17.000000\n",
      "INFO:tensorflow:Step 660, mean_score: -18.000000\n",
      "INFO:tensorflow:Step 665, mean_score: -18.000000\n",
      "INFO:tensorflow:Step 670, mean_score: -18.000000\n",
      "INFO:tensorflow:Step 675, mean_score: -18.000000\n",
      "INFO:tensorflow:Step 680, mean_score: -18.000000\n",
      "INFO:tensorflow:Step 685, mean_score: -18.000000\n",
      "INFO:tensorflow:Step 690, mean_score: -18.000000\n",
      "INFO:tensorflow:Step 695, mean_score: -19.000000\n",
      "INFO:tensorflow:Step 700, mean_score: -19.000000\n",
      "INFO:tensorflow:Step 705, mean_score: -19.000000\n",
      "INFO:tensorflow:Step 710, mean_score: -19.000000\n",
      "INFO:tensorflow:Step 715, mean_score: -19.000000\n",
      "INFO:tensorflow:Step 720, mean_score: -19.000000\n",
      "INFO:tensorflow:Step 725, mean_score: -19.000000\n",
      "INFO:tensorflow:Step 730, mean_score: -20.000000\n",
      "INFO:tensorflow:Step 735, mean_score: -20.000000\n",
      "INFO:tensorflow:Step 740, mean_score: -20.000000\n",
      "INFO:tensorflow:Step 745, mean_score: -20.000000\n",
      "INFO:tensorflow:Step 750, mean_score: -20.000000\n",
      "INFO:tensorflow:Step 755, mean_score: -20.000000\n",
      "INFO:tensorflow:Step 760, mean_score: -20.000000\n"
     ]
    }
   ],
   "source": [
    "!python -m tensor2tensor.rl.evaluator \\\n",
    "  --loop_hparams_set=rlmf_tiny \\\n",
    "  --hparams=game=pong \\\n",
    "  --policy_dir=mf_pong \\\n",
    "  --debug_video_path=mf_pong \\\n",
    "  --num_debug_videos=4 \\\n",
    "  --eval_metrics_dir=mf_pong/full_eval_metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "mDoR0C0ZKCOn",
    "colab_type": "code",
    "outputId": "aba41a4d-2957-4ea0-d511-eae7ea4e238e",
    "executionInfo": {
     "status": "ok",
     "timestamp": 1.553254513355E12,
     "user_tz": -60.0,
     "elapsed": 3908.0,
     "user": {
      "displayName": "Piotr Kozakowski",
      "photoUrl": "",
      "userId": "01014928596539690143"
     }
    },
    "colab": {
     "resources": {
      "http://localhost:8080/nbextensions/vid.mp4": {
       "data": "[encoded data, 92548 characters]",
       "ok": true,
       "headers": [
        [
         "content-type",
         "video/mp4"
        ]
       ],
       "status": 200.0,
       "status_text": ""
      }
     },
     "base_uri": "https://localhost:8080/",
     "height": 501.0
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <video width=\"640\" height=\"480\" controls>\n",
       "      <source src=\"/nbextensions/vid.mp4\" type=\"video/mp4\">\n",
       "    </video>\n",
       "  "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 31,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "play_video('mf_pong/0.avi')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NQmZEVKGF4Hh",
    "colab_type": "text"
   },
   "source": [
    "# Model-based training\n",
    "\n",
    "The `rl` package offers many more features, including model-based training. For instructions on how to use them, go to our [README](https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/rl/README.md)."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "hello_t2t-rl.ipynb",
   "version": "0.3.2",
   "provenance": [
    {
     "file_id": "1nQvfx1EzY3ElJUy-FVF1G16okSbkeUa2",
     "timestamp": 1.553274233669E12
    }
   ],
   "collapsed_sections": []
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3"
  },
  "accelerator": "GPU"
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
