I Love wasting the only 90min a day I have debugging dumb shit 🙃.

METRA On-policy built on top of pufferlib. Pure skill discovery. Trained on 4 discrete skills in the convert_circle. Env action space is multi discrete and obs are 28 dim. Running exp on harder envs (non locomotion-based & partial obs). From @seohong_park & @_oleh, @svlevine

Start building RL with puffer on GPU for brookies today:

Implementing LSD from @seohong_park. Seems to work well on tasks where exploration is "coordinate" distance based but not much on partial one-hot obs. going to dive deeper in this, maybe switch to discrete skills?.

RT @spenccheng:

RT @jsuarez5341:

It's so fun and always amazed me to watch such videos

RT @spenccheng:

I think next-gen exploration in RL has to be a function of your goals somehow.

What are the limitations of exploration in RL? Most-count based methods pushes the agent to discover new states, but that's not what you should do in all envs. Dynamic based exploration is useless in very simple envs where the env dynamics are easy to compute.

That doesn't matter as long as your model ENCODES A & B differently, but what if it doesn't? That's why it's probably worth taking some time looking at how encoding is made - can prob make a huge difference.

Example: .A: You're approaching a green light at a busy intersection, no pedestrians in sight. B: You're approaching a green light at the same intersection, but now a pedestrian is about to cross the street. C: You’re driving on a straight empty highway, far from any intersection.

There are scenarios where you might actually want to act very differently between A and B, but very similarily between A and C. In this case encoding might fuck you up and your model will always act similarly between A and B and very differently in C.

Let's say you have 3 points in high dimensional data where you can look at some form of distance between them: A,B and C. A & B are very close to each other, and C is further ahead.

Nothing here, just all converges. I guess puffer is too good

We better understand what's going on by looking at the average food amount in the envs. Same pattern

Experiments when agents die & respawn immediately, with truncated envs after 2k timesteps. Agents clearly learn to take all the food, then keeps dying until the env reset. At least, they learn fast. Let's see if sweeping gamma & lambda help them improve.

Solving for Ax=b is an optimization problem. Usefull to think this way, this means we can use optimization techniques.

never use memset for floats or ints if not setting to 0. memset in C does byte level assignement, so setting all bytes to -1 doesn’t have the desired effect.

It's crazy how much time you can waste looking at your sweeps running while doing absolutely nothing for hours starring at charts printing on the screen.