Dota 2 and OpenAI

I'm a big DOTA 2 fan. I've been aware of it since it's conception as a Warcraft III mod. I was aware of the WC3 custom game arena back in the day, but spent most of my time playing both the campaigns and online matches of WC3. (And time well spent, imo--WC3 is still one of my all-time favorite video games.)

My younger brothers, though, played DOTA (the Defense of the Ancients custom game created from the WC3 editor) a lot. I looked over their shoulders on occasion and while it looked interesting, there was something about the game that just didn't draw me in.

Flash forward to about ten years later--about 2012--where my siblings and I are playing Starcraft 2 on a pretty regular basis. Most of that involved 4v4 matches, but on occasion my sister's husband wanted to join us, and that meant we had to find a 5-player game. We turned to some SC2 custom games, which included Aeon of Storms, as it was called at the time (another MOBA). We played this for a while, and then found out DOTA 2 was going to be released soon for free on Steam, and, well, the rest is history.

I play the occasional action, role-playing, or survival horror game on PS4, but DOTA 2 has been my go-to game for the past few years, now. It's fun, it's incredibly complicated, and it's very competitive. (It also notoriously has one of the worst-mannered gaming communities in existence, but you win some you lose some, ya know?)

So, DOTA 2 is cool. If you're curious, Juggernaut is one of my all-time favorite heroes, but lately I've also been playing a lot of Pudge.

This screenshot is from a year ago or so.

This one is pretty recent.

Anyway, with all that being said, there are some interesting developments involving DOTA 2 and one of my other current interests, AI.

An interesting exhibition match happened last year at The International 7 last year (the most prestigious DOTA 2 tournament, held every year, which consistently has the highest prize-pools in e-sports history). Dendi, one of the best and most famous DOTA 2 players in the short history of the game, faced off in a one-on-one scenario (DOTA 2 is usually 5v5) against a special opponent: an OpenAI bot--or, in other words, an artificial intelligence developed by the OpenAI research company, funded in large part by Elon Musk.



In short, the bot dominated.

OpenAI has continued working hard on their DOTA 2 program because they feel that the DOTA 2 video game presents a particular challenge for an AI. It is one of the most complicated and nuanced video games in existence. But don't take my word for it, take OpenAI's (from their recent blog post):

Dota 2 is a real-time strategy game played between two teams of five players, with each player controlling a character called a “hero”. A Dota-playing AI must master the following:

• Long time horizons. Dota games run at 30 frames per second for an average of 45 minutes, resulting in 80,000 ticks per game. Most actions (like ordering a hero to move to a location) have minor impact individually, but some individual actions like town portal usage can affect the game strategically; some strategies can play out over an entire game. OpenAI Five observes every fourth frame, yielding 20,000 moves. Chess usually ends before 40 moves, Go before 150 moves, with almost every move being strategic.

• Partially-observed state. Units and buildings can only see the area around them. The rest of the map is covered in a fog hiding enemies and their strategies. Strong play requires making inferences based on incomplete data, as well as modeling what one’s opponent might be up to. Both chess and Go are full-information games.

• High-dimensional, continuous action space. In Dota, each hero can take dozens of actions, and many actions target either another unit or a position on the ground. We discretize the space into 170,000 possible actions per hero (not all valid each tick, such as using a spell on cooldown); not counting the continuous parts, there are an average of ~1,000 valid actions each tick. The average number of actions in chess is 35; in Go, 250.

• High-dimensional, continuous observation space. Dota is played on a large continuous map containing ten heroes, dozens of buildings, dozens of NPC units, and a long tail of game features such as runes, trees, and wards. Our model observes the state of a Dota game via Valve’s Bot API as 20,000 (mostly floating-point) numbers representing all information a human is allowed to access. A chess board is naturally represented as about 70 enumeration values (a 8x8 board of 6 piece types and minor historical info); a Go board as about 400 enumeration values (a 19x19 board of 2 piece types plus Ko).

So, this time around, OpenAI's goal is to have a team of five AI bots go up against five professionals at The International 8, happening in just a few weeks. The OpenAI system has already defeated a team of five former professional players, so things are looking pretty scary for The International this year.

It's all pretty exciting.

At least it is for me :-). I recognize these are two very niche subjects that, when combined, form even more of a niche subject, and I'm okay with that. But, hey, I find it fascinating.

I have a lot of other thoughts about the machine learning processes OpenAI has experimented with to get to the point they are at today in DOTA 2. I have even more thoughts about the implications of such things--that is, after all, in large part what my next project is about!

But, for now, it's fun to sit back, watch, and learn.