The next website submit, except if otherwise mentioned, was penned by a member of Gamasutras community.
The feelings and thoughts expressed are all those of the author and not Gamasutra or its dad or mum company.
In 2009 I printed a paper on predicting create orders in StarCraft: Brood War using unique classification algorithms. I made the data set available for other researchers to check out, and some academic projects have made use of this data set. Nonetheless, I never ever made the source code offered for the experiments and inevitably dropped it, equivalent to Blizzard and the source code for the original StarCraft. Since transitioning to the game titles sector, I’ve begun advocating for my coworkers to create reproducible investigate. This indicates authoring code that outputs final results and that gets rid of manual methods, and using environments that can be reproduced on unique devices and platforms.
There is been renewed desire in StarCraft AI investigate because the announcement of StarCraft Remastered and Google’s DeepMind being used to StarCraft II. Now that just about a 10 years has passed because I initial printed this investigate, I was curious if I could reproduce the experiment, and see if much more the latest strategies these kinds of as XGBoost would outperform the former classification algorithms I examined.
My unique setup for these experiments was a blend of Java code and the Wekalibrary. I initial examined out unique classifiers using the interactive Weka GUI, and then switched to using the Weka API at the time I desired to operate experiments. My Java scripts would operate experiments and output the final results to text files, which I manually copied into excel when making charts. I ran experiments on both my Home windows laptop and the university’s Unix servers.
The important experiment I desired to reproduce was predicting opponent create orders at unique simulated occasions through the recreation. The output of this experiment from my paper is demonstrated over. As much more information and facts is made offered to the unique algorithms, they grow to be better at determining the opponent’s create purchase. The 1000+ replays had been labeled with 6 unique create orders, and my tactic for labeling the create orders in covered in much more depth in the paper.
My new tactic for operating this experiment in a reproducible way is to use R Markdown, which is a markdown language that can embed R code and visualizations. The purpose of this tactic is to take away all manual methods when operating experiments. The markdown script downloads the vital data files from github, operates the experiments, and renders the final results to a pdf document or html file. The script is available here, and the final results are offered in pdf and html formats. Just one of the outputs of this script is the figure demonstrated down below.
The script loads the data frame for Protoss vs Terran create orders straight from Github. This eliminates the need to copy files locally for the experiment.
df <- read.csv("https://github.com/bgweber/StarCraftMining/
The experiment starts at 12 minutes game time, and then steps backwards to simulate different time steps during the games. I used this approach to avoid having to reload the data frame multiple times. The segment below shows how the data frame (df) is filtered to a specific game time (frame).
for (i in colnames(df))
if (i != 'label')
df[i][index] <- 0
I applied classifiers from the following packages: nearest neighbor (class), decision tree (rpart), neural networks (nnet), and xgboost(xgboost).
predict <- knn(train, test, train$label, use.all = FALSE)
reg <- nnet::multinom(label ~ ., data=train, trace = FALSE)
model<- rpart(factor(label) ~ ., data = train)
bst <- xgboost(data = trainM, label = train$label, ... )
And plot the results using ggplot2, which produces the figure above:
ggplot(data=melt(results, id="minutes"), aes(x=minutes, y=value,
colour=variable)) + geom_line() + ...
The results show that xgboost does outperform the best performing algorithm from the paper, LogitBoost, at 10 minutes game time with an accuracy of 96% vs 94%. However, the original finding of nearest neighbor approaches outperforming the other classifiers during the early game still stands.
The script also generates a second chart, which is the importance of different features at 8 minutes into the game. This is a type of analysis that I did not include in the original paper, but is interesting to include here because it shows what players should look out for when scouting an opponent. The importance of features changes over time, based on the trained model. The visualization below was generated using the xgboost functions xgb.importanceand xgb.plot.importance.
I wrote the R Markdown script on my Windows laptop for testing. To make the experiment reproducible across different machines and environments I used Docker. I started with the CentOS image, installed R, setup the libraries and other dependencies such as pandoc, and then ran the script which outputs an html file with the visualizations. I spun up a EC2 instance on AWS, using the Amazon Linux instance, and ran the following commands. Some of the commands are highlighted in the segment below:
# install and start docker
sudo yum -y install docker
sudo service docker start
# get the centos image, and start it in interactive mode
sudo docker pull centos
sudo docker run -t -i centos /bin/bash
# install R, which requires updating the repo
yum -y install R
# install required R packages
R -e "install.packages('caTools', repos='http://cran.us.r-project.org')"
# download the RMD file
# run the script!
Rscript -e "rmarkdown::render('SC_Data_Mining.Rmd')"
For this example, I ran the container in interactive mode. If I wanted to persist the environment for running the experiments in the future, I would create and apply a docker file and save the resulting image to a docker registry. This would ensure that changes to the R libraries or other dependencies do not impact the experiment environment.
If I were to rewrite this research paper today, I’d recommend using tools that make the experiments repeatable. Using R Markdown and Docker is one approach that can help accomplish this goal.
Reposted from my Medium Blog. The Science Team at Twitch is growing!