Color Botting Basics

Our JavaScript RuneScape bot continues with part 3. Here we learn how to automate dropping logs from our inventory, how to use basic computer vision to find our targets to click on, how to randomize our clicks, and how to automate keyboard key presses to rotate our camera when we can't find any more trees to click on. This video covers the key concepts you need to know to build a video game bot with RobotJS.

Links

GitHub repo: https://github.com/learncodebygaming/woodcutter (view source code here)
RobotJS Documentation: http://robotjs.io/docs/syntax
Eloquent JavaScript: Free Online Edition / Paperback or Ebook Edition
W3Schools: https://www.w3schools.com/js/default.asp

Ok, so we've got a lot of things to cover in this part of the tutorial. We want to be able to move to different trees and chop them down, instead of waiting for the same tree to respawn. We also want to automatically drop the logs out of our inventory, so that it doesn't fill up with useless logs. And once we've done these things the painful way, I'm going to show you how to do pixel color matching, so that our code can actually see what's going on inside the game and it can try to find tree locations for us.

When we left off, our bot could simply click over and over again in the same location to chop down a single tree. Now let's use the same techniques to click on, and chop down, multiple trees.

var first_tree_x = 790;
var first_tree_y = 466;
var second_tree_x = 1352;
var second_tree_y = 515;

while (true) {
	robot.moveMouse(first_tree_x, first_tree_y);
	robot.click();
	sleep(8000);

	robot.moveMouse(second_tree_x, second_tree_y);
	robot.click();
	sleep(8000);
}

You'll find it's quite difficult to find two pixel coordinate positions that will consistently cut down two trees, because as your character moves the pixel location of the tress also moves. I recommend cutting down two trees manually a few times until you get a consistent pattern. After you cut down tree #2, take a screenshot and use that to get the location of tree #1. Then manually cut down tree #1, take a screenshot, and use that to get the location of tree #2. The manually cut down tree #2 once more, so that your character is in the correct starting position for our code (because it will move our character from tree #2 to tree #1 as it's first step).

This is of course very fragile. We'll come back to this in a little bit.

As we have been developing our code, our inventory has been filling up with logs whenever we cut down trees. So far we've needed to discard those logs manually, which is pretty annoying, so let's write an automation to do that for us. To do this, first make sure your inventory is open, and then we need to determine the pixel position where those logs appear in our inventory on the screen. Once we've determined that, we can have RobotJS move our mouse to that position, then do a right click on it to open the item options dialog. From the current mouse position, we can then move down some pixels to the "Drop Logs" choice and click that. I found this position to be 70 pixels lower on the screen from the position of our initial click, so we can simply add 70 to the Y coordinate to get that position.

You could take this code and put it after each time we cut down a tree, but whenever you start duplicating code like that it's usually a good indication that you should create a function instead. So let's put it in a function called dropLogs(), which we can call in our main loop.

function dropLogs() {
	var inventory_x = 1882;
	var inventory_y = 830;
	// drop logs from the inventory
	robot.moveMouse(inventory_x, inventory_y);
	robot.mouseClick('right');
	robot.moveMouse(inventory_x, inventory_y + 70);
	robot.mouseClick();
	sleep(1000);
}	

Now let's talk about simple computer vision using pixel color matching. In the RobotJS documentation, check out the screen.capture() method. This allows our code to take a screenshot at any point in time. That screenshot is returned as a Bitmap object which, if you read further down in the documenation, has a method on it called colorAt(x, y). When we call this method, it will return the color found in the screenshot at that pixel coordinate location, formatted as a hex color code.

To make sure we understand how to use these methods, let's write a quick test function.

function testScreenCapture() {
    // taking a screenshot
    var img = robot.screen.capture(0, 0, 1920, 1080);

    var pixel_color = img.colorAt(30, 18);
    console.log(pixel_color);
}

This code takes a screenshot of the whole screen (if your resolution is 1080p), and I'm checking a pixel near the upper left corner. I chose this position because for me I expect it to have a distinct blue color that I can easily recognize. To run this test function, you can comment out the call to main(); at the bottom of our code and call testScreenCapture(); instead (this is another advantage to using a main function in our code). In your log you should see a hex code, like 23a9f2.

Now that we understand better how the screen methods in RobotJS work, let's use them to find trees to click on our screen.

Before creating a new function, I first like to write the code where I call that function, pretending that it already exists. This helps to inform what the function should look like, in terms of what parameters it should have and what types of values it should return. So in our main function, I no longer want to hardcode any pixel locations. Instead, I want to call a function that will return the pixel position of a tree, and then I can simply click on that tree to chop it down.

function main() {
    console.log("Starting...");
    sleep(4000);

    // infinite loop. use ctrl+c in terminal to stop the program
    while (true) {
        var tree = findTree();
        // if we can't find a tree, write an error message and exit the loop
        if (tree == false) {
            console.log('Could not find a tree');
            break;
        }

        // chop down the tree we found
        robot.moveMouse(tree.x, tree.y);
        robot.mouseClick();
        sleep(8000);

        dropLogs();
    }
}

In the code I came up with, I'd like to have a function called findTree() that takes no arguments. If it can't find a tree it should return false, and in that case I will print out an error message to the console and then exit the loop using a break; statement. But if it does find a tree, I want it to return that as an object with an x and y property. I can then pass those properties along to the moveMouse() function.

So with our main function defined, now we just need to write that findTree() function.

To find a tree, the first step will be to take a screenshot of the game. Instead of taking a screenshot of the entire screen, though, I'm going to crop it to just capture the middle of the screen where trees are likely to be. This way the map and the rest of the interface won't interfere with our pixel search.

function findTree() {
    var x = 300, y = 300, width = 1300, height = 400;
    var img = robot.screen.capture(x, y, width, height);
    // TODO - finish the rest of this function
}

Now that we have a screen capture, the next thing we need to do is figure out what pixel color we're going to look for. Inside the game, we know we want to click on a tree trunk, which is one of several possible shades of brown. Using the eye dropper tool in my photo editing software, I sampled several of these colors to get the hex codes for a few different possible tree trunk colors that we can search for.

Back in our code, instead of saving each of these color hex codes to their own variable, this is a perfect use case for a data structure called an array. In JavaScript, you can create an array like this:

var tree_colors = ["5b462a", "60492c", "6a5130", "705634", "6d5432", "574328"];

This should give us enough brown options for our code to find a matching pixel. The next thing we need to do is start searching our screen capture for a pixel that is one of these colors in our array.

To do that we could use nested classic for loops to check every pixel in the screenshot image.

for (var i = 0; i < width; i++) {
	for (var j = 0; j < height; j++) {
		var sample_color = img.colorAt(i, j);
		console.log(sample_color);
	}
}

This code will look at every pixel in the image, starting in the upper left and scanning down each pixel column from left to right. This code would work, but it will preferentially find trees on the left side of the screen. This is also a good chance to add some randomness to our clicking code. So instead of doing it this way, let's search random pixels instead. To do that, we'll first need a way to get a random number between two values. A simple Google search found this function on StackOverflow:

function getRandomInt(min, max) {
    min = Math.ceil(min);
    max = Math.floor(max);
    return Math.floor(Math.random() * (max - min + 1)) + min;
}

We can use this to check the color of random pixels in our screen capture:

for (var i = 0; i < 100; i++) {
	var random_x = getRandomInt(0, width-1);
	var random_y = getRandomInt(0, height-1);
	var sample_color = img.colorAt(random_x, random_y);
	console.log(sample_color);
}

Here I'm checking 100 random pixel locations, and printing out to the console each color it finds. I'm subtracting 1 off the width and height for our max possible values because this getRandomInt() function is inclusive of both the given min and max. And in an image that is 1300 pixels wide, because we start counting the first column at 0, in the last column x is actually 1299.

So how do we check to see if any of these sampled colors is one of the brown colors in our array? In JavaScript, arrays all have a method on them called includes() that returns true if the argument given exists as a value inside the array.

if (tree_colors.includes(sample_color)) {
	// will enter this loop if the sample color exists in the tree_colors array
}

When we find a matching pixel, then we want to return an object with the x and y screen coordinates for that pixel. Becuase our screen capture is cropped, we need to add back in the exclude x and y pixels to get the final screen position.

If we didn't find a matching pixel color in any of our samples, we want our function to return false.

function findTree() {
    var x = 300, y = 300, width = 1300, height = 400;
    var img = robot.screen.capture(x, y, width, height);

    var tree_colors = ["5b462a", "60492c", "6a5130", "705634", "6d5432", "574328"];

    for (var i = 0; i < 100; i++) {
        var random_x = getRandomInt(0, width-1);
        var random_y = getRandomInt(0, height-1);
        var sample_color = img.colorAt(random_x, random_y);

        if (tree_colors.includes(sample_color)) {
            var screen_x = random_x + x;
            var screen_y = random_y + y;

            console.log("Found a tree at: " + screen_x + ", " + screen_y + " color " + sample_color);
            return {x: screen_x, y: screen_y};
        }
    }
    
    // did not find the color in our screenshot
    return false;
}

Now's a good time to test the code in the game again. Your character should now be finding and clicking on random trees!

If you let this code run for long enough, you'll eventually find a time where it fails to find a tree and your script exits. If there are trees on the screen when it exits, you might consider increasing the amount of sampled pixels from 100 to something higher (maybe 1000 or even more), or adjusting the crop on the screenshot you're taking.

But even after those adjustments, your character will eventually get itself into a position where there are no more trees on the screen. Right now our code simply stops in this situation, but let's try having our script rotate the screen instead. This will hopefully move our character back into the forest. We'll make a new function called rotateCamera() for this, and instead of breaking out of the loop, we'll use continue; to return to the top of the loop.

var tree = findTree();
// if we can't find a tree, rotate the camera
if (tree == false) {
    rotateCamera();
    continue;
}

To rotate the camera in RuneScape, you simply press the arrow key on your keyboard. So in our rotate camera function, we can use RobotJS again to simulate that key press. Looking at the RobotJS documentation, you'll find a keyToggle() method that can change a keyboard button state to pressed down, or to let up. So we'll use that to hold the arrow key for one second.

function rotateCamera() {
    console.log("Rotating camera");
    robot.keyToggle('right', 'down');
    sleep(1000);
    robot.keyToggle('right', 'up');
}

Now when our script can't find a tree, rather than giving up it will turn the camera to look for more trees!

Our bot has made a lot of progress, but it's still very fragile and prone to errors. In the final part of this series, we're going to address all of the defects and really polish this into a strong, robust automation.

The key programming concepts we discussed in this video are arrays, traditional for loops, function return statements, if statements, and loop break and continue statements. I encourage you to read more on those topics to solidify your understanding.