Let's explore using SendMessage to send automated inputs to multiple windows at once, or to windows that are minimized or in the background. I'll share my progress and discuss the suitability of using this method for botting.
Grab the code on GitHub: https://github.com/learncodebygaming/multiple-minimized-windows
Today I want to answer the most asked question on my channel:
I'm guessing most of you want to know how to do this because you've written a bot, and you want to run multiple bots at once, or you want your bot to run in the background while you continue using your computer.
And my initial reaction when I started getting this question was: just use virtual machines. You could have multiple VM's running in the background and those would be isolated from whatever you're doing on your main desktop. I've confirmed that this does work, but the problem is the performance of virtual machines is typically not very good. At least with Hyper-V, even basic games show considerable lag. You might be able to find better performance by setting up hardware graphics acceleration for your VM, but that's not the route I decided to explore today.
So what other options do we have?
About a year ago now, as I was working on my OpenCV series, I got a tip from a viewer telling me about this SendMessage() function he was using to send inputs to a window in the background. It's a Windows API function he was calling via pywin32, and it allows you to send messages, including mouse and keyboard inputs, to a specific window. And there's a couple StackOverflow discussions about SendMessage(), but other than that, there really isn't a whole lot I could find about working with this. So let's try it out and see what we can do with it.
I first tested a simple example with Notepad, and this gave me more problems than I was expecting.
from time import sleep
import win32gui, win32ui, win32con, win32api
def main():
window_name = "Untitled - Notepad"
hwnd = win32gui.FindWindow(None, window_name)
hwnd = get_inner_windows(hwnd)['Edit']
win = win32ui.CreateWindowFromHandle(hwnd)
#win.SendMessage(win32con.WM_CHAR, ord('A'), 0)
#win.SendMessage(win32con.WM_CHAR, ord('B'), 0)
#win.SendMessage(win32con.WM_KEYDOWN, 0x1E, 0)
#sleep(0.5)
#win.SendMessage(win32con.WM_KEYUP, 0x1E, 0)
win32api.SendMessage(hwnd, win32con.WM_KEYDOWN, 0x41, 0)
sleep(0.5)
win32api.SendMessage(hwnd, win32con.WM_KEYUP, 0x41, 0)
def list_window_names():
def winEnumHandler(hwnd, ctx):
if win32gui.IsWindowVisible(hwnd):
print(hex(hwnd), '"' + win32gui.GetWindowText(hwnd) + '"')
win32gui.EnumWindows(winEnumHandler, None)
def get_inner_windows(whndl):
def callback(hwnd, hwnds):
if win32gui.IsWindowVisible(hwnd) and win32gui.IsWindowEnabled(hwnd):
hwnds[win32gui.GetClassName(hwnd)] = hwnd
return True
hwnds = {}
win32gui.EnumChildWindows(whndl, callback, hwnds)
return hwnds
main()
The Notepad window has an inner window called 'Edit' that you need to target if you want to send inputs to the main text area. Even after I solved that, I wasn't able to get any results using WM_KEYDOWN
and WM_KEYUP
. Only by using WM_CHAR
with ord()
was I able to get text to appear in Notepad. Both SendMessage()
and PostMessage()
work this way.
Next I decided to take this to a simple browser game (Mario). I found that WM_KEYDOWN
and WM_KEYUP
did work in Chrome, and there weren't any inner windows to worry about either.
# https://supermarioemulator.com/mario.php
import sched
from time import sleep, time
import win32gui, win32ui, win32con, win32api
# http://www.kbdedit.com/manual/low_level_vk_list.html
VK_KEY_W = 0x57
VK_KEY_A = 0x41
VK_KEY_S = 0x53
VK_KEY_D = 0x44
VK_KEY_P = 0x50
VK_SHIFT = 0xA0
def main():
# init window handle
window_name = "Super Mario Bros in HTML5 - Google Chrome"
#hwnd = win32gui.FindWindow(None, window_name)
hwnds = find_all_windows(window_name)
# bring each window to the front
for hwnd in hwnds:
win32gui.SetForegroundWindow(hwnd)
sleep(1.0)
s = sched.scheduler(time, sleep)
offset_secs = 1.0
for hwnd in hwnds:
press_key(hwnd, s, VK_KEY_P, 0.1 + offset_secs, 0.1)
press_key(hwnd, s, VK_KEY_D, 0.6 + offset_secs, 1.95)
press_key(hwnd, s, VK_KEY_W, 2.5 + offset_secs, 0.9)
press_key(hwnd, s, VK_KEY_D, 3.3 + offset_secs, 1.05)
press_key(hwnd, s, VK_KEY_W, 3.5 + offset_secs, 0.8)
offset_secs += 3.31
s.run()
# send a keyboard input to the given window
def press_key(hwnd, s, key, start_sec, hold_sec):
priority = 2
foreground_time = 0.15
duration = start_sec + hold_sec
s.enter(start_sec - foreground_time, priority, win32gui.SetForegroundWindow,
argument=(hwnd,))
s.enter(start_sec, priority, win32api.SendMessage,
argument=(hwnd, win32con.WM_KEYDOWN, key, 0))
s.enter(duration - foreground_time, priority, win32gui.SetForegroundWindow,
argument=(hwnd,))
s.enter(duration, priority, win32api.SendMessage,
argument=(hwnd, win32con.WM_KEYUP, key, 0))
# win32gui.SetForegroundWindow(hwnd)
# win32api.SendMessage(hwnd, win32con.WM_KEYDOWN, key, 0)
# sleep(sec)
# win32api.SendMessage(hwnd, win32con.WM_KEYUP, key, 0)
def list_window_names():
def winEnumHandler(hwnd, ctx):
if win32gui.IsWindowVisible(hwnd):
print(hex(hwnd), '"' + win32gui.GetWindowText(hwnd) + '"')
win32gui.EnumWindows(winEnumHandler, None)
def get_inner_windows(whndl):
def callback(hwnd, hwnds):
if win32gui.IsWindowVisible(hwnd) and win32gui.IsWindowEnabled(hwnd):
hwnds[win32gui.GetClassName(hwnd)] = hwnd
return True
hwnds = {}
win32gui.EnumChildWindows(whndl, callback, hwnds)
return hwnds
def find_all_windows(name):
result = []
def winEnumHandler(hwnd, ctx):
if win32gui.IsWindowVisible(hwnd) and win32gui.GetWindowText(hwnd) == name:
result.append(hwnd)
win32gui.EnumWindows(winEnumHandler, None)
return result
main()
#list_window_names()
The first challenge I faced here was how to press down multiple keys at the same time. If you sleep()
between WM_KEYDOWN
and WM_KEYUP
that's blocking, so other code can't execute at the same time. You could solve this by using threading, but this time I decided to use sched to schedule each command in queue, and then run that queue once it's been built. This worked really well for one game, so now I wanted to scale it up to play games in multiple windows.
Unfortunately in Chrome, I couldn't get SendMessage to work when the window wasn't focused. So I ended up using SetForegroundWindow()
to quickly bring focus to the window that I'm sending the next command to. This does work, but because setting the foreground window takes a little bit of time you have to be careful to not have two commands run at the same time. I wasn't very careful about how I avoided those collisions here, so that's something you'll want to develop if you plan on using this method of quickly swapping between windows. The more copies of the game you're running, the more likely you are to run into these conflicts.
As a side note, if you're going to be swapping window focus anyway, you might find better performance using SendInput()
, or even just PyAutoGUI or PyDirectInput as you normally would. The only benefit that SendMessage()
or PostMessage()
give us is that we can target a specific window handle with them.
So now I kinda have a solution for botting in multiple windows at once, but we still have the issue of wanting to send inputs to minimized or unfocused windows so that we can run our bots in the background.
I found that how SendMessage and PostMessage behave are very dependent on what program you're trying to automate. For example with Firefox, I was able to send inputs to Mario even when the browser was minimized or in the background. But this wasn't quite a perfect solution because Firefox was laggy compared to Chrome, and I couldn't get SendMessage to work with multiple tabs or multiple windows in Firefox unless I resorted back to our SetForegroundWindow method.
Hopefully these working examples can help you on your project. Again, your success with SendMessage will vary wildly depending on your use case, so you might find better luck than I did. In the end, we have a bit of an operating system problem. Windows isn't really designed for a user to be able to give simultaneous inputs to multiple windows. So we can try to get around that by simulating multiple machines with VMs, or we can have our code quickly switch between multiple windows (but that's not really simultaneous), or we can try to co-opt this SendMessage/PostMessage API that really wasn't designed for time-sensitive inputs.
Good luck on your project, and let us know if you find a better solution!