How To Send Inputs to Multiple Windows and Minimized Windows with Python

Let's explore using SendMessage to send automated inputs to multiple windows at once, or to windows that are minimized or in the background. I'll share my progress and discuss the suitability of using this method for botting.

Links

Grab the code on GitHub: https://github.com/learncodebygaming/multiple-minimized-windows

Today I want to answer the most asked question on my channel:

• How do I send mouse and keyboard inputs to multiple windows at once,
• or how do I send inputs to a minimized window,
• or how can I keep using my mouse and keyboard while also sending automated inputs?

I'm guessing most of you want to know how to do this because you've written a bot, and you want to run multiple bots at once, or you want your bot to run in the background while you continue using your computer.

And my initial reaction when I started getting this question was: just use virtual machines. You could have multiple VM's running in the background and those would be isolated from whatever you're doing on your main desktop. I've confirmed that this does work, but the problem is the performance of virtual machines is typically not very good. At least with Hyper-V, even basic games show considerable lag. You might be able to find better performance by setting up hardware graphics acceleration for your VM, but that's not the route I decided to explore today.

So what other options do we have?

About a year ago now, as I was working on my OpenCV series, I got a tip from a viewer telling me about this SendMessage() function he was using to send inputs to a window in the background. It's a Windows API function he was calling via pywin32, and it allows you to send messages, including mouse and keyboard inputs, to a specific window. And there's a couple StackOverflow discussions about SendMessage(), but other than that, there really isn't a whole lot I could find about working with this. So let's try it out and see what we can do with it.

I first tested a simple example with Notepad, and this gave me more problems than I was expecting.

from time import sleep
import win32gui, win32ui, win32con, win32api


def main():
    window_name = "Untitled - Notepad"
    hwnd = win32gui.FindWindow(None, window_name)
    hwnd = get_inner_windows(hwnd)['Edit']
    win = win32ui.CreateWindowFromHandle(hwnd)

    #win.SendMessage(win32con.WM_CHAR, ord('A'), 0)
    #win.SendMessage(win32con.WM_CHAR, ord('B'), 0)
    #win.SendMessage(win32con.WM_KEYDOWN, 0x1E, 0)
    #sleep(0.5)
    #win.SendMessage(win32con.WM_KEYUP, 0x1E, 0)
    win32api.SendMessage(hwnd, win32con.WM_KEYDOWN, 0x41, 0)
    sleep(0.5)
    win32api.SendMessage(hwnd, win32con.WM_KEYUP, 0x41, 0)


def list_window_names():
    def winEnumHandler(hwnd, ctx):
        if win32gui.IsWindowVisible(hwnd):
            print(hex(hwnd), '"' + win32gui.GetWindowText(hwnd) + '"')
    win32gui.EnumWindows(winEnumHandler, None)


def get_inner_windows(whndl):
    def callback(hwnd, hwnds):
        if win32gui.IsWindowVisible(hwnd) and win32gui.IsWindowEnabled(hwnd):
            hwnds[win32gui.GetClassName(hwnd)] = hwnd
        return True
    hwnds = {}
    win32gui.EnumChildWindows(whndl, callback, hwnds)
    return hwnds

main()

The Notepad window has an inner window called 'Edit' that you need to target if you want to send inputs to the main text area. Even after I solved that, I wasn't able to get any results using WM_KEYDOWN and WM_KEYUP. Only by using WM_CHAR with ord() was I able to get text to appear in Notepad. Both SendMessage() and PostMessage() work this way.

Next I decided to take this to a simple browser game (Mario). I found that WM_KEYDOWN and WM_KEYUP did work in Chrome, and there weren't any inner windows to worry about either.

# https://supermarioemulator.com/mario.php

import sched
from time import sleep, time
import win32gui, win32ui, win32con, win32api

# http://www.kbdedit.com/manual/low_level_vk_list.html
VK_KEY_W = 0x57
VK_KEY_A = 0x41
VK_KEY_S = 0x53
VK_KEY_D = 0x44
VK_KEY_P = 0x50
VK_SHIFT = 0xA0

def main():
    # init window handle
    window_name = "Super Mario Bros in HTML5 - Google Chrome"
    #hwnd = win32gui.FindWindow(None, window_name)
    hwnds = find_all_windows(window_name)

    # bring each window to the front
    for hwnd in hwnds:
        win32gui.SetForegroundWindow(hwnd)
    sleep(1.0)

    s = sched.scheduler(time, sleep)

    offset_secs = 1.0
    for hwnd in hwnds:
        press_key(hwnd, s, VK_KEY_P, 0.1 + offset_secs, 0.1)
        press_key(hwnd, s, VK_KEY_D, 0.6 + offset_secs, 1.95)
        press_key(hwnd, s, VK_KEY_W, 2.5 + offset_secs, 0.9)
        press_key(hwnd, s, VK_KEY_D, 3.3 + offset_secs, 1.05)
        press_key(hwnd, s, VK_KEY_W, 3.5 + offset_secs, 0.8)

        offset_secs += 3.31
   
    s.run()

    
# send a keyboard input to the given window
def press_key(hwnd, s, key, start_sec, hold_sec):
    priority = 2
    foreground_time = 0.15
    duration = start_sec + hold_sec

    s.enter(start_sec - foreground_time, priority, win32gui.SetForegroundWindow, 
            argument=(hwnd,))
    s.enter(start_sec, priority, win32api.SendMessage, 
            argument=(hwnd, win32con.WM_KEYDOWN, key, 0))
    s.enter(duration - foreground_time, priority, win32gui.SetForegroundWindow, 
            argument=(hwnd,))        
    s.enter(duration, priority, win32api.SendMessage, 
            argument=(hwnd, win32con.WM_KEYUP, key, 0))

    # win32gui.SetForegroundWindow(hwnd)
    # win32api.SendMessage(hwnd, win32con.WM_KEYDOWN, key, 0)
    # sleep(sec)
    # win32api.SendMessage(hwnd, win32con.WM_KEYUP, key, 0)


def list_window_names():
    def winEnumHandler(hwnd, ctx):
        if win32gui.IsWindowVisible(hwnd):
            print(hex(hwnd), '"' + win32gui.GetWindowText(hwnd) + '"')
    win32gui.EnumWindows(winEnumHandler, None)


def get_inner_windows(whndl):
    def callback(hwnd, hwnds):
        if win32gui.IsWindowVisible(hwnd) and win32gui.IsWindowEnabled(hwnd):
            hwnds[win32gui.GetClassName(hwnd)] = hwnd
        return True
    hwnds = {}
    win32gui.EnumChildWindows(whndl, callback, hwnds)
    return hwnds


def find_all_windows(name):
    result = []
    def winEnumHandler(hwnd, ctx):
        if win32gui.IsWindowVisible(hwnd) and win32gui.GetWindowText(hwnd) == name:
            result.append(hwnd)
    win32gui.EnumWindows(winEnumHandler, None)
    return result


main()
#list_window_names()

The first challenge I faced here was how to press down multiple keys at the same time. If you sleep() between WM_KEYDOWN and WM_KEYUP that's blocking, so other code can't execute at the same time. You could solve this by using threading, but this time I decided to use sched to schedule each command in queue, and then run that queue once it's been built. This worked really well for one game, so now I wanted to scale it up to play games in multiple windows.

Unfortunately in Chrome, I couldn't get SendMessage to work when the window wasn't focused. So I ended up using SetForegroundWindow() to quickly bring focus to the window that I'm sending the next command to. This does work, but because setting the foreground window takes a little bit of time you have to be careful to not have two commands run at the same time. I wasn't very careful about how I avoided those collisions here, so that's something you'll want to develop if you plan on using this method of quickly swapping between windows. The more copies of the game you're running, the more likely you are to run into these conflicts.

As a side note, if you're going to be swapping window focus anyway, you might find better performance using SendInput(), or even just PyAutoGUI or PyDirectInput as you normally would. The only benefit that SendMessage() or PostMessage() give us is that we can target a specific window handle with them.

So now I kinda have a solution for botting in multiple windows at once, but we still have the issue of wanting to send inputs to minimized or unfocused windows so that we can run our bots in the background.

I found that how SendMessage and PostMessage behave are very dependent on what program you're trying to automate. For example with Firefox, I was able to send inputs to Mario even when the browser was minimized or in the background. But this wasn't quite a perfect solution because Firefox was laggy compared to Chrome, and I couldn't get SendMessage to work with multiple tabs or multiple windows in Firefox unless I resorted back to our SetForegroundWindow method.

Hopefully these working examples can help you on your project. Again, your success with SendMessage will vary wildly depending on your use case, so you might find better luck than I did. In the end, we have a bit of an operating system problem. Windows isn't really designed for a user to be able to give simultaneous inputs to multiple windows. So we can try to get around that by simulating multiple machines with VMs, or we can have our code quickly switch between multiple windows (but that's not really simultaneous), or we can try to co-opt this SendMessage/PostMessage API that really wasn't designed for time-sensitive inputs.

Good luck on your project, and let us know if you find a better solution!