Input from code: I am struggling

[LeannAlexandra]

How can I simulate user-input from another project programatically?

firstly: if this is the wrong platform- please direct me to the right person to talk to.

Background

Hi, I wrote some base code for an ai to play games using RetroArch. So far I have a foundation program in c++ using OpenCV to extract the matrix, Tesseract for scores & timers (to rate the progress) and planning on tensorflow to be the Neural Network to process the frames: Here is the current state of progress

What I have tried

My current solution bogs down my system to simulate global input - but it can not run in the background, and I cannot do anything else. So here are my current considerations:

Current Options

1. Make an abomination of my fork of RetroArch to make it unplayable by human. - (not inline with goals)
   1. This might even help my interpretation for assisted AI by using the information needed before drawing the background, which takes away from the "ooooooh loook my shiny smart ai" hype
   2. This can also help in quickly iterating to train ai's
2. simulate global input, run in a virtual machine or docker image - feels like unnecessary layer of virtualization. - will use more resources than previous method.
3. ask for help from forums to achieve the first point for training but still stay true to the original idea (fully knowing I'd be looking like an idiot)*

Current Challenges

• My knowledge of C is limited
• I need to understand how to use a possible API-call to simulate input, even if the window is in the background.

Desired Outcome

• Create General AI that can play games based on visual input - because studying programming leaves me without time to play the games, therefore I need an AI that can play games in my stead.
• Seriously I would like to know how to turn the nobs under the hood - to one day create better ai and npc's in the future - but till then, I just want to learn and contribute

How to reach me? - github, email

[hizzlekizzle]

RetroArch has a command interface where you can send packets via UDP socket to make stuff happen (load games, save state, etc). I don't think inputs are included currently, but that may be an adequate path for your purposes.

[LeannAlexandra]

I checked it out, not exactly what I needed. - I think I'll need a completely virtual joystick

[LeannAlexandra]

update: I did write a virtual controller that is registered globally as if a physical device is plugged in that can be controlled by the AI thanks for the assistance

[LeannAlexandra]

CLOSING REMARK: Wrote a global virtual Controller to do as intended.

[hizzlekizzle]

Very cool. Is that published publicly anywhere?

[LeannAlexandra]

Yes, Yes it does

there is a general version of it here: virtualController - just a few dents nicks and scratches need to be polished out - after this repository proved my concept I moved on to the project using it. [edit: aiplayercontroller.hpp is the best place to start]

'works on my system' - Manjaro

1. Part 1: main.c duplicates an existing controller (not dynamically sourced yet, but I could, and did use libudev to source it previously), the other emulator (mGBA) would snatch my device off the eventX list (basically leaving the device there but without the "eventX handle" then in that case, libudev & EVIOCGRAB would not suffice in stopping this from happening. - It works fine with libretro (on my system). with a little more elbow grease I could potentially read a second controller, and program the xbox controller button to switch modes nand, or, xor and send those out via my upgraded concept -> part two:
2. Part 2: (the solution)
   This is now a library class, which can be used to create a controller without input device cloning -
   src/virtualcontroller.cpp (and header include/virtualcontroller.hpp) registers a global device (literally a virtual clone of an xbox controller even in vendor and product id )- but it registers as "AI Player Controller" which can be seen in linux inputs too and now exist as a systemwide (virtual) plugged in device.

To be honest, there is much we can 'add and refine' from it - and this version is terrible - it doesn't really exist as an api yet, because a neural network doesn't really care about button 'labels' so it exists as a struct of 8 abs values and 11 buttons - I am pretty sure a thing like a switch would have some more buttons, and motion capture too, so from the basis of this class, you'd be close enough to make it work.

The Bigger Picture

I am actually more exited about the bigger project that is using this class than I am actually expanding this class, as I feel in the bigger picture the sub part becomes novel*

The idea was to jump the hype-train of ai, and make one that can eventually learn to copilot something like ice climber. Think of it as your little sibling watching you play, you give them this controller (and that is their world) and when you step away this little ai steps up and completely sucks (at first) but the more they watch you play the better they become.
Not a bad concept ? - I just cannot get past my (core dumped) obsession it seems. (my focus now is pytorch (c++) the final 'piece' of the puzzle, then I'll make it public too: aiplayer

[hizzlekizzle]

That sounds awesome man :D