Automate My Eyes, Please

[rberq]

ME does with the keyboard and mouse pretty much all that I can do with them. What ME does not do well, is what I can do with my eyes. For example, there are many forum questions about how to locate a control on a screen, when the control is not of the type where “GET CONTROL” or “CAPTURE CONTROL” will work. This is especially a problem with web forms, where the browser may format the screen somewhat unpredictably.

 

I’d love to see a Macro Express scanning feature to simulate what we easily do by sight. This would consist of a “Capture Image Fragment” command, where you would mouse-outline a piece of the screen, and ME would save the bitmap or pixel map of the outlined area. Then at run time, there would be an equivalent “Find Image Fragment” to scan the screen and return the x- and y-coordinates of the fragment. Then you would simply move your mouse to the coordinates, click to get the text cursor there for typing, double-click if appropriate, and so on.

 

When I searched the Internet, I found “Macro Scheduler”, sold by mjtnet. It has an Image Recognition feature that does this pretty well. Macro Scheduler can create executables that will run on any PC without the base product being installed. I am tempted to buy a copy for image recognition alone. It could be used from ME by running the executables with the Launch Program command. But it would be a lot easier to have an integrated ME feature.

 

Does anyone know of a similar independent product that could be used with ME?

[Yehnfikm8Gq]

Isn't the pixel mapping that has been mentioned numerous times in the forum the same thing? All those image recognition algorithms will work in a similar way I'm sure. Look for a basic pattern first and then see if the total fits. For those that have their own pixel mapping, I can't see any advantage. I don't know if there are any mathematical methods of finding pixels quickly.

 

The mouse outline could be simulated by allowing multiple entries of pixel mapping (say for diagonal features). You would map one area and then another area say diagonal down/right. Repeat ad infinitum if desired! Ditto for any other part of the image whether connected or not. If the dialogs are based on a grid starting at the mouse location it should be easy to map an area or two. You could even click the mouse at multiple points, gathering info using a fixed grid at each point. All you are really doing with the image map is increasing the chance you have the right place. A decent grid should be reliable too.

 

To answer the question, No!

 

(edit) If you are looking for resampled image recognition you could probably do that with a pixel map and something like AutoIt's pixel search which has a pixel "range of agreement". You would need a lot more pixels though. Looking at forms with uniform colour entry boxes should be easier to handle.

Edited February 25, 2010 by JohnS

[rberq]

Isn't the pixel mapping that has been mentioned numerous times in the forum the same thing?

No, it is related, but not the same at all. With a pixel map, you can look at a spot (or many spots) and see whether what is there is what you expected to be there. That's kind of the opposite of knowing that what you want is (or may be) somewhere on the screen, but you don't know where.

 

Granted, you could write a macro that prompts you to select two opposite corners of a rectangular area, then saves a pixel map of the rectangle at whatever resolution you choose. That is easy. Then, in theory at least, another macro could scan a screen matching the original pixel map against every same-sized rectangle to see if the map matches a section of the screen. But my PC is not a Cray-15 or an IBM Sequoia. As a practical matter, the matching has to be done by a fast, pre-compiled program. A run-time-interpreted script like Macro Express is orders of magnitude too slow to do the task in a reasonable time.

[Yehnfikm8Gq]

A pixel map is simply a set of colours at coordinates. There are at least two ways of using one. I would use a moving map. Assumed a rectangular map say 4x4, 16 points, spaced 1 or more pixels. I would take the top left pixel, search for that, if found check the bottom right. If not correct continue search with top left pixel. Repeat until bottom right is correct. Then check entire grid of 16. Experience will show if checking with top right pixel is preferred (and moving on) before doing full check of 16 points. Experience may also require two 4x4 grids.

 

As was pointed out in another thread, pixel searching is slow with ME. The script I did with AutoIt and my old PC scans my screen in 2 secs. A map is going to take a bit longer.

[Yehnfikm8Gq]

I got a pixel mapping macro working fine. Used the same principles as detailed previously.

 

"Set" macro maps colours of grid of 16 pixels spaced 4x4, 1 or more pixels apart - choose to suit graphic

Top left of grid set by mouse position prior to sample

Store data in ASCII file

 

"Search" macro:

Reads ASCII file for all data including pixel spacing

Searches for first occurrence of top left pixel matching pixel on the "new" graphic/webpage

If top left pixel correct checks bottom right pixel

 

If bottom right pixel incorrect resumes top left pixel search

If bottom right pixel correct checks all 16 pixels

 

If all 16 pixels don't match resumes top left pixel search

If all 16 pixels match returns top left coordinates

 

In order to test a failure on the 16 pixel match and find another I had to make a special graphic. On normal graphics it was always correct (ie the two pixel match found the correct rectangle). Webpages with more uniform colours may be more tricky and need the full 16 match to be sure. It should be easy to piggyback a second rectangle for difficult images. Work in progress.

 

The search is very slow using ME as rberg predicted (no prizes!) although can be considerably sped up with reduced search area or larger grid. I may try and modify my AutoIt-assisted ME Pixel Search macro to do this function.

[rberq]

That's an clever approach, John. It will be interesting to see whether the AutoIT script runs fast enough to be practical.

 

One thing you might think about: I have noticed that some Web page renderings, even though they are "identical" from one day to the next, seem to be off by a pixel here and there, at the boundaries between two colors. If true, this makes your absolute reliance on the corner pixels a weak point. I wonder how it would work if you chose random pixels to match, rather than specific (corner) ones. The random sample could be chosen either when developing the original map, or at search time. Probably it's more efficient to choose the sample only once, when developing the original map.

 

Then when searching you would not require 100 percent matches between map and screen -- to allow for the "expected" tiny differences from day to day in screen formatting. Probably matching 90 to 98 percent of the pixels would be enough to make the identification.

[Yehnfikm8Gq]

Pixel changes are a good point. I had thought about how a webpage can change. I don't see a problem with dialogs etc assuming the page is viewed at the same zoom level - this method has all the problems of Mouse Move. I was more concerned with any graphics that may be resampled differently and things like Clear Type. It's possible that colour ghosting we see on the screen are not there as far as the PC is concerned (it's a monitor distortion). I remember others commenting on how things shift 1 pixel. I presently have 3 pixel spacing. It would be hit and miss if you landed on an aberration.

 

The grid is essentially random when used on the later page. When "Set", the mouse is put in an approximate location. The fact that top left corner is the main search point makes it no different to using any of the others. There could be good reason to line up the "Set" accurately to land later on a particular pixel but that makes it deliberately non-random.

 

It would not be difficult to do voting on the 16 pixel check and or a colour tolerance but that may be missed if the 2 pixel check fails. Colour tolerance on the 1 and 2 pixel searches will slow things down significantly.

 

As it is, the extra checks should not add significantly to the time. The 2 pixel check is allowing the correct pixel to be chosen and the 16 pixel check is only done occasionally. Switching to a 1 then 16 pixel check may add quite a bit of time. Also, there is a lot more logic than a simple search so my AutoIt script modified may not be quick enough. Being able to restrict the search to even 1/4 the screen speeds things up 4x. I just thought of another problem - AutoIt works in decimal RGB not BGR per ME. BGRrrrr! I think I wrote a script for that recently.

 

I modified the macros because the "Search" does not need to knows the X,Y coords of the "Set", it only needs to know the spacing of the grid. Naturally that introduced errors that took forever to find! My limit on this project is likely to be getting the AutoIt working in a practical fashion and leaving custom settings to others.

 

(edit) I wrote the "Search" macro with the 2 and 16 pixel comparisons as separate "Subroutines". That made it so much easier to follow the flow of the program and to jump back from those comparisons to resuming the 1 pixel search. Full marks to ISS for adding the GoTo command.

[Cory]

Would adding a tolerance help you?

.......................................................

Look back awhile and you will find one of my posts talking about this. I studied it all very carefully and came up with a macro that would calculate the individual RGB values from decimal. It works very quick and allows one to have a tolerance. EG subtract the value in the macro from the value the macro found and if the difference is within the tolerance specified allow it. Later I found a programmatic way to do it negating the need but I did play with it long enough to prove the concept.

[Yehnfikm8Gq]

Thanks for the reference. I have written a number of colour conversion macros from other threads, I have no idea about speed. The problem is that ME only reads in decimal BGR so you would have to convert to Hex BGR (RGB is an unnecessary step). The tolerances on the B, G and R would have to be calculated in Hex which takes time in ME. I have no idea how you would do a comparison - it would have to be done on each colour component separately since tolerances on the combined number are meaningless. You could also convert further to decimal format eg 255, 255, 255.

 

The macro I'm starting using AutoIt already has a colour tolerance parameter. I have to look into how that works. I doubt I will bother with tolerance but will write so others could mess with it.

[Cory]

Converting decimal to hex is not difficult.

...........................................................

Simply divide by 16 taking the remainder several times. Then the interesting bit is that you literally split the string into the three chunks. You see when they make a decimal value they're literally taking the three as if they were one number. The analogy is how a SSN is actually 3 numbers but you can represent it as an integer. And of course when you have the hex value you can convert it back to integer to apply a tolerance. And math functions seem to work pretty quick in MEP. But I'm sure if you were scanning millions of pixels it might be slow. But in that case I'd probably employ a binary search.

[Yehnfikm8Gq]

Agreed the Hex conversion is easy. It's just a bunch more code. Once it is sent to AutoIt it gets converted to decimalRGB. AutoIt's colour tolerance is a decimal number for each 0-255 colour value. That's pre-programmed as a parameter:

PixelSearch ( left, top, right, bottom, color [, shade-variation [, step [, hwnd]]].

 

The PixelSearch in AutoIt is very fast but it is designed to scan a rectangle and find the first occurrence. I'm having a few debugging problems getting it to work with multiple occurrences. For comparison, in AutoIt if I increment x and y to do a search it takes 13 secs for the screen. PixelSearch does it in less than one second. If running from ME, it takes about one second for the AutoIt exe to start. I expect to have some time delays handshaking between the two apps. Work still in progress.

[Yehnfikm8Gq]

Here's an update on my multi-point pixel match macros (not that anyone probably cares!). I have the macro working well, running PixelSearch as a standalone AutoIt exe in conjunction with ME. Here's what I've developed/found:

 

Mapping

 

A hotkey ME macro maps the 4x4 grid (presently 3 pixel spacing) to the right and below the chosen pixel. A Multi-Choice Display gives the option of entering X and Y for the reference point (followed by Variable Set Integer dialogs) or using the mouse position. A dialog tells you to place the mouse in the spot you want to find. Hit OK and it records the colour value at each point of the grid into a text file, together with the grid spacing. Also the x, y of the mouse location for reference and testing only. If part of the grid is off the screen at the edge, it substitutes -1 in place of the colour value and tells you that it will now be a 12 point match (or less) rather than the full 16. At present this will not occur left and top edges because the grid is right and below the mouse point.

 

Using

 

A separate macro is used for the search running an AutoIt exe (AiT). At the moment the search area is set via dialog but could be written into the macro. Presently based on screen position only, not window position. The macro reads the map and puts the colours into an array. The target colour is converted from Decimal BGR to Hex RGB and into an ini file with other info. It starts up the AutoIt exe file which reads the search area boundaries, the grid spacing, the colour to search for and colour variation (currently 0). It takes about 1 sec on my old PC to start up the exe, less than 1 sec to scan the screen if nothing found.

 

The exe searches for the first colour occurrence. If not found it tells ME via the ini file the search failed. If found, the X and Y are passed to ME which checks the colour at the bottom right corner of the grid (2 point match). If not a match, ME tells AiT to find the next occurrence. If there is a 2 point match, it then does the 16 point match. If matched, a dialog tells you a match is found; whether it's 16 point or less and the X,Y location. If not matched ME tells AiT to continue the single pixel search. This could all be done in AiT but I have chosen to put as much in ME as possible.

 

Where the grid goes off the screen, all the comparison values that were set to -1 during mapping are ignored (ME puts them equal to a full match)

 

Usage Issues

 

1. As rberg intimated, sometimes using the top left grid pixel is not ideal. You may want to use another, or even random. That complicates the search macro considerably. Where part of the grid is offscreen right or bottom edge, it would be better to use bottom right rather than top left as the grid reference. Also, scanning backwards from the bottom right corner may be desirable. That is easy to do in AiT. Were I doing this for commercial reasons I may have incorporated these. It's too much work when doing for fun.

 

2. To quote the obvious, the larger the search area the longer it may take to scan. I found that searches can be very long if you choose the wrong pixel. Very quick if the pixel is unique or has one or two occurrences. Every time a pixel is matched, AiT tells ME, ME does a check, tells AiT to continue all via ini. Using the Registry is about twice the speed on my PC. If the entire row is the same colour, AiT is using PixelSearch pixel-by-pixel which is very inefficient. Choice of target is important. See "Additional Tools" para.

 

3. If you incorporated colour variation, the search could be very long. At that stage I would do all the number crunching in AiT since it has the built-in colour variation parameter. Doing it in ME would be a pain.

 

4. To go to search by window would not be that difficult but is simply a lot more code to write so I have not bothered. Pixel location macros are best done with a fixed display for repeatability eg application fullscreen.

 

5. Forms and icons are more problematic since colours tend to be uniform and appear many times. Reducing the search area helps considerably.

 

Additional Tools

 

I found that moving the reference point may change the number of pixel matches from 1200 to 1. Clearly it helps to use a unique pixel. I wrote a macro that you would use prior to any mapping. You put the mouse at what you think is a good location. ME starts an AiT External Script which quickly scans the screen from 0,0 to the reference point and tells you how may occurrences (and presently/optionally lists an array with their X,Y). Currently it stops at 500 occurrences (to save time) and tells you it's not a good choice. For less than 500, the X,Y are copied to the Clipboard.

 

I have to tidy up the code and will probably post. There's a heck of lot of it.