Search for a pixel

[acantor]

I have a challenge for you all!

 

There are occasions when the most reliable way to target a screen object is to "hunt" for a pixel colour that appears only in that object. For example, in the ME Pro Script Editor, in the default "M" icon to the left of the "Macro Nickname:" label, there is one pixel with colour = 7004146. (The "Mouse Locator" shows positions of the pointer as well as pixel color.)

 

The AutoHotKey "PixelSearch" command is lightening fast, and easy to use. There is no equivalent command in Macro Express. I have put in a feature request. Hopefully, our friends at Insight will come through with a new command. But in the meantime, I need a way to search for a pixel in Macro Express.

 

It was not horribly complicated to write a ME script to search for a pixel colour within a specified region (e.g., a rectangle). But my script runs slowly. For example, on my PC it takes 15 seconds to test a 400 by 400 pixel rectangle. AutoHotKey's PixelSearch chews through 1280 by 1024 pixels in under a second.

 

I would welcome suggestions on how to make this kind of script run faster.

 

If you are going to experiment with my script:

 

1. My script contains several disabled lines that I enable for testing. However, these lines are not disabled in the codebox. I suggest that you import the attachment. Otherwise, the script will run even more slowly than usual.

 

2. The macro that I exported is activated by Shift + Ctrl + Alt + \, which you may want to change.

 

// Search a rectangular region of the screen for a pixel colour
// Start at the top-left. Move right along a horizontal line, testing each pixel. At end of row, go to start of next line, and try again.
// (Enable the disabled lines for testing and debugging)

// Specify pixel colour to search for. Use "Mouse Locator" to obtain value, or use one of the values below...
Variable Set Integer %PixelTarget% to 7004146 // This colour appears in the ME Pro Script Editor, in the "M" icon to the left of "Macro Nickname:"
Variable Set Integer %PixelTarget% to -1 // This colour never appears. Enable this line to test for case when pixel cannot be found

// Specify (x, y) of the top-left corner of the search rectangle...
Variable Set Integer %xStart% to 0
Variable Set Integer %yStart% to 0

// Specify (x, y) of the bottom-right corner of search rectangle...
Variable Set Integer %xEnd% to 400
Variable Set Integer %yEnd% to 400

// Calculate length and height of search rectangle:
Variable Modify Integer: %xLength% = %xEnd% - %xStart%
Variable Modify Integer: %yHeight% = %yEnd% - %yStart%

// Set coordinate to test. (Start in upper-left corner)...
Variable Set Integer %xCurrent% to %xStart%
Variable Set Integer %yCurrent% to %yStart%

// Start with the first "row" of pixels. We test up to yHeight rows...
Repeat Start (Repeat %yHeight% times)
// Test each pixel along a horizontal line xLength pixels long...
 Repeat Start (Repeat %xLength% times)
Get Pixel Color at (%xCurrent%, %yCurrent%) Relative to Current Window into %PixelColour%
Mouse Move: %xCurrent%, %yCurrent% Relative to Current Window // Use for debugging. Lets you see search progress, but it's slow!
If Variable %PixelColour% Equals "%PixelTarget%"
// Pixel found...
  Mouse Move: %xCurrent%, %yCurrent% Relative to Current Window
  Text Box Display: Found!
  Macro Stop
Else
// Increment x so we test next pixel to the right...
  Variable Modify Integer %xCurrent%: Increment
End If
// Continue until the entire row is tested...
 End Repeat
 // Reset x to its original value, and increment y so we can test pixels in the next row...
 Variable Set Integer %xCurrent% to %xStart%
 Variable Modify Integer %yCurrent%: Increment
End Repeat

// Pixel not found...
Mouse Move: %xEnd%, %yEnd% Relative to Current Window
Text Box Display: Not Found!

Pixel_Search.mex

Edited January 22, 2010 by Alan
changed 'codebox' tags to 'code' tags to eliminate horizontal scrolling

[Yehnfikm8Gq]

Most people would find other ways of identifying screen changes or, given the utility you mentioned, invoke that with MEP to do the legwork. MEP is great for ease of use but where it is slow, it is better to use it to control other apps. Not a big deal IMO. Now that MEPro can run external scripts, that's another route - minus the ease of programming!

 

You have to be quite desperate to be searching for pixel colours generally. I've usually found non-pixel ways of doing it. You may be looking at pixel colour to tell if a box has appeared but that's not one-pixel sensitive. My main pixel colour usage is in telling whether Firefox has loaded a page. I went the route of the custom throbber which you design to give a predictable area of uniform colour. I cannot envisage any method for speeding up the macro you specified other than if some parts of the rectangle are more likely to contain the pixel than others (have MEP build a history of past search successes and start there!)

[acantor]

Most people would find other ways of identifying screen changes or, given the utility you mentioned, invoke that with MEP to do the legwork. MEP is great for ease of use but where it is slow, it is better to use it to control other apps. Not a big deal IMO. Now that MEPro can run external scripts, that's another route - minus the ease of programming!

 

You have to be quite desperate to be searching for pixel colours generally. I've usually found non-pixel ways of doing it. You may be looking at pixel colour to tell if a box has appeared but that's not one-pixel sensitive. My main pixel colour usage is in telling whether Firefox has loaded a page. I went the route of the custom throbber which you design to give a predictable area of uniform colour. I cannot envisage any method for speeding up the macro you specified other than if some parts of the rectangle are more likely to contain the pixel than others (have MEP build a history of past search successes and start there!)

 

I use "pixel hunting" strategies as a last ditch effort, when all else has failed, to locate, click on, or to give focus to screen objects. Recently I developed over 50 scripts for a non-standard application that had no detectable controls, and displayed no cursor changes. But there were unique colours in predictable areas of the screen, and I developed many scripts that took advantage of the pixel colours that were visible, or could be made visible via scrolling. To my surprise, these macros have proven to be extremely reliable and stable. I could find no other way to get the job done. It was not possible, for example, to move the mouse cursor to a specific coordinate, because the location of the target changed in response to the user's input.

 

Normally, my pixel colour macros do not search large regions. They usually check pixels along a single vertical or horizontal line, so run reasonably fast.

 

But recently, I encountered a situation where the target colours could appear anywhere on the screen. Macro Express was not up to the task. I ended up using AutoHotkey for this script, but I wished that I could do the same thing in Macro Express.

[Yehnfikm8Gq]

That really is desperation! I've been learning AutoIt so this was an opportunity to try out my code on others. Below is code that seems to work. You would need to have AutoIt installed. I thought about making a standalone exe file but this is it for now. Takes about a second on my pc if nothing found (whole screen searched). I did not try your totally-MEPro script so have no comparison. You will need to change input and output presentations to suit your needs. It will return the X,Y coords or a message "Color not found".

 

// Rectangle Coords
Variable Set Integer %left% to 0
Variable Set Integer %top% to 0
Variable Set Integer %right% to 1024
Variable Set Integer %bottom% to 768
// Color variation 0-255   None=0
Variable Set Integer %colorvar% to 0
// Step  1=every pixel, 2=every 2 etc
Variable Set Integer %step% to 1
// Color as Decimal or Hexadecimal
Variable Set Integer %pixelcolor% to 0x990066
External Script: AutoIt // Output in %coordxy%
Text Box Display: Result of Pixel Search,   X,Y

 

<COMMENT Value="Rectangle Coords"/>
<VARIABLE SET INTEGER Option="\x00" Destination="%left%" Value="0"/>
<VARIABLE SET INTEGER Option="\x00" Destination="%top%" Value="0"/>
<VARIABLE SET INTEGER Option="\x00" Destination="%right%" Value="1024"/>
<VARIABLE SET INTEGER Option="\x00" Destination="%bottom%" Value="768"/>
<COMMENT Value="Color variation 0-255   None=0"/>
<VARIABLE SET INTEGER Option="\x00" Destination="%colorvar%" Value="0"/>
<COMMENT Value="Step  1=every pixel, 2=every 2 etc"/>
<VARIABLE SET INTEGER Option="\x00" Destination="%step%" Value="1"/>
<COMMENT Value="Color as Decimal or Hexadecimal"/>
<VARIABLE SET INTEGER Option="\x00" Destination="%pixelcolor%" Value="0x990066"/>
<EXTERNAL SCRIPT Language="AutoIt" Dest="%coordxy%" Script="; Find pixel entered as decimal or hexadecimal in defined rectangle\r\n$left = \"%left%\"\r\n$top = \"%top%\"\r\n$right = \"%right%\"\r\n$bottom = \"%bottom%\"\r\n$colorvar = \"%colorvar%\";color variation 0-255\r\n$step = \"%step%\" ;steps, optional, 1=every pixel\r\n$pixelcolor = \"%pixelcolor%\"\r\n\r\n$coordxy = PixelSearch( $left, $top, $right, $bottom, $pixelcolor, $colorvar, $step)\r\n\r\nIf Not @error Then\r\n;Optional message from AutoIt\r\n;MsgBox(0, \"X and Y\", $coordxy[0] & \",\" & $coordxy[1])\r\n	ConsoleWrite($coordxy[0] & \",\" & $coordxy[1])\r\nElse\r\n;Optional message from AutoIt\r\n;MsgBox(0, \"Warning\", \"Color not found\")\r\n	ConsoleWrite(\"Color not found\")\r\nEndIf\r\n" _COMMENT="Output in %coordxy%"/>
<TEXT BOX DISPLAY Title="Result of Pixel Search,   X,Y" Content="{\\rtf1\\ansi\\ansicpg1252\\deff0\\deflang1033{\\fonttbl{\\f0\\fnil\\fcharset0 Tahoma;}{\\f1\\fnil Tahoma;}}\r\n\\viewkind4\\uc1\\pard\\qc\\lang4105\\f0\\fs16 Color %pixelcolor%\r\n\\par %coordxy%\\lang1033\\f1 \r\n\\par }\r\n" Left="Center" Top="Center" Width="177" Height="107" Monitor="0" OnTop="FALSE" Keep_Focus="TRUE" Mode="\x00" Delay="0"/>

[Yehnfikm8Gq]

If you are not interested in installing AutoIt, I have written a standalone exe. Unfortunately, although it works fine, I can't test it as one would wish since I have AutoIt installed and I can't tell if it's using that by some roundabout means. Suggestions from experts?

 

I have a second MEPro script similar to the previous one. Instead of calling the script it does the following:

 

Writes all the parameters to a file c:\temp\pixelsearch.ini

Runs an AutoIt file PixelSearchfromIni.exe

That imports data from the ini file

Does the pixel search

Exports the info back to the ini file

The MEPro script reads the X, Y, and whether the colour was found

Deletes the ini file

 

It takes a fraction of a second longer than the scripted version. I can post if you are interested.

[paul]

If you are not interested in installing AutoIt, I have written a standalone exe. Unfortunately, although it works fine, I can't test it as one would wish since I have AutoIt installed and I can't tell if it's using that by some roundabout means. Suggestions from experts?

If you're asking whether an executable, created from an AutoIt script, will run on a machine where AutoIt is not installed, then the answer is Yes.

[paul]

One technique I'd strongly recommend to you is to isolate the pixel colours you're interested in from the script, and have a separate script for defining them.

Pixel colours are absolutely dependent on the video card you're using, and possibly the OS too. Setting the pixel colours in a separate macro (which you call from any macro that uses pixel colours) makes it far easier to change your code when you change your video card, or want to run your macros on a different machine. This is a technique I've learnt from bitter experience!

[Yehnfikm8Gq]

If you're asking whether an executable, created from an AutoIt script, will run on a machine where AutoIt is not installed, then the answer is Yes.

 

I realize that. The problem I have is that when I test on my PC where AutoIt is installed, I don't know whether the exe is running correctly or whether it is using the installed AutoIt files. Ideally you need to test on a PC where AutoIt is not installed. Then you know if it works as standalone.

 

As far as colours go I'd have to disagree. I'm not sure what experience would lead you to think otherwise. When pixel colours are grabbed it should be from the data going to the graphic card. Otherwise the standard ME colour tools would not work. If you take standard Windows grey 131060660, that is always the same. Doesn't matter what PC, monitor or graphic card you use, it's the same as far as the system is concerned. One thing you cannot do is to zoom in or out, say on an image after the sampling. The pixel colour will normally change, if not by simple displacement, by resampling.

 

I'm not sure what happens with overlays - I would guess you could only measure the background colour which is normally black. Tested that with my TV app, #100010 or 16,0,16 RGB, curious. I'm not sure about when applications use colour management. For the sort of tasks Alan is most likely doing, that is irrelevant. If you are doing screen calibrations with photometric devices they will be affected by all parameters.

 

I'd be interested in your experience. I do a huge amount of graphics processing and I've never seen variations between colour picker results and applications. What would happen with websafe colours?

 

The script as shown is outline only. As it's written, you change values in the script and save - test only. As I pointed out to Alan, he will have to sort out his own I/O depending on how he uses it. I would assume for automatic operation you would store the required colour as a variable and use the fact the colour has been found or not to do the necessaries.

 

I have run into a really peculiar problem. After all my successful testing I found it did not seem to work. It was easy to find out why but I have not resolved the issue yet:

 

Mouse Locator gives two values. One is a Windows colour number which seem to be (BBGGRR). The GRRRRR is poignant! The other is the HTML Hexadecimal value (#RRGGBB) minus the # of course. AutoIt works with Decimal or Hexadecimal (HTML) colours but not Windows colours. For example, the pale yellow information area in ME Explorer is as follows:

 

Mouse Locator 14811135 (BGR 225,255,255)(E1FFFF in hex) and 0xFFFFE1 (RGB 255,255,225)(16777185 in decimal)

AutoIt PixelSearch will work with 0xFFFFE1 or 16777185

 

That makes it more awkward to use because the Mouse Locator only returns the Windows colour. You would have to manually enter the hexadecimal value in the macro. I have not tackled the conversion yet but maybe PGM have!

 

(Edit) I've made a few more scripts to assist with colour handling, both in dialogued and blind (no user involvement) versions:

Decimal to Hex conversion

Decimal Windows BGR to Hex RGB conversion

The latter would be used if obtaining colours from ME and the mouse, then using the data for PixelSearch or other apps needing Hex RGB. What a pain.

Edited January 23, 2010 by JohnS

[paul]

As far as colours go I'd have to disagree. I'm not sure what experience would lead you to think otherwise. When pixel colours are grabbed it should be from the data going to the graphic card. Otherwise the standard ME colour tools would not work. If you take standard Windows grey 131060660, that is always the same. Doesn't matter what PC, monitor or graphic card you use, it's the same as far as the system is concerned. One thing you cannot do is to zoom in or out, say on an image after the sampling. The pixel colour will normally change, if not by simple displacement, by resampling.

In the days when ME3 had no "Move Mouse to Tray Icon" command, I wrote a macro to search for a target tray icon by starting at the right-most bottom pixel of the tray area and searching for a predefined block of pixels matching my target icon. For example, I do some processing with Roboform, for which I need to right-click on the Roboform icon.

Having written a set of macros that worked reliably on my home machine, I then upgraded my machine, ending up, of course, with a different video card (actually plural because I work with 3 monitors). And I discovered, to my considerable surprise, that the coordinates of the tray area had changed by a single pixel, and that all the colous of my predefined icons had changed.

I then tested my macros on several other machines, and in all cases all colours were different.

 

Of course I don't use that technique any longer because of the presence of the "Move Mouse to Tray Icon" command. But I have developed several other macros for use with MS Access to allow me to change window heights, for which I need to use the various colours of several components of Access windows, e.g. title bar, resizing bar, inactive areas on the status bar, etc. And once again, these colours differ from one machine to another (but note they all use the same Windows scheme, and they all have precisely the same RGB colours for the various components of the desktop).

 

And having recently moved up to Windows 7 x64 (which of course required a complete reinstall), I find that some colours are different to before on the same machine using the same 2 video cards.

[rberq]

Having written a set of macros that worked reliably on my home machine, I then upgraded my machine, ending up, of course, with a different video card (actually plural because I work with 3 monitors). And I discovered, to my considerable surprise, that the coordinates of the tray area had changed by a single pixel, and that all the colous of my predefined icons had changed.

I have macros that identify windows by checking color "maps" with a few dozen pixel locations in each map. Like you, I also have seen that pixel locations can change slightly -- even within the same PC from one day to the next! -- so I am careful to map to the middle of fields, and don't check too close to the edges. Also, color value changes on a machine running 16-bit vs. 32-bit color, so I have two sets of color maps. But I have not seen any problems from machine to machine, other than those two differences. The macros run on dozens of PCs of different varieties.

 

P.S. (Oh, yes. Pixel location may change, depending on the application, if you change screen resolution. For some applications it matters, for some it doesn't.)

[acantor]

I have macros that identify windows by checking color "maps" with a few dozen pixel locations in each map. Like you, I also have seen that pixel locations can change slightly -- even within the same PC from one day to the next! -- so I am careful to map to the middle of fields, and don't check too close to the edges.

 

Whether or not colours vary with OS, display settings, video card, or whatever, this sounds like a situation where pixel hunting might reduce complexity. Although it might be necessary to manually set a few values, once running, a macro hunting for pixel colours is an accurate way to differentiate windows. For example, I have scripts that do things like this:

 

A. Search along a horizontal line for a pixel of one colour.

B. Search along a vertical line for a pixel of another colour.

C. Determine the pixel colour at position x, y.

 

If A and B are found, and C = abc, assume this is screen 1

If A and B are found, and C = def, assume this is screen 2

If only A is found, assume this is screen 3.

If only B is found, assume this is screen 4.

Otherwise, signal that the screen is unknown.

 

Macros like these are never *truly* elegant. As I mentioned earlier, I use techniques like these as a last ditch effort. But once you have a handle on the techniques, they can be powerful ways to zero in on screen objects, or infer information about a window.

 

Maybe the question should be this: If Macro Express had a fast pixel search command, do you think you would find ways to use it?

[Yehnfikm8Gq]

Re Paul and rberg,

 

My PCs are similar, XP, same res, different graphics, different graphic card settings. One has changes to brightness and gamma within the card settings to give similar naked eye appearance. I don't think global utilities like Adobe Gamma would affect the measured colour (by ME or other screen colour picker) nor graphics editors like ThumbsPlus where you can adjust gamma for the app. I'm guessing that colour management is a background activity. It still uses the same pixel value but when it comes to doing something like printing, it says "the monitor looks like this but the printer does this, so change the colour to print to this"). IMO the graphics card should make no difference (to static displays) assuming Windows setup is the same. It may look different to the naked eye.

 

On my PCs, I can find no differences on a per pixel basis in areas where one would expect similarity. In the tray for example, the x coords are going to vary due to which icons are where but on a pixel-to-pixel basis my icons are identical in content and y location.

 

If I were going to x64, I would anticipate there could be differences. 131060660 is so much last year's grey! Could be the same as 32 bit but I would check. If screen colour depth is lowered you would certainly see differences. The icons for example would use the lower res versions and graphics would be posterized - colour puddling for high res pics.

 

I can't explain all of the variations you guys are seeing. Title bars with gradients are iffy. I never use them other than the window disappears and there is a different colour behind. On my setup, the title bar is constant colour for about the first 20-30 pixels from left then the gradient kicks in (depends on window width). My older version of Access seems to follow the same rules as any other window.

 

The tray icons changing colour seems strange. When I was compiling the PixelSearch script I looked at modifying an AutoIt icon to make it unique. Too much work to edit all! The icon stack consists of:

16x16 16 colours

16x16 256 colours

32x32 16 colours

32x32 256 colours

48x48 16 colours

48x48 256 colours

In my setup it would choose the 16x16 256 colours for tray and file listings. My setup would only use others if I used "large icons" in particular circumstances.

 

If I really had to keep settings for each PC and there were lots of PCs, I would use text or ini files for the settings. Way easier to handle for reports and incidental viewing. Getting info out of ME macros is a bit of a pain.

 

I'll bear your comments in mind for the future.

[Yehnfikm8Gq]

I should elaborate on overlays for anyone trying to Get Pixel Color with ME. You can't do it and it's one scenario where what you see is entirely from the graphics card, not from any Windows action. I'm not familiar with the mechanics of it but here's a simple explanation.

 

If you have a video application such as TV, movie or video game that uses Overlay, the graphics tells Windows to put a black (or similar) rectangle on the screen. All Windows sees is the black. The graphics card puts its display "behind" and everywhere the screen is black, the video shows through. They are normally the same size and location. If you put a colour picker on top, it will only see black (from Windows).

 

Overlays can give some peculiar effects. If you have your TV app running but are viewing photos in a graphic application over the top, you will not normally see the TV picture. However, anywhere the photo is black the TV picture will show through.

[terrypin]

I should elaborate on overlays for anyone trying to Get Pixel Color with ME. You can't do it and it's one scenario where what you see is entirely from the graphics card, not from any Windows action. I'm not familiar with the mechanics of it but here's a simple explanation.

 

If you have a video application such as TV, movie or video game that uses Overlay, the graphics tells Windows to put a black (or similar) rectangle on the screen. All Windows sees is the black. The graphics card puts its display "behind" and everywhere the screen is black, the video shows through. They are normally the same size and location. If you put a colour picker on top, it will only see black (from Windows).

 

Overlays can give some peculiar effects. If you have your TV app running but are viewing photos in a graphic application over the top, you will not normally see the TV picture. However, anywhere the photo is black the TV picture will show through.

 

Another common operation which displays this effect is when trying to take screenshots by 'standard' mehods while playing a video, rather than using the player's special capture tool (if it has one). But it's strangely inconsistent. (Or at least, I haven't deduced the 'rules'.)

 

For example, I can play an MPG in MediaPlayer Classic, hit Prt Scr, and paste the snapshot into IrfanVew to get an accurate, stable result. But if I try that from say VLC Player or WMP 11, it displays the overlay effect, changing as I move the IrfanVew window around the screen.

 

------------

 

Going OT for a minute, I've been trying in vain to post this from Firefox. But I cannot type into the message composition window. Same with a new topic, where I can enter into the Subject and the Topic Description, but not the main window. I tried 5 other forums and could type correctly in each of their message windows. So maybe it's a problem confined to this forum? If so, it only happens with FF. I've opened an IE window, where all works normally, allowing me to to send this.

 

Anyone have any ideas or seen similar behaviour please?

 

FWIW, I've had the text cursor disappear in FF before on several occasions. But I could always still type, even if I had to guess where I was.

 

--

Terry, East Grinstead, UK

[Yehnfikm8Gq]

Another common operation which displays this effect is when trying to take screenshots by 'standard' mehods while playing a video, rather than using the player's special capture tool (if it has one). But it's strangely inconsistent. (Or at least, I haven't deduced the 'rules'.)........

 

What you are seeing is exactly what I described. The issues that you will get with ME and Get Pixel Color are the same as screen captures. Digressing a bit, screen caps, grabs etc are terms freely abused. I differentiate into two types - screen captures and video captures but these are only my usage.

 

Screen capture is capturing what Windows puts on the monitor screen. For most folk making a "screen grab", they are capturing what Windows tells the graphics to display. Capturing the screen on a 1:1 basis gives a capture say 1024x768. If the media player is in "Normal" mode, any playing video will also be captured. Windows is putting it there. Get Pixel Color will also work.

 

If the media player is in "Overlay" mode, where the video should be playing, you get the black rectangle when you do a screen capture. That is all that Windows sees (even though we see video on the screen). The video is being intercepted between Windows and the monitor by the graphics card. The black rectangle is replaced with the generated video (could be a game, movie, TV show).

 

Video capture is capturing the video at source. If you capture video at source size from a playing dvd or TV, NTSC format, the capture will always be 720x480. Video games vary to settings. Most capture programs can also resize the capture so you can have it any size including 1024x768.

 

If you want to access "Overlay" video, you have to use video capture (WMP may be an exception, I don't use it). The capture is usually done with a utility that comes with the graphics card, or by a utility that can figure out how to access the video stream directly. As an example, my ATi graphics has a TV tuner and it also has a function ATi Rage Video Capture. The ATi interface can capture the TV video or stills, but I also use another application that taps into the same video capture stream. I've never come across a color picker that provides this access and most "screen capture" utilities do not. Try and capture with Prt Scr and you get black.

 

It's something of a minefield, but generally speaking failure to register the pixels is due to Overlay mode. Media Player Classic will not capture images if in Overlay; you have to change the Playback Output option.

 

Re OT I've yet to have problems with FF and this forum. I'm often compose in Notepad and copy into the forum message box.

[moombexpecy]

I apologise, but, in my opinion, you commit an error. I can defend the position.