Finding A Javascript Button In A Webpage?

[mcrae]

after reading the forum, the only possible way to do this that i can see is to scan the screen, pixel by pixel, left to right, and when the macro detects a certain combination of pixel colours in a row, it can record where on the screen those pixels are.

 

is this possible?

is there a more efficient method?

 

how exactly do i write a script liek this that will scan the screen looking for a certain set of pixels?

[ol3ears]

Hi

 

To efficiently scan a screen try...www.users.on.net/~brae/

There is a screen view application that I have written to do just this

I use if for web and citrix applications

 

I would however recommend you refer to the Me and Javascript thread and simply call the onclick function directly.

 

Regards

[mcrae]

Thanks for the quick answer!

 

However... maybe I wasn't clear enough. The web page isn't mine, and i can't edit it. From what i saw in the Me and Javascript thread (I know nothing about programing html/java), that code is meant to put on a webpage.

 

I simply want to go to a webpage, find a button (which isn't always in the same place), and click it.

[Cory]

I think it would be fairly easy to scan all the pixels in a window line by line hunting for a series of pixel colors. As it scans evaluate the last series of pixels of a certain width and when you it matches the pattern you recorded earlier use those coordinates to click. I think it would be pretty straight forward.

 

Also you might do a more clever search. Let’s say it’s a gray button on a green background. Do a loose grid search like playing “Battleship” where the grid is smaller in X and Y than the button to get you in the ballpark then to amore refined search if necessary. Just depends on how detailed the background and button are.

[mcrae]

The onclick function that ol3ears mentioned sounds promising, I wonder if you could tell me how to use it?

 

And Cory, could you explain how I would do what you suggest? What commands would I use?

[joe]

Hello ol3ears!

 

To efficiently scan a screen try...www.users.on.net/~brae/

I've downloaded and have been looking through the ScreenView executable documentation and library. Specifically the Hot.bmp and Hot.txt example. I understand the concept of what you are doing, but some of the details escape me.

 

Please correct me if I have any of this wrong:

 

ScreenView takes its comparison instructions from the Hot.txt file, one line (instruction) at a time. In your example it scans the whole screen looking for a pixel that matches the color of the point at 8,7 in the Hot.bmp file. When it finds it, it then attempts the next comparison, looking for a "cross" shape on the screen (relative to the point it just found) that matches the cross at 31,14 in the Hot.bmp file. This cross shape is a 3x3 pattern that demands a comparison of 5 pixels. The center of the cross being at 31,14. It then compares another cross at a different set of coordinates, and finally it compares 330 pixels in a 30x11 "block" shape at coordinates 3,5. If each comparison is true, then ScreenView found what it is looking for and returns the X and Y screen coordinates to a Registry string along with the width and height values of the graphic contained in the Hot.bmp file.

 

Some other questions:

• Is the OffCross pattern a 3x3 pattern that compares the 4 points on the corners?
  
• What are the details of the Star pattern?
  
• Is the "delta" parameter of the HLine and VLine patterns the line length?
  

One more thing. Although I understand the how for the BGRrange, I don't understand the why. Is there something that tells ScreenView that the pixel it is comparing is a background pixel as opposed to a foreground pixel?

[Cory]

I’m sorry but I don’t have time to go into detail as I need to get back to billing some hours but I think I would get the screen size of the window first as a limit and write that to integer variables. I would then create a nested repeat loop for the X and Y values. I would scan in every pixel color at every coordinate across but only maintain the width of the button signature. IOW if the button signature was 100 pixels (could be smaller probably) wide I would only maintain that many in the string var. Write a tab to separate each color and every loop you append to the end and trim off from the front. You can use the tab to determine the position of the first one and trim off that many characters. Each time you test to see if that var matches your signature. When it does execute a mouse move and click at that position. You might need to modify the coordinates to add an offset to hit the center of the button.

 

But let me get back to the signature. I would first write a special macro to get the signature. I would ask the user to center their mouse over the button and then I would read the pixels starting 50 from the left and counting over 100 (in this example). Then hard code this into the macro as the signature.

 

I just had another thought. Check to see if there are any unique colors in the button. IOW a color that doesn’t exist anywhere else on the page. Let’s say it was a gray button on a blue background. Then this would all be a lot simpler. Just scan line by line for that color and fire a mouse click at it when it’s found. You could start by logging all the pixel colors to a file and looking where the button is to see if there are any unique. Oh, here’s what I would do… Do a screen capture and use a simple graphics editor like Irfanview (Free!) . Create two files and on one cut out the button so it’s just black there. Then conversely clip out the button and paste it to a new file. Then have ME scan every pixel and log every unique color in each to separate files. Then have ME look to see if there are any pixels in the button set that do not occur in the screen set. Seems likely you might find one. Then you can use this as your signature pixel color and avoid having to look for patterns.

[mcrae]

Thanks for your time and effort Cory, this is much more complex than I thought it would be, so I'm going to give up. If I do ever get around to it though... I'll be back!

 

Have a good one, and thanks

[Cory]

It's not as bad as it sounds but it would probably take a couple of hours so as always you have to weigh if it's worth the time. I've been wanting to try this out now for some time as often people post this same sort of problem but I've never found an application that was worth the time.

[ol3ears]

Hi Joe

 

Screen view will scan the whole screen unless you give it a set of constraints

All .txt coordinates are relative to the top left of the .bmp file

 

The return coordinates are also relative to the position of the top left of the .bmp file per the screen position the image was found at. Screen view returns the height and width of the bitmap so that the api can calculate the centre position of the bitmap (ie the centre position of a button is usually more useful than its top left corner)

 

Is the OffCross pattern a 3x3 pattern that compares the 4 points on the corners?

Yes and the cente point

 

What are the details of the Star pattern?

combination of cross and offcross

9 points in a 3*3 block

 

Is the "delta" parameter of the HLine and VLine patterns the line length?

Yes (base 0 numbering) so line length +1

It doesn't matter if you go off the boundaries of the screen or the .bmp

 

A lot of the commands were from early versions before I put the block command in. They are there to minimise the time ScreenView spends scanning irrelevant regions of the screen. For most fast PC's it might be easier to forget them and just use a point command for a key colour to get you in the neighbourhood and then use the block command.

 

Although I understand the how for the BGRrange, I don't understand the why

This was to solve a problem I had reading a yellow javascript menu on a graduated blue background. While it was possible just to read the yellow text, without the background colour giving the text context it was too easy to get false positives. However there was no guarantee what the background colour would be in the various menu states. Hence the bgrrange command

 

This stands for blue green red range - it takes two 24 bit bgr values as decimal integers. Range low and range high. The mechanism to tell screen view to use the range value or not is that all points already defined that fall within the bgrrange are determined to be bgrrange colours and will return true in comparisons if the points fall within the colour range

 

Regards

Bruce

[lemming]

after reading the forum, the only possible way to do this that i can see is to scan the screen, pixel by pixel, left to right, and when the macro detects a certain combination of pixel colours in a row, it can record where on the screen those pixels are.

 

is this possible?

is there a more efficient method?

 

how exactly do i write a script liek this that will scan the screen looking for a certain set of pixels?

I've written a macro that does exactly that.

 

It's probably not as fast as ol3's program, but it has the advantage of being a "pure" Macex script. Plus it's fairly easy to understand, IMHO.

 

I've uploaded it into the Third Party Tools section

 

-Lemming.