Clicking a button on a web page with unreliable location

[wzeller]

I'm writing a macro that interfaces with a website and loads several pages and processes a bunch of stuff.

 

And, just when I thought I was almost finished, the very last button to click has a special challenge for me.

 

There are a variable number of pop-up ads on the page at both the top and bottom (between 0 and 2, inclusively), and each of those will have at least one link. Which means that both tabbing from the top of the page down and shift-tabbing from the bottom up are unreliable ways to get the focus onto the button. The presence or absence of the top of the page popups also change the physical location of the button vertically so I can't use mouse motion.

 

From what I've read in the forum here, the whole browser window is one big control, so using a control variable is probably not an option either.

 

The browser I'm using is Google Chrome, and I did have one idea that might work but it seems ludicrously roundabout: I could hit Ctrl-F and then type in text matching a label that sits a fixed distance and direction from the button. That will perform a "find on page" search and highlight the label in orange. Then I could search a rectangular area for the first orange pixel and put the mouse there, and then move the mouse in the right direction for the right distance, and then click.

 

But there has just GOTTA be a better way than that.

 

(I'm already running ad blocking software, and if that was working perfectly then the pop-ups would be gone and the problem with them. But some get through, thus the challenge. I even tried disabling the popup blocker figuring at least I'd have a constant number of ads - but it turns out that the number of ads shown really is different from time to time.)

 

Very frustrating.

 

I've spent quite some time going through these forums but haven't found a solution.

 

Any ideas/pointers?

 

w

[Yehnfikm8Gq]

Not seeing the webpage makes it very hard to suggest the right approach. Buttons on webpages may or may not be Controls. I assume you already tried.

 

A few recent contributions have noted the use of search for pixel where there is a unique colour. It was discussed in this thread I don't use pixel search although I wrote a script for ME using AutoIt which works fine. If you want details beyond what I wrote in that post let me know. I haven't got round to making it more generic (standalone exe or data stored under username). Very quick, even faster if you can narrow the search area down. The technique may be less applicable to a widely changing page.

 

What came out of the discussion in that thread for me, and applies to many ME applications, is that the PC doesn't care about what it does. If it works, that's all we care about. Don't dismiss your suggestion just because it's ugly to you.

 

You did not comment as to whether the popups can be terminated. They may have titles/part titles that can be used or the fact they are not the URL/page title you addressed, or they are on top etc etc.

[rberq]

Do you know, in one dimension, where the button is? Here's a method I have used a number of times:

 

Let's say you know the button is at coordinates 123,xxx -- that is you know its horizontal displacement but not its vertical displacement. You move the mouse to something like 123,100, then in a REPEAT loop you increment (or decrement) the y-coordinate a few pixels at a time to move the mouse down (or up) the screen. At each mouse step you do IF MOUSE CURSOR = INTERNET NAVIGATE to see if the cursor has changed to the hand with finger ready to click, meaning the mouse is now on the button.

 

You can search right or left, up or down, or scan a rectangular area depending on how you increment or decrement the coordinates. The biggest problem with this technique is, there may be other objects on the page that will also turn the mouse cursor into the NAVIGATE format. If you run into one of those before you find the button, then you are out of luck, unless you get REALLY fancy and calculate the size of the found object by moving the mouse up-down-left-right, or if you can rely on a color or something else to see if you have the right object.

[wzeller]

Do you know, in one dimension, where the button is? Here's a method I have used a number of times:

 

Let's say you know the button is at coordinates 123,xxx -- that is you know its horizontal displacement but not its vertical displacement. You move the mouse to something like 123,100, then in a REPEAT loop you increment (or decrement) the y-coordinate a few pixels at a time to move the mouse down (or up) the screen. At each mouse step you do IF MOUSE CURSOR = INTERNET NAVIGATE to see if the cursor has changed to the hand with finger ready to click, meaning the mouse is now on the button.

 

You can search right or left, up or down, or scan a rectangular area depending on how you increment or decrement the coordinates. The biggest problem with this technique is, there may be other objects on the page that will also turn the mouse cursor into the NAVIGATE format. If you run into one of those before you find the button, then you are out of luck, unless you get REALLY fancy and calculate the size of the found object by moving the mouse up-down-left-right, or if you can rely on a color or something else to see if you have the right object.

 

 

That's beautiful. And exactly what I'll do. Thanks so much!!!

[Yehnfikm8Gq]

A cruder method than rberg's is to pepper the potential area with mouse clicks. You could use several parameters to determine success; change in pixel colour at a particular coordinate, different mouse cursor, new webpage URL/title. Rberg's method is very neat and can be extended by right-click, Copy Link Location (shortcut may depend on browser). Compare to expected partial if predictable. That will add a time delay but more certainty.

 

Depending on layout variation and code you can click the mouse in a pop-up free area near the button and tab or back-tab from there. You can also click and drag the mouse (right or down), copy text and see how it compares with expected text. That will indicate you are in the ballpark.

[acantor]

You can combine these techniques, as well. For example...

 

Check horizontally, left to right, one pixel at a time, starting at (100, 0) for pixel colour 1234.

 

Let's say it's found at Y = 200.

 

Check vertically, moving upward one pixel at a time, starting at (100, 200) for cursor shape XYZ.

 

Click!

 

---

 

I have written many macros that perform pixel by pixel hunts, and I can tell you that the macros are either extremely reliable or extremely unreliable. I use these techniques when it is not possible to gather underlying information from the application. The scripts infer what is happening "under the hood" by inspecting the user interface, and gathering clues. So there is a lot that can go wrong.

 

I keyboard-enabled Hotmail and Goggle Maps a few years ago, and both worked well until Microsoft updated the look-and-feel of Hotmail, and Google added new widgets. In both cases, it was back to the drawing board.

[Cory]

CTRL+F not so bad.

......................................

I've done many macros with the CTRL+F method and using controls it can go so fast it's nearly invisible to the user. Usually after a CTRL+F I do a Shift+Tab > Tab to highlight the button or whatever and Enter. The exact method will vary depending on your specific circumstances. The important part is that there's no mouse involved.

 

But pixel hunts are a good method and I have done many before that work very well. With a little clever math and logic it can be very efficient. Beware that if you are in Vista or 7 Aero can slow it down a lot.

[wzeller]

Thank you for all the replies!

 

I actually got it working perfectly.

 

Once I realized that I knew one dimension of where it would be and also knew within 200 pixels where it would be in the other dimension, I just set it to move the mouse to the upper-most possibility and then come down one at a time and check pixel color until it finds the color that Google Chrome outlines buttons with on a hover. That's the extreme edge of the button, so I skip down 6 more pixels for safety and left click. The hunt happens so fast you can't even see it.

 

For others in the future that find this thread and decide to do the same thing, I should give one warning: On my first test I had the color number wrong and the mouse zoomed to the bottom of the screen and I couldn't break the loop. I ended up having to kill it with Task Manager. I solved that by making it read the mouse position into a variable each time through the loop and adding an "If Variable" that issues a "Repeat Exit" if the Y coordinate gets too high.

 

So, for the benefits of those future thread finders:

 

  Mouse Move: 748, 379 Relative to Screen
 Repeat Until %COLOR% Equals "15462385"
Mouse Move: 0, 1 Relative to Last Position
Get Pixel Color from Beneath the Mouse into COLOR
Get Mouse Position into (%mousex%, %mousey%) Relative to Screen
If Variable %mousey% Is Greater Than "600"
  Repeat Exit
End If
 End Repeat
 Mouse Move: 0, 6 Relative to Last Position
 Mouse Left Click

 

Thanks guys!

w

[paul]

Once I realized that I knew one dimension of where it would be and also knew within 200 pixels where it would be in the other dimension, I just set it to move the mouse to the upper-most possibility and then come down one at a time and check pixel color until it finds the color that Google Chrome outlines buttons with on a hover. That's the extreme edge of the button, so I skip down 6 more pixels for safety and left click. The hunt happens so fast you can't even see it.

There's no need to move the mouse - the Get Pixel command also works with specific coordinates relative to the screen or to the current window.

And actually, in some cases, moving the mouse can fail where Get Pixel works because the mouse can cause colours to change!

...........

Some moments later:

I see that, in your case, you do need to move the mouse because you do want the underlying colour to change.

In such cases, I think it's good practice to return the mouse to its original location where you're done.

 

And, yes, checking for a max value is a really good idea, as I've learnt from bitter experience (just as you did!). Once you've exited the repeat, you should again check whether the mouse position is within limits, otherwise you're clicking on some point you might not want to click on.

[wzeller]

There's no need to move the mouse - the Get Pixel command also works with specific coordinates relative to the screen or to the current window.

And actually, in some cases, moving the mouse can fail where Get Pixel works because the mouse can cause colours to change!

 

I'm not sure I follow. If the mouse doesn't move then how can Get Pixel know where to get the color from? If you mean I should grab the color from "Specific Coordinates" instead of "Beneath the Mouse", then how would I know what coordinates to use? (If I knew that, then I'd just move the mouse there and click and wouldn't care about the color.)

 

I want it to be as reliable and fast as possible, so I definitely want to understand what you mean, but I think I'm missing something.

 

Thanks,

w

[paul]

I'm not sure I follow. If the mouse doesn't move then how can Get Pixel know where to get the color from? If you mean I should grab the color from "Specific Coordinates" instead of "Beneath the Mouse", then how would I know what coordinates to use? (If I knew that, then I'd just move the mouse there and click and wouldn't care about the color.)

I want it to be as reliable and fast as possible, so I definitely want to understand what you mean, but I think I'm missing something.

You set 2 variables (X and Y) to your start position, use those variables in your Get Pixel command, and increment the X or Y variable after each Get Pixel command, as in:

Repeat Until %iPixel% Equals "%iFrameEdge%" -- iFrameEdge is predefined elsewhere
 Variable Modify Integer %i%: Decrement
 Get Pixel Color at (%i%, %iControlY%) Relative to Screen into %iPixel%
End Repeat

[acantor]

Something I have found when hunting for mouse pointer changes. There are situations when the only way I was able to make the script work was to "wiggle" the mouse pointer before testing its shape:

 

  // Check mouse cursor...
 // But the cursor must be "wiggled" before testing! (Delays alone are not effective)
Mouse Move: 2, 0 Relative to Last Position // Wiggle starts now...
Mouse Move: -2, 0 Relative to Last Position // Wiggle some more...
Mouse Move: 0, 2 Relative to Last Position // Wiggle more...
Mouse Move: 0, -2 Relative to Last Position // Last wiggle...
Delay: 5 milliseconds
If Mouse Cursor: Arrow
  Mouse Left Click
  Repeat Exit
Else
  Continue
End If

[wzeller]

<slap location="forehead" ownership="own">Doh!</slap>

 

I get my brain into a mode sometimes where the blinders come out and the tunnel vision turns on.

 

Thanks for the tip!

 

Marshall

[Yehnfikm8Gq]

Something I implied in another thread on web page search was to put webpage methods in one place. I think it may have been Kevin who said he'd done something in the past but could not spend the time retrieving. This subject, together with methodology for storing information (registry/text/ini/enviro vars) and any other faves could be put into Pinned posts. It's pointless regurgitating the same stuff over and over unless there is a specific issue. A responder could just refer the poster to the Pinned post.

[rberq]

Something I have found when hunting for mouse pointer changes. There are situations when the only way I was able to make the script work was to "wiggle" the mouse pointer before testing its shape:

Yes! I have seen the same thing, but I thought it was unique to the weird application I was working with.

The other issue I ran into was, on an extra-fast computer, by the time Windows (or IE?) changed the mouse pointer shape, the pointer had zipped past the target and the click fell on dead space. Had to set Mouse Speed to a few milliseconds to slow it down a bit.