How to select a button?

[John Welch]

I'm finding the simplest thing the hardest to do. I have tried to select a button with coordinates, but the mouse cursor goes to the wrong spot on the screen. I used GetControl but the index randomly changes between uses of the button. There are absolutely no shortcuts on the window to start from. I cannot rely on the mouse cursor being in any one postion or on any one control before activating the macro. 

 

It seems that the simple way of using coordinates needs to be debugged. So my question is: Why can't I set the position of the mouse where I want it to go? In the screen shot attached, I am trying to select the "META-CORECTION" button 5 times -- once to start the process and then again for four more times every time the process completes

[rberq]

If the Meta-Correction button is always at the same spot on the screen, could you do something as simple as this?  (Use the macro editor's Mouse Locator tool to find the screen coordinates of the button.)

 

Mouse Move: 1279, 221 Relative to Screen
Mouse Left Click

 

[John Welch]

That's what I thought too, but it didn't work. The mouse was taken to the wrong place by MouseMove. Then I discovered that the position is being reported incorrectly by the utility. I used the dialog to get the correct coordinates and it all works now. What the image below is showing is that my cursor is sitting right where  (not in view) it needs to be and the mouse locator is reporting it at 1258, 192. However, the position that actually works and was captured by ctrl-space in 1573,239. The boxes with the triangle on the are correct and the Mouse Locator is wrong. And, no, my mouse did not move somewhere else on the screen when I captured this. Looks like a bug.

 

[rberq]

Odd.  The only problem I have had with the mouse locator is when I have accidentally used window-relative instead of screen-relative, or vice versa.

Glad you have it working now. 

[Cory]

It's been a while since I've messed with it, but there are multiple ways to get a control. If I recall, one can get the control by the control text. I'll look at it later when I have time. If nothign else, one could go through every control to find the correct one. 

[acantor]

It looks like you may have recorded the initial script. Recording steps is, generally, an extremely unreliable method. You are more likely to get something to work if you ditch the recording, and start from scratch by inserting instructions one after another.

I can see a number of potential methods that would allow script to "click" on buttons. The simplest way might involve pressing Tab or Shift+Tab to navigate around the screen. You'll need to experiment with these keys, and perhaps other standard keyboard navigation keys. If you discover a sequence of steps, they might be relatively straightforward to automate via Macro Express.

 

The solution that is most likely to work is more complex: "hunting" for pixels of particular colours, and performing actions based on moving the mouse pointer to the location of those found pixels. For example, the script might perform these steps:

1. Check every pixel diagonally from the upper left corner of the window until you reach a white pixel.

2. Check every pixel vertically up until you reach the first non-white pixel.

3. Calculate the number of pixels along the Y axis between the two points, and divide that number in half.

4. Move the mouse pointer down by the previous value.

5. Check every pixel horizontally to the right until you reach the unique grey coloured pattern in the first button.

6. Move the mouse pointer there, and click.

 

I've also used this approach in combination with my first suggestion (i.e., using standard keyboard navigation keys).

Scripts like these are not easy to develop, but if you persist, the scripts can be remarkably fast and reliable. They are definitely not programmatic, but I consider them quasi-programmatic as the scripts interact with colours that the application developers "baked into" the program's user-interface.

[John Welch]

 

Thank you everyone for your advice and ideas. I have this macro nearly complete and running the way I want. I am fine tuning bits right now like adding in the estimated time to completion and reading setup data from files and so forth. (That's actually done but I am getting one odd number that needs to be debugged.) I've already used it to save a heck of a lot of time where I do not have to sit in front of the computer waiting for long process to end just so I can start them again. (Sound of hair being pulled out.)

 

My next addition will be to find a way to tell Alexa to tell me that the macro has completed. I'm thinking email trigger or the like. I haven't explored this yet.

[terrypin]

Hi John,

 

That would be a fun project to instruct Alexa (if you don't have an Echo/Dot near to the computer). But why is she needed? Just send yourself an email with the single command Email Send (under the Internet menu).

 

 

[John Welch]

Because it will be fun to work it out. Actually, I'd like to send an SMS to my phone.

 

Last night I ran into a new problem: The macro stopped working. It had been selecting the button (which is also a drop down), then preforming the drop down, descending two menu items and then pressing enter. As of last night, it no longer selected the button. I got a different set of coordinates from the locator utility. I changed them while thinking that something else must be wrong. Then the button was selected, but the arrow keys would no longer select the menu items. Frustrating.

 

Does ME have any problems with memory leaks that could be causing this? I ran it on and off all day yesterday while developing and then a batch of live tests -- so it had about a 12 hour workout. That's more than enough time for memory leaks to start effecting calculations.

[terrypin]

1. Have you radically edited your code since recording it, as acantor recommended? The last screenshot you showed still had evidence of the original, such as redundant delays, etc.

2. Show us your full macro and its code.

 

3. Place Pauses or Macro Stops at key points and carefully check that

- your mouse cursor is where you expected

- correct items in the drop-down menu are displayed

 

4. Check that the Meta-Correction button and its drop-down inset are consistently at the same screen position under all circumstances. For example, no downward displacement due to additional content appearing above.

 

5. Clarify the point of your last screenshot which shows contradictory entries: 1573 v 1258, and 239 v 192. That looks like you may have moved the mouse since last entering the values?

 

 

[Samrae]

2 hours ago, John Welch said:

Because it will be fun to work it out. Actually, I'd like to send an SMS to my phone.

Here is an article about how have a macro send a text / SMS message to your phone: Send Text Messages to Mobile Phones

[Samrae]

On 2/27/2020 at 7:32 AM, rberq said:

If the Meta-Correction button is always at the same spot on the screen, could you do something as simple as this?  (Use the macro editor's Mouse Locator tool to find the screen coordinates of the button.)

 

Mouse Move: 1279, 221 Relative to Screen
Mouse Left Click

 

 

22 hours ago, rberq said:

The only problem I have had with the mouse locator is when I have accidentally used window-relative instead of screen-relative, or vice versa.

 

It might work best to use Mouse Move: Relative to Current Window. I find it better to work with positions relative to the active window because I rarely have my applications at full screen. When they come up they are usually in slightly different positions.

 

Another thing to consider is if the button moves when the active window is resized. If this is the case then your macro would be more reliable if you set the window to a specific size. Here is an actual example from one of my macros:

Window Move and Size: Move "CMS: *Defect" to (635,320) and size to 990 x 750
Mouse Move: 90, 456 Relative to Current Window

 

[John Welch]

57 minutes ago, terrypin said:

1. Have you radically edited your code since recording it, as acantor recommended? The last screenshot you showed still had evidence of the original, such as redundant delays, etc.

2. Show us your full macro and its code.

 

3. Place Pauses or Macro Stops at key points and carefully check that

- your mouse cursor is where you expected

- correct items in the drop-down menu are displayed

 

4. Check that the Meta-Correction button and its drop-down inset are consistently at the same screen position under all circumstances. For example, no downward displacement due to additional content appearing above.

 

5. Clarify the point of your last screenshot which shows contradictory entries: 1573 v 1258, and 239 v 192. That looks like you may have moved the mouse since last entering the values?

 

 

1. The only code I recorded was the original cursor positioning, arrow key movements and enter key. All the rest I did. I don't know what the delays are for, but I made sure I them with each movement of the mouse or keys. This is the first ME macro I have written.

2. Note: This code is working for now. I changed the mouse to a left click because I found that I can left click anywhere on the button to get the drop down. Then just arrow down to where I want. Ran it today on a 1 1/2 hr job. Code is attached.

3. I've done this. It is how I found out that the positions that were working had stopped working.

4. Yes, they are fixed.

5. Look at the attached jpeg. It shows where the cursor is reported to be by the locator. However, if I position the corsor over the middle of the Meta-Corection" button and press ctrl-space, the numbers in the dialog box are what I get and what works. At first, not knowing about the ctrl-space facility, I was using the numbers from the locator, and that, of course, did not work.

 

macex.mex Meta-Correction Macro.txt

[John Welch]

30 minutes ago, Samrae said:

 

 

It might work best to use Mouse Move: Relative to Current Window. I find it better to work with positions relative to the active window because I rarely have my applications at full screen. When they come up they are usually in slightly different positions.

 

Another thing to consider is if the button moves when the active window is resized. If this is the case then your macro would be more reliable if you set the window to a specific size. Here is an actual example from one of my macros:

Window Move and Size: Move "CMS: *Defect" to (635,320) and size to 990 x 750
Mouse Move: 90, 456 Relative to Current Window

 

 

Window or screen make no difference in this case. The programmers coded it to not have any resize and it is always full screen. They don't even let the taskbar pop up if the mouse goes to the bottom of the screen. Bottom line is the button is always in exactly the same place.

[John Welch]

47 minutes ago, Samrae said:

Here is an article about how have a macro send a text / SMS message to your phone: Send Text Messages to Mobile Phones

Thanks

[John Welch]

19 hours ago, acantor said:

It looks like you may have recorded the initial script. Recording steps is, generally, an extremely unreliable method. You are more likely to get something to work if you ditch the recording, and start from scratch by inserting instructions one after another.

I can see a number of potential methods that would allow script to "click" on buttons. The simplest way might involve pressing Tab or Shift+Tab to navigate around the screen. You'll need to experiment with these keys, and perhaps other standard keyboard navigation keys. If you discover a sequence of steps, they might be relatively straightforward to automate via Macro Express.

 

The solution that is most likely to work is more complex: "hunting" for pixels of particular colours, and performing actions based on moving the mouse pointer to the location of those found pixels. For example, the script might perform these steps:

1. Check every pixel diagonally from the upper left corner of the window until you reach a white pixel.

2. Check every pixel vertically up until you reach the first non-white pixel.

3. Calculate the number of pixels along the Y axis between the two points, and divide that number in half.

4. Move the mouse pointer down by the previous value.

5. Check every pixel horizontally to the right until you reach the unique grey coloured pattern in the first button.

6. Move the mouse pointer there, and click.

 

I've also used this approach in combination with my first suggestion (i.e., using standard keyboard navigation keys).

Scripts like these are not easy to develop, but if you persist, the scripts can be remarkably fast and reliable. They are definitely not programmatic, but I consider them quasi-programmatic as the scripts interact with colours that the application developers "baked into" the program's user-interface.

 

I think I've covered this pretty well already -- that I cannot use the tab key, shortcut keys or any other reliable user navigation means to get to the button I want. The user could be anywhere on the screen when the macro is started. I do not know what you mean by, "The solution that is most likely to work is more complex." Why would I not be able to simply tell the cursor where to go. It seems dead simple to me. ME should be able to reliably place a cursor with simple commands. E.G.:  "Mouse Move: 1502, 221 Relative to Screen." While this position worked yesterday it did not today. Today I had to change it to: "Mouse Move: 1582, 227 Relative to Screen." The position of the button did not move so why would the coordinates be different?
 

[terrypin]

Because of the excessive delay time (nearly a full second) after moving the mouse, perhaps vibration or unintended movement has shifted the mouse cursor's position?  A mere 0.1 s delay would usually be OK. Often zero too, but I usually err on the cautious side. And I would use the plain Delay command. So I would have written that section thus:

  Mouse Move: 1520, 220 Relative to Screen
  Delay: 0.1 seconds
  Mouse Left Click
 

And the code:

<MOUSE MOVE Option="\x01" X="1520" Y="220" _PROMPT="0x000A"/>
<DELAY Flags="\x01" Time="0.1"/>
<MOUSE LEFT CLICK/>

 

BTW, that's the way you would normally post the macro and its code. Use the 'Code' tool above, '<>'. No need to attach separate files usually. Although if it's an accessible macro (that others could run with no other apps or files needed) then an MEX is a welcome extra.

 

As the button is fixed and you are clicking well within it, I can think of no other cause for the inconsistency. And if you've paused or stopped the macro at that point for a visual check it's strange that you didn't spot anything amiss.

 

One other possibility is that your app may be one of the few I recall that need a particularly 'smooth' approach in a mouse move. Or even simply slowing that down with a Mouse Speed command set to say 5 ms.

 

 

 

 

[John Welch]

14 minutes ago, terrypin said:

Because of the excessive delay time (nearly a full second) after moving the mouse, perhaps vibration or unintended movement has shifted the mouse cursor's position?  A mere 0.1 s delay would usually be OK. Often zero too, but I usually err on the cautious side. And I would use the plain Delay command. So I would have written that section thus:

  Mouse Move: 1520, 220 Relative to Screen
  Delay: 0.1 seconds
  Mouse Left Click
 

And the code:

<MOUSE MOVE Option="\x01" X="1520" Y="220" _PROMPT="0x000A"/>
<DELAY Flags="\x01" Time="0.1"/>
<MOUSE LEFT CLICK/>

 

BTW, that's the way you would normally post the macro and its code. Use the 'Code' tool above, '<>'. No need to attach separate files usually. Although if it's an accessible macro (that others could run with no other apps or files needed) then an MEX is a welcome extra.

 

As the button is fixed and you are clicking well within it, I can think of no other cause for the inconsistency. And if you've paused or stopped the macro at that point for a visual check it's strange that you didn't spot anything amiss.

 

One other possibility is that your app may be one of the few I recall that need a particularly 'smooth' approach in a mouse move. But check out the other things first.

 

 

 

 

 

 

Thanks. I wasn't sure how to do that with the macro and code.

 

My mouse cursor was not moving. I need you t get that I really really checked that out. Did you see my picture of the elephant in the room? The mouse locator does not agree with the capture tool. Does yours?

 

I will reduce the delays. It is possible that the mouse cursor is drifting with so many near 1 sec delays. I'll try them at your recommended 100ms delay.

[terrypin]

Did you see my picture of the elephant in the room? The mouse locator does not agree with the capture tool. Does yours?

 

As I suggested in my earlier reply, that screenshot appears to show that at the moment you took it the mouse cursor was in a different position to where it was when you used the capture tool to take a snapshot.

Try it again a few times: move the mouse, release it,  press Ctrl+Space ... and the values entered into the command should exactly match those you see in the Mouse Locator. If not then I'm at a loss to explain it as it always has agreed on all my PCs and in all versions of Macro Express/Macro Express Pro.

[John Welch]

Sorry. I get what you are after, but no. I was very very certain about the position of the mouse cursor. That is why I posted an image which clearly shows that the cursor is all the way to the right of the screen and shows the same (nearly) position as the one that I  captured that is working to press the button. They cannot both be position 1504. If I move that cursor back over onto the button and press ctrl-space, it will capture the coordinates 1504, 221 (or thereabouts) as the middle of the button.

[terrypin]

To make sure I’ve correctly understood:

1. What is your screen resolution?

2. Is the middle of the target button, which you’re clicking, at roughly 1520, 220, using Tools > Mouse Locator?

3. Using my amended code for the move, with those coordinates, does the button now get activated correctly and consistently?

4. If not, what exactly is the behaviour you get?

5. If your mouse is battery-powered, is it in need of replacement?

6. Are there any other clues from mouse or graphics quirky behaviour in other apps?

[Cory]

You might see if IFTTT has something to Alexa. I see this page with Alexa stuff. 

I'm thinking they probably have a Web API so you could use the web page commands in MEP. 

[John Welch]

Cool. Thanks for the lead.

[acantor]

If the button is always in the same position, try moving the pointer and clicking relative to the window instead of relative to the screen.

[John Welch]

Yes, thanks. In this case window and screen are always the same.