Clicking places that are ever changing...

[UAMike]

I use MEP to automate tasks at work. One of the primary apps that my macros run in is a Java program in IE. For some reason, whoever is in charge of maintaining this application periodically makes slight, seemingly useless modifications to the layout of the program and as a result, all of my macros that require a click will be off by a few pixels. Sometimes this makes no difference, but other times it's the difference between clicking on a button or not clicking on a button. Needless to say, it is infuriating because it renders most of macros useless. 

I've tried to use the 'Get Control' function, but the utility is not able to recognize any buttons in the application that I am using. In addition, sometimes I am not wanting to click on a button at all. 

Is there anything that you might suggest I think about when addressing this problem?

[Cory]

In a web browser the entire displayed page is one control. The web form controls within it are not accessible.

I too had this same problem. Often it's environmental as well. The user zooms in the web browser, has different borders, and about 99 other things that cause the controls to change position slightly. For that reason I completely abandoned the technique of mouse clicks. Instead I relied completely on tabbing or other keystroke combinations. I would even maintain tables of tab counts externally as a map for all the pages and controls so I didn't have to change my macro as often. It's far more reliable.

[rberq]

I have done the same as Cory on some screens.  Sometimes on a web page I will start with Alt-d to highlight the URL, because that also positions the cursor at the URL and gives a known starting point for tabbing.  Of course, whoever maintains the page might also change the number of tabs needed.  But you can diagnose and fix that fairly quickly when it happens, with the judicious use of Macro Stop statements inserted temporarily to see where the tabbing has left off.  

Sometimes if I can position the mouse fairly close above or below where I know the button will be, I write a short Repeat loop to move the mouse down or up a few pixels at a time, checking after each move to see if the mouse pointer has changed from an arrow to a finger.  That works pretty well but sometimes fails due to timing, Windows busy doing something else so not changing the pointer quickly enough for example. 

[acantor]

If you sink enough time and effort into the problem, you may come up with a solution that almost always works. But you gotta love experimenting and tweaking, or you will likely find the experience a miserable drudge!

I use a combination of techniques, including:

- Clicking on a "neutral" part of the screen that resets the tab order. Then send tabs/shift tabs to the control.

- Searching for a pixel colour on a control, or to something near the target, and then tabbing and/or clicking.

- Searching for text on the screen. Then give the word or phrase focus, and then tab to the target.

- Searching for text on the screen and then selecting it. Then search for the pixel colour of the selection,and click.

Some of my Macro Express scripts perform thousands of actions to zero in on a target on a web app, and are almost 100% reliable. But they were labours of love. I was willing to put in the time to get them to work. And some of them are a bit brittle.

[rberq]

acantor sounds pessimistic, but he is absolutely right.  The best Macro Express can do with Windows and especially web forms, is to be almost 100% reliable.  It is great for automating personal tasks at work -- I always said it was worth half a person as a time saver.  You can even roll it out to dozens or hundreds of users, but not to dummies.  You/they have to watch carefully as the macros run and recognize when ME has gone wrong. 

[UAMike]

Thank you for the feedback. I was recently reminiscing about playing games on my old Tandy 2000 and how you would have to calibrate the joystick every time you use it. I used this idea to set specific anchor points from which my mouse will move relative to that point. My thought is that when the elements deviate slightly from where they used to be, I will just have a script (haven't go that this part yet) to quickly change the coordinates of 4-5 anchor points.

I would have liked to incorporate more tabbing into my macros, but it was not working as I had hoped and this new solution seems fine for the moment.

I appreciate all the input. 

[rberq]

Anchor points -- excellent idea.  

The script to record anchor points shouldn't be too hard to write.  I have done something vaguely similar to record pixel colors of unique points on a series of identically-named web screens, so a subsequent macro can determine which specific screen is currently present.  I found the easiest method was to position the mouse by hand, then trigger the recording macro via a keyboard key.  If I positioned the mouse then clicked, the clicking jiggled the mouse just enough that I often recorded the wrong location. 

[Cory]

I had a macro once where the user could train the macro. It stored all the coordinates in a text file. It was simple. 

[rberq]

1 hour ago, Cory said:

I had a macro once where the user could train the macro. It stored all the coordinates in a text file. It was simple. 

Yes, same here.  Then a macro ran more or less constantly scanning screens to see whether the pixel colors/positions matched with the coordinate file.  That would identify which screen was visible, and call other macros to process the screen.  Users with slower computers told me they didn't like it because whenever that application was running the CPU cooling fan would come on and make too much noise.   

[UAMike]

On 6/29/2017 at 8:01 PM, acantor said:

If you sink enough time and effort into the problem, you may come up with a solution that almost always works. But you gotta love experimenting and tweaking, or you will likely find the experience a miserable drudge!

I use a combination of techniques, including:

- Clicking on a "neutral" part of the screen that resets the tab order. Then send tabs/shift tabs to the control.

- Searching for a pixel colour on a control, or to something near the target, and then tabbing and/or clicking.

- Searching for text on the screen. Then give the word or phrase focus, and then tab to the target.

- Searching for text on the screen and then selecting it. Then search for the pixel colour of the selection,and click.

Some of my Macro Express scripts perform thousands of actions to zero in on a target on a web app, and are almost 100% reliable. But they were labours of love. I was willing to put in the time to get them to work. And some of them are a bit brittle.

I'm trying to revisit some of the ideas from this post so that I can make my macros run with greater reliability. This post mentions "searching for text on the screen" followed by a way to get to that text using tabs or pixel colors. I like this idea, but I cannot figure out how to implement it. My macros primarily run in Chrome and I could perform a search for specific text using Ctrl+F, but how would I then get to that word? The only method I can think of is to have the mouse move a few pixels at a time until it finds the right color, but I can see this being a pain because of 1) false positives and 2) potentially having to look all over the screen for the right color. To address the former problem, I imagine that there's some extension available to change the highlighted text from orange/yellow (default in chrome) to a more unique color.

 

Any advice is appreciated.

[acantor]

I'm trying to revisit some of the ideas from this post so that I can make my macros run with greater reliability. This post mentions "searching for text on the screen" followed by a way to get to that text using tabs or pixel colors. I like this idea, but I cannot figure out how to implement it. My macros primarily run in Chrome and I could perform a search for specific text using Ctrl+F, but how would I then get to that word?

 

Assuming that Chrome handles searches similarly to Firefox or IE, the steps will look something like this (in pseudo code). In this example, I am navigating two web controls beyond the "Contact us" link.

Text Type "{Ctrl+f}" // Start a search

Text Type "Contact us{Enter}"  // Search term

Text Type "{Esc}" // Cancel the search, but the "Contact us" link is now focused

Text Type "{Tab}{Tab}" // Navigate to a web control that is  two tab stops beyond the "Contact us" link

Text Type "{Enter}" // Activate the link

[rberq]

Thanks for the tip.  I just found that will work for me on a web page that changes frequently.  It's also considerably faster than the technique I have been using.

[acantor]

I have learned a lot of your posts over the years, so I am glad to be able to reciprocate!

The technique is not 100% reliable, but considering the range of macro scripting techniques that involve interacting with the user interface, it's pretty good. Inserting short delays may make the technique less apt to fail when web traffic, downloads, and low resources slow down a system.

[UAMike]

On 11/12/2017 at 9:37 AM, acantor said:

I have learned a lot of your posts over the years, so I am glad to be able to reciprocate!

The technique is not 100% reliable, but considering the range of macro scripting techniques that involve interacting with the user interface, it's pretty good. Inserting short delays may make the technique less apt to fail when web traffic, downloads, and low resources slow down a system.

Indeed Chrome does work in the same fashion as Firefox and your notes proved to be very helpful for navigating a web page. 

Following your instructions will allow me to search for a button and then press on it using the Enter or Space key. There are a few instances where I haven't gotten this to work... if there is an element on the screen that is clickable, but not really a button per se, then pressing Enter or Space will not do the trick. I cannot figure this one out. Am I missing something really obvious?

[acantor]

Can you navigate to these "buttons" by tabbing to them? Is it clear the buttons have focus?

If it's clear these non-responsive buttons have keyboard input focus, and if pressing Enter or spacebar have no effect, you may be dealing with a scripted control that does not obey standard interaction rules.

If you post a sample of the source code for the control (which you can expose in Chrome by pressing Ctrl + U), there may be hints on ways to activate the control via keyboard (and therefore via keystrokes sent via Macro Express). Or it may be obvious the developers have broken keyboard interaction!

If the developers have designed the page for mouse-only interaction, I would encourage you to write them and report a bug -- and an area of non conformance with WCAG 2.0, the de facto international standard for web content accessibility.

https://www.w3.org/TR/UNDERSTANDING-WCAG20/keyboard-operation-keyboard-operable.html

 

[UAMike]

2 hours ago, acantor said:

Can you navigate to these "buttons" by tabbing to them? Is it clear the buttons have focus?

Sort of... What it is is a "box" (looks like two columns in a spreadsheet)  with several phrases from which you can pick one or more options. When you click on one of these phrases, it has a downstream effect on what else gets displayed on the page. I'm not sure if that constitutes a button, but is certainly an element of the page that I can interact with as opposed to a plain string of text. From what I can tell, the box itself can have focus, but not the individual phrases within the box.

Here is a short sample of the source code, which is about 90% foreign to me.

id='sidebar-region-one' class='screen-layout-region screen-layout-region-main screen-layout-region-main-sidebar1 screen-layout-region-resize' aria-label='Left Panel' role='complementary'><div class='screen-layout-region-content'><div  class='sectionDivStyle' id='RULE_KEY' node_type='MAIN_RULE' node_name='PrismPortalNav' data-node-id='PrismPortalNav' version='1' objclass='Rule-HTML-Section'  pyclassname='Data-Portal' readonly='false'  NAME='BASE_REF' BASE_REF=''  expandRL='' uniqueID='SID1510616504615' ><span class='inspector-span' ><div class='layout layout-noheader layout-noheader-section_divider'><div section_index='1' class='layout-body clearfix' ><div  class='sectionDivStyle' id='RULE_KEY' node_type='MAIN_RULE' node_name='PrismActionItemLinks' data-node-id='PrismActionItemLinks' version='1' objclass='Rule-HTML-Section'  pyclassname='Data-Portal' readonly='false'  NAME='BASE_REF' BASE_REF=''  expandRL='' uniqueID='SID1510616504616' ><div   class='layout layout-outline layout-outline' id='EXPAND-OUTERFRAME' PARAM_NAME="EXPANDEDSubSectionPrismActionItemLinksB"><div class='collapsible Expanded ' id='EXPAND-PLUSMINUS' name='' aria-live='polite' ><div class='header header-bar clearfix' tabindex=0 role='heading' aria-expanded='true' aria-label='Hide Action' id='RULE_KEY' node_type='HEADER'   onkeydown='expandHeader(this,event,null,true);' onclick="setUserStart(id);expandHeader(this,event);"><div class='header-left' title='Hide Action'><i class='icon icon-openclose'></i></div><div class='header-content'><h2  class='header-title' aria-level='2' id=''>Action</h2><nobr></nobr><span class='header-element header-title-table'><table cellspacing='0' cellpadding='0'><tr><td nowrap='nowrap' class='tdRightStyle'><table cellpadding='0' cellspacing='0' align='right'><tr><span style="display:none;"> <input type=hidden id=SpellChecker value= "Spell Checker" /> <input type=hidden id=More value= "More" /> <input type=hidden id=Edit value= "Edit" /> <input type=hidden id=Revert value="Revert  " /> <input type=hidden id=RevertTo value= "Revert to" /> <input type=hidden id=IgnoreAll value= "Ignore all" /> <input type=hidden id=AddToDictionary value= "Add to Dictionary" /> <input type=hidden id=ChangeAll value= "Change All" /> <input type=hidden id=SpellChecking value= "Spell Checking" /> <input type=hidden id=NoSpellChecker value= "No SpellChecker Available" /> <input type=hidden id=NoSuggestions value= "No Suggestions" /> </span>     <script language="javascript"> var haveActionChoices = false; function initActionChoices () { gStrSpellChecker = document.getElementById("SpellChecker").value; gStrMore = document.getElementById("More").value; gStrEdit = document.getElementById("Edit").value; gStrRevert = document.getElementById("Revert").value; gStrRevertTo = document.getElementById("RevertTo").value; gStrIgnoreAll = document.getElementById("IgnoreAll").value; gStrAddToDictionary = document.getElementById("AddToDictionary").value; gStrChangeAll = document.getElementById("ChangeAll").value; gStrSpellChecking = document.getElementById("SpellChecking").value; gStrNoSpellChecker = document.getElementById("NoSpellChecker").value; gStrNoSuggestions = document.getElementById("NoSuggestions").value; } var gStrSpellChecker = null; var gStrMore = null; var gStrEdit = null; var gStrRevert = null; var gStrRevertTo = null; var gStrIgnoreAll = null; var gStrAddToDictionary = null; var gStrChangeAll = null; var gStrSpellChecking = null; var gStrNoSpellChecker = null; var gStrNoSuggestions = null; </script> <td id='HeaderButtonIconsTDId'><script>try {var gHeaderButtonHTML=document.getElementById("HEADER_BUTTON_HTML");if(!pega.u.d.bModalDialogOpen){document.getElementById('HeaderButtonIconsTDId').innerHTML = gHeaderButtonHTML.innerHTML;}} catch(e) {}</script></td><td><input type="hidden" id="HeaderButtonSectionIndicator" value="SubSectionPrismActionItemLinksB" /></td></tr></table></td></tr></table></span></div><div class='header-right'></div></div></div><div section_index='1' id='EXPAND-INNERDIV' class='layout-body clearfix'  ><div bSimpleLayout='true'   class='content   layout-content-simple_list content-simple_list  ' ><div  style=''  class='content-item content-field item-1   ' STRING_TYPE='field' RESERVE_SPACE='false'><div class='content-inner '><div class='field-item dataValueWrite'><script type="text/javascript">/**/</script><div id='div-menu-1510616504617' style='display:none;'></div><script

[acantor]

Although the code is (mostly) unintelligible to me, I did have a few concerns that might account for the inability to interact with the screen elements via keyboard:
 

Quote

  onclick="setUserStart(id);expandHeader(this,event);"

Interaction via keyboard is improved by relying on "onfocus" instead of "onclick" events.

Quote

if(!pega.u.d.bModalDialogOpen)

A modal dialog is a secondary window that opens in front of the main window. For example, when you close a word processing document without having saved it, the program pops up a window asking you if you want to save the document. You must interact with the dialog box before you return to main window.

In Windows, standard dialog boxes are keyboard accessible by default.  In web apps,  dialog boxes can be keyboard accessible, but it's up to the developer to ensure keyboard interactivity.

Quote

NOT! <a href="index.html" accesskey="h">Home</a>

There are no accesskey attributes in the code sample. Developers can provide mechanisms for interacting via keyboard by incorporating access keys.  In the above example, a user would be able to go to the home page by pressing Alt + h (in Internet Explorer and Chrome) or Shift + Alt + h (in Firefox). In Chrome, the access key is either Alt + h or Shift+ Alt + h, depending on the circumstances.

[Cory]

These are the things that drove me bonkers. I decided to write simple programs that send HTTP requests and receive HTTP Responses. IE no web browser. Most forms use the POST method for forms so all I do is copy the POST data and modify it myself. In most browsers hit F12 and you can see the requests and responses. Right after you click you will likely see a POST request. Look at the POST (form) data. Usually it's something like this: "name=Fed&email=Fred@SlateRockAndGravel.com&message=Yabba%20Dabba%20Dooo". This returns the source HTML of the resulting web page. And super fast. Typically 3 per second. Then I get what i need from there. One can navigate many pages and never actually render a page in a browser. If anyone is interested i could create a helper app for MEP to do this method. Or if they would like to learn to do it my way.