Jump to content
Macro Express Forums

Ideas for clicking text fields in Chrome


Recommended Posts

My employer started using a new web-based app and I need help on how I can focus on specific text fields. My go-to method in the past was to use the search function in Chrome to find specific text and then tab to the correct place from there, but it doesn't seem that this method will work in this new app. The reason this is not working for me is that the webpage has a massive embedded field where all the individual text boxes are located (Not sure what the right term for this is... but it has it's own scroll bar if that helps explain). So, when I search for specific text, as soon as I try to tab somewhere, it shoots me back to the first text field within this larger embedded box. I've tried a few other things so far...

 

  • Just tabbing to the right field: I think this won't work because certain fields are sometimes greyed out and I can't predict when this will be the case. I might need to further explore this method to see if I can determine if this is possible.
  • Using controls in MEP: I've never actually done this before, but following the MEP help file, I don't think it's possible because MEP isn't recognizing individual text boxes when I used the crosshairs to set the control. In my first attempt, I thought I had it working because it jumped to the right field, but I wasn't able to recreate this when I picked a different field to click on... not really sure what to make of this.
  • Searching for pixel colors: I've used this in the past and it's pretty slow and unreliable. I know some people in this forum can attest to that. I wouldn't mind using this method if there just one or two fields I needed to get to, but that isn't the case here. Plus, I think the fact that there's a scrollbar involved would only further complicate things.

 

Are there any other methods I should explore?

Link to comment
Share on other sites

Maybe you could go to the end and tab backward?

Controls commands in MEP are for WinForm controls only. In your browser the rendered document is in one large WebBrowser control. Hence the MEP commands are worthless. 

I do not recommend pixel scanning. 

 

Link to comment
Share on other sites

Sorry. I clicked submit accidently.

If there is a label to some kind of unique text near the TextBox you desire do a find for it. Ctrl+f > search for term > Escape > tab X times. When you hit excape the found term will be highlighted and the documents cursor position will remain there. So maybe you have a label left of the field you want named "Date of Hire". Find it, escape, it will now be highlighted, and tab once to get into your field. 

Will that work?

Link to comment
Share on other sites

14 minutes ago, Cory said:

Sorry. I clicked submit accidently.

If there is a label to some kind of unique text near the TextBox you desire do a find for it. Ctrl+f > search for term > Escape > tab X times. When you hit excape the found term will be highlighted and the documents cursor position will remain there. So maybe you have a label left of the field you want named "Date of Hire". Find it, escape, it will now be highlighted, and tab once to get into your field. 

Will that work?

The Ctrl+f is the first thing I tried. That's my bread and butter for most things I try to do inside a web browser. It doesn't work because after I press ESC, pressing tab or shift-tab will bring me to the first or last field respectively. 

Link to comment
Share on other sites

Bummer. I can't think of anything else, but if I do, I'll post. 

 

It was many things like this that led me away from MEP for web automation. MEP is great for windows apps, but it often is incapable of doing things like this. In order to meet my client's needs I went and learned .NET and have been doing all my web automation that way. Once I got over the learning period, I found that so many things that I struggled with were incredibly simple in .NET. Mostly I communicate directly with the web server with HTTP requests like your web browser issues. All the data you put into forms and selections you make will eventually be put into a simple string of text and sent to the server in a POST request. So I have my program create the string from my data source and fire off a bunch of those requests. Often I can do 10 or more a second. Or sometimes I'll create my own web browser program. Takes about 5 minutes. That way the WebBrowser control is an object I can programmatically interact with with ease while allowing for user observation and interaction. And timing issues are non-existent when done programmatically. But with .NET or any of the web automation tools, there's a lot of learning that needs to be done. 

Link to comment
Share on other sites

Is it possible to show us the web app? Otherwise mainly guesswork. If confidentiality precludes that, then can you find another accessible web site that gives you the same problem?

Link to comment
Share on other sites

An approach that I use for especially nasty web applications involves identifying a "neutral zone" that when clicked, one or several presses of the Tab key (or Shift + Tab) gives focus to a link or control that is close to where I'm trying to get.

 

Neutral areas are sometimes quite large (although not every web app has one). The only way I know to discover neutral zones is through trial and error experimentation. It's hard to tell from looking at the page. When I'm looking for one, I click everywhere on the screen where nothing is clickable, and then press the Tab key and Shift + Tab. If there is a neutral zone, the focus always goes to the same object on the page that is close to where I want to go. So a script might do something like this to get to the target:

 

1. Find the neutral zone.

2. Click.

3. Tab Tab Tab Tab Tab

 

I frequently find a neutral zone near the left margin of a webpage. (But not in Outlook.com's Inbox; but there is a large neutral zone underneath the list of messages. When the message list extends to the bottom of the screen, I find that I need to decrease page magnification, sometimes significantly, to be able to click in the area below all messages.)

 

These neutral zones tend to be large, so knowing the exact pixel is not important. I typically use pixel colour searches to zero in on them, sometimes checking every 50 or 100 pixels, which I find is fast enough. Here's an excerpt from a Macro Express script for a web app. The macro searches diagonally south-east in 100 pixel increments for a white pixel.

 

Repeat Start (Repeat 100 times) // First, search diagonally from the top left corner of the window for a white pixel
  Get Pixel Color from Beneath the Mouse into %PixelColour%
  If Variable %PixelColour% Equals "16777215" // We have reached a white pixel, which is hopefully within the white box in the Sign In window
    Repeat Exit
  End If
  Mouse Move: 100, 100 Relative to Last Position
End Repeat

Mouse Left Click

 
In summary, sometimes a single, strategic click can get you close to your target, and then you can Tab or Shift + Tab the rest of the way to the target.

 

Macros that ferret out neutral zones tend to be complex, clunky, and hard to maintain. I avoid them if there are alternatives. But sometimes there are no alternatives.

 

Unfortunately, not every web app has neutral zones. And if that's the case, although automating a task with macros might be possible, it's probably not going to be easy!

Link to comment
Share on other sites

I mentioned in my previous post that I typically use pixel colour searches to find and click on neutral zones. But there are times it's better to monitor changes in the shape of the mouse pointer.

 

Mouse pointer changes have always been slower than pixel colour searches, but in Windows 10, they are especially molasses-like. Yet the technique can still be made to work by inserting delays, and by increasing the interval between mouse moves.

 

In this example, the macro checks the mouse cursor every 50 pixels until it changes to the "Internet Navigate" cursor.

 

// Hunt for "Account Manager" button
Repeat Start (Repeat 10 times)
  If Mouse Cursor: Internet Navigate
    Mouse Left Click
    Delay: 100 milliseconds // Allow time for the "Sign out" button to appear
    Repeat Exit
  End If
  Mouse Move: 0, 50 Relative to Last Position
  Delay: 50 milliseconds // Allow time for the mouse cursor to settle in its new location
End Repeat

 

Link to comment
Share on other sites

On 5/20/2021 at 2:22 PM, terrypin said:

Is it possible to show us the web app? Otherwise mainly guesswork. If confidentiality precludes that, then can you find another accessible web site that gives you the same problem?

Yes confidentiality is an issue... I wish I could show you so you'd have a better idea, but the best I could do right now is a few screenshots taken with my phone. I can't think off the top of my head of a web site that behaves like this... probably because it's not best practice to build sites this way.

 

On 5/20/2021 at 5:50 PM, acantor said:

In summary, sometimes a single, strategic click can get you close to your target, and then you can Tab or Shift + Tab the rest of the way to the target.

 

Macros that ferret out neutral zones tend to be complex, clunky, and hard to maintain. I avoid them if there are alternatives. But sometimes there are no alternatives.

 

Unfortunately, not every web app has neutral zones. And if that's the case, although automating a task with macros might be possible, it's probably not going to be easy!

 

This is a good idea and I will try to experiment with neutral zones some more. In the attached screenshots, you'll see that I've clicked in such a way as to outline the outer border of this big box, which I now realize is embedded within a larger box that takes up the entire browser window. There is no clickable left margin outside of the outer box, and the slimmest of margins on the right side, although I think this is where the browser's scroll bar would be if the page was long enough. 

 

One new discovery is that there are buttons on the top and bottom of the page that are findable and clickable using the ctrl+f method. Since they're only at the top and bottom, though, it's they're not all that useful for tabbing to somewhere in the middle of the page.

 

 

Link to comment
Share on other sites

1 hour ago, UAMike said:

One new discovery is that there are buttons on the top and bottom of the page that are findable and clickable using the ctrl+f method. Since they're only at the top and bottom, though, it's they're not all that useful for tabbing to somewhere in the middle of the page.

 

For at least one badly-behaved app that I've automated, I was forced to Tab (or Shift+Tab) 10, 15, 20, maybe 25 times to reach some targets. It wasn't ideal, but the macros worked reliably.

Link to comment
Share on other sites

With the app maximised, try methodically establishing the locations (x,y coordinates) of points in the middle of all the boxes into which you expect the macro to enter text. Do so under the differing conditions that you actually encounter. If/when you find an inconsistency (when a box location has changed), try to identify exactly why. As just one example, it may be lower down because extra material  (such as tabs in your we browser) have been generated for some reason at the top. 

 

You may then be able to determine what that ‘extra’ is by some fairly simple logic test, typically with pixel colour testing at a known location. And then, with a little maths, scroll up or down a compensating amount, so that you can click directly on the target.

 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
 Share

×
×
  • Create New...