Jump to content
Macro Express Forums

Capture Data from Web Page


Recommended Posts

I am trying to capture data from a webpage. On the page there is a record counter that tells you record 1 of 251. After capturing a record I want to move on to the next record. I know I can emulate a click on the button for the next record but how will Macro Express know that the record has actually changed. The site we are getting the data from is very data heavy so the pages take a little while to load. I thought maybe if I could edit the text or manually enter the record number that would help but I do not have that option and I suppose that would produce the same result as far as knowing when the new record is being displayed.

 

This is the website I am searching: https://propertyinfo.revenue.wi.gov/iasWorld/Search/AdvSearch.aspx?mode=Advanced

1. Under Critera Select County Name, type in Washington then click Add.

2. Under Critera Select Date Recorded in From enter 01/01/2011 and in To field enter 02/01/2011 and click Add.

3. Then click on Search.

4. You will see a list of records, click on the 1st one in the list.

 

Now you should be view the record. At the top left of the record you will see a box that says Current record with arrows for navigating through the records. When you click on the arrow to change records, how can Macro Express know when the next record is being displayed?

 

I hope I explained this well enough, thank you in advance for any help you can give.

 

Bob

Link to comment
Share on other sites

There are 2 main approaches to consider:

- You can write a macro that is activated when you click your mouse to navigate to the next record. Each time the macro runs it needs to read in the record number last used (from the registry, or from a file, or from an environment variable), increment that counter by one, and then do whatever processing you have determined is needed.

- Or you can write a macro that remains in control of the whole process of navigating your records; in this case it must navigate to the next record by clicking the mouse in the relevant area of your screen; you could trigger this next record activity by having the macro wait for a specified keystroke - e.g. N for Next record, or PgDn - and then take the appropriate action, or by presenting a menu - e.g. Next Record, Previous Record perhaps?, etc.

 

The first method results in your macro running each time you navigate to the next record, while the second approach has your macro being run only once, remaining in control until you exit it, e.g. using the Esc key if you adopt the menu approach. In the second approach your macro can navigate to the relevant web page, etc.

Link to comment
Share on other sites

I know I can emulate a click on the button for the next record but how will Macro Express know that the record has actually changed.

 

Hi Bob,

 

I'm not sure if I've misunderstood, but if you're using IE then ME Pro recognises automatically when the next page has finished downloading. For Firefox, which I use, additional code is necessary. But in either case you would just leave it to run automatically. If you want to be present at the screen when each new record arrives (perhaps to bypass processing, or instigate some other special action) then have the macro ring a bell or something.

 

But even simpler, just open the record required by using its URL!

 

https://propertyinfo.revenue.wi.gov/iasWorld/Forms/Datalets.aspx?mode=TR&sIndex=0&idx=10

 

will open record 10 of 251, and

 

https://propertyinfo.revenue.wi.gov/iasWorld/Forms/Datalets.aspx?mode=TR&sIndex=0&idx=250

 

will open record 250, etc.

 

The following macro processes all the records. (I stopped it manually after half a dozen.)

 

Variable Set Integer %Record% to 1

Repeat Until %Record% Equals "252"

Variable Modify Integer %Record%: Convert to Text String (%tRecord%)

Web Site, "https://propertyinfo.revenue.wi.gov/iasWorld/Forms/Datalets.aspx?mode=TR&sIndex=0&idx=%tRecord%", using Default Web Browser

// The following Macro Run command is necessary ony if the browser is not IE. In my case it's Firefox.

Macro Run: Wait for FF web page ready

Text Box Display: Record downloaded

// PROCESS THE RECORD

Delay: 3 seconds

Variable Modify Integer %Record%: Increment

End Repeat

 

--

Terry, East Grinstead, UK

Link to comment
Share on other sites

  • 4 weeks later...

If it were me I’d look for a way to download them directly. I do this thing all the time and by analyzing the URL one can often control which page opens. EG I recently wrote a macro to search Google for a guy and one can control what ‘page’ of results to go to and how many to display. But here’s the cool bit. I use a simple VBScript to download the raw HTML to a text file then load that text file into a variable in MEP and process it. The entire process is fast (doesn’t load advertising or run scripts), invisible, and 100% immune to timing issues. I know a lot of people prefer being able to see what’s going on but in my experience it’s always a nest of vipers to get anything more than a couple of page loads to run reliably.

 

 

Link to comment
Share on other sites

I think many new users have a conceptual problem manipulating variables. I think if he used the F5 window or even popped open the occasional text box to see what's going on with the variables he'd be more comfortable with them. But you can only lead a horse to water.

 

So do you use my VBScript method to download your web pages to file directly?

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...