Questions On Me Internet Capabilities

[HeyJim]

I just finished my first efforts at writing a macro which will enable me to manipulate some desktop software and build webpages quicker than I could ever do it myself. Well, big deal. That's what ME is all about. (Actually it IS a big deal to me. The first time the macro ran successfully I jumped up, whooped, and scared a co-worker half to death.)

 

There are some other things I'd like to do in the near future but suddenly I'm having doubts based on some older posts I've found in the forum. Now I'm concerned that maybe the internet capabilities of ME are limited and it might be wiser for me to invest my learning time elsewhere. Frankly, I hope not. I really like what I've seen so far.

 

So lets say I wanted to write a macro that would query Google for sites on widgets, visit each of the pages showing in the serps and record any instances of text found on the site that are in bold type. Can I do that with ME?

 

Using ME is it possible to create a macro that will visit a specific website and follow each link until it finds a link to my website or reports "all pages visited and link is not found?

 

In other words, I know there are some internet/web functions but now I'm getting concerned that ME is a little bit limited in this area. Or, did I just work too many hours today.

 

How would you rank ME's internet capabilities?

[joe]

Hello HeyJim!

 

It is true that Macro Express has no specific built-in commands to perform the tasks in your example. It is also true that there are no built-in commands to find and change specific words in a Word document or specific values in an Excel spreadsheet. These things can be done of course, but it requires that you and I build macros to do them. So to answer your last question, I would rank ME's internet capabilities the same as those for Microsoft Word, Excel, or any other software app which needs automation.

 

Everything that you want to do in your examples can be done by having Macro Express generate and control Windows Scripting files. The same can be said, for any other Office application.

[HeyJim]

Thanks, Joe.

I feel alot better about getting more involved in ME then.

 

So, to make sure I understand you properly, if I learn enough about building macros with ME then I can query Google, visit the sites returned in a search and mine them for information. THATS exactly what I've been hoping to find and with a shorter learning curve then trying to become proficient in a programing language.

 

A lifetime ago I did some elementary programing for my Commodore VIC20 in Basic but I've probably forgotten everything I every learned. I may be way off base but I'm seeing ME as something of a shortcut for accomplishing some of the things I want to do without having to learn a programing language from scratch.

[joe]

No, sorry. No shortcuts. It's more than learning just Macro Express. If you want to effectively do data mining on the internet, you will also need to learn about Windows Scripting, the DOM (Document Object Model), a scripting language, and maybe some DHTML for good measure.

 

If you wanted to control Excel using Macro Express you would want to learn something about Excel and its application interface, wouldn't you?. Same with Microsoft Word and any other application automation task. The point is: knowing just Macro Express to automate an application is never enough. You must also know the application and the language that controls it. In your case, your target application is your browser and the DOM (a web page).

 

As developers, we face this situation every time a client asks us to generate macros to do this or that in application X or Y. We certainly know Macro Express, but in many cases have never heard of application X or application Y. So, we must learn something about them in order to generate macros for them.

 

As to data mining on the internet, tools are available that are designed for mining web pages, and more than likely, you already have them. Microsoft's Windows Scripting Host is built in to Windows XP, 2000, and 98. To use it effectively, you will need to learn about DOMs, which you can do from many, many different web sites including Microsoft's. And you will need to learn a scripting language. VBScript, JScript, Javascript are all languages that the Windows Scripting Host uses, understands, and is driven by. Which means that you already have them too because they are part of the Windows Scripting Host.

 

Others in this forum, such as Randallc and Floyd, have used Windows Scripting Host files and have provided examples on how to populate and control them through Macro Express.

 

Internet data mining is a subject near and dear to our hearts at Professional Grade Macros because lately we have been working on generic internet data mining tool macros. We started with macros that return to Macro Express (in the form of CSV files) arrays of different objects on any given web page. Objects include table data, links, and so forth. Also, a very sweet macro that generates other macros to automatically log in to your password-protected web sites without using the Text Type command. It uses some data mining techniques instead. Very cool, indeed!

[HeyJim]

Ouch. But, very much appreciated. I prefer "eyes wide open" and the challenge to me is a bit more (well, a lot more) than what I'd expected. However, I'll do what's necessary. I'm not even sure what many of my future projects will involve. Basically, I just want to find ways to provide unique content for a wide variety of sites.

 

I'll find my way. And, now, ME will be a large part of the process.

[Nicolas]

if you want to follow links, maybe you could display the source code of the Internet page you're processing, copy everything in the clipboard, look at the relevant HTML tags, copy links in a file and process these links

 

of course you can't use Google highlight colors in this case (source code is plain text), but it could give you some good ideas

[HeyJim]

I have a couple of other macro projects that are taking priority at the moment but, yes, what you suggest is something that I've given a little bit of thought to over the last couple of days.

 

Something along the lines of plugging in my search term in Google and then run my macro which would do a right click and view source. Run through the html code and I'm sure there's some way to pattern match links and copy them to the clipboard, text file, or csv.

 

Then the same macro could load each url in turn, load the page, view code, collect the info I'm looking for, etc.

 

Sound like a viable plan? I mean, I should be able to script a macro that will do all this. Certainly it would take a lot more macro knowledge than I have but it seems workable.

 

Ha! Then beyond that my imagination becomes challenged as to what I can then do to help me build quality websites for marketing purposes; something special that would help me be different and give me an edge.

[mebrown1110]

I wonder if some of the Copernic search tools would deliver what the original poster is requesting.

 

So Copernic Agent (there's a free version) could search Google for keywords, search within those results, and then track changes to the Google search if new pages are found. Then you could use Copernic Summarizer on the returned pages.

 

http://www.copernic.com/

 

No affiliation with Copernic; in fact, I've never tried the software before. But it looks as if it will do most of the heavy-lifting that was requested.

 

mike

[pcarlow]

Have you figured out how to do this yet? I'm trying to do the same thing.

[lemming]

pcarlow and HeyJim, I've posted a macro which can extract all urls from a Google search page.

 

http://pgmacros.com/community/index.php?showtopic=1043

 

Just to show what Macex is capable of.

 

-Lemming