Jump to content
Macro Express Forums

Speed Of Text File Process


Les Hazlett
 Share

Recommended Posts

Hi,

 

I am in the process of integrating two macros. I assumed that it would be faster to work with the data from the first macro without writing it to a file and reading it back with the second. However, it seems that using Text File Processing is so much faster than manipulating the large string, that it is actually slower.

 

Perhaps someone can advise.

 

The data (file) structure is lines ending with CRLF. I replaced the Text File Begin Process/Text File End Process loop with a Repeat Until/Repeat End loop which works fine. I capture the lines with the following instructions:

 

Variable Set Integer %N5% with Postion of Text in Variable

Variable Modify Integer: Dec (%N5%)

Variable Modify String: Copy Par of %T98% to %T95%

Variable Modify Integer: %N5% - %N5% + 2

Variable Modify String: Delete Part of %T98%

 

This works but is slower than writing and reading text files.

 

Thanks for any help,

 

Les

Link to comment
Share on other sites

Hello Les!

 

It is possible that working with a string in this manner can be slower than processing a loop. You are using five lines of "character-by-character" code to get to, and strip, the CR/LF as compared to a single line of code that reads in the string and removes the CR/LF all at once. How big is the string?

 

Do I understand that you are combining macros? There are no CR/LFs in native Macro Express code. What am I missing?

 

Also, can you post the native code of the example shown here? It might help.

Link to comment
Share on other sites

Hi, Les,

 

Yes, I agree that the text file process seems to be heaps faster for huge strings, and Floyd has posted his revised sort program using the vbscript equivalent of text file process, making things so much faster.

I have posted the csv file processor which reads the whole file as one line (slows things down in vbscript too), and I have got the text file process (vbscript) going for text files, 1.8Mb file loads, sorts, saves, in 3 secs! (same process as Floyd is now using, and I think his would do the same). I'll post it if you want.

 

Best, randall

Link to comment
Share on other sites

Joe,

 

Thanks for your response.

 

>>Also, can you post the native code of the example shown here? It might help.

 

No one would want to look at the two huge macros I am integrating but here are the example commands.

 

<IVAR2:05:13:98:%T13%><NMVAR:09:05:0:0000001:0:0000000><TMVAR2:10:95:98:001:N05:><NMVAR:01:05:1:0000005:2:0000002><TMVAR2:11:98:00:001:N05:>

 

>>Do I understand that you are combining macros? There are no CR/LFs in native Macro Express code. What am I missing?

 

When integrated into one job, the first macro runs the second macro. The CRLFs are in the data being processed, not in the ME code. When run separately, the first macro writes a CRLF delimited file and the second one reads it. I thought that it would be faster to leave the large text string in RAM than writing the file and reading it back with Text File Processing loop. It turned out to be slower. The strings are 2-5 megabytes.

 

In a repeat loop, the five instructions that I posted do the eqivalent of the Text File Process instruction loop. It puts the data into T98, one line at a time, and deletes the line from the input string. Instruction 1 finds the position of the first CRLF. The text of the CRLF is held in a text variable T13. 2. Decrimenting N5 makes it point at the last character in the first line of data. 4. Increasing N5 by 2 makes it point to the LF so that the first line including the CRLF can be deleted by instruction 5. Oh how I wish there was a command for Text String Process.

 

1 Variable Set Integer %N5% with Postion of Text in Variable %T98%

2 Variable Modify Integer: Dec (%N5%)

3 Variable Modify String: Copy Part of %T98% to %T95%

4 Variable Modify Integer: %N5% = %N5% + 2

5 Variable Modify String: Delete Part of %T98%

 

I don't understand how block deletion of the CRLFs would help me in any way.

 

Randall responded with concurrance that the Text File Processing is faster for large strings. - - - I wonder if anyone else has any knowledge of this - either measured or anecdotal - - - - .

 

Les

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
 Share

×
×
  • Create New...