Speed Of Text File Process

[Les Hazlett]

Hi,

 

I am in the process of integrating two macros. I assumed that it would be faster to work with the data from the first macro without writing it to a file and reading it back with the second. However, it seems that using Text File Processing is so much faster than manipulating the large string, that it is actually slower.

 

Perhaps someone can advise.

 

The data (file) structure is lines ending with CRLF. I replaced the Text File Begin Process/Text File End Process loop with a Repeat Until/Repeat End loop which works fine. I capture the lines with the following instructions:

 

Variable Set Integer %N5% with Postion of Text in Variable

Variable Modify Integer: Dec (%N5%)

Variable Modify String: Copy Par of %T98% to %T95%

Variable Modify Integer: %N5% - %N5% + 2

Variable Modify String: Delete Part of %T98%

 

This works but is slower than writing and reading text files.

 

Thanks for any help,

 

Les

[joe]

Hello Les!

 

It is possible that working with a string in this manner can be slower than processing a loop. You are using five lines of "character-by-character" code to get to, and strip, the CR/LF as compared to a single line of code that reads in the string and removes the CR/LF all at once. How big is the string?

 

Do I understand that you are combining macros? There are no CR/LFs in native Macro Express code. What am I missing?

 

Also, can you post the native code of the example shown here? It might help.

[randallc]

Hi, Les,

 

Yes, I agree that the text file process seems to be heaps faster for huge strings, and Floyd has posted his revised sort program using the vbscript equivalent of text file process, making things so much faster.

I have posted the csv file processor which reads the whole file as one line (slows things down in vbscript too), and I have got the text file process (vbscript) going for text files, 1.8Mb file loads, sorts, saves, in 3 secs! (same process as Floyd is now using, and I think his would do the same). I'll post it if you want.

 

Best, randall

[Les Hazlett]

Joe,

 

Thanks for your response.

 

>>Also, can you post the native code of the example shown here? It might help.

 

No one would want to look at the two huge macros I am integrating but here are the example commands.

 

<IVAR2:05:13:98:%T13%><NMVAR:09:05:0:0000001:0:0000000><TMVAR2:10:95:98:001:N05:><NMVAR:01:05:1:0000005:2:0000002><TMVAR2:11:98:00:001:N05:>

 

>>Do I understand that you are combining macros? There are no CR/LFs in native Macro Express code. What am I missing?

 

When integrated into one job, the first macro runs the second macro. The CRLFs are in the data being processed, not in the ME code. When run separately, the first macro writes a CRLF delimited file and the second one reads it. I thought that it would be faster to leave the large text string in RAM than writing the file and reading it back with Text File Processing loop. It turned out to be slower. The strings are 2-5 megabytes.

 

In a repeat loop, the five instructions that I posted do the eqivalent of the Text File Process instruction loop. It puts the data into T98, one line at a time, and deletes the line from the input string. Instruction 1 finds the position of the first CRLF. The text of the CRLF is held in a text variable T13. 2. Decrimenting N5 makes it point at the last character in the first line of data. 4. Increasing N5 by 2 makes it point to the LF so that the first line including the CRLF can be deleted by instruction 5. Oh how I wish there was a command for Text String Process.

 

1 Variable Set Integer %N5% with Postion of Text in Variable %T98%

2 Variable Modify Integer: Dec (%N5%)

3 Variable Modify String: Copy Part of %T98% to %T95%

4 Variable Modify Integer: %N5% = %N5% + 2

5 Variable Modify String: Delete Part of %T98%

 

I don't understand how block deletion of the CRLFs would help me in any way.

 

Randall responded with concurrance that the Text File Processing is faster for large strings. - - - I wonder if anyone else has any knowledge of this - either measured or anecdotal - - - - .

 

Les

[Les Hazlett]

Randall,

 

I would like to see your code, but as I said, I won't get to look at it for several weeks.

 

Thanks,

 

Les

[joe]

Hello Les!

 

If the Text File Begin/End Process command is the faster of the two options, and if speed is the primary concern, then why not use it?