Best Practice 4 Developing In Multiuser Environmnt

[Cory]

My hope is that Kevin will mainly answer this as I would like to know what the design intent was but of course all are welcome. I would ask him offline but I think many of us could benefit from the answer.

 

Derfel brought a question about synchronization in “Line Numbers” that I too would like an answer to but since it was wondering off topic I thought I would ask it in a new post. Can you tell me what the best practice is for developing macros in a multi-user network environment?

 

I was considering this the other day as I was once working on a macro while users were running it and weird things started happening for them so I swore off doing that. I was going to look into the implications of the sync feature but haven't gotten there yet.

 

As it is now I have a couple of users on a network and we all access the same file. Sync is off. Currently I copy the macro in the macro file to work on while the users keep using the other. IOW I do not create another macro file. It seems stable but if I roll out a new change be deleting the old macro and renaming my development copy I have to have the users reload the MEX to realize the change. Since we are in a sort of trial stage this is no big deal but soon I’ll have a dozen or more so I’ll need another solution.

 

I’ve heard of people copying macros to user’s machines, loading different MEX files from macros, using MXE file and all sorts of different things. What is best? I think that if users were only playing in this environment it would be best to have one file and use sync. But how and where should I do my development and how best should I implement my changes?

 

One concern I have about sync is what functionally happen when it syncs? What if I change a macro while a user is running it and the sync kicks in? Maybe Kevin could explain when and how ME reads the lines of the macro to play, that would explain a lot. IOW does ME load the entire macro at the start of play or does it read it line by line as it’s being executed? That sort of thing.

 

Many thanks in advance.

[kevin]

I agree. It is good to start a new topic to discuss this. To answer some of your questions I am reposting the following that was originally posted in the 'Line Numbers' topic:

 

 

The feature to share a macro file works fine and is very convenient in situations like yours. But it will not be reliable unless you enable the Network option File Locking of Macro Files. As a convenience we also suggest that you enable the Automatic Synchronization Update.

 

But, if your network or your server(s) are not reliable and go down from time to time, and if most of the applications used by your users are on their local computers, then you may want to consider having the macro file on each individual computer. This allows the users to continue to work when the network or a server goes down.

 

However, if your network and server(s) are very reliable or if most of the applications used by your users require the network and server(s), then it doesn't matter if the macro file is in a central location. In this senario, if the network goes down, they can't work anyway so it doesn't matter if Macro Express is not working.

 

 

Some of the customers who are updating their macro files using the techniques described previously do not have permanent connection to their server. Some of them are updating their macro file via an FTP file transfer.

[Cory]

Thanks for the quick response Kevo but some of my questions are still unanswered. In my case the server and network are sound. Let’s assume that I enable the sync and locking, now how should I work on this macro?

 

Also could you explain how the locking and sync works?

 

Finally and if still relevant could you explain how and when ME reads the lines of a macro file to execute. Does it read it once at startup and never again? Or maybe it reads the entire macro file once each time the macro is run. Or maybe it only reads one line at a time as it is executing. I know it’s rather technical but I’m curious and it would help me get a grip on how best to develop. For instance if it only read the code of one macro when it was run I would be free to work on another macro in that file without getting it confused.

 

Hope you have a great weekend.

 

Thanks!

[kevin]

As it is now I have a couple of users on a network and we all access the same file. Sync is off. Currently I copy the macro in the macro file to work on while the users keep using the other.

In this situation, the critial setting is File Locking of Macro Files. This forces Macro Express to check to see if another process (another copy of Macro Express or a file copy) is making changes to the macro file when Macro Express wants to update the macro file. If more than one copy of Macro Express is updating the macro file, then the second one to attempt to update will wait until the first is finished.

 

If two copies of Macro Express happen to update the macro file at the same time then your macro file is going to get trashed. We cannot predict what will happen when a macro in the trashed file runs. To be safe you need to enable the File Locking of Macro Files feature.

 

...what functionally happen when it syncs?

Macro Express compares the macro file's size and date/time stamp with the size and date/time stamp of the file when it was loaded into memory. If the macro file is newer it loads it again. This process flushes the macro cache. Up until now we haven't discussed the macro cache. The macro cache is a critical piece of this discussion.

 

The first time a macro is activated, it is loaded into the macro cache just before it runs. The next time the macro runs, if it is in the macro cache, it runs without accessing the macro file. If the cache is too small to hold all of the macros, as macros run, the most recently used macros are stored in the macro cache and less recently used macros are dumped.

 

Macro Express allows you to set the size of the macro cache. See the Macro Caching setting under Options, Preferences, Caching. To use a minimum amount of memory you can disable macro caching altogether. If macro caching is disabled then every time a macro is activated, the macro will be loaded from the macro file.

 

What if I change a macro while a user is running it and the sync kicks in?

The File Locking of Macro Files feature described above protects the macro file. The File Locking logic would detect that you are currently writing to the macro file and hold the sync up for a moment. By the way, all this locking and syncing happens at computer speeds. It is hardly noticable.

 

In summary: If you are sharing a macro file you must enable File Locking of Macro Files on each computer running Macro Express or your macro file is at risk.

[Cory]

That was an excellent answer, thanks!

 

I had a misconception about locking. In my experience a lock means you can’t access it. Like AutoCAD.

 

I did not know about the cache. This explains a lot.

 

Now I’m still a little fuzzy on the locking. You’re explanation is good I’m just slow to learn. Is it fair to say this only applies to editing? And if there is only one person doing the editing would this not apply except for updating the cache?

 

Given your explanation of caching and how locking works with the cache when a macro is being run it seems that I want to enable locking and sync. The cache setting is irrelevant. This way I can edit the macro file everyone is using by copying the used macro to a separate WIP macro (in the same file) and work on that one. This is important as I need to access other macros as subroutines. Then when I’m done I can rename it to the original name and hotkey and the next time a user runs it the macro will load. Also if a user is running the macro when I do this it’s no big deal as they will continue to execute the older version in cache which won’t be updated until the next time they run it. Did I get this right?

 

One last question: What if you have a macro that runs for a long time? Something like a macro that loops indefinitely waiting for something to happen? Maybe it would be best to call that macro from a parent macro to force it to check periodically? Or run it as a scheduled item instead? What are your thoughts?

[kevin]

I had a misconception about locking. In my experience a lock means you can’t access it.

Yes, the macro file is locked and prevents any process other than the one that locked it from accessing the file. Unless something goes wrong the file is locked for a very short amount of time. Then the next process can access it.

 

Is it fair to say this only applies to editing?

The macro file can be updated by running a macro within it. For example, enabling or disabling a macro will update the macro file.

 

if there is only one person doing the editing would this not apply except for updating the cache?

No. The macro file can be updated at any time.

 

it seems that I want to enable locking and sync.

Yes, unless you want to risk losing your macro file.

 

Also if a user is running the macro when I do this it’s no big deal as they will continue to execute the older version in cache which won’t be updated until the next time they run it.

Not necessarily. If they haven't run that macro yet or if they just launched Macro Express or if their cache got full or if the synchronization routine detected a change or any number of other senarios that I can't think of right now, then the new version of the macro file will run. We have made every effort to make sure that the new version of a macro will run.

 

If you want to develop a new version of a macro then copy it, develop it, and debug it. Once you update the orginal macro, you can bet that Macro Express will try to use the new version of the macro. It may take a little while but you cannot count on it being very long.

 

Better yet, develop your macros in a separate macro file. This may not be possible if several people are writing macros at the same time. But even in that case you could develop the macro in a separate macro file and then export the macro from the development file and import it into the production macro file.

 

What if you have a macro that runs for a long time?

Since only one macro can run at a time I do not recommend that macros run for a long period of time. How you break things up depends on your environment.

 

The check to see if the macro file needs to be synchronized occurs at macro activation. So while a single macro is running, it will not be updated. However, if it uses the Macro Run command to run other macros, they may be updated.

[Cory]

Great, I'm crystal clear now.

[wbeldman]

The explanation above is very helpful but I'd change one bit of this scenario...

 

Instead of modifying a macro in the .mex file which is in use on several production systems, I'd rather have a separate prod.mex and a dev.mex (and maybe a test.mex as well). The production PCs use prod.mex and all my modifications are made in dev.mex.

 

To put an update into production, I would like to open prod.mex and import the update. But that causes two macros with duplicate names to show in the MacEdit's macro list! At least that's happening now, before we bring this locking into practice...

 

Is it possible to turn this option off, prevent it in any way from working or do you have another solution??

 

TIA

[kevin]

Currently, there is no option to import and replace macros. We are working on a solution, however.

[wbeldman]

Now on import both the old, existing macro and the newly imported one are listed in the Macro Explorer list:

 

MyTestMacro        No Activation  Global  02-02-07 15:21

MyTestMacro        No Activation  Global  02-02-07 15:31

 

 

 

Can we be sure that all Macro Run commands reference to the oldest macro?

 

And that by deleting that macro all references point to to the newer copy automatically (then the only copy)?

 

 

In that case we have a safe two step Import & Replace procedure...

[kevin]

When a macro executes the 'Macro Run' command it looks through the list of macros for a match. The first one it finds is the one that runs. Normally this would be the oldest macro but that is not guaranteed. Deleting the extra macros and leaving only the newest macro will work properly.