Re: [coldsync-hackers] ColdSync 1.5.0 snapshot (and RFC)

["J.D. Smith" <jdsmith@astro.cornell.edu>]

>         - Added John-David Smith's Install conduits. These provide a
>           hook for actions to be taken when installing a database.
>           (When you're looking at the code, bear in mind that he wrote
>           the code before I ripped main() apart and rewrote it from
>           scratch. So John-David Smith gets credit for writing this
>           code, and I get the blame for breaking it.)

Thanks for including this, Andrew.  I've been using it with success for
over a month (though I've yet to test if your main() redesign might have
broken anything).  

> * Install conduits:
> 
>         The way Install conduits currently work (or should work) is as
>   follows: ColdSync looks at each file in ~/.palm/install in turn, and
>   runs all applicable Install conduits defined in .coldsyncrc . An
>   Install conduit can modify its input database, upload records
>   directly to an existing database on the Palm, add records to
>   databases in the backup directory, delete the input database, or
>   anything else it likes.
>         After the Install conduits have run, ColdSync takes a second
>   look at ~/.palm/install, and uploads anything that's still there.
>         The intent is to allow Install conduits to control how a given
>   database gets installed. For instance, if you want to incorporate
>   all of your friend's addresses, you can just plunk his AddressDB.pdb
>   into ~/.palm/install , and a conduit will add its contents to
>   ~/.palm/backup/AddressDB.pdb, then delete
>   ~/.palm/install/AddressDB.pdb .
> 
>         Question: Let's say that your site administrator has just
>   dropped a copy of the corporate directory in your
>   ~/.palm/install/AddressDB.pdb . However, you prefer to use the
>   SuperAddressBook application, so you write an Install conduit to
>   convert ~/.palm/install/AddressDB.pdb to
>   ~/.palm/install/SuperAB.pdb. After ColdSync runs the Install
>   conduits for AddressDB.pdb, should it also run the Install conduits
>   for SuperAB.pdb ? Or should it assume that if you just created it,
>   then you know what you're doing?

The latter, absolutely.  Otherwise you could end up with loops.  This is
nicely clarified by calling the install phase the "pre-install" phase,
with the install phase (as before) simply stupidly copying everything in
.palm/install/ to the unit (well, not totally stupidly, the mod_num is
checked).  The pre-install phase can in fact do the full-up SPC record
based install, if it likes, but that will be rare.  Pre-install conduits
have many choices, among which are:

1)  Update the file in place, leaving it in install/.  This isn't much
different from a fetch conduit, which can modify the version in backup/,
except that it can be made either to strictly overwrite the palm's
version of that database, or only update it based on the modification
number.  Also, it runs even if no backed-up version yet exists.  No
record-level syncing is done on these records.

2)  Merge the file's records into a database in backup/, and delete it. 
No install will be done, but record-based syncing will occur.  The
example of merging address books is valid, but this could be achieved of
course with a user-level tool, unassociated with the sync.  This may be
more convenient, if you're always adding people's databases in.  A
better example is some process which requires knowing certain records
are on the device.  At install, you can gather those records and write
them out, let them get merged in or installed otherwise to the Palm, and
then, in a complimentary post-dump conduit, ensure the records actually
made it.  This is most relevant for databases which are automatically
put in install/, but which may or may not be installed (because the user
forgets to sync for a week, say), or because an install error occured. 
Note that it's *not* enough to check whatever is in *backup*, since this
can in principle be different from what's on the Palm, if record level
syncing ever ran afoul.

3) Update the file and change its name/database type, writing it out
again into install/.  This is another interesting variant, an in-place
conversion.  The resulting database should be directly installed, to
prevent loops.

> 
>         Question: one thought that crossed my mind was that of adding
>   another built-in conduit, "[install]", which installs its database,
>   and make it a default conduit, so that by default, ColdSync will
>   install new files in the normal way. This would allow you to do
>   something like:
>         conduit install {
>                 type: Supe/Addr;
>                 path: /conduits/address-preprocessor;
>         }
>         conduit install {
>                 type: Supe/Addr;
>                 path: [install];
>         }
>         conduit install {
>                 type: Supe/Addr;
>                 path: /conduits/address-postprocessor;
>         }
>   The danger here is that if you forget to invoke "[install]"
>   explicitly after your own conduit, the file will still be there, and
>   ColdSync will try to reinstall it the next time you sync.

This is a danger, I think.  This makes install much more like "sync",
and thus more difficult for the user to implement without messing up.  
I think keeping the install-phase a default, and letting the conduit
writer delete whichever files needn't be installed anymore will work
more easily.  In many cases, all installable files will be processed,
deleted, and then no *actual* install will occur at all.

> 
>         Question: currently, if an Install conduit deletes its input
>   database, that means that it has taken care of installing it (or
>   whatever). What if you have two conduits:
>         conduit install {
>                 type: Foox/Barx;
>                 path: /conduits/first;
>         }
>         conduit install {
>                 type: Foox/Barx;
>                 path: /conduits/second;
>         }
>   If /conduits/first deletes its input database, should
>   /conduits/second still run?


Maybe try:


this-round=all install conduits
LOOP UNTIL this-round is empty, or no files remain:
Clear "next-round".
Loop over install/ files (generate the list anew each time):
1) if install conduit from "this-round" matches file, run it
2) if another install conduit from "this-round" matches, don't run it,
but instead save it to a list of further conduits to run (next-round).
Set this-round to next-round. 
END LOOP

This is pseudo-logic only.  You should be able to put it more elegantly,
perhaps with a maximum depth to avoid runaway loops (a smaller danger
here, but potential).  Simply rescanning the directory, and running
subsequent install conduits on remaining files is the idea.  

Another possibility: let the conduit writers deal with this.  Of course,
the user may just chain together two install conduits he shouldn't
have... it's somewhat of a tough call.

JD

-- 
 J.D. Smith                  |   WORK: (607) 255-6263
 Cornell Dept. of Astronomy  |         (607) 255-5842
 304 Space Sciences Bldg.    |    FAX: (607) 255-5875  
 Ithaca, NY 14853            |

-- 
This message was sent through the coldsync-hackers mailing list.  To remove
yourself from this mailing list, send a message to majordomo@thedotin.net
with the words "unsubscribe coldsync-hackers" in the message body.  For more
information on Coldsync, send mail to coldsync-hackers-owner@thedotin.net.