Re: [coldsync-hackers] syncing of creation/modification dates

[Pierre Phaneuf <pp at ludusdesign dot com>]

David A. Desrosiers wrote:

 >> Ok, agreed that its not a cache file, because it also contains
 >> other "non-critical" data, like you mentioned (also of interest is
 >> the list of visited links). That's why I'd rather not pre-emptively
 >> delete the metadata file, unless I changed anything.
 >
 > Ah, but if you sync a new version of MyDB.pdb, you _have_ changed
 > something, in fact everything, even if MyDB.pdb contains the exact
 > same webpages. Are you sure that the remote site didn't change a link
 > on the site? Or fix a typo? Or move a link from page 1 to page 2?
 > Something as simple as this would invalidate your "seen" links as
 > recorded in the Plkr-MyDB.pdb metadata file, right?

Here's my algorithm:

For every record in the PDB, I check whether they have a counterpart in 
the cache directory. If not, I mark the record as deleted and set a 
"modified" flag.

For every file in the cache directory, I add or update the corresponding 
record in the PDB (setting the "modified" flag again). If the record 
already existed and had the exact same data, I do not update it (and do 
avoid setting the "modified" flag).

If the "modified" flag is set, I update the creation and modification 
time to the current time.

So if I do "plucker-build -c", then I sync twice, the second time I 
sync, the creation/modification times will *not* be changed. Supposing 
that ColdSync's generic conduit would transfer the timestamps, if I had 
browsed the document a bit, all of my "seen" links would be just fine, 
they wouldn't be invalidated.

If I did "plucker-build -c" and everything is *exactly* the same, my 
"seen" links are still fine. If something was different, even only one 
page, it would be invalidated (which I think is fair).

 >> I don't use the "plucker-build -f" mode, I have "plucker-build -q
 >> -c" in a cron job, every hour, and I *update* (never create, this
 >> being a "fetch" conduit) a previously installed PDB. I created the
 >> PDB using "plucker-build -f", though.
 >
 > I'm confused, you just said two contradictory things. You said you
 > never use plucker-build -f, but instead use the -c to create the
 > cache of items in ~/.plucker/cache. Which piece of this puzzle
 > actually concatenates those records into the pdb, and which tool is
 > used to install it on the Palm?

My "plucker-fetch" conduit. I updated the PDB that is in my 
~/.palm/backup directory, and I then let the ColdSync generic conduit do 
its magic and update the corresponding database on the Palm in a minimal 
way.

 >> IMHO, the "-f" mode of Plucker is next to useless for something
 >> like updating from a ColdSync conduit, since you have to delete the
 >> PDB on the Palm before installing a new one.
 >
 > A Coldsync limitation? I never have to delete my pdb on the Palm when
 > I install a new version of Plucker documents, even from the same
 > content or the same filename. Then again, I don't use Coldsync
 > normally.

Yes, could very be a ColdSync limitation (or safety). If you drop a file 
in the ~/.palm/install directory, it will be installed on the next sync 
by the generic install conduit, but the generic install conduit will not 
overwrite an existing file of the same name.

I don't remember pilot-xfer's behavior when trying to install a file 
that already exists on the Palm.

My first idea was exactly to have a cron job keep an up-to-date PDB in 
the ~/.palm/install directory (using "plucker-build -f"), but I think 
that even if this would have worked, I like my current method better, 
because I get to leave the file pristine if the content of the cache 
directory didn't really change (overwriting the database would 
invalidate the "seen" links, even if the file is perfectly identical, 
hardly ideal).

 >> I also try to make things as fast as possible (if some records are
 >> completely identical, I don't touch them, so only the
 >> deleted/modified/new records get synced).
 >
 > Are you creating a stream resource database? Meaning, are you doing
 > basically what the Plucker 0.03 perl conduit did, creating the pdb
 > "on the fly" when you sync the Palm? There were several reasons why
 > this approach was deprecated a couple of years ago. Or are you doing
 > something completely different, creating pdb on disk, and sync'ing
 > one file to the PDA?

What are these reasons?

I'm not updating the PDB using DLP calls, if that's what you're 
wondering. I'm updating the back up copy on the workstation, which 
ColdSync then syncs with the PDA. This is what all ColdSync "fetch" 
conduits do (they can't talk to the PDA).

 >> It works perfect, except for ColdSync not updating the creation
 >> time (which I'm careful not to change if I didn't actually change
 >> the PDB, and would thus clear out the metadata correctly).
 >
 > Sounds like a Coldsync problem, overall. Either delete the metadata
 > when you know the data has changed (i.e. always), or use Andrew's
 > suggestion from earlier, which will require a bit more checking in
 > your code to make sure the .pdb exists before you try to update the
 > creation time of it.

I'm trying another option right now. I'm adding the following to my 
.coldsyncrc:

conduit sync {
         path: "/home/pp/bin/conduits/plucker-sync";
         type: Plkr/Data;
}

conduit sync {
         path: [generic];
         type: Plkr/Data;
}

This plucker-sync conduit sets the creation/modification times on the 
PDA's  PDB to be identical to the one in the local file (if it exists). 
I have to put the silly second entry so that the generic conduit still 
runs (it'd be nice if there's was a way to do this without having the 
extra config, say "fetch" conduits being able to send DLP requests or 
"sync" conduits being able to tell ColdSync that it should still run the 
generic conduit).

 >> Of course, I could change the whole thing into a sync conduit
 >> rather than a fetch conduit, but having ColdSync's generic conduit
 >> do the grunt work for me is quite attractive... :-)
 >
 > You are sync'ing, so using a 'fetch' method on this probably isn't
 > going to scale in the long run. Once I figure out why Coldsync
 > doesn't work with m505 devices under BSD (but pilot-link does, go
 > figure), I'll give it a try.

Why wouldn't using a fetch conduit scale? The 'fetch' method seems to 
have been created exactly for this purpose, in fact!

-- 
Pierre Phaneuf
http://advogato.org/person/pphaneuf/


-- 
This message was sent through the coldsync-hackers mailing list.  To remove
yourself from this mailing list, send a message to majordomo@thedotin.net
with the words "unsubscribe coldsync-hackers" in the message body.  For more
information on Coldsync, send mail to coldsync-hackers-owner@thedotin.net.