Re: [coldsync-hackers] Design logic of Sync conduits

[Marco van Beek <marco at 84andahalf dot com>]

>     "When a record is edited on the Palm (which I will now refer
>     to as Handheld ), what really happens is that the old record
>     is marked as deleted, and the new record assumes a new,
>     hopefully unique, {id} number."
>
> It _can_ happen this way, but it usually doesn't. When the record on the 
> Palm is changed, it's marked as dirty. But the record ID is usually 
> unchanged. Some apps might handle updates by deleting the old record and 
> adding a new one, but I've never seen it happen with any of the standard 
> PalmOS apps. It certainly isn't a good idea to have the record ID change 
> constantly since it introduces huge amounts of complexity into the whole 
> sync protocol, but maybe you've been working with an application that does 
> this?

Thanks Cristophe. You are, of course, right. I suspect that it only 
happens if a key field is changed (perhaps a field it uses for an 
index). I am working with the built-in applications, and I spotted this 
happening, and thought it was standard.

Since my current conduits always update the server regardless of {dirty} 
status, I generally only noticed the "new" records going through.

I have updated the document with your input, and thanks for the help.

I also have a question about the SPC stuff. Once I open the DB (using 
dlp_OpenDB), can I write to the data in the same way I would with Fetch 
& Dump conduits?

The Palm::SPC man page gives the impression that I have to read the data 
in by using dlp_ReadAppBlock, and then write it with dlp_WriteAppBlock, 
but then all the writing would be done in one go, and fastsync's 
wouldn't work.

If I use $PDB = &dlp_OpenDB(xxx), does it work as a handle in the same 
way as $PDB->load(xxx)? Am I just making this way too difficult by 
misunderstanding / over complicating SPC? Is in fact my problem that I 
need to open the "live" PDB rather than the copy in the .backup directory?

Regards,

Marco

Title: Multi User Conduits in Coldsync

Multi User Conduits in Coldsync

Draft Two

One of Coldsync's many capabilities includes being able to share Palm data with many users. However, when the relationship between the desktop and the palms becomes a one-to-many relationship, the underlying design logic of the conduit has to take this into account.

The only piece of data that the user cannot change on the Palm is the {id} field. This means that in order to syncronise across many Palm's, we have to use the {id} as the only reliable piece of information we can obtain from the Palm to match records. However, there may be a sting in the tail. When a record is edited on the Palm (which I will now refer to as Handheld ), it is possible that the old record is marked as deleted, and the new record assumes a new, hopefully unique, {id} number. This would seem to happen when primary index fields are changed, and is unverified. This gives us one good and one bad issue. First of all, we already have a way of avoiding the desktop ( which I will now refer to as Server ) overwriting a record that has also been changed on the Handheld (or vice versa) as they would have different {id} numbers. Secondly, there is a remote chance that two Handhelds would create a new record with the same {id}, so the Server MUST control the issuing of {id} numbers.

There are 8 possible scenarios when we syncronise a record.

Scenario	Handheld Record	Server Record	Task
1	Unchanged	Unchanged	Nothing to do
2	New		Add record to Server
3	Updated	Unchanged	Update record on Server
4	Deleted	Unchanged	Delete record on Server
5		New	Add record to Handheld
6	Unchanged	Updated	Update record on Handheld
7	Unchanged	Deleted	Delete record on Handheld
8	Updated	Updated	Conflict

Based on the information above, we can make the following observations.

• Scenario 2 will need the Master {id} from the server written back to the Handheld.
• Scenario 3 is sometimes a combination of 2 + 4, since we have no easy way of deciding which "new" Handheld record is really an edited record.
• Scenario 8 is sometimes a combination of 2 + 5.

So, logically, we need to look at the Master database first, and force any changes into the Handheld. We can do this because we know that any changed record on the Handheld will no longer have the same {id} number. As long as the new Handheld {id} range is different to the Server {id} range, we should never have a problem. We may be able to force this by using the {appinfo}{lastUniqueID} field in each of the Handhelds' databases.

1. Open connections and lock Server database tables from being written to.
2. Record current date/time.*
3. Check & update (if necessary) current date/time on Handheld.
4. Loop through Handheld records by {id} looking for Server match:
   1. If found:
      1. Check Handheld {dirty} flag:
         1. Handheld has changed: check Server timestamp against last sync
            1. Newer: Scenario 8: Conflict resolution needed
            2. Older: Scenario 3: Update record on Server (and clear Handheld {dirty} flag**)
         2. Handheld record has not changed: Scenario 1: No changes
   2. If not found:
      1. Check {dirty} flag:
         1. Set: Scenario 2: Add to Server and force Handheld {id} field.
         2. Not Set: Scenario 7: Delete from Palm.
5. Loop through Server records by {id} looking for Handheld match:
   1. If found:
      1. Check Server record timestamp:*
         1. Server has changed: check {dirty} flag
            1. Handheld has changed: Scenario 8: Conflict resolution needed
            2. Handheld not changed: Scenario 6: Update record on Handheld.
         2. Server record has not changed: Scenario 1: No changes
   2. If not found:
      1. Check Server record timestamp against last sync:
         1. Newer: Scenario 5: Add to Handheld and force {id} field.
         2. Older: Scenario 4: Delete from Server.
6. Update {dbinfo}{baktime} on Handheld (if not already done)
7. Unlock Server database tables and close all connections.

* What we really mean here is "Is it between when I last sync and before I started this process?". Because we may have already updated some Server records, they will show a newer Timestamp, which would confuse the issue. By recording the time after we lock the tables but before we change any records, any records newer than that must have been changed by us.

** If we can, by clearing the {dirty} flag once we have updated the Server record, we avoid confusing the second part of the sync.

Conflict Resolution: There are 4 possible outcomes:

1. Do nothing: leave both records as they are: The wimp-out option!
2. Overwrite the Server with the Handheld record: Generally a bad thing to do.
3. Overwrite the Handheld with the Server record: Also generally a bad thing to do.
4. Add the records to both by creating a new record on both the Server and the Handheld:

Having duplicate entries is the safest option, as long as the user knows which records have been duplicated so that the issue can be resolved. Having coldsync running on a server means that a summary and/or log file could be emailed to the user.

It is also very important to be able to exclusively lock the appropriate Server database tables while the sync process is going on, otherwise external changes can be made that will cause chaos! If two users try to sync at the same time, which gets to the conduit first will get a lock, and the other will have to wait in the queue.

Unanswered Questions:

Can we write to the HotSync log from inside the conduit?

Do we need to reset the dirty flags manually?

Marco van Beek - February 2004.

Written while coding SSiS: The Single Source information Server.