Re: [coldsync-hackers] Automated Daemon with different actions

["Bron Gondwana" <coldsync@brong.net>]

Andrew Arensburger said:
> On Thu, Nov 15, 2001 at 01:05:01AM +1100, Bron Gondwana wrote:
>> I have maybe 200 Handspring Visors out in the field.  They all have
>> modems in them, and are used to collect data.  This data is for many
>> different projects.  At the moment, they dial a modem on a Win98 box
>> running the standard hotsync software, which hopefully has accounts
>> for all of them, and they sync data.  If this fails, they drop the
>> connection unhappily.
>
> 	To start off with, it sounds as if you want to run ColdSync in
> daemon mode.

I have played with that.  It does look like what I want except for the
"reads all databases on the palm" bit.  Also, it has to run as root and
change users.  I would prefer to run as another user (and I've noticed
in the comments here that it's possible to run as another user so long
as all the per-palm options are for that same user).  I'll give it
another try though.

>> What I'd like is the following couple of bits of magic:
>>
>> * ID either purely on username, or on username and user_id (only
>> username
>>   if no user_id has been entered in the config file)
>
> 	Since the Visor doesn't have a usable serial number[1], they
> all show up with the dummy serial number "*Visor*", and ColdSync
> decides which one it's dealing with based on the username and user ID.
> 	The reason for using both is that multiple people can have the
> same name. It wouldn't do to have two users named John Smith share a
> configuration, just because they share a name.

Sure, though UID should help here too.  I presume that UID is set randomly
by the Windows HotSync software?  I wouldn't mind some way of finding out
what the UIDs out there are.  Specifically, some of these palms are already
in the field, and I don't know how to find out their UID.  I want to match
only on the username (and possibly treat the UID as a password type thing
once it is known).

> [1] Unless Handspring has fixed this in newer models.

The models we have haven't had it fixed.

>> * Action taken depends on the username, i.e. backup, restore, full
>> sync.
>
> 	You can specify a .coldsyncrc-like config file in
> /usr/local/etc/palms . This allows you to specify a set of conduits to
> run. If you don't want to perform a full sync, you can just override
> this.

Hmm.. I guess I can live with that.  The .coldsynrc-like file would be
generated from something else of course, but I can live with that.

> 	Currently, there is no way to say "make a backup of FooDB" or
> "restore /data/BarDB.pdb". You can, of course, write an external
> Sync-flavored conduit for this, but perhaps a better way would be to
> reuse the backup and restore code, and add two built-in conduits. Grep
> for "DummyConduit" in src/conduit.c to see how to do this.

Sure.  It seems a little messy to do it this way, but such is life.  I also
need to know the type, rather than the name of the database?  It appears
that way from the examples I've seen.

> 	On the other hand, this is probably suboptimal for your
> purposes: a full backup does just that: it copies the entire database
> from the Palm to the desktop, so if a user has a 3000-record database,
> then all 3000 records are transferred.
> 	What would be better for your purposes would be one-way
> variants of the generic conduit, which implement
> palm-overwrites-desktop and desktop-overwrites-palm. I think the
> easiest way to do this would be to subclass GenericConduit. You should
> be able to inherit everything, and only override the SyncRecord
> method.
> 	This way, if the user has a 3000-record database and only
> changes one of them, then only that record is transferred.

Sounds reasonable.  Of course it has to deal with the other end palm dying
or being lost, and replaced in the field.  I like 'copy everything every
time and sort it all out in the wash' because that way I have a history of
changes pre-parsing.  I don't think the databases are likely to get that
big on these trials anyway.

>> * Options are also available in the config.  I.E. for palms on which
>> all
>>   data is being collected into 'Foo' database, I would like to only
>>   backup data in 'Foo', then disconnect the modem (some people are
>>   dialing from Mexico to Australia, so line time is a premium).
>
> 	You can do this with a configuration similar to the following:
>
> 	conduit sync {
> 		type: XXXX/YYYY;	# Creator/type for "Foo" database
> 		path: /path/to/foo-conduit;
> 	}
>
> 	conduit sync {
> 		type: */*;		# Matches everything
>
> 		# No "path:" line, so this does nothing.
> 	}

Yep, though this suffers from the 'have to know type rather than name'
issue I pointed out above.  Specificially, what if I have two databases
with known names, but the same type and creator?

> 	There is currently no way to prevent ColdSync from fetching
> the list of all databases on the Palm.

At least in daemon mode, which is another reason not to use it.

>> * If a username is not recognised, a default behaviour should be
>> possible,
>>   either a single default, or based on username (i.e. FooP234 username
>>   will take the action for Foo).
>
> 	ColdSync doesn't do this, mainly because I don't like the
> security implications.

I don't particularly worry because I'm never giving them any information,
and the database I dump to is transation based, with 'most recent matching
transaction is the value', so I just join against the valid palms list at
the end and all is happy.

> 	If you look at the "while ((palment = getpalment())" loop in
> src/coldsync.c, you'll see how it determines whether an entry in
> /usr/local/etc/palms matches the Palm at the other end of the current
> connection or not. This code really ought to be put into a separate
> function, and is known to be buggy, so feel free to suggest
> improvements.

I'll have another look at it.

> 	For your purposes, I think the thing to do would be to declare
> that a field whose value is "+" (by analogy with the truck-sized
> security hole in Solaris) is a wildcard. Thus, the entry
> 	+|+|+|dataguy|+|/usr/local/libdata/collector.rc
> would mean "if no other entry has matched so far, setuid() to user
> `dataguy' and use /usr/local/libdata/collector.rc as the config file."
> 	However, bear in mind that doing this means that anyone can
> dial in to your modem and poison your database with bogus data. If you
> know who the data collectors are, which Palms they're using, and what
> their user names and IDs are, you can have one entry per person.

I can know them at the end of a project, but not always during.  In a
perfect world, I'd get all the information in the correct order.  I'm
quickly learning that medical data collection is not a perfect world,
and all sorts of strange things will be sent in.  The important thing is
that someone out in the field gets the behaviour they're expecting (my
end sucks the data off their palm) rather than an obscure error message
which generates a support phone call in the best case (within Australia),
and long delays in the worst case (Mexico connections for now).

> 	Oh, one more thing: currently, the config file format doesn't
> allow you to include other config files. In other words, if twenty
> people use a particular conduit to synchronize the "Foo" database, and
> you want to change that, you need to edit twenty config files.
> 	Until this is fixed, you can always use m4 to generate
> people's config files from a template.

I'll probably get something written in Perl to spit them out.  All my
other data handling stuff is already written in Perl, so I have libraries
that can grab the list of known palms from the distribution database.

>> My question for the list - has anyone done anything this?  I've read
>> through quite a lot of the coldsync code over the past few days, and
>> played with a couple of options.  Unfortunately, the way the code is
>> now means that coldsync is quite committed to exactly what it wants to
>> do _before_ it knows who the palm is - or even opens a connection.
>
> 	I'm not sure what you mean. What does ColdSync want to do that
> it shouldn't be doing?

It decides what action (from the command line switches) to take before it
knows what the remote palm is.  I was originally looking at branching from
Daemon mode to Backup mode if I found a matching palm.

>> I guess the other question - is it worth me trying to re-architect
>> this for everyone else (would anyone else be likely to use per-palm
>> actions in a server only environment)?
>
> 	Yes, I think so. I'd like for ColdSync to be something you
> could reasonably use to provide centralized syncing services for a
> whole company.
>
>> My other option is basically to murder coldsync.c.  I really don't
>> like parser.y and the structs.
>
> 	What don't you like? If you have a better suggestion, I'm all
> ears.

It's very nice code - I was actually quite impressed with it, just
that you need to make changes in coldsync.h, plus a couple of spots
in parser.y, just to add a single value to the config.  It's more a
problem I have with the low-level coding required in C.  I'm sure
there are generic config-file parsers for C available, that expose
a logical API which allows new options to be added without multiple
edits.  I guess I'm just a Perl programmer at heart ;)

>> Of course what would be left would be just the backup section, but
>> written to assume options in the config file instead of on the
>> command line.
>
> 	Take a look at the v2.2.4 snapshot, which has an options{}
> section in the config file.

Cool.  I've had segfault problems (parsing %s in printf statements by
the look of things) with installing straight from source on Debian, but
I haven't taken the time to check exactly why yet.  I've been using
the .deb package in Debian unstable.

> 	I hope this answers some of your questions. If not, feel free
> to ask.
> 	I do have one request, though: you are, of course, free to rip
> out big honking chunks of ColdSync and replace them with your own. But
> if you make your version incompatible with the standard version at
> ooblick.com and distribute it to third parties, I'd like to ask that
> you call your version something else, so that people don't mistakenly
> assume that the two versions will work the same way.

Oh, I certainly would either call it coldsync-$version-brong or rename
it if the changes were massive enough.  I'm more likely to try and keep
compatability and provide a patch against a particular version than
distribute anyway.

> 	(In the best of all possible worlds, of course, we'd work out
> some way of making the standard version do what you need it to, and
> avoid gratuitous forking.)

Sure, and I'm happy to put time into it too.  I can't give you too much
time at the moment because I'm massively over-worked trying to convert
myself into a department of 4 people over the next month.  One of these
people is also a good C programmer and will be taking over most of the
palm programming work, so I'll probably work with him to get the server
end happy.  The other programmer also has palm experience, so she'll
hopefully be helpful too.. who knows, we may even come up with some
patches that others can use.

Anyway, I'm probably going to be freezing work on this for about a week
while I deal with some more urgent issues.

I guess the issue is what level it's best to deal with this - minimum
disruption to coldsync and work out how to make conduits handle the
work I need, or just hack coldsync into something more like my idea of
the ideal app, while breaking all compatability.  Either way, Open
Source makes it easier to do things.

( of course I _should_ write down all the things I've come across (I
  believe the 'snum: "*Visor*-G" checksum issue that caused me trouble
  for a while is fixed now), and write documentation patches - coldsync
  is better documented than a lot of opensource stuff I've used, but
  could probably do with more - no such thing as too much doco! )

Bron.


This message was sent through the coldsync-hackers mailing list.  To remove
yourself from this mailing list, send a message to majordomo@thedotin.net
with the words "unsubscribe coldsync-hackers" in the message body.  For more
information on Coldsync, send mail to coldsync-hackers-owner@thedotin.net.