Re: [coldsync-hackers] comparing pdb->DBF (4000 records)

["Bron Gondwana" <coldsync@brong.net>]

Wojciech Godynski said:
> Hi.
> I'm not sure if I adress this question right.
> I have to compare pdb file with dbf file. DB contains about 4000
> records. I'm doing it in perl,using XBase module and Palm::PDB.
> The point is to make pdb have the same records as dbf, (dbf sometimes
> changes,pdb never).

Can you afford the memory to keep them all in memory at once?  Ok, silly
question, you already do (use of "foreach" below).

> Here is my algorithm:
>
> 1 I walk through dbf ,look for  correspond record in pdb. If  I find
> one,check if it was changed.If don't find one I add the record from dbf
>  to pdb.
> 2 Then I have to walk through pdb looking for correspond record in dbf.
>  If I don't find one I have to delete it from pdb.

Walk through DBF, build a hash table of all entries.
Walk through PDB.  For each entry,
if (entry_is_in_hash_table) {
    delete_entry_from_hash_table
} else {
    delete_entry_from_pdb
}
For all remaining entries in hash table, add to PDB.

This will do both operations, and reduce the load considerably.  Also
consider storing all your 'delete from pdb' operations until the end,
that way you don't need to restart indexing the pdb (if that happens).

> First step takes reasonable time (3 min)
> The second one takes not reasonable time.
>
> Here is my code;
>  my $cennik2=new XBase "cennik" or die "450 Nie przygotowano nowych
> towarow 2!\n";
>  my $it2=0;  my @data;  my $it1;  my $it3=0;  my $it4;
>  my
>
($id_tow,$nazwa,$skrot,$vat_grupa,$jm,$szt_op,$cena_s2,$cena_n2,$stan2,$skro
tp,$nic,$sww);
>  foreach(@{$PDB->{"records"}})
>    {
>    $found="false";
>    $it3=0;
>    $it4=$cennik2->last_record;
>
>
($id_tow,$nazwa,$skrot,$vat_grupa,$jm,$szt_op,$cena_s2,$cena_n2,$stan2,$skro
tp,$nic,$sww)=unpack("a15
>  a40 a15 n a4 N N N N a20 a a15",$_->{data});
>    $id_tow=&Polish2($id_tow);$it1=index($id_tow,"\000");
>    if($it1 != -1) {$id_tow=substr($id_tow,0,$it1);}
> #    $cursor=$cennik2->prepare_select();
>    for($it3;$it3<$it4;$it3++)
>    {
>        @data=$cennik2->get_record($it3,"ID_TOW");
> #        print STDERR (@data,"\n");
>    if($id_tow eq $data[1])
>        {
>        $found="true";
>        last;
>        }
>    }
>    if($found ne "true")
>    {
>    $PDB->delete_Record($_,"true");
>    print DIFF "-",$data[1],"\n";
>    }
>    print STDERR $it2++,"\n";
>    }
>  $cennik2->close();


This is untested, but it should do both what your code below did, a little
quicker:

# Here is a list of the names of each element in the palm record.  Since
# I don't know what they're for, I'll keep them here for reference
# id_tow nazwa skrot vat_grupa jm szt_op cena_s2 cena_n2 stan2 skrotp nic
sww

use constant ID_TOW => 0;

# Open the XBase database
my $cennik2 = new XBase "cennik" or die "450 Nie przygotowano nowych
towarow 2!\n";
my %records;

foreach my $record (@{$PDB->{"records"}}) {
    my @data = unpack("a15 a40 a15 n a4 N N N N a20 a a15", $record->
{data});
    my $id_tow = &Polish2($data[ID_TOW]);
    #$it1 = index($id_tow, "\000");
    #if($it1 != -1) {
    #    $id_tow = substr($id_tow, 0, $it1);
    #}
    # There is a shorter way of doing this
    $id_tow =~ s/\0.*$//;  # will remove everything from the first \000
    $records{$id_tow} = \@data;  # copy data into the hash
}

my $deleted = 0;

my $size = $cennik2->last_record - 1;
my %to_add;
foreach my $id (0..$size) {
    my @cennik_data = $cennik2->get_record($id, "ID_TOW");
    if ($records{$cennik_data[1]}) {
        $PDB->delete_Record($id, "true");
        print DIFF "-", $id, "\n";
        print STDERR $deleted++,"\n";
    } else {
        $to_add{$cennik_data[1]} = \@cennik_data;
    }
}

# finally, this bit will add the bits to be added from the PDB, if you
# want that too.  Note that I'm not sure on the syntax for this because
# I haven't seen your code.

foreach my $id (keys %to_add) {
    # presuming that same pack is used?
    my $record = pack("a15 a40 a15 n a4 N N N N a20 a a15", @{$to_add
{$id}});
    $PDB->add_Record($id, $record);
}

$cennik2->close();


-------------------------

Hope this helps you.

Regards,

Bron ( list virgin! )


This message was sent through the coldsync-hackers mailing list.  To remove
yourself from this mailing list, send a message to majordomo@thedotin.net
with the words "unsubscribe coldsync-hackers" in the message body.  For more
information on Coldsync, send mail to coldsync-hackers-owner@thedotin.net.