×

Discussion Board

Results 1 to 8 of 8
  1. #1
    Registered User
    Join Date
    Jun 2005
    Posts
    16

    Question Regression? UTF-8 problem in 1.1.5, not present in 1.1.3

    Hello,

    with this script I can import vcards in an utf-8 file (with umlauts etc.) with PyS60 1.1.3pre (Series 60 1.2):

    Code:
    import re
    import contacts
    import codecs
    import graphics
    import e32
    import appuifw
    
    expremptyline = re.compile(r"\r\n\r\n", re.IGNORECASE & re.MULTILINE)
    exprswitchtel = re.compile(r"TEL\;TYPE=", re.IGNORECASE & re.MULTILINE)
    
    def exportforseries60(vcard):
     return exprswitchtel.sub(r"TEL;", vcard)
    
    def readfile(filename):
     file = codecs.open(filename,"r","utf8")
     inhalt = file.read()
     file.close()
     return inhalt
    
    def writetext(s,clear = True,x = 2, y = 12):
     if clear: draw.clear()
     draw.text((x,y),s,0x008000)
    
    def carryon(event):
     global lockjc
     lockjc.signal()
    
    def quit():
     global lockjc
     lockjc.signal() #so it won't be locked in any case
     appuifw.app.exit_key_handler=None #don't know if this is necessary
     appuifw.app.set_exit()
     
    appuifw.app.title=u"Import VCards"
    canvas=appuifw.Canvas(event_callback=carryon)
    appuifw.app.body=canvas
    draw=graphics.Draw(canvas)
    
    lockjc=e32.Ao_lock()
    appuifw.app.exit_key_handler=quit
    
    vcardstrings = expremptyline.split(readfile(r"e:\python\addressbook.vcf")) #last line is an empty newline (ist Kontact schuld)
    writetext(u"%s VCards aus Datei geöffnet"%(len(vcardstrings)-1))
    
    contactdb = contacts.open()
    allenamen = contactdb.keys()
    laengeallenamen = len(allenamen)
    writetext(u"Es sind %s Kontakte im Telefonbuch vorhanden."%(laengeallenamen),False,2,25)
    lockjc.wait()
    for i in range(laengeallenamen):
     writetext(u"Lösche Kontakt Nr.: %s/%s"%(i+1, laengeallenamen))
     e32.ao_yield()
     del contactdb[allenamen[i]] #Missing here is the compact method to get the db-file small again (see other post of mine)
    
    #appuifw.note(u"Alle %s Kontakte gelöscht..."%(laengeallenamen),"info")
    #contactdb.compact()
    print "Alle gelöscht"
    
    laengevcardstring = len(vcardstrings)
    for g in range(laengevcardstring):
     writetext(u"Importiere VCard Nr.: %s/%s"%(g, laengevcardstring-1))
     contactdb.import_vcards(exportforseries60(vcardstrings[g]))
     e32.ao_yield()
    
    allenamen = contactdb.keys()
    laengeallenamen = len(allenamen)
    writetext(u"Komprimiere Telefonbuch...",False,2,25)
    #contactdb.compact()
    contactdb.close()
    appuifw.note(u"%s Kontakte importiert!"%(laengeallenamen),"info")
    This is not possible anymore with 1.1.5 (with the exact same vcf-file), the vcard_import method fails with
    Code:
    Traceback (most recent call last):
      File "E:\System\Apps\Python\my\syncandrunclient.py", line 154, in run
        execfile(target, g)
      File "e:\python\vcardimportseries60.py", line 66, in ?
        contactdb.import_vcards(exportforseries60(vcardstrings[g]))
      File "E:\system\libs\contacts.py", line 157, in import_vcards
        return [self[x] for x in self._db.import_vcards(unicode(vcards),vcard_flags)]
    UnicodeError: ASCII encoding error: ordinal not in range(128)
    I think the relevant parts are the method where I read the file via codecs.open(f,"r","utf8") and the method to import the vcard in the database.
    Is there any change between 1.1.3pre and 1.1.5 concerning the handling of unicode data?

    Thanks,

    Joost

  2. #2
    Super Contributor
    Join Date
    Dec 2004
    Posts
    643
    The default character set was changed back from 'latin1' back to 'ascii' between 1.1.3 and 1.1.5, but the reason for the breakage you see is actually a bug in contacts.py. Changing unicode(vcards) to vcards on contacts.py line 157 should fix this.

  3. #3
    Registered User
    Join Date
    Jun 2005
    Posts
    16
    Hello,

    sorry, but that doesn't fix the problem. The new error is:
    Code:
    Alle gelöscht
    Traceback (most recent call last):
      File "E:\System\Apps\Python\my\syncandrunclient.py", line 154, in run
        execfile(target, g)
      File "e:\python\vcardimportseries60.py", line 65, in ?
        #contactdb.import_vcards(exportforseries60(vcardstrings[g]))
      File "E:\system\libs\contacts.py", line 157, in import_vcards
        return [self[x] for x in self._db.import_vcards(vcards,vcard_flags)]
    UnicodeError: ASCII encoding error: ordinal not in range(128)
    Hoping for your next suggestion...
    Bye,
    Joost

  4. #4
    Registered User
    Join Date
    Jun 2005
    Posts
    16
    Hi,

    I tend to think that this problem is related to the latin1 -> ascii switch, as I could import äöüß in 1.1.3, but no € sign. And as I understand it, äöüß are covered by latin1, but not the € sign, and none of them is covered by ascii. So it seems to me that I never imported in unicode, but always in latin1 (which perhaps does not work anymore due to the switch to ascii).

    Bye,

    Joost

  5. #5
    Registered User
    Join Date
    Aug 2004
    Posts
    8

    Any solution?

    I have the same problem when opening files with Non-Ascii characters like "茅", "胈" and so on. With unicode conversion i get the same error message, without all UI elements show only rectangles instead of letters.
    I also tried converting the text file to unicode and opening them with different codec types from the codecs module with the same result.
    Thanks a lot

  6. #6
    Registered User
    Join Date
    Jul 2005
    Posts
    33
    I don't have experience with vCard.
    But ,when I read or write Thai-char in Unicode,I use this code.
    It worked fine for me.

    n=appuifw.query(u'Name of file E:\\','text',nm)
    if n is not None:
    f=codecs.open(("E:\\"+n),mode='w',encoding='utf_8')
    f.write(t.get())
    f.close()
    appuifw.note(u"save file E:\\"+n,'info')

    or

    n=appuifw.query(u'Name of file E:\\','text',nm)
    if n is not None:
    f=codecs.open(("E:\\"+n),mode='r',encoding='utf_8')
    t.clear()
    t.add(f.read()+u'')
    f.close()

    or (from screen to file)

    n=appuifw.query(u'Name of file E:\\','text',nm)
    if n is not None:
    f=open(("E:\\"+n),mode='w')
    f.write(t.get().encode('utf8').replace("\xE2\x80\xA9","\x0D\x0A"))
    f.close()
    appuifw.note(u"save file E:\\"+n,'info')

    May help...
    You may try my simple Unicode Editor
    http://www.isiam.info/dl/knotepad.zip

  7. #7
    Regular Contributor
    Join Date
    Jan 2005
    Location
    Bangkok, Thailand
    Posts
    148
    Glad to know that there is another Thai python programmer here.

  8. #8
    Registered User
    Join Date
    Jun 2005
    Posts
    16
    Cool, thanks for the code! I will try this, of course, sadly I'm quite busy with an exam right now.

    Bye,

    Joost

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
×