Hello, I have been using freeze.py on 32 bit linux distributions without a problem. But recently I tried to do the same on RHEL5 x86_64 and ran into some issues.
1) When I ran the script, I got Error: needed directory /usr/lib/python2.4/config not found
2) Then I "yum install python-devel" which installed python-devel.i386 and python-devel.x86_64. Then when I tried freezing, it worked but make barfed up a bunch of errors during ld like
/usr/bin/ld: warning: i386 architecture of input file `/usr/lib/python2.4/config/libpython2.4.a(rangeobject.o)' is incompatible with i386:x86-64 output
and
/usr/lib/python2.4/config/libpython2.4.a(posixmodule.o): In function `posix_getcwd': (.text+0x53a1): undefined reference to `PyEval_RestoreThread'
3) I tried uninstalling python-devel.i386 (retaining only the x86_64) and freeze fails with the same error message as in 1
On Tue, 03 Nov 2009 11:15:39 -0800, Girish Venkatasubramanian
<giris...@gmail.com> wrote: > Hello, > I have been using freeze.py on 32 bit linux distributions without a > problem. But recently I tried to do the same on RHEL5 x86_64 and ran > into some issues.
> 1) When I ran the script, I got > Error: needed directory /usr/lib/python2.4/config not found
I don't know anything about freeze.py but on 64-bit Red Hat distros (RHEL, Fedora, etc) it should be /usr/lib64/python2.4/config :-)
-- Rami Chowdhury "Never attribute to malice that which can be attributed to stupidity" -- Hanlon's Razor 408-597-7068 (US) / 07875-841-046 (UK) / 0189-245544 (BD)
Hi Rami, Thanks for pointing this out. I did see that point - but apart from installing python-devel (which has created and populated /usr/lib64/python2.4/...) I am not sure what I should do - is there some setting in python where I can ask it to look at lib64 instead of lib? Thanks.
<rami.chowdh...@gmail.com> wrote: > On Tue, 03 Nov 2009 11:15:39 -0800, Girish Venkatasubramanian > <giris...@gmail.com> wrote:
>> Hello, >> I have been using freeze.py on 32 bit linux distributions without a >> problem. But recently I tried to do the same on RHEL5 x86_64 and ran >> into some issues.
>> 1) When I ran the script, I got >> Error: needed directory /usr/lib/python2.4/config not found
> I don't know anything about freeze.py but on 64-bit Red Hat distros (RHEL, > Fedora, etc) it should be /usr/lib64/python2.4/config :-)
> -- > Rami Chowdhury > "Never attribute to malice that which can be attributed to stupidity" -- > Hanlon's Razor > 408-597-7068 (US) / 07875-841-046 (UK) / 0189-245544 (BD)
Rami Chowdhury wrote: > On Tue, 03 Nov 2009 11:15:39 -0800, Girish Venkatasubramanian > <giris...@gmail.com> wrote:
>> Hello, >> I have been using freeze.py on 32 bit linux distributions without a >> problem. But recently I tried to do the same on RHEL5 x86_64 and ran >> into some issues.
>> 1) When I ran the script, I got >> Error: needed directory /usr/lib/python2.4/config not found
> I don't know anything about freeze.py but on 64-bit Red Hat distros > (RHEL, Fedora, etc) it should be /usr/lib64/python2.4/config :-)
This sounds a lot like a missing Python devel RPM.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Source (#1, Nov 03 2009)
Hey Marc-Andre, Ummm - I have installed python-devel.x86_64 and checked that the /usr/lib64/python2.4/ is populated - anything else I can/shuld do to check/ensure the the devel rpm is installed? Thanks.
On Tue, Nov 3, 2009 at 11:32 AM, M.-A. Lemburg <m...@egenix.com> wrote: > Rami Chowdhury wrote: >> On Tue, 03 Nov 2009 11:15:39 -0800, Girish Venkatasubramanian >> <giris...@gmail.com> wrote:
>>> Hello, >>> I have been using freeze.py on 32 bit linux distributions without a >>> problem. But recently I tried to do the same on RHEL5 x86_64 and ran >>> into some issues.
>>> 1) When I ran the script, I got >>> Error: needed directory /usr/lib/python2.4/config not found
>> I don't know anything about freeze.py but on 64-bit Red Hat distros >> (RHEL, Fedora, etc) it should be /usr/lib64/python2.4/config :-)
> This sounds a lot like a missing Python devel RPM.
> -- > Marc-Andre Lemburg > eGenix.com
> Professional Python Services directly from the Source (#1, Nov 03 2009) >>>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________
> ::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/
Girish Venkatasubramanian wrote: > Hey Marc-Andre, > Ummm - I have installed python-devel.x86_64 and checked that the > /usr/lib64/python2.4/ is populated - anything else I can/shuld do to > check/ensure the the devel rpm is installed?
If you have the config/ sub-dir in there, things should be fine.
However, it's possible that you need to tweek the freeze.py script a little, since RedHat chose to split the Python installation on x64 in two parts and they may have missed patching freeze.py as well:
The platform independent parts are in /usr/lib, whereas the platform dependent parts are in /usr/lib64.
Python normally doesn't support this. It only has a prefix and an exec_prefix, but those only allow to do things like prefix=/usr and exec_prefix=/usr64, not changing the lib/ part in /usr/lib/.
> On Tue, Nov 3, 2009 at 11:32 AM, M.-A. Lemburg <m...@egenix.com> wrote: >> Rami Chowdhury wrote: >>> On Tue, 03 Nov 2009 11:15:39 -0800, Girish Venkatasubramanian >>> <giris...@gmail.com> wrote:
>>>> Hello, >>>> I have been using freeze.py on 32 bit linux distributions without a >>>> problem. But recently I tried to do the same on RHEL5 x86_64 and ran >>>> into some issues.
>>>> 1) When I ran the script, I got >>>> Error: needed directory /usr/lib/python2.4/config not found
>>> I don't know anything about freeze.py but on 64-bit Red Hat distros >>> (RHEL, Fedora, etc) it should be /usr/lib64/python2.4/config :-)
>> This sounds a lot like a missing Python devel RPM.
>> -- >> Marc-Andre Lemburg >> eGenix.com
>> Professional Python Services directly from the Source (#1, Nov 03 2009) >>>>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >> ________________________________________________________________________
>> ::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> http://www.egenix.com/company/contact/
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Source (#1, Nov 03 2009)
I checked and ls /usr/lib64/python2.4/config/ returns config.c config.c.in install-sh libpython2.4.a Makefile makesetup python.o Setup Setup.config Setup.local
so I am guessing the python-devel installation went off OK, from what you say.
I looked at the freeze.py code and I see your point. But for tweaking, I would need to know what modules should get included from lib64 instead of lib and hardcode them .... Maybe tinkering around with the makefile which is produced by freeze is a better way to go?
But since I need to freeze this python code by today, can you suggest any other tool to do this?
On Tue, Nov 3, 2009 at 11:47 AM, M.-A. Lemburg <m...@egenix.com> wrote: > Girish Venkatasubramanian wrote: >> Hey Marc-Andre, >> Ummm - I have installed python-devel.x86_64 and checked that the >> /usr/lib64/python2.4/ is populated - anything else I can/shuld do to >> check/ensure the the devel rpm is installed?
> If you have the config/ sub-dir in there, things should be > fine.
> However, it's possible that you need to tweek the freeze.py > script a little, since RedHat chose to split the Python > installation on x64 in two parts and they may have missed > patching freeze.py as well:
> The platform independent parts are in /usr/lib, whereas the > platform dependent parts are in /usr/lib64.
> Python normally doesn't support this. It only has a > prefix and an exec_prefix, but those only allow to do > things like prefix=/usr and exec_prefix=/usr64, not > changing the lib/ part in /usr/lib/.
>> Thanks.
>> On Tue, Nov 3, 2009 at 11:32 AM, M.-A. Lemburg <m...@egenix.com> wrote: >>> Rami Chowdhury wrote: >>>> On Tue, 03 Nov 2009 11:15:39 -0800, Girish Venkatasubramanian >>>> <giris...@gmail.com> wrote:
>>>>> Hello, >>>>> I have been using freeze.py on 32 bit linux distributions without a >>>>> problem. But recently I tried to do the same on RHEL5 x86_64 and ran >>>>> into some issues.
>>>>> 1) When I ran the script, I got >>>>> Error: needed directory /usr/lib/python2.4/config not found
>>>> I don't know anything about freeze.py but on 64-bit Red Hat distros >>>> (RHEL, Fedora, etc) it should be /usr/lib64/python2.4/config :-)
>>> This sounds a lot like a missing Python devel RPM.
>>> -- >>> Marc-Andre Lemburg >>> eGenix.com
>>> Professional Python Services directly from the Source (#1, Nov 03 2009) >>>>>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>>>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>>>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >>> ________________________________________________________________________
>>> ::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
>>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >>> Registered at Amtsgericht Duesseldorf: HRB 46611 >>> http://www.egenix.com/company/contact/
> -- > Marc-Andre Lemburg > eGenix.com
> Professional Python Services directly from the Source (#1, Nov 03 2009) >>>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________
> ::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/
<giris...@gmail.com> wrote: > I checked and ls /usr/lib64/python2.4/config/ returns > config.c config.c.in install-sh libpython2.4.a Makefile makesetup > python.o Setup Setup.config Setup.local
> so I am guessing the python-devel installation went off OK, from what > you say.
> I looked at the freeze.py code and I see your point. But for tweaking, > I would need to know what modules should get included from lib64 > instead of lib and hardcode them .... > Maybe tinkering around with the makefile which is produced by freeze > is a better way to go?
> But since I need to freeze this python code by today, can you suggest > any other tool to do this?
I believe /usr/lib64 on a 64-bit RHEL will contain everything you need ; can you try just changing the directory freeze.py looks at to 'lib64', and see if the freeze works?
> On Tue, Nov 3, 2009 at 11:47 AM, M.-A. Lemburg <m...@egenix.com> wrote: >> Girish Venkatasubramanian wrote: >>> Hey Marc-Andre, >>> Ummm - I have installed python-devel.x86_64 and checked that the >>> /usr/lib64/python2.4/ is populated - anything else I can/shuld do to >>> check/ensure the the devel rpm is installed?
>> If you have the config/ sub-dir in there, things should be >> fine.
>> However, it's possible that you need to tweek the freeze.py >> script a little, since RedHat chose to split the Python >> installation on x64 in two parts and they may have missed >> patching freeze.py as well:
>> The platform independent parts are in /usr/lib, whereas the >> platform dependent parts are in /usr/lib64.
>> Python normally doesn't support this. It only has a >> prefix and an exec_prefix, but those only allow to do >> things like prefix=/usr and exec_prefix=/usr64, not >> changing the lib/ part in /usr/lib/.
>>> Thanks.
>>> On Tue, Nov 3, 2009 at 11:32 AM, M.-A. Lemburg <m...@egenix.com> wrote: >>>> Rami Chowdhury wrote: >>>>> On Tue, 03 Nov 2009 11:15:39 -0800, Girish Venkatasubramanian >>>>> <giris...@gmail.com> wrote:
>>>>>> Hello, >>>>>> I have been using freeze.py on 32 bit linux distributions without a >>>>>> problem. But recently I tried to do the same on RHEL5 x86_64 and ran >>>>>> into some issues.
>>>>>> 1) When I ran the script, I got >>>>>> Error: needed directory /usr/lib/python2.4/config not found
>>>>> I don't know anything about freeze.py but on 64-bit Red Hat distros >>>>> (RHEL, Fedora, etc) it should be /usr/lib64/python2.4/config :-)
>>>> This sounds a lot like a missing Python devel RPM.
>>>> -- >>>> Marc-Andre Lemburg >>>> eGenix.com
>>>> Professional Python Services directly from the Source (#1, Nov 03 >>>> 2009) >>>>>>> Python/Zope Consulting and Support ... >>>>>>> http://www.egenix.com/ >>>>>>> mxODBC.Zope.Database.Adapter ... >>>>>>> http://zope.egenix.com/ >>>>>>> mxODBC, mxDateTime, mxTextTools ... >>>>>>> http://python.egenix.com/ >>>> ________________________________________________________________________
>>>> ::: Try our new mxODBC.Connect Python Database Interface for free ! >>>> ::::
>>>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >>>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >>>> Registered at Amtsgericht Duesseldorf: HRB 46611 >>>> http://www.egenix.com/company/contact/
>> -- >> Marc-Andre Lemburg >> eGenix.com
>> Professional Python Services directly from the Source (#1, Nov 03 2009) >>>>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >> ________________________________________________________________________
>> ::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> http://www.egenix.com/company/contact/
-- Rami Chowdhury "Never attribute to malice that which can be attributed to stupidity" -- Hanlon's Razor 408-597-7068 (US) / 07875-841-046 (UK) / 0189-245544 (BD)
<rami.chowdh...@gmail.com> wrote: > On Tue, 03 Nov 2009 11:57:17 -0800, Girish Venkatasubramanian > <giris...@gmail.com> wrote:
>> I checked and ls /usr/lib64/python2.4/config/ returns >> config.c config.c.in install-sh libpython2.4.a Makefile makesetup >> python.o Setup Setup.config Setup.local
>> so I am guessing the python-devel installation went off OK, from what you >> say.
>> I looked at the freeze.py code and I see your point. But for tweaking, >> I would need to know what modules should get included from lib64 >> instead of lib and hardcode them .... >> Maybe tinkering around with the makefile which is produced by freeze >> is a better way to go?
>> But since I need to freeze this python code by today, can you suggest >> any other tool to do this?
> I believe /usr/lib64 on a 64-bit RHEL will contain everything you need ; can > you try just changing the directory freeze.py looks at to 'lib64', and see > if the freeze works?
>> On Tue, Nov 3, 2009 at 11:47 AM, M.-A. Lemburg <m...@egenix.com> wrote:
>>> Girish Venkatasubramanian wrote:
>>>> Hey Marc-Andre, >>>> Ummm - I have installed python-devel.x86_64 and checked that the >>>> /usr/lib64/python2.4/ is populated - anything else I can/shuld do to >>>> check/ensure the the devel rpm is installed?
>>> If you have the config/ sub-dir in there, things should be >>> fine.
>>> However, it's possible that you need to tweek the freeze.py >>> script a little, since RedHat chose to split the Python >>> installation on x64 in two parts and they may have missed >>> patching freeze.py as well:
>>> The platform independent parts are in /usr/lib, whereas the >>> platform dependent parts are in /usr/lib64.
>>> Python normally doesn't support this. It only has a >>> prefix and an exec_prefix, but those only allow to do >>> things like prefix=/usr and exec_prefix=/usr64, not >>> changing the lib/ part in /usr/lib/.
>>>> Thanks.
>>>> On Tue, Nov 3, 2009 at 11:32 AM, M.-A. Lemburg <m...@egenix.com> wrote:
>>>>> Rami Chowdhury wrote:
>>>>>> On Tue, 03 Nov 2009 11:15:39 -0800, Girish Venkatasubramanian >>>>>> <giris...@gmail.com> wrote:
>>>>>>> Hello, >>>>>>> I have been using freeze.py on 32 bit linux distributions without a >>>>>>> problem. But recently I tried to do the same on RHEL5 x86_64 and ran >>>>>>> into some issues.
>>>>>>> 1) When I ran the script, I got >>>>>>> Error: needed directory /usr/lib/python2.4/config not found
>>>>>> I don't know anything about freeze.py but on 64-bit Red Hat distros >>>>>> (RHEL, Fedora, etc) it should be /usr/lib64/python2.4/config :-)
>>>>> This sounds a lot like a missing Python devel RPM.
>>> ::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
>>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >>> Registered at Amtsgericht Duesseldorf: HRB 46611 >>> http://www.egenix.com/company/contact/
> -- > Rami Chowdhury > "Never attribute to malice that which can be attributed to stupidity" -- > Hanlon's Razor > 408-597-7068 (US) / 07875-841-046 (UK) / 0189-245544 (BD)
Having an issue with elementtree XML() in python 2.6.4.
This code works fine:
from xml.etree import ElementTree as et getResponse = u'''<?xml version="1.0" encoding="UTF-8"?> <customer><shipping><state>bobble</state><city>head</ city><street>city</street></shipping></customer>''' theResponseXml = et.XML(getResponse)
This code errors out when it tries to do the et.XML()
from xml.etree import ElementTree as et getResponse = u'''<?xml version="1.0" encoding="UTF-8"?> <customer><shipping><state>\ue58d83\ue89189\ue79c8C</state><city> \ue69f8f\ue5b882</city><street>\ue9ab98\ue58d97\ue58fb03</street></ shipping></customer>''' theResponseXml = et.XML(getResponse)
In my real code, I'm pulling the getResponse data from a web page that returns as XML and when I display it in the browser you can see the Japanese characters in the data. I've removed all the stuff in my code and tried to distill it down to just what is failing. Hopefully I have not removed something essential.
Why is this not working and what do I need to do to use Elementtree with unicode?
> Having an issue with elementtree XML() in python 2.6.4.
> This code works fine:
> from xml.etree import ElementTree as et > getResponse = u'''<?xml version="1.0" encoding="UTF-8"?> > <customer><shipping><state>bobble</state><city>head</ > city><street>city</street></shipping></customer>''' > theResponseXml = et.XML(getResponse)
> This code errors out when it tries to do the et.XML()
> from xml.etree import ElementTree as et > getResponse = u'''<?xml version="1.0" encoding="UTF-8"?> > <customer><shipping><state>\ue58d83\ue89189\ue79c8C</state><city> > \ue69f8f\ue5b882</city><street>\ue9ab98\ue58d97\ue58fb03</street></ > shipping></customer>''' > theResponseXml = et.XML(getResponse)
> In my real code, I'm pulling the getResponse data from a web page that > returns as XML and when I display it in the browser you can see the > Japanese characters in the data. I've removed all the stuff in my code > and tried to distill it down to just what is failing. Hopefully I have > not removed something essential.
> Why is this not working and what do I need to do to use Elementtree with > unicode?
et expects bytes as input, not unicode. You're decoding too early (decoding early is good, but not in this case, because et does the work for you). Either feed et.XML with the bytes before decoding, or reencode the received xml text in UTF-8 (since this is the declared encoding).
On Nov 3, 2009, at 4:44 PM, Gabriel Genellina wrote:
> En Tue, 03 Nov 2009 21:01:46 -0300, Kee Nethery <k...@kagi.com> > escribió:
>> I've removed all the stuff in my code and tried to distill it down >> to just what is failing. Hopefully I have not removed something >> essential.
Sounds like I did remove something essential.
> et expects bytes as input, not unicode. You're decoding too early > (decoding early is good, but not in this case, because et does the > work for you). Either feed et.XML with the bytes before decoding, or > reencode the received xml text in UTF-8 (since this is the declared > encoding).
Here is the code that hits the URL: getResponse1 = urllib2.urlopen(theUrl) getResponse2 = getResponse1.read() getResponse3 = unicode(getResponse2,'UTF-8') theResponseXml = et.XML(getResponse3)
So are you saying I want to do: getResponse1 = urllib2.urlopen(theUrl) getResponse4 = getResponse1.read() theResponseXml = et.XML(getResponse4)
The reason I am confused is that getResponse2 is classified as an "str" in the Komodo IDE. I want to make sure I don't lose the non- ASCII characters coming from the URL. If I do the second set of code, does elementtree auto convert the str into unicode? How do I deal with the XML as unicode when I put it into elementtree as a string?
> Having an issue with elementtree XML() in python 2.6.4.
> This code works fine:
> from xml.etree import ElementTree as et > getResponse = u'''<?xml version="1.0" encoding="UTF-8"?> > <customer><shipping><state>bobble</state><city>head</ > city><street>city</street></shipping></customer>''' > theResponseXml = et.XML(getResponse)
> This code errors out when it tries to do the et.XML()
> from xml.etree import ElementTree as et > getResponse = u'''<?xml version="1.0" encoding="UTF-8"?> > <customer><shipping><state>\ue58d83\ue89189\ue79c8C</state><city> > \ue69f8f\ue5b882</city><street>\ue9ab98\ue58d97\ue58fb03</street></ > shipping></customer>''' > theResponseXml = et.XML(getResponse)
> In my real code, I'm pulling the getResponse data from a web page that > returns as XML and when I display it in the browser you can see the > Japanese characters in the data. I've removed all the stuff in my code > and tried to distill it down to just what is failing. Hopefully I have > not removed something essential.
> Why is this not working and what do I need to do to use Elementtree > with unicode?
On Nov 4, 11:01 am, Kee Nethery <k...@kagi.com> wrote:
> Having an issue with elementtree XML() in python 2.6.4.
> This code works fine:
> from xml.etree import ElementTree as et > getResponse = u'''<?xml version="1.0" encoding="UTF-8"?> > <customer><shipping><state>bobble</state><city>head</ > city><street>city</street></shipping></customer>''' > theResponseXml = et.XML(getResponse)
> This code errors out when it tries to do the et.XML()
> from xml.etree import ElementTree as et > getResponse = u'''<?xml version="1.0" encoding="UTF-8"?> > <customer><shipping><state>\ue58d83\ue89189\ue79c8C</state><city> > \ue69f8f\ue5b882</city><street>\ue9ab98\ue58d97\ue58fb03</street></ > shipping></customer>''' > theResponseXml = et.XML(getResponse)
> In my real code, I'm pulling the getResponse data from a web page that > returns as XML and when I display it in the browser you can see the > Japanese characters in the data. I've removed all the stuff in my code > and tried to distill it down to just what is failing. Hopefully I have > not removed something essential.
> Why is this not working and what do I need to do to use Elementtree > with unicode?
What you need to do is NOT feed it unicode. You feed it a str object and it gets decoded according to the encoding declaration found in the first line. So take the str object that you get from the web (should be UTF8-encoded already unless the header is lying), and throw that at ET ... like this:
| Python 2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit (Intel)] on win32 | Type "help", "copyright", "credits" or "license" for more information. | >>> from xml.etree import ElementTree as et | >>> ucode = u'''<?xml version="1.0" encoding="UTF-8"?> | ... <customer><shipping> | ... <state>\ue58d83\ue89189\ue79c8C</state> | ... <city>\ue69f8f\ue5b882</city> | ... <street>\ue9ab98\ue58d97\ue58fb03</street> | ... </shipping></customer>''' | >>> xml= et.XML(ucode) | Traceback (most recent call last): | File "<stdin>", line 1, in <module> | File "C:\python26\lib\xml\etree\ElementTree.py", line 963, in XML | parser.feed(text) | File "C:\python26\lib\xml\etree\ElementTree.py", line 1245, in feed | self._parser.Parse(data, 0) | UnicodeEncodeError: 'ascii' codec can't encode character u'\ue58d' in position 69: ordinal not in range(128) | # as expected | >>> strg = ucode.encode('utf8') | # encoding as utf8 is for DEMO purposes. | # i.e. use the original web str object, don't convert it to unicode | # and back to utf8. | >>> xml2 = et.XML(strg) | >>> xml2.tag | 'customer' | >>> for c in xml2.getchildren(): | ... print c.tag, repr(c.text) | ... | shipping '\n' | >>> for c in xml2[0].getchildren(): | ... print c.tag, repr(c.text) | ... | state u'\ue58d83\ue89189\ue79c8C' | city u'\ue69f8f\ue5b882' | street u'\ue9ab98\ue58d97\ue58fb03' | >>>
By the way: (1) it usually helps to be more explicit than "errors out", preferably the exact copied/pasted output as shown above; this is one of the rare cases where the error message is predictable (2) PLEASE don't start a new topic in a reply in somebody else's thread.
On Nov 4, 12:14 pm, Kee Nethery <k...@kagi.com> wrote:
> On Nov 3, 2009, at 4:44 PM, Gabriel Genellina wrote:
> > En Tue, 03 Nov 2009 21:01:46 -0300, Kee Nethery <k...@kagi.com> > > escribió:
> >> I've removed all the stuff in my code and tried to distill it down > >> to just what is failing. Hopefully I have not removed something > >> essential.
> Sounds like I did remove something essential.
No, you added something that was not only inessential but caused trouble.
> > et expects bytes as input, not unicode. You're decoding too early > > (decoding early is good, but not in this case, because et does the > > work for you). Either feed et.XML with the bytes before decoding, or > > reencode the received xml text in UTF-8 (since this is the declared > > encoding).
> Here is the code that hits the URL: > getResponse1 = urllib2.urlopen(theUrl) > getResponse2 = getResponse1.read() > getResponse3 = unicode(getResponse2,'UTF-8') > theResponseXml = et.XML(getResponse3)
> So are you saying I want to do: > getResponse1 = urllib2.urlopen(theUrl) > getResponse4 = getResponse1.read() > theResponseXml = et.XML(getResponse4)
You got the essence. Note: that in no way implies any approval of your naming convention :-)
> The reason I am confused is that getResponse2 is classified as an > "str" in the Komodo IDE. I want to make sure I don't lose the non- > ASCII characters coming from the URL.
str is all about 8-bit bytes. Your data comes from the web in 8-bit bytes. No problem. Just don't palpate it unnecessarily.
> If I do the second set of code, > does elementtree auto convert the str into unicode?
Yes. See the example I gave in my earlier posting:
| ... print c.tag, repr(c.text) | state u'\ue58d83\ue89189\ue79c8C'
That first u means the type is unicode.
> How do I deal with > the XML as unicode when I put it into elementtree as a string?
That's unfortunately rather ambiguous: (1) put past/present? (2) string unicode/str? (3) what is referent of "it"?
All text in what et returns is unicode [*] so you read it out as unicode (see above example) or written as unicode if you want to change it:
your_element.text = u'a unicode object'
[*] As an "optimisation", et stores strings as str objects if they contain only ASCII bytes (and are thus losslessly convertible to unicode). In preparation for running your code under Python 3.X, it's best to ignore this and use unicode constants u'foo' (if you need text constants at all) even if et would let you get away with 'foo'.
> On Nov 4, 11:01 am, Kee Nethery <k...@kagi.com> wrote: >> Having an issue with elementtree XML() in python 2.6.4.
>> This code works fine:
>> from xml.etree import ElementTree as et >> getResponse = u'''<?xml version="1.0" encoding="UTF-8"?> >> <customer><shipping><state>bobble</state><city>head</ >> city><street>city</street></shipping></customer>''' >> theResponseXml = et.XML(getResponse)
>> This code errors out when it tries to do the et.XML()
>> from xml.etree import ElementTree as et >> getResponse = u'''<?xml version="1.0" encoding="UTF-8"?> >> <customer><shipping><state>\ue58d83\ue89189\ue79c8C</state><city> >> \ue69f8f\ue5b882</city><street>\ue9ab98\ue58d97\ue58fb03</street></ >> shipping></customer>''' >> theResponseXml = et.XML(getResponse)
>> In my real code, I'm pulling the getResponse data from a web page >> that >> returns as XML and when I display it in the browser you can see the >> Japanese characters in the data. I've removed all the stuff in my >> code >> and tried to distill it down to just what is failing. Hopefully I >> have >> not removed something essential.
>> Why is this not working and what do I need to do to use Elementtree >> with unicode?
> On Nov 4, 11:01 am, Kee Nethery <k...@kagi.com> wrote: >> Having an issue with elementtree XML() in python 2.6.4.
>> This code works fine:
>> from xml.etree import ElementTree as et >> getResponse = u'''<?xml version="1.0" encoding="UTF-8"?> >> <customer><shipping><state>bobble</state><city>head</ >> city><street>city</street></shipping></customer>''' >> theResponseXml = et.XML(getResponse)
>> This code errors out when it tries to do the et.XML()
>> from xml.etree import ElementTree as et >> getResponse = u'''<?xml version="1.0" encoding="UTF-8"?> >> <customer><shipping><state>\ue58d83\ue89189\ue79c8C</state><city> >> \ue69f8f\ue5b882</city><street>\ue9ab98\ue58d97\ue58fb03</street></ >> shipping></customer>''' >> theResponseXml = et.XML(getResponse)
>> In my real code, I'm pulling the getResponse data from a web page >> that >> returns as XML and when I display it in the browser you can see the >> Japanese characters in the data. I've removed all the stuff in my >> code >> and tried to distill it down to just what is failing. Hopefully I >> have >> not removed something essential.
>> Why is this not working and what do I need to do to use Elementtree >> with unicode?
> What you need to do is NOT feed it unicode. You feed it a str object > and it gets decoded according to the encoding declaration found in the > first line.
That it uses "the encoding declaration found in the first line" is the nugget of data that is not in the documentation that has stymied me for days. Thank you!
The other thing that has been confusing is that I've been using "dump" to view what is in the elementtree instance and the non-ASCII characters have been displayed as "numbered entities" (<city>柏市</city>) and I know that is not the representation I want the data to be in. A co-worker suggested that instead of "dump" that I use "et.tostring(theResponseXml, encoding='utf-8')" and then print that to see the characters. That process causes the non-ASCII characters to display as the glyphs I know them to be.
If there was a place in the official docs for me to append these nuggets of information to the sections for "xml.etree.ElementTree.XML(text)" and "xml.etree.ElementTree.dump(elem)" I would absolutely do so.
> So take the str object that you get from the web (should > be UTF8-encoded already unless the header is lying), and throw that at > ET ... like this:
> | Python 2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit > (Intel)] on win32 > | Type "help", "copyright", "credits" or "license" for more > information. > | >>> from xml.etree import ElementTree as et > | >>> ucode = u'''<?xml version="1.0" encoding="UTF-8"?> > | ... <customer><shipping> > | ... <state>\ue58d83\ue89189\ue79c8C</state> > | ... <city>\ue69f8f\ue5b882</city> > | ... <street>\ue9ab98\ue58d97\ue58fb03</street> > | ... </shipping></customer>''' > | >>> xml= et.XML(ucode) > | Traceback (most recent call last): > | File "<stdin>", line 1, in <module> > | File "C:\python26\lib\xml\etree\ElementTree.py", line 963, in XML > | parser.feed(text) > | File "C:\python26\lib\xml\etree\ElementTree.py", line 1245, in > feed > | self._parser.Parse(data, 0) > | UnicodeEncodeError: 'ascii' codec can't encode character u'\ue58d' > in position 69: ordinal not in range(128) > | # as expected > | >>> strg = ucode.encode('utf8') > | # encoding as utf8 is for DEMO purposes. > | # i.e. use the original web str object, don't convert it to unicode > | # and back to utf8. > | >>> xml2 = et.XML(strg) > | >>> xml2.tag > | 'customer' > | >>> for c in xml2.getchildren(): > | ... print c.tag, repr(c.text) > | ... > | shipping '\n' > | >>> for c in xml2[0].getchildren(): > | ... print c.tag, repr(c.text) > | ... > | state u'\ue58d83\ue89189\ue79c8C' > | city u'\ue69f8f\ue5b882' > | street u'\ue9ab98\ue58d97\ue58fb03' > | >>>
> By the way: (1) it usually helps to be more explicit than "errors > out", preferably the exact copied/pasted output as shown above; this > is one of the rare cases where the error message is predictable (2) > PLEASE don't start a new topic in a reply in somebody else's thread.
On Nov 4, 1:06 pm, Kee Nethery <k...@kagi.com> wrote:
> On Nov 3, 2009, at 5:27 PM, John Machin wrote:
> > On Nov 4, 11:01 am, Kee Nethery <k...@kagi.com> wrote: > >> Why is this not working and what do I need to do to use Elementtree > >> with unicode?
> > What you need to do is NOT feed it unicode. You feed it a str object > > and it gets decoded according to the encoding declaration found in the > > first line.
> That it uses "the encoding declaration found in the first line" is the > nugget of data that is not in the documentation that has stymied me > for days. Thank you!
And under the "don't repeat" principle, it shouldn't be in the Elementtree docs; it's nothing special about ET -- it's part of the definition of an XML document (which for universal loss-free transportability naturally must be encoded somehow, and the document must state what its own encoding is (if it's not the default (UTF-8))).
> The other thing that has been confusing is that I've been using "dump" > to view what is in the elementtree instance and the non-ASCII > characters have been displayed as "numbered > entities" (<city>柏市</city>) and I know that is not the > representation I want the data to be in. A co-worker suggested that > instead of "dump" that I use "et.tostring(theResponseXml, > encoding='utf-8')" and then print that to see the characters. That > process causes the non-ASCII characters to display as the glyphs I > know them to be.
> If there was a place in the official docs for me to append these > nuggets of information to the sections for > "xml.etree.ElementTree.XML(text)" and > "xml.etree.ElementTree.dump(elem)" I would absolutely do so.
I don't understand ... tostring() is in the same section as dump(), about two screen-heights away. You want to include the tostring() docs in the dump() docs? The usual idea is not to get bogged down in the first function that looks at first glance like it might do what you want ("look at the glyphs") but doesn't (it writes a (transportable) XML stream) but press on to the next plausible candidate.
En Tue, 03 Nov 2009 23:06:58 -0300, Kee Nethery <k...@kagi.com> escribió:
> If there was a place in the official docs for me to append these nuggets > of information to the sections for "xml.etree.ElementTree.XML(text)" and > "xml.etree.ElementTree.dump(elem)" I would absolutely do so.
On Nov 3, 2009, at 7:06 PM, Gabriel Genellina wrote:
> En Tue, 03 Nov 2009 23:06:58 -0300, Kee Nethery <k...@kagi.com> > escribió:
>> If there was a place in the official docs for me to append these >> nuggets of information to the sections for >> "xml.etree.ElementTree.XML(text)" and >> "xml.etree.ElementTree.dump(elem)" I would absolutely do so.
I've submitted documentation bugs in the past and no action was taken on them, the bugs were closed. I'm guessing that information "that everyone knows" not being in the documentation is not a bug. It's my fault I'm a newbie and I accept that. Thanks to you two for helping me get past this block.
> On Nov 4, 12:14 pm, Kee Nethery wrote: >> The reason I am confused is that getResponse2 is classified as an >> "str" in the Komodo IDE. I want to make sure I don't lose the non- >> ASCII characters coming from the URL.
> str is all about 8-bit bytes.
True in Py2.x, false in Py3.
What you mean is the "bytes" type, which, sadly, was named "str" in Python 2.x.
The problem the OP ran into was due to the fact that Python 2.x handled "ASCII characters in a unicode string" <-> "ASCII encoded byte string" conversion behind the scenes, which lead to all sorts of trouble to loads of people, and was finally discarded in Python 3.0.
On Nov 5, 12:35 am, Stefan Behnel <stefan...@behnel.de> wrote:
> John Machin, 04.11.2009 02:56:
> > On Nov 4, 12:14 pm, Kee Nethery wrote: > >> The reason I am confused is that getResponse2 is classified as an > >> "str" in the Komodo IDE. I want to make sure I don't lose the non- > >> ASCII characters coming from the URL.
> > str is all about 8-bit bytes.
> True in Py2.x, false in Py3.
And the context was 2.x.
> What you mean is the "bytes" type, which, sadly, was named "str" in Python 2.x.
What you mean is the "bytes" concept.
> The problem the OP ran into was due to the fact that Python 2.x handled > "ASCII characters in a unicode string" <-> "ASCII encoded byte string" > conversion behind the scenes, which lead to all sorts of trouble to loads > of people, and was finally discarded in Python 3.0.
What you describe is the symptom. The problems are (1) 2.X ET expects a str object but the OP supplied a unicode object, and (2) 2.X ET didn't check that, so it accidentally "worked" provided the contents were ASCII-only, and otherwise gave a novice-mystifying error message.
In article <c3604abd-677a-428a-94fa-c0e3f5a22...@a21g2000yqc.googlegroups.com>, Mike Driscoll <kyoso...@gmail.com> wrote:
>Something that you might want to try in the future is GUI2Exe, which >allows you to play with a whole slew of freezing modules:
Does GUI2Exe work from just the command-line? I spent a fair amount of time getting rid of the Mac GUI .pkg creator and I sure don't want to introduce more GUI into our ant build process. -- Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/
[on old computer technologies and programmers] "Fancy tail fins on a brand new '59 Cadillac didn't mean throwing out a whole generation of mechanics who started with model As." --Andrew Dalke
|
