python sitemap_gen.py MemoryError - Sitemap Protocol

Message from discussion python sitemap_gen.py MemoryError

cristina

View profile

More options Jul 3, 8:31 am

From: cristina

Date: Wed, 2 Jul 2008 15:31:17 -0700 (PDT)

Local: Thurs, Jul 3 2008 8:31 am

Subject: Re: python sitemap_gen.py MemoryError

Forward | Print | View thread | Report this message | Find messages by this author

Can you run the sitemap generator more than once
for different config.xml files with different
settings for the <directory> node,
just to break the sitemaps for different sub-folders,
to check if indeed the problem is memory leak
because of the large number of URLs,
and not some problem because of file system walking.

For example first time run the sitemap generator
for the directory where you got the error
to check that this directory can be walked OK

change default_file to index.shtml
if the default home page is index.shtml
After that run the sitemap generator for other
non-overlapping directories,
you can use if you want the <sitemap>
nodes as well to aggregate sitemaps
(you can use <sitemap> nodes in version 1.4,
I am not sure if you can use them in version 1.5)

It is not great, just to check that the
problems are indeed because of memory leaks
caused by the large number of URLs.

Cristina.

On Jul 2, 9:39 pm, BadXAsh wrote:

> Man I was very confident in that working. I Changed the verbose
> attribute of the site nod in config.xml to 3, and it does say
> something about the directory being walked at the very beginning of
> the process, but then around sitemap 54 I received this message:
> ---
> URL: loc=[http://www.diyautoparts.com/search/parts/1985/dodge/aries/
> air-check-valve.shtml] lastmod=[2008-01-17T16:25:38Z] changefreq=[]
> priority=[]
> URL: loc=[http://www.diyautoparts.com/search/parts/1985/dodge/aries/
> air-conditioning-accumulator.shtml] lastmod=[2008-01-17T16:25:38Z]
> changefreq=[] priority=[]
> Traceback (most recent call last):
> File "sitemap_gen.py", line 2206, in ?
> sitemap.Generate()
> File "sitemap_gen.py", line 1778, in Generate
> input.ProduceURLs(self.ConsumeURL)
> File "sitemap_gen.py", line 979, in ProduceURLs
> os.path.walk(self._path, PerDirectory, None)
> File "/usr/lib/python2.4/posixpath.py", line 298, in walk
> walk(name, func, arg)
> File "/usr/lib/python2.4/posixpath.py", line 298, in walk
> walk(name, func, arg)
> File "/usr/lib/python2.4/posixpath.py", line 298, in walk
> walk(name, func, arg)
> File "/usr/lib/python2.4/posixpath.py", line 298, in walk
> walk(name, func, arg)
> File "/usr/lib/python2.4/posixpath.py", line 298, in walk
> walk(name, func, arg)
> File "/usr/lib/python2.4/posixpath.py", line 290, in walk
> func(arg, top, names)
> File "sitemap_gen.py", line 974, in PerDirectory
> PerFile(dirpath, name)
> File "sitemap_gen.py", line 959, in PerFile
> consumer(url, False)
> File "sitemap_gen.py", line 1839, in ConsumeURL
> self._urls[hash] = 1
> MemoryError
> ---

> I am using the directory node this is what i have:

> ---
> <directory
> path="(My Site Path)"
> url="http://www.diyautoparts.com/"
> default_file="index.html"
> />
> ---

> (My Site Path) Is my long path that i actually have typed in but I'm
> so paranoid i took it out just in case haha. Anyways, I do believe my
> Python is up to date as my web hosting company takes care of that
> server side software.

> And yes those pages are all different one for each type of part we
> carry, which is a lot, so they are not the same page over and over, if
> that's what you mean.

> Thanks for your help so far!

Forward