python sitemap_gen.py MemoryError - Sitemap Protocol

Message from discussion python sitemap_gen.py MemoryError

cristina

View profile

More options Jul 10, 5:03 am

From: cristina

Date: Wed, 9 Jul 2008 12:03:22 -0700 (PDT)

Local: Thurs, Jul 10 2008 5:03 am

Subject: Re: python sitemap_gen.py MemoryError

Forward | Print | View thread | Report this message | Find messages by this author

You can filter out directories from the sitemap
with the filter nodes.
The filter nodes apply to URLs, so to filter out
the /parts/ directory you can add to your
config.xml file something like

If you want, for example, to include only the
/parts/ sub-directories from the 1970s
and remove all the other /parts/ sub-folders,
try something like

or you could try the type="regexp" filters.

You can see from the comments in the example_config.xml file that
the filters are applied in the order they appear in the config.xml
file,
and a pass filter shortcuts any other later filter that match.

Cristina.

On Jul 8, 2:30 pm, BadXAsh wrote:

> Well i ran just the '85 dodge aries folder and it ran perfectly,
> finished in the blink of an eye. I also tried to run just the /parts/
> directory, which is the single largest directory on my site. A page
> for each and every part for every vehicle from 1965 thru 2007, so as
> you can imagine it's rather large. Just walking along that directory
> Sitemapping it alone failed in MemoryError as well. Though it went
> past the '85 dodge aries it crashed out in the '86 year, Something
> similar though is that it crashed when it reached mid page 55 of the
> Sitemaps.

> So I'm wondering can i set up a filter to break down the /parts/
> directory and Map it in sections, say like in 10 to 20 years
> increments, i.e. /parts/1965 - 1985 I'm not to clear on the FILTERS
> rules, it seems like i can't really specify directories i want
> filtered. And alternitively, can i leave out the /parts/ directory
> once i have that mapped so i can map the rest of the site and leave
> out that directory?

> Or should i just abandon all hope?? hehe Thank you for your help so
> far Cristina!

> On Jul 2, 6:31 pm, cristina wrote:

> > Can you run the sitemap generator more than once
> > for different config.xml files with different
> > settings for the <directory> node,
> > just to break the sitemaps for different sub-folders,
> > to check if indeed the problem is memory leak
> > because of the large number of URLs,
> > and not some problem because of file system walking.

> > For example first time run the sitemap generator
> > for the directory where you got the error
> > to check that this directory can be walked OK

> > <directory
> > path="/main_path/search/parts/1985/dodge/aries"
> > url="http://www.diyautoparts.com/search/parts/1985/dodge/aries/"
> > default_file="index.html"
> > />

> > change default_file to index.shtml
> > if the default home page is index.shtml
> > After that run the sitemap generator for other
> > non-overlapping directories,
> > you can use if you want the <sitemap>
> > nodes as well to aggregate sitemaps
> > (you can use <sitemap> nodes in version 1.4,
> > I am not sure if you can use them in version 1.5)

> > It is not great, just to check that the
> > problems are indeed because of memory leaks
> > caused by the large number of URLs.

> > Cristina.

Forward