> The sitemap as such isn't important except to the extent that it
> should not contain outdated information about pages removed or renamed
> since last produced, in particular if your server is not producing a
> 404 or a 301 for them - as that would keep them in the index.
> For moved or renamed urls you should implement 301 redirections from
> old to new url. For deleted yrls they should produce a 410 or 404
> server response.
> If Xenu finds over 13000 urls yet you say your site contains just
> abtou 1000 pages there's something wrong, unlesss all the extras Xenu
> found are supposed to be blocked in robots.txt. Xenu does not read
> robots.txt - you have to manually input any such crawling restrictions
> to it.
> On Nov 27, 8:53 am, Matty2 wrote:
> > We've quite a lagre site with way over 1000 pages - I did our sitemap
> > back in August and I think it needs updating. Is there an easy way to
> > do this without trawling throught the files to see which are there and
> > which aren't? I've recently discovered Xenu which came back with
> > 13,274 URL's surely they don't all need mapping?
> > Finally, I used a sitemap generator to do the first map but looking
> > through it, there are definately some pages missing. If I completely
> > resubmit the map do I run the risk of those pages that are currently
> > indexed being indexed (around 90% of the original map) being removed.
> > I.e am I best just updating what I have or starting again?- Hide quoted text -
> - Show quoted text -
|
