We have optimized our web crawling system to improve efficiency and indexing speed. Web crawlers that use sitemap triggers will now crawl URLs added or modified since the last successful run, based on the sitemap's lastmod date.
This improvement provides the following benefits:
Quick updates - receive up-to-date search results and easily identify content changes.
Faster indexing - the crawler only includes URLs modified after the previous successful run, reducing the number of indexed items and speeding up the indexing process.
Manual control - you can now trigger a full crawl from the Sitecore Search interface whenever you want.
Predictable full crawls - URLs without a
lastmoddate are still crawled during periodic full crawls, ensuring no updates are missed.