British Library develops web crawling system for p

Anon E. Mouse writes Pocket-lint.co.uk: The internet is an ever-changing repository of information, where pages disappear as quickly as new ones appear. The British Library is part of a group of institutions working to preserve digital data presented on the internet for future generations and to make it available for research.

The British Library co-funded with The National Library of New Zealand the development of a web harvesting management system called the web curator tool. It uses software to crawl through a specified section on the web and gather snapshots of various sites.”