This is a CLI tool for recursively crawling websites.
Clone this repository and follow these simple steps:
First, create an empty database (the crawler will create the tables automatically):
echo "CREATE DATABASE your_db_name" |mysql
Then modify lib/db/connect.js to suit your MySQL setup (user/password and database name)
Now the obligatory
Now run the crawler with:
npm run demo
You can hit Ctrl+C to stop crawling and wait about 2 seconds for the script to finish the exit routines.
npm run demo once more will resume the crawling.
The script will auto-restart itself every 100 URLs to work around a memory leak in cheerio
See lib/constants.js for more settings regarding crawl delay, in-memory LRU cache size, user agent and others
Please don't abuse the demo configuration, write your own (for example
./config/my_config.js) in and run it with
node runner.js --preset my_config