deep-scrape
Scrape and crawl pages with io.js and get a whole lot of meta data. Shows; headers, Ajax requests/responses, rendered html, Javascript AST's, dependencies, console events, and a whole lot more. Crawl sites, or scrape a single page. Add cookies or proxy re
Last updated 3 years ago by zacharyiles .
MIT · Repository · Bugs · Original npm · Tarball · package.json
$ cnpm install deep-scrape 
SYNC missed versions from official npm registry.

Deep Scrape

npm version

Scrape and crawl pages with io.js and get a whole lot of meta data. Shows; headers, Ajax requests/responses, rendered html, Javascript AST's, dependencies, console events, and a whole lot more. Crawl sites, or scrape a single page. Add cookies or proxy requests. Fingerprints common javascript libraries, and allows you to write your own.

Installation

This was tested on node 0.12.x. It can be run as a module export, or a command line script.

npm install deep-scrape
// or clone the repository and run it as a script.

Use Case

  • You are scraping websites with lots of javascript (Angular, Ember, Browserfy).
  • You don't mind trading a bit of performance for more detailed scraping data.
  • You would like to find potential DOM sinks and sources on your pages (Possibly for vulnerability scanning).
  • You need the most detailed metadata, metrics, and analyitics on your scraped pages.
  • You would like to fingerprint possible technologies a certain site or page uses.

Current Tags

  • 0.0.5                                ...           latest (3 years ago)

5 Versions

  • 0.0.5                                ...           3 years ago
  • 0.0.4                                ...           3 years ago
  • 0.0.3                                ...           3 years ago
  • 0.0.2                                ...           4 years ago
  • 0.0.1                                ...           4 years ago
Maintainers (1)
Downloads
Today 5
This Week 5
This Month 10
Last Day 0
Last Week 5
Last Month 0
Dependencies (16)
Dev Dependencies (2)
Dependents (1)

Copyright 2014 - 2016 © taobao.org |