Simple crawler for Single Page Applications
Last updated 5 years ago by arboleya .
Repository · Bugs · Original npm · Tarball · package.json
$ cnpm install snapshooter 
SYNC missed versions from official npm registry.


Simple crawler for Single Page Applications.

Dependency Status NPM version


Snapshooter will load a URL, wait the javascript to render and save it as plain HTML.

This will be done recursively until all found links in the start URL is saved.


Do not hesitate to open a feature request or a bug report.


## Requirements

You'll need PhantomJS installed (v 1.9 or greater) in order to use this package.

## Installing
npm install -g snapshooter
## Help
  snapshooter [options] [params]

  -i, --input            Input url to index                                                 
  -o, --output           Output folder to save indexed files                                
  -e, --exclude          Regex pattern for excluding files (pass between quotes)            
  -p, --pretty           Output indexed files in a pretty fashion way                       
  -s, --server           Start a server for previewing indexed content                      
  -P, --port             Preview server port                                                  [default: 8080]
  -f, --forward          Avoid indexing links up to the initial url folder                  
  -t, --timeout          Time limit (in seconds) to wait for a page to render                 [default: 15]
  -m, --max-connections  Max connections limit, use with care                                 [default: 10]
  -l, --log              Show 'console.log' messages (try disabling it if phantom crashes)  
  -L, --live             Creates a "live" tunnel, crawling will happen on demand            
  -O, --once             Avoid recursivity, index only the given url and nothing else       
  -S, --stdout           Prints indexed content to stdout (auto-set -O=true -l=false)       
  -V, --verbose          Shows info logs about files skipped                                
  -D, --delete           Automatically delete destination folder before writing new files   
  -X, --overwrite        Automatically overwrite destination folder with new files          
  -H, --hidden           Doesn't inject the `window.snapshooter=true` on pages being indexed
  -v, --version          Shows snapshooter version                                          
  -h, --help             Shows this help screen                                             

  snapshooter -i <> -o <local-folder>
  snapshooter -i <> -o <local-folder> -p
  snapshooter -i <> -o <local-folder> -ps [-P 3000] [-e '/\.exe$/m'] [-t 20000]
### Integration

A very tiny bit of integration is needed in order for it to effectively wait until all javascript opterations is done, such as data loadings, template rendering etc.

Considering you have a Single Page Application I bet you have also some render method, and possibly another in and out too for handling transitions.

Well, the only matter here is to inform Snapshooter that the page has finish rendering. It's achieved by setting the property window.crawler.is_rendered.

window.crawler.is_rendered = true

Snapshooter will keep waiting for the page until this variable gets true and then the rendered DOM will be saved as a plain html file.

### Filtering the DOM before saving to file

We got ourselves many times having to "clean the DOM" before saving to file or just wanting to save the content without header and footer.

In order to achieve that we created the option to write a coffee file and filter the source just before it gets written to the HTML file.

In order to achieve filtering you must create a coffee file with a before_save method and specify that file with the "-k" option on the command line.

The method will be called and will receive a jQuery object which you can use to manipulate the DOM, you should return the piece of HTML you want to save.

# removing script tags from the DOM

exports.before_save = ( $ ) ->
  $.find( 'script' ).remove()

  return $.html()

# returning just the content div

exports.before_save = ( $ ) ->
  return $.find( '#content' ).html()

then from the command line

snapshooter -i <> -o <local-folder> -k
## Contributing

Download the repo and have fun, pull requests are more than welcome.

Setting up

  git clone git://
  cd snapshooter
  npm link


To build, just run:

make build

During develop you may prefer:

make watch


Do not mess with version number.

Powered by

Current Tags

  • 0.3.9                                ...           latest (5 years ago)

26 Versions

  • 0.3.9                                ...           5 years ago
  • 0.3.8                                ...           5 years ago
  • 0.3.7                                ...           5 years ago
  • 0.3.5                                ...           6 years ago
  • 0.3.4                                ...           6 years ago
  • 0.3.3                                ...           6 years ago
  • 0.3.2                                ...           6 years ago
  • 0.3.1                                ...           6 years ago
  • 0.3.0                                ...           6 years ago
  • 0.2.15                                ...           6 years ago
  • 0.2.14                                ...           6 years ago
  • 0.2.13                                ...           6 years ago
  • 0.2.12                                ...           6 years ago
  • 0.2.11                                ...           6 years ago
  • 0.2.10                                ...           6 years ago
  • 0.2.9                                ...           6 years ago
  • 0.2.8                                ...           6 years ago
  • 0.2.7                                ...           6 years ago
  • 0.2.6                                ...           6 years ago
  • 0.2.5                                ...           6 years ago
  • 0.2.4                                ...           6 years ago
  • 0.2.3                                ...           6 years ago
  • 0.2.2                                ...           6 years ago
  • 0.2.1                                ...           6 years ago
  • 0.2.0                                ...           6 years ago
  • 0.1.1                                ...           7 years ago
Maintainers (2)
Today 0
This Week 0
This Month 0
Last Day 0
Last Week 0
Last Month 0
Dependencies (9)
Dev Dependencies (0)
Dependents (2)

Copyright 2014 - 2016 © |