pika-web-crawler-test
Web crawler test script
Last updated a year ago by pikamachu .
MIT · Repository · Bugs · Original npm · Tarball · package.json
$ cnpm install pika-web-crawler-test 
SYNC missed versions from official npm registry.

Pika web crawler test

Introduction

Web crawler test script for web testing with snapshot comparison.

Installing / Getting started

To install the package execute:

npm install -g pika-web-crawler-test

After installation, tou will have access to the 'pika-web-crawler-test' binary in your command line. You can check help with this command:

pika-web-crawler-test --help

Developing

Built With

Prerequisites

The following software must be installed

Folder structure

  • root: Contains the README.md, the main configuration to execute the project such as package.json or any other configuration file.
  • bin: Contains the oclif script run.
  • scripts: Contains the scripts to start/stop Chrome in headless mode or any other system script required to run the tests.
  • src: Contains the source code for crawler test script.
  • node_modules: Contains third party JS libraries used in this project

Setting up Dev

Download the code

git clone https://github.com/pikamachu/pika-web-crawler-test.git
cd pika-web-crawler-test

Install dependencies

npm install

Run web crawler test. Parameters:

  • initial-url: Initial Url to open and parse links.
  • chunks: Number of concurrent url calls.
bin/run [initial-url] [chunks]

Pika commands

All previous command can be executed using pika script

Usage: pika [command]

where [command] is one of:
   run [initial-url] [chunks] - run the crawler test - also starts and stops chrome
     - initial-url: Initial Url to open and parse links.
     - chunks: Number of concurrent url calls. Default 2
   format - auto format project code using prettier

Windows Environment

  • If the project is ran in a Windows environment use the script files to start and stop chrome.
  • Chrome should be installed in c:\Program Files (x86)\Google\Chrome\Application, if it is not the case update the path in processes/start-chrome-headless.bat.

Current Tags

  • 0.1.1                                ...           latest (a year ago)

5 Versions

  • 0.1.1                                ...           a year ago
  • 0.3.0                                ...           a year ago
  • 0.2.1                                ...           a year ago
  • 0.2.0                                ...           a year ago
  • 0.1.0                                ...           a year ago
Maintainers (1)
Downloads
Today 0
This Week 0
This Month 0
Last Day 0
Last Week 0
Last Month 0
Dependencies (7)
Dev Dependencies (10)
Dependents (0)
None

Copyright 2014 - 2017 © taobao.org |