mwoffliner
Mediawiki HTML/ZIM scraper
Last updated a month ago by isnit0 .
GPL-3.0 · Repository · Bugs · Original npm · Tarball · package.json
$ cnpm install mwoffliner 
SYNC missed versions from official npm registry.

MWoffliner

MWoffliner is a tool for making a local offline HTML snapshot of any online Mediawiki instance. It goes through all articles (or a selection if specified) and writes the HTML/images to a local directory. It has mainly been tested against Wikimedia projects like Wikipedia, Wiktionary, ... But it should also work for any recent Mediawiki.

It can write the raw HTML/JS/CSS/PNG... files to the filesystem or pack them all in a highly compressed ZIM file.

Read CONTRIBUTING.md to know more about MWoffliner development.

NPM

npm Docker Build Status Build Status codecov CodeFactor NPM

Prerequisites

  • *NIX Operating System (GNU/Linux, macOS, ...)
  • NodeJS
  • Redis
  • Libzim (On linux we automatically download binaries)
  • Various build tools that are probably already installed on your machine (libjpeg, gcc)

See Environment setup hints to know more about how to install them.

Usage

To install MWoffliner globally:

npm i -g mwoffliner

You might need to run this command with the sudo command, depending how your npm is configured.

Then to run it:

mwoffliner --help

API

MWoffliner provides also an API and therefore can be used as a NodeJS library. Here a stub example:

const mwoffliner = require('mwoffliner');
const parameters = {
    mwUrl: "https://es.wikipedia.org",
    adminEmail: "foo@bar.net",
    verbose: true,
    format: "nozim",
    articleList: "./articleList"
};
mwoffliner.execute(parameters); // returns a Promise

Background

Complementary information about MWoffliner:

  • MediaWiki software is used by dozen of thousands of wikis, the most famous ones being the Wikimedia ones, including Wikipedia.
  • MediaWiki is a PHP wiki runtime engine.
  • Wikitext is the name of the markup language that MediaWiki uses.
  • MediaWiki includes a parser for WikiText into HTML, and this parser creates the HTML pages displayed in your browser.
  • There is another WikiText parser, called Parsoid, implemented in Javascript/NodeJS. MWoffliner uses Parsoid.
  • Parsoid is planned to eventually become the main parser for MediaWiki.
  • MWoffliner calls Parsoid and then post-processes the results for offline format.

Environment setup hints

macOS

Install NodeJS:

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh | bash && \
source ~/.bashrc && \
nvm install stable && \
node --version

Install Redis:

brew install redis

Install libzim: Read these instructions

GNU/Linux - Debian based distributions

Install NodeJS:

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh | bash && \
source ~/.bashrc && \
nvm install stable && \
node --version

Install Redis:

sudo apt-get install redis-server

Releasing

  1. Update package.json
  2. Commit :package: Release version vX.X.X
  3. Run git tag vX.X.X
  4. Run git push origin master --tags

License

GPLv3 or later, see LICENSE for more details.

Current Tags

  • 1.9.14-1576229974020                                ...           dev (14 hours ago)
  • 1.9.13                                ...           latest (a month ago)
  • 1.9.5-rc1                                ...           rc (4 months ago)

174 Versions

  • 1.9.14-1576229974020                                ...           14 hours ago
  • 1.9.14-1576057165100                                ...           3 days ago
  • 1.9.14-1575970712126                                ...           4 days ago
  • 1.9.14-1575797794379                                ...           6 days ago
  • 1.9.14-1575624958588                                ...           8 days ago
  • 1.9.14-1575538593881                                ...           9 days ago
  • 1.9.14-1575365724638                                ...           11 days ago
  • 1.9.14-1575106529989                                ...           14 days ago
  • 1.9.14-1575020021519                                ...           15 days ago
  • 1.9.14-1574858736227                                ...           16 days ago
  • 1.9.14-1574760695143                                ...           18 days ago
  • 1.9.14-1574674268980                                ...           19 days ago
  • 1.9.14-1574534413202                                ...           20 days ago
  • 1.9.13-1574534374084                                ...           20 days ago
  • 1.9.13-1574501388071                                ...           21 days ago
  • 1.9.13-1574328496684                                ...           23 days ago
  • 1.9.13-1573982887208                                ...           a month ago
  • 1.9.13-1573896383591                                ...           a month ago
  • 1.9.13-1573723594519                                ...           a month ago
  • 1.9.13-1573550715650                                ...           a month ago
  • 1.9.13-1573464310268                                ...           a month ago
  • 1.9.13-1573377857734                                ...           a month ago
  • 1.9.13-1573291379360                                ...           a month ago
  • 1.9.13-1573204957268                                ...           a month ago
  • 1.9.13-1573118510484                                ...           a month ago
  • 1.9.13-1573032082774                                ...           a month ago
  • 1.9.13-1572945691387                                ...           a month ago
  • 1.9.13-1572807806804                                ...           a month ago
  • 1.9.13-1572444713722                                ...           a month ago
  • 1.9.13-1572444669768                                ...           a month ago
  • 1.9.13                                ...           a month ago
  • 1.9.13-1572426655403                                ...           a month ago
  • 1.9.13-1572382849640                                ...           a month ago
  • 1.9.13-1572254141629                                ...           2 months ago
  • 1.9.13-1572167902339                                ...           2 months ago
  • 1.9.13-1572081359667                                ...           2 months ago
  • 1.9.13-1571822182750                                ...           2 months ago
  • 1.9.13-1571662747940                                ...           2 months ago
  • 1.9.12-1571649092595                                ...           2 months ago
  • 1.9.12-1571487002584                                ...           2 months ago
  • 1.9.12-1571389781455                                ...           2 months ago
  • 1.9.12-1571303328185                                ...           2 months ago
  • 1.9.12-1571130449810                                ...           2 months ago
  • 1.9.12-1571044016579                                ...           2 months ago
  • 1.9.12-1570871190887                                ...           2 months ago
  • 1.9.12-1570698436571                                ...           2 months ago
  • 1.9.12-1570525545082                                ...           2 months ago
  • 1.9.12-1570266263868                                ...           2 months ago
  • 1.9.12-1570093377033                                ...           2 months ago
  • 1.9.12-1569926573594                                ...           2 months ago
  • 1.9.12-1569926263812                                ...           2 months ago
  • 1.9.12                                ...           2 months ago
  • 1.9.10-1569924205073                                ...           2 months ago
  • 1.9.10-1569924040269                                ...           2 months ago
  • 1.9.10-1569834108145                                ...           2 months ago
  • 1.9.10-1569661305842                                ...           3 months ago
  • 1.9.10-1569488458972                                ...           3 months ago
  • 1.9.10-1569399760960                                ...           3 months ago
  • 1.9.10-1569238196681                                ...           3 months ago
  • 1.9.10-1569229759862                                ...           3 months ago
  • 1.9.10-1569142752793                                ...           3 months ago
  • 1.9.10-1568969949266                                ...           3 months ago
  • 1.9.10-1568883513672                                ...           3 months ago
  • 1.9.10-1568710675042                                ...           3 months ago
  • 1.9.10-1568537864719                                ...           3 months ago
  • 1.9.10-1568365045867                                ...           3 months ago
  • 1.9.10-1568132671130                                ...           3 months ago
  • 1.9.10-1568105654099                                ...           3 months ago
  • 1.9.10-1567932786521                                ...           3 months ago
  • 1.9.10-1567762134551                                ...           3 months ago
  • 1.9.10                                ...           3 months ago
  • 1.9.9-1567759003981                                ...           3 months ago
  • 1.9.9                                ...           3 months ago
  • 1.9.8-1567690473934                                ...           3 months ago
  • 1.9.8                                ...           3 months ago
  • 1.9.7-1567689693497                                ...           3 months ago
  • 1.9.7-1567673573679                                ...           3 months ago
  • 1.9.7-1567614717007                                ...           3 months ago
  • 1.9.7-1567596239875                                ...           3 months ago
  • 1.9.7-1567505094404                                ...           3 months ago
  • 1.9.7-1567439650581                                ...           3 months ago
  • 1.9.7-1567360386708                                ...           3 months ago
  • 1.9.7-1567327582618                                ...           3 months ago
  • 1.9.7-1567159368082                                ...           3 months ago
  • 1.9.7-1567159210515                                ...           3 months ago
  • 1.9.7-1567159134548                                ...           3 months ago
  • 1.9.7-1567158316939                                ...           3 months ago
  • 1.9.7-1567157553811                                ...           3 months ago
  • 1.9.7-1567102177532                                ...           3 months ago
  • 1.9.7-1567008774699                                ...           4 months ago
  • 1.9.7-1566918684732                                ...           4 months ago
  • 1.9.6-1566909989981                                ...           4 months ago
  • 1.9.6-1566908441305                                ...           4 months ago
  • 1.9.6-1566827388065                                ...           4 months ago
  • 1.9.6-1566636353560                                ...           4 months ago
  • 1.9.6-1565968461694                                ...           4 months ago
  • 1.9.6-1565534303357                                ...           4 months ago
  • 1.9.6                                ...           4 months ago
  • 1.9.5-1565504480621                                ...           4 months ago
  • 1.9.5-1565257579465                                ...           4 months ago
  • 1.9.5-1565256582919                                ...           4 months ago
  • 1.9.5-1565255129327                                ...           4 months ago
  • 1.9.5-1564802207572                                ...           4 months ago
  • 1.9.5-1564676702010                                ...           4 months ago
  • 1.9.5                                ...           4 months ago
  • 1.9.5-rc1                                ...           4 months ago
  • 1.9.4-1564673279716                                ...           4 months ago
  • 1.9.5-rc1-1564672653355                                ...           4 months ago
  • 1.9.4-1564671403069                                ...           4 months ago
  • 1.9.4-1564653534747                                ...           4 months ago
  • 1.9.4-1564598710475                                ...           4 months ago
  • 1.9.4-1564591940813                                ...           4 months ago
  • 1.9.4-1564154025042                                ...           5 months ago
  • 1.9.4-1564123991543                                ...           5 months ago
  • 1.9.4-1564065577735                                ...           5 months ago
  • 1.9.4-1563952613266                                ...           5 months ago
  • 1.9.4-1563806962808                                ...           5 months ago
  • 1.9.4-1563806395093                                ...           5 months ago
  • 1.9.4-1563532785491                                ...           5 months ago
  • 1.9.4-1563180432331                                ...           5 months ago
  • 1.9.4-1563180279660                                ...           5 months ago
  • 1.9.4-1563180091203                                ...           5 months ago
  • 1.9.4                                ...           5 months ago
  • 1.9.4-rc1                                ...           5 months ago
  • 1.9.3                                ...           6 months ago
  • 1.9.3-rc4                                ...           6 months ago
  • 1.9.3-rc3                                ...           6 months ago
  • 1.9.3-rc2                                ...           6 months ago
  • 1.9.3-rc1                                ...           6 months ago
  • 1.9.2                                ...           6 months ago
  • 1.9.2-rc3                                ...           6 months ago
  • 1.9.2-rc2                                ...           6 months ago
  • 1.9.2-rc1                                ...           6 months ago
  • 1.9.1                                ...           6 months ago
  • 1.9.0-rc1                                ...           6 months ago
  • 1.8.6                                ...           7 months ago
  • 1.8.5                                ...           7 months ago
  • 1.8.4                                ...           8 months ago
  • 1.8.3                                ...           8 months ago
  • 1.8.2                                ...           8 months ago
  • 1.8.1                                ...           9 months ago
  • 1.8.0                                ...           9 months ago
  • 1.7.1                                ...           10 months ago
  • 1.7.0                                ...           a year ago
  • 1.6.0                                ...           a year ago
  • 1.5.0                                ...           a year ago
  • 1.4.1                                ...           2 years ago
  • 1.4.0                                ...           2 years ago
  • 1.3.7                                ...           2 years ago
  • 1.3.6                                ...           2 years ago
  • 1.3.5                                ...           2 years ago
  • 1.3.4                                ...           2 years ago
  • 1.3.3                                ...           2 years ago
  • 1.3.2                                ...           2 years ago
  • 1.3.1                                ...           2 years ago
  • 1.3.0                                ...           2 years ago
  • 1.2.7                                ...           3 years ago
  • 1.2.6                                ...           3 years ago
  • 1.2.5                                ...           3 years ago
  • 1.2.4                                ...           3 years ago
  • 1.2.3                                ...           3 years ago
  • 1.2.2                                ...           3 years ago
  • 1.2.1                                ...           3 years ago
  • 1.2.0                                ...           3 years ago
  • 1.1.6                                ...           3 years ago
  • 1.1.5                                ...           3 years ago
  • 1.1.4                                ...           3 years ago
  • 1.1.3                                ...           3 years ago
  • 1.1.2                                ...           3 years ago
  • 1.1.1                                ...           3 years ago
  • 1.0.5                                ...           3 years ago
  • 1.0.4                                ...           3 years ago
  • 1.0.3                                ...           3 years ago
  • 1.0.2                                ...           3 years ago
Downloads
Today 0
This Week 47
This Month 101
Last Day 16
Last Week 52
Last Month 337
Dependencies (47)
Dev Dependencies (11)
Dependents (0)
None

Copyright 2014 - 2017 © taobao.org |