/OpenPension forked from SoapSeller/OpenPension

Open Pension parsing framework
  1. JavaScript 99.6%
  2. Other 0.4%
JavaScript Other
Find file
Pull request Compare This branch is 27 commits ahead, 1 commit behind SoapSeller:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
common @ 5894cb0 update common Jan 14, 2016
excel 2016 3 Dec 15, 2016
sql Issuer number is string, Jan 19, 2016
tests WIP: Tests for different usages of postgres. Mar 10, 2014
xlsx use patched xlsx parser until new release comes Mar 23, 2015
.gitignore Merge remote-tracking branch 'origin/2015_3' Jan 19, 2016
.gitmodules fix submodule Feb 15, 2014
.travis.yml fix seg fault Sep 7, 2014
CSVWriter.js Create indices Feb 5, 2016
Cakefile +fix number parsing bug;+fix bug when file is called twice Jun 9, 2014
LICENSE License credit update. Jun 2, 2014
LevDistance.js string cleanup func Jun 15, 2013
README.md update readme Feb 6, 2016
config.json Update config.json Dec 14, 2015
db.js Release db client May 8, 2016
dbLoader.js Use old bluebird Feb 1, 2016
detectors.js new logger Dec 18, 2015
dirs.js create dirs if dont exist Jan 4, 2016
elasticLoader.js copy only last 4 quarters to elastic Feb 6, 2016
fetcher.common.js Add error message on error May 8, 2016
fetcher.harel.js Harel-fetcher: Fix half-auto fetching of problematic funds. Jun 2, 2014
fetcher.js some tweaks for 2016 Oct 13, 2016
files_loader.js some tweaks for 2016 Oct 13, 2016
genericImporter.js some tweaks for 2016 Oct 13, 2016
logger.js add logger level trace Dec 27, 2015
op.js copy only last 4 quarters to elastic Feb 6, 2016
package.json copy only last 4 quarters to elastic Feb 6, 2016
quarter.js Create needed dirs on startup Jan 20, 2015
test.js add index of sheet to output (1-based) Apr 30, 2013
utils.js Merge remote-tracking branch 'origin/2015_3' Jan 19, 2016
validator.js Issuer number is string, Jan 19, 2016
xlsxparser.js Skip Meta table sheet Jan 18, 2016

README.md

Open Pension

Build Status

Import pension "single asset" files into unified DB.

Installation

Prerequisites

ssconvert

sudo apt-get install gnumeric

Clone and initialize submodules

git clone ...
git submodule init
git submodule update
npm install

Commands

Usage: $ node op.js [command] [options]

  Commands:

    convert-files [options]     convert excel files to csv
    db-create-table [options]   create table in database
    db-empty-table [options]    truncate table in database
    db-load-files [options]     load csv files to database
    fetch-google [options]      download files in Google Doc.
    recreate-elastic-indices    copy database data to elastic search.

Convert Excel files to CSV

Usage: $ node op.js convert-files [options]

  Options:

    -y, --year <year>         year
    -q, --quarter <quarter>   quarter
    -b, --body <body>         body
    -f, --fund <fund number>  fund
    -s, --srcdir <name>       path of Excel files, default:./excel/
    -t, --trgdir <name>       path of CSV files, default:./csv/
    -o, --overwrite           overwrite existing CSV files, default: false

Create Empty DB Table

Usage: $ node op.js db-create-table [options]

  Options:

    -t, --table <name>  table name

Truncate DB Table

Usage: $ node op.js db-empty-table [options]

  Options:

    -t, --table <name>  table name

Load CSV Files to Database

Usage: $ node op.js db-load-files [options]

  Options:

    -y, --year <year>           year
    -q, --quarter <quarter>     quarter
    -b, --body <body>           body
    -f, --fund <fund number>    fund number
    -t, --table <name>          table name
    -s, --srcdir <name>         path of CSV files, default:./csv/
    -c, --concurrency <number>  number of concurrent DB connections, defaults to 4

Fetch Excel files from Google Sheet

Usage: $ node op.js fetch-google [options]

  Options:

    -y, --year <year>         year
    -q, --quarter <quarter>   quarter
    -b, --body <body>         body
    -f, --fund <fund number>  fund number
    -t, --trgdir <name>       path of Excel files, default:./excel/
    -o, --overwrite           overwrite existing Excel files, default: false

License

See LICENSE file.