github recommendation version 3

idea: download data from bigquery github table, analyze the data and visualize the data.

1. Use github.com/ferryzhou/gcutil

2. Download pemission json file and save it to ~/.config/gcloud/application_default_credentials.json, see https://godoc.org/golang.org/x/oauth2/google#DefaultTokenSource

3. Enable storage JSON API

4. Setup bigquery project, dataset and storage bucket

5. Download data

./run.sh get_bq_data

1. Result Data
2. repos.csv repo_url, name, owner, created_at, watchers, language, description, ...
3. recs.csv repo1_url, repo2_url, count

raw data is large and we don't need them all. here we sequencing the url and truncate recommendations data.

map shortPath to int and vice-versa recs[i] is a slice of

./run.sh process_data

1. Prerequisite

//install postgrest

1. Serve repos data: /repos?
2. Load data to postgres ./run.sh csv2db
3. Run postgrest ./run.sh serve_repos

./run.sh serve

1. Test api

./run.sh test_api

1. http://postgrest.com/api/reading/