openaustralia / alaveteli_versions

Scrapes the version of every Alaveteli deployment

README.md

This is a scraper that runs on Morph.

To get started see the documentation

Development

Set up a local development copy of the scraper by running:

bash git clone https://github.com/openaustralia/alaveteli_versions cd alaveteli_versions bundle

Then run regression tests and audit the scraper style with:

bash bundle exec rake

Contributors

Last run completed successfully about an hour ago.

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Discovering deployments... Saving 0 records Done.

Data

Downloaded 52 times by

data

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (13 KB) Use the API

rows 10 / 36

alaveteli_version	ruby_version	visible_request_count	alaveteli_git_commit	confirmed_user_count	name	url	version_url	error	visible_public_body_count	visible_comment_count	track_thing_count	widget_vote_count	public_body_change_request_count	request_classification_count	visible_followup_message_count	date_checked
alaveteli_version 0.20.0.14	ruby_version 1.8.7	visible_request_count 29	alaveteli_git_commit 823e58dc69960c600230b10604a0051359173f85	confirmed_user_count 50	name Sobunakirwa	url https://sobanukirwa.rw/	version_url https://sobanukirwa.rw/version.json	error	visible_public_body_count	visible_comment_count	track_thing_count	widget_vote_count	public_body_change_request_count	request_classification_count	visible_followup_message_count	date_checked
alaveteli_version 0.20.0.14	ruby_version 1.9.3	visible_request_count 17	alaveteli_git_commit 48f07a68cb7373a9a032009ab760b5f4d156d6cd	confirmed_user_count 39	name Diritto Di Sapere	url https://chiedi.dirittodisapere.it/	version_url https://chiedi.dirittodisapere.it/version.json	error	visible_public_body_count	visible_comment_count	track_thing_count	widget_vote_count	public_body_change_request_count	request_classification_count	visible_followup_message_count	date_checked
alaveteli_version	ruby_version	visible_request_count	alaveteli_git_commit	confirmed_user_count	name Je Veux Savoir	url http://www.jeveuxsavoir.org/	version_url http://www.jeveuxsavoir.org/version.json	error 404 => Net::HTTPNotFound for http://www.nordouvert.ca/version.json/2015/06/15/je-veux-savoir-fin-d-une-belle-aventure.html -- unhandled response	visible_public_body_count	visible_comment_count	track_thing_count	widget_vote_count	public_body_change_request_count	request_classification_count	visible_followup_message_count	date_checked
alaveteli_version 0.19	ruby_version 1.8.7	visible_request_count 991	alaveteli_git_commit d12c236fbd606fb38c01aebb39694a568740de06	confirmed_user_count 939	name Queremos Saber	url http://queremossaber.org.br/	version_url http://queremossaber.org.br/version.json	error	visible_public_body_count	visible_comment_count	track_thing_count	widget_vote_count	public_body_change_request_count	request_classification_count	visible_followup_message_count	date_checked
alaveteli_version 0.21.0.26	ruby_version 1.8.7	visible_request_count 10	alaveteli_git_commit 5e2f00176f73736929fde4e1ab09f594ba5252cf	confirmed_user_count 4	name Marsoum 41	url http://www.marsoum41.org/en/	version_url http://www.marsoum41.org/en/version.json	error	visible_public_body_count	visible_comment_count	track_thing_count	widget_vote_count	public_body_change_request_count	request_classification_count	visible_followup_message_count	date_checked
alaveteli_version 0.12	ruby_version 1.8.7	visible_request_count	alaveteli_git_commit ad820340ad8bebffbe453ae337a4ec49dec6022f	confirmed_user_count	name Tu Derecho A Saber	url http://tuderechoasaber.es/	version_url http://tuderechoasaber.es/version.json	error	visible_public_body_count	visible_comment_count	track_thing_count	widget_vote_count	public_body_change_request_count	request_classification_count	visible_followup_message_count	date_checked
alaveteli_version	ruby_version	visible_request_count	alaveteli_git_commit	confirmed_user_count	name Da Zanamo Svi	url http://daznamosvi.rs/sr/	version_url http://daznamosvi.rs/sr/version.json	error 404 => Net::HTTPNotFound for http://daznamosvi.rs/sr/version.json -- unhandled response	visible_public_body_count	visible_comment_count	track_thing_count	widget_vote_count	public_body_change_request_count	request_classification_count	visible_followup_message_count	date_checked
alaveteli_version	ruby_version	visible_request_count	alaveteli_git_commit	confirmed_user_count	name Nós Queremos Saber	url http://nosqueremossaber.org/	version_url http://nosqueremossaber.org/version.json	error Didn't get JSON from API.	visible_public_body_count	visible_comment_count	track_thing_count	widget_vote_count	public_body_change_request_count	request_classification_count	visible_followup_message_count	date_checked
alaveteli_version	ruby_version	visible_request_count	alaveteli_git_commit	confirmed_user_count	name Pravo Da Znam	url http://www.pravodaznam.ba/	version_url http://www.pravodaznam.ba/version.json	error Net::HTTP::Persistent::Error: too many connection resets (due to Net::ReadTimeout - Net::ReadTimeout) after 0 requests on 47044091722280, last used 1531865782.47403 seconds ago	visible_public_body_count	visible_comment_count	track_thing_count	widget_vote_count	public_body_change_request_count	request_classification_count	visible_followup_message_count	date_checked 2018-07-17
alaveteli_version	ruby_version	visible_request_count	alaveteli_git_commit	confirmed_user_count	name RightToKnow	url http://foi.sinarproject.org/	version_url http://foi.sinarproject.org/version.json	error Net::HTTP::Persistent::Error: too many connection resets (due to Net::ReadTimeout - Net::ReadTimeout) after 0 requests on 46942657109440, last used 1551708277.0665338 seconds ago	visible_public_body_count	visible_comment_count	track_thing_count	widget_vote_count	public_body_change_request_count	request_classification_count	visible_followup_message_count	date_checked 2019-03-04

Statistics

Average successful run time: 2 minutes

Total run time: about 1 month

Total cpu time used: about 1 hour

Total disk space used: 139 KB

History

Auto ran revision 4a0c1181 and completed successfully about an hour ago.

run time 3 s

nothing changed in the database
Auto ran revision 4a0c1181 and completed successfully about 13 hours ago.

run time 2 s

nothing changed in the database
Auto ran revision 4a0c1181 and completed successfully a day ago.

run time 2 s

nothing changed in the database
Auto ran revision 4a0c1181 and completed successfully 3 days ago.

run time 2 s

nothing changed in the database
Auto ran revision 4a0c1181 and completed successfully 3 days ago.

run time 2 s

nothing changed in the database
...
Created on morph.io 10 years ago

Show complete history

Scraper code

Ruby

alaveteli_versions / scraper.rb

git clone URL