austccr / link_and_attachment_archiver

Parse links out of HTML in a morph.io scraper and do some archiving actions on each of them. This is kind of a meta-archiver really.


Contributors equivalentideas

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling... [1G [1G-----> Ruby app detected [1G-----> Compiling Ruby [1G-----> Using Ruby version: ruby-2.6.3 [1G-----> Installing dependencies using bundler version 1.17.2 [1G Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment [1G Warning: the running version of Bundler (1.17.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. [1G Fetching gem metadata from https://rubygems.org/...... [1G Fetching https://github.com/openaustralia/scraperwiki-ruby.git [1G Using bundler 1.17.2 [1G Fetching coderay 1.1.2 [1G Fetching ffi 1.10.0 [1G Fetching httpclient 2.8.3 [1G Installing ffi 1.10.0 with native extensions [1G Installing coderay 1.1.2 [1G Installing httpclient 2.8.3 [1G Fetching method_source 0.9.2 [1G Fetching mini_portile2 2.4.0 [1G Installing mini_portile2 2.4.0 [1G Fetching sqlite3 1.4.0 [1G Installing method_source 0.9.2 [1G Installing sqlite3 1.4.0 with native extensions [1G Fetching nokogiri 1.10.1 [1G Installing nokogiri 1.10.1 with native extensions [1G Fetching pry 0.12.2 [1G Installing pry 0.12.2 [1G Fetching sqlite_magic 0.0.6 [1G Installing sqlite_magic 0.0.6 [1G Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176) [1G Fetching ethon 0.12.0 [1G Installing ethon 0.12.0 [1G Fetching typhoeus 1.3.1 [1G Installing typhoeus 1.3.1 [1G Bundle complete! 8 Gemfile dependencies, 13 gems now installed. [1G Gems in the groups development and test were not installed. [1G Bundled gems are installed into `./vendor/bundle` [1G Removing bundler (1.15.2) [1G Bundle completed (29.03s) [1G Cleaning up the bundler cache. [1G Warning: the running version of Bundler (1.17.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. [1G-----> Detecting rake tasks [1G [1G [1G-----> Discovering process types [1G Procfile declares types -> scraper Injecting scraper and running... Searching for records at https://api.morph.io/austccr/bca_media_releases_scraper/data.json Requesting records 1 to 5 with 'select * from "data" limit 5 offset 0' /app/vendor/ruby-2.6.3/lib/ruby/2.6.0/json/common.rb:156:in `parse': 767: unexpected token at '' (JSON::ParserError) from /app/vendor/ruby-2.6.3/lib/ruby/2.6.0/json/common.rb:156:in `parse' from scraper.rb:29:in `archive_links_from_morph_results' from scraper.rb:81:in `work_through_morph_results' from scraper.rb:95:in `block in <main>' from scraper.rb:90:in `each' from scraper.rb:90:in `<main>'

Data

Downloaded 0 times

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (3.89 MB) Use the API

rows 10 / 2710

url syndication errors source_url archived_at
2019-03-19 02:47:06 UTC
2019-03-24 02:15:41 UTC
302: Found
2019-03-24 02:15:42 UTC
2019-04-02 22:26:39 UTC
502: Bad GatewayLiveDocumentNotAvailableException: https://minerals.org.au/sites/default/files/190402%20Australia%27s%20mining%20industry%20delivers%20Budget%20surplus%20and%20a%20stronger%20economy.pdf: live document unavailable: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX
2019-04-02 22:26:40 UTC
2019-04-17 17:48:14 UTC
502: Bad GatewayLiveDocumentNotAvailableException: https://minerals.org.au/sites/default/files/190519%20Australia%27s%20minerals%20industry%20will%20work%20with%20Coalition%20government%20for%20a%20stronger%20Australia.pdf: live document unavailable: javax.net.ssl.SSLHandshakeException: sun.security.validator.Valid
2019-05-20 20:25:51 UTC
2019-06-07 04:10:03 UTC
400: Invalid URI: noSlash
2019-06-12 05:11:00 UTC
502: Bad GatewayLiveDocumentNotAvailableException: https://minerals.org.au/sites/default/files/181012%20Commodity%20Insights%20Met%20Coal%20Report.pdf: live document unavailable: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath
2019-07-01 06:12:15 UTC

Statistics

Average successful run time: about 3 hours

Total run time: about 1 month

Total cpu time used: about 1 hour

Total disk space used: 4.14 MB

History

  • Auto ran revision bc449ac8 and failed .
    nothing changed in the database
  • Auto ran revision bc449ac8 and failed .
    nothing changed in the database
  • Auto ran revision bc449ac8 and failed .
    nothing changed in the database
  • Auto ran revision bc449ac8 and failed .
    nothing changed in the database
  • Auto ran revision bc449ac8 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history