Google First-Page Job Search (Ruby on Morph.io)
Scrapes the first Google results page for a boolean query that targets Lever job posts, then saves results to Morph.io SQLite via scraperwiki.
Heads-up: Google can block automated requests or change markup. This script is best-effort. If you hit consent/CAPTCHA pages, consider a search API (e.g., SerpAPI) or a different engine.
Default query
Defined in scraper.rb:
Last run failed
with status code 998.
Console output of last run
Injecting configuration and compiling...
[1G [1G-----> Ruby app detected
[1G-----> Compiling Ruby
[1G-----> Using Ruby version: ruby-2.6.6
[1G-----> Installing dependencies using bundler version 1.17.2
[1G Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment
[1G Warning: the running version of Bundler (1.17.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.
[1G Fetching gem metadata from https://rubygems.org/......
[1G Using bundler 1.17.2
[1G Fetching mutex_m 0.3.0
[1G Fetching racc 1.8.1
[1G Fetching sqlite3 1.5.4 (x86_64-linux)
[1G Installing mutex_m 0.3.0
[1G Installing racc 1.8.1 with native extensions
[1G Fetching httpclient 2.9.0
[1G Installing sqlite3 1.5.4 (x86_64-linux)
[1G Installing httpclient 2.9.0
[1G Fetching sqlite_magic 0.0.6
[1G Installing sqlite_magic 0.0.6
[1G Fetching scraperwiki 3.0.2
[1G Installing scraperwiki 3.0.2
[1G Fetching nokogiri 1.13.10 (x86_64-linux)
[1G Installing nokogiri 1.13.10 (x86_64-linux)
[1G Bundle complete! 3 Gemfile dependencies, 8 gems now installed.
[1G Gems in the groups development and test were not installed.
[1G Bundled gems are installed into `./vendor/bundle`
[1G Removing bundler (1.15.2)
[1G Bundle completed (2.38s)
[1G Cleaning up the bundler cache.
[1G Warning: the running version of Bundler (1.17.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.
[1G-----> Detecting rake tasks
[1G
[1G [1G-----> Discovering process types
[1G Procfile declares types -> scraper
Injecting scraper and running...
Saved 0 results from the first page.
Scraper didn't create an SQLite database in your current working directory called
data.sqlite. If you've just created your first scraper and not edited the code yet
this is to be expected.
To fix this make your scraper write to an SQLite database at data.sqlite.
Statistics
Total run time: 2 minutes
Total cpu time used: less than 5 seconds
Total disk space used: 78.6 KB