ianheggie-oaf / wollondilly
Wollondilly Shire Council Development Applications
Wollondilly
Wollondilly now has a custom web application which has an API. So, we're using that.
This is a scraper that runs on Morph. To get started see the documentation
Add any issues to https://github.com/planningalerts-scrapers/issues/issues
Set MORPH_AUSTRALIAN_PROXY to an Australian proxy, preferably one that rotates its IP address
Expected Output
Storing S68/2026/29/1 - 2 Jan Street PICTON NSW
Storing S68/2026/30/1 - 350 Spring Creek Road MOUNT HUNTER NSW
...
Storing CDC/2026/158/1 - 50 Koolahs Street APPIN NSW
Storing CDC/2026/161/1 - 4 Joseph Lane WILTON NSW
Finished - processed 133 records
Expected runtime: ~ 10 seconds
To run the scraper
bundle exec ruby scraper.rb
To run style and coding checks
bundle exec rubocop
To check for security updates
gem install bundler-audit
bundle-audit
Contributors
Last run failed
with status code 1.
Console output of last run
Injecting configuration and compiling...
Injecting scraper and running...
Using Australian proxy...
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:229:in `rbuf_fill': Net::ReadTimeout with #<TCPSocket:(closed)> (Net::ReadTimeout)
from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/protocol.rb:164:in `read'
from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http/response.rb:583:in `block in read_body_0'
from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http/response.rb:539:in `inflater'
from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http/response.rb:573:in `read_body_0'
from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http/response.rb:344:in `read_body'
from /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:928:in `response_read'
from /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:287:in `block in fetch'
from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1873:in `block in transport_request'
from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http/response.rb:301:in `reading_body'
from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1872:in `transport_request'
from /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb:1826:in `request'
from /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.8/lib/net/http/persistent.rb:958:in `block in request'
from /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.8/lib/net/http/persistent.rb:662:in `connection_for'
from /app/vendor/bundle/ruby/3.2.0/gems/net-http-persistent-4.0.8/lib/net/http/persistent.rb:952:in `request'
from /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize/http/agent.rb:284:in `fetch'
from /app/vendor/bundle/ruby/3.2.0/gems/mechanize-2.8.5/lib/mechanize.rb:465:in `get'
from scraper.rb:35:in `run'
from scraper.rb:65:in `<main>'
Statistics
Total run time: 8 days
Total cpu time used: 8 minutes
Total disk space used: 347 KB