jamezpolley / newcastle_atdis_test

Newcastle Development (ATDIS) (TEST ONLY)

README.md

Newcastle ATDIS scraper

  • Server - Technology One
  • ATDIS feed. Very simple to scrape. Thank you Newcastle Council!
  • JSON - Yes - Yay....

For simple local testing, simply run make on a machine with docker and docker-compose installed. A docker image which (roughly) matches what morph.io will run in production will be created and executed.

If the scraper run was successful, the resulting data.sqlite will be extracted from the container and left in the current directory. The next time make is run, this will be moved to prev-data.sqlite

Enjoy

Contributors henare LoveMyData jamezpolley equivalentideas JasonThomasData mercuryfrost nik3daz CloCkWeRX mlandauer

Last run failed with status code 255.

Console output of last run

Injecting configuration and compiling... [1G [1G-----> Ruby app detected [1G-----> Installing bundler 1.17.3 [1G-----> Removing BUNDLED WITH version in the Gemfile.lock [1G-----> Compiling Ruby [1G-----> Using Ruby version: ruby-3.0.3 [1G-----> Installing dependencies using bundler 1.17.3 [1G Running: BUNDLE_WITHOUT='development:test' BUNDLE_PATH=vendor/bundle BUNDLE_BIN=vendor/bundle/bin BUNDLE_DEPLOYMENT=1 BUNDLE_GLOBAL_PATH_APPENDS_RUBY_SCOPE=1 bundle install -j4 [1G You are trying to install in deployment mode after changing [1G your Gemfile. Run `bundle install` elsewhere and add the [1G updated Gemfile.lock to version control. [1G [1G If this is a development machine, remove the /tmp/build/Gemfile freeze [1G by running ``. [1G [1G The list of sources changed [1G [1G You have added to the Gemfile: [1G * source: https://github.com/planningalerts-scrapers/atdisplanningalertsfeed.git [1G [1G You have deleted from the Gemfile: [1G * source: git://github.com/planningalerts-scrapers/atdisplanningalertsfeed.git [1G Bundler Output: You are trying to install in deployment mode after changing [1G your Gemfile. Run `bundle install` elsewhere and add the [1G updated Gemfile.lock to version control. [1G [1G If this is a development machine, remove the /tmp/build/Gemfile freeze [1G by running ``. [1G [1G The list of sources changed [1G [1G You have added to the Gemfile: [1G * source: https://github.com/planningalerts-scrapers/atdisplanningalertsfeed.git [1G [1G You have deleted from the Gemfile: [1G * source: git://github.com/planningalerts-scrapers/atdisplanningalertsfeed.git [1G [1m[31m [1G ! [1G ! Failed to install gems via Bundler. [1G ![0m

Data

Downloaded 7 times by jamezpolley

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (180 KB) Use the API

rows 10 / 395

council_reference address description info_url comment_url date_scraped date_received on_notice_from on_notice_to
DA2019/00513
26 Meya Crescent, Fletcher, NSW 2287
Dwelling house and associated retaining walls
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00513
2019-07-05
2019-05-08
DA2019/00261
2 Lloyd Street, Merewether, NSW 2291
Demolition of dwelling and erection of two storey dwelling
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00261
2019-07-05
2019-03-14
DA2019/00578
11 Mackie Avenue, New Lambton, NSW 2305
Swimming pool
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00578
2019-07-05
2019-05-29
DA2019/00547
346 Maitland Road, Mayfield, NSW 2304
Demolition of outbuilding erection of garage
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00547
2019-07-05
2019-05-20
DA2019/00393
163 Christo Road, Waratah, NSW 2298
Alterations and additions to dwelling
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00393
2019-07-05
2019-04-11
DA2019/00581
4 Gavey Street, Mayfield, NSW 2304
Demolition of outbuilding erection of garage
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00581
2019-07-05
2019-05-24
DA2019/00118
119 Bruce Street, Cooks Hill, NSW 2300
Dwelling, alterations and additions to
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00118
2019-07-05
2019-02-08
DA2019/00400
9 Pembroke Street, Stockton, NSW 2295
Demolition of outbuilding erection of garage and storage area
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00400
2019-07-05
2019-04-12
DA2019/00468
116 Young Road, Lambton, NSW 2299
Dwelling house, alterations and additions to
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00468
2019-07-05
2019-05-06
DA2019/00360
3 Karoola Road, Lambton, NSW 2299
Single storey dwelling
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00360
2019-07-05
2019-04-03

Statistics

Average successful run time: 3 minutes

Total run time: about 12 hours

Total cpu time used: 16 minutes

Total disk space used: 236 KB

History

  • Auto ran revision 295d6737 and failed .
    nothing changed in the database
  • Auto ran revision 295d6737 and failed .
    nothing changed in the database
  • Auto ran revision 295d6737 and failed .
    nothing changed in the database
  • Auto ran revision 295d6737 and failed .
    nothing changed in the database
  • Auto ran revision 295d6737 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

newcastle_atdis_test / scraper.rb