jamezpolley / newcastle_atdis_test

Newcastle Development (ATDIS) (TEST ONLY)

README.md

Newcastle ATDIS scraper

  • Server - Technology One
  • ATDIS feed. Very simple to scrape. Thank you Newcastle Council!
  • JSON - Yes - Yay....

For simple local testing, simply run make on a machine with docker and docker-compose installed. A docker image which (roughly) matches what morph.io will run in production will be created and executed.

If the scraper run was successful, the resulting data.sqlite will be extracted from the container and left in the current directory. The next time make is run, this will be moved to prev-data.sqlite

Enjoy

Contributors henare LoveMyData jamezpolley equivalentideas JasonThomasData mercuryfrost nik3daz CloCkWeRX mlandauer

Last run failed with status code 255.

Console output of last run

Injecting configuration and compiling... [1G [1G-----> Ruby app detected [1G-----> Installing bundler 1.17.3 [1G-----> Removing BUNDLED WITH version in the Gemfile.lock [1G-----> Compiling Ruby [1G-----> Using Ruby version: ruby-3.0.3 [1G-----> Installing dependencies using bundler 1.17.3 [1G Running: BUNDLE_WITHOUT='development:test' BUNDLE_PATH=vendor/bundle BUNDLE_BIN=vendor/bundle/bin BUNDLE_DEPLOYMENT=1 BUNDLE_GLOBAL_PATH_APPENDS_RUBY_SCOPE=1 bundle install -j4 [1G You are trying to install in deployment mode after changing [1G your Gemfile. Run `bundle install` elsewhere and add the [1G updated Gemfile.lock to version control. [1G [1G If this is a development machine, remove the /tmp/build/Gemfile freeze [1G by running ``. [1G [1G The list of sources changed [1G [1G You have added to the Gemfile: [1G * source: https://github.com/planningalerts-scrapers/atdisplanningalertsfeed.git [1G [1G You have deleted from the Gemfile: [1G * source: git://github.com/planningalerts-scrapers/atdisplanningalertsfeed.git [1G Bundler Output: You are trying to install in deployment mode after changing [1G your Gemfile. Run `bundle install` elsewhere and add the [1G updated Gemfile.lock to version control. [1G [1G If this is a development machine, remove the /tmp/build/Gemfile freeze [1G by running ``. [1G [1G The list of sources changed [1G [1G You have added to the Gemfile: [1G * source: https://github.com/planningalerts-scrapers/atdisplanningalertsfeed.git [1G [1G You have deleted from the Gemfile: [1G * source: git://github.com/planningalerts-scrapers/atdisplanningalertsfeed.git [1G [1m[31m [1G ! [1G ! Failed to install gems via Bundler. [1G ![0m

Data

Downloaded 7 times by jamezpolley

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (180 KB) Use the API

rows 10 / 395

council_reference address description info_url comment_url date_scraped date_received on_notice_from on_notice_to
council_reference
DA2019/00513
address
26 Meya Crescent, Fletcher, NSW 2287
description
Dwelling house and associated retaining walls
info_url
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00513
comment_url
date_scraped
2019-07-05
date_received
2019-05-08
on_notice_from
on_notice_to
council_reference
DA2019/00261
address
2 Lloyd Street, Merewether, NSW 2291
description
Demolition of dwelling and erection of two storey dwelling
info_url
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00261
comment_url
date_scraped
2019-07-05
date_received
2019-03-14
on_notice_from
on_notice_to
council_reference
DA2019/00578
address
11 Mackie Avenue, New Lambton, NSW 2305
description
Swimming pool
info_url
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00578
comment_url
date_scraped
2019-07-05
date_received
2019-05-29
on_notice_from
on_notice_to
council_reference
DA2019/00547
address
346 Maitland Road, Mayfield, NSW 2304
description
Demolition of outbuilding erection of garage
info_url
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00547
comment_url
date_scraped
2019-07-05
date_received
2019-05-20
on_notice_from
on_notice_to
council_reference
DA2019/00393
address
163 Christo Road, Waratah, NSW 2298
description
Alterations and additions to dwelling
info_url
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00393
comment_url
date_scraped
2019-07-05
date_received
2019-04-11
on_notice_from
on_notice_to
council_reference
DA2019/00581
address
4 Gavey Street, Mayfield, NSW 2304
description
Demolition of outbuilding erection of garage
info_url
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00581
comment_url
date_scraped
2019-07-05
date_received
2019-05-24
on_notice_from
on_notice_to
council_reference
DA2019/00118
address
119 Bruce Street, Cooks Hill, NSW 2300
description
Dwelling, alterations and additions to
info_url
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00118
comment_url
date_scraped
2019-07-05
date_received
2019-02-08
on_notice_from
on_notice_to
council_reference
DA2019/00400
address
9 Pembroke Street, Stockton, NSW 2295
description
Demolition of outbuilding erection of garage and storage area
info_url
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00400
comment_url
date_scraped
2019-07-05
date_received
2019-04-12
on_notice_from
on_notice_to
council_reference
DA2019/00468
address
116 Young Road, Lambton, NSW 2299
description
Dwelling house, alterations and additions to
info_url
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00468
comment_url
date_scraped
2019-07-05
date_received
2019-05-06
on_notice_from
on_notice_to
council_reference
DA2019/00360
address
3 Karoola Road, Lambton, NSW 2299
description
Single storey dwelling
info_url
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00360
comment_url
date_scraped
2019-07-05
date_received
2019-04-03
on_notice_from
on_notice_to

Statistics

Average successful run time: 3 minutes

Total run time: about 12 hours

Total cpu time used: 16 minutes

Total disk space used: 236 KB

History

  • Auto ran revision 295d6737 and failed .
    nothing changed in the database
  • Auto ran and failed .
  • Auto ran and failed .
  • Auto ran and failed .
  • Auto ran and failed .
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

newcastle_atdis_test / scraper.rb