What's New

Here we show you the changes that you can see. We update this page every few weeks. There's also lots of stuff that changes under the hood. If you're interested in seeing those too, look at the full list of commits.

Summary for May & April 2015

It's been a busy few weeks as we're working hard to further improve the user experience by tweaking lots of small things and adding a few much-needed features.

The buildpacks support, that allows anybody to install their own libraries, is also now activated for everyone.

April 30

Everyone can now easily install their own libraries and set their language versions. Up till now this has only been available through the "experimental Buildpack support".

Unless you set your own libraries you should notice no difference to how your scrapers runs except a small change to the console output.

April 28

  • In lists of scrapers show the language of the scraper
  • Simplify the compile output of buildpacks
  • Reverse the order of scrapers in email alerts so that most recently errored scrapers appear at the top
  • Added scraper creation to history

April 24

  • Don't render arbitrary html in tables anymore

April 22

Several significant improvements to search:

  • Search now works across users too
  • What you searched for is now highlighted in the results
  • Search results are now paginated And also
  • Simplified the display of users and organisations in lists

April 18

  • Show the meta (or title) information for domains scraped on the scraper page
  • Data table on scraper page is now responsive (works well across screen sizes) and has improved display when table is wider than current screen width

April 17

  • Put "add new scraper button" on organisation page

April 16

  • When using buildpacks create template scraper that includes configuration files

April 15

  • Fixed command-line tool (morph-cli gem) which had been broken since the addition of secret environment variable support to
  • Fixed command-line tool with buildpacks
  • Improve display of database size on scraper page
  • If local sqlite database gets corrupted let the user know and let them know how to fix it.
  • Improvements to layout for navigation bar, including search and menu toggler icon

April 14

  • A brand new landing page! (visible when you're not logged in)

April 13

  • Fix issue with symlinks in repos with buildpacks

April 10

  • Improve loading of small avatar images

April 8

  • With buildpacks scraper output now streams in real time to the console

April 7

  • Improve console output when outputting utf-8
  • Show domains (and number of pages scraped) for each scraper

May 30

  • Improve reliability of how stopped scrapers are cleaned up which reduces occurences of "stuck" scrapers
  • Add basic search over scrapers

May 26

  • Keep people signed in on Firefox browser

Feb 17

  • If an update to the scraper code has been force pushed to github, make pick up the changes correctly

And further back

If you're interested in looking further back then see the full list of commits.