omorilewa / healthtools_ke

[morph] Scrapers for the HealthTools Kenya data.

Contributors Gathondu DavidLemayian RyanSept andela-mabdussalam omorilewa

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... [1G [1G-----> Python app detected [1G-----> Installing python-2.7.9 [1G $ pip install -r requirements.txt [1G Collecting appdirs==1.4.3 (from -r /tmp/build/requirements.txt (line 1)) [1G Downloading appdirs-1.4.3-py2.py3-none-any.whl [1G Collecting backports.ssl-match-hostname==3.5.0.1 (from -r /tmp/build/requirements.txt (line 2)) [1G Downloading backports.ssl_match_hostname-3.5.0.1.tar.gz [1G Collecting beautifulsoup4==4.5.3 (from -r /tmp/build/requirements.txt (line 3)) [1G Downloading beautifulsoup4-4.5.3-py2-none-any.whl (85kB) [1G Collecting boto3==1.4.4 (from -r /tmp/build/requirements.txt (line 4)) [1G Downloading boto3-1.4.4-py2.py3-none-any.whl (127kB) [1G Collecting botocore==1.5.27 (from -r /tmp/build/requirements.txt (line 5)) [1G Downloading botocore-1.5.27-py2.py3-none-any.whl (3.4MB) [1G Collecting bs4==0.0.1 (from -r /tmp/build/requirements.txt (line 6)) [1G Downloading bs4-0.0.1.tar.gz [1G Collecting certifi==2017.4.17 (from -r /tmp/build/requirements.txt (line 7)) [1G Downloading certifi-2017.4.17-py2.py3-none-any.whl (375kB) [1G Collecting click==6.7 (from -r /tmp/build/requirements.txt (line 8)) [1G Downloading click-6.7-py2.py3-none-any.whl (71kB) [1G Collecting docutils==0.13.1 (from -r /tmp/build/requirements.txt (line 9)) [1G Downloading docutils-0.13.1-py2-none-any.whl (537kB) [1G Collecting elasticsearch==5.4.0 (from -r /tmp/build/requirements.txt (line 10)) [1G Downloading elasticsearch-5.4.0-py2.py3-none-any.whl (58kB) [1G Collecting Flask==0.12.1 (from -r /tmp/build/requirements.txt (line 11)) [1G Downloading Flask-0.12.1-py2.py3-none-any.whl (82kB) [1G Collecting futures==3.0.5 (from -r /tmp/build/requirements.txt (line 12)) [1G Downloading futures-3.0.5-py2-none-any.whl [1G Collecting gunicorn==19.7.1 (from -r /tmp/build/requirements.txt (line 13)) [1G Downloading gunicorn-19.7.1-py2.py3-none-any.whl (111kB) [1G Collecting itsdangerous==0.24 (from -r /tmp/build/requirements.txt (line 14)) [1G Downloading itsdangerous-0.24.tar.gz (46kB) [1G Collecting Jinja2==2.9.6 (from -r /tmp/build/requirements.txt (line 15)) [1G Downloading Jinja2-2.9.6-py2.py3-none-any.whl (340kB) [1G Collecting jmespath==0.9.2 (from -r /tmp/build/requirements.txt (line 16)) [1G Downloading jmespath-0.9.2-py2.py3-none-any.whl [1G Collecting MarkupSafe==1.0 (from -r /tmp/build/requirements.txt (line 17)) [1G Downloading MarkupSafe-1.0.tar.gz [1G Collecting nose==1.3.7 (from -r /tmp/build/requirements.txt (line 18)) [1G Downloading nose-1.3.7-py2-none-any.whl (154kB) [1G Collecting packaging==16.8 (from -r /tmp/build/requirements.txt (line 19)) [1G Downloading packaging-16.8-py2.py3-none-any.whl [1G Collecting progressbar2==3.30.2 (from -r /tmp/build/requirements.txt (line 20)) [1G Downloading progressbar2-3.30.2-py2.py3-none-any.whl [1G Collecting pyparsing==2.2.0 (from -r /tmp/build/requirements.txt (line 21)) [1G Downloading pyparsing-2.2.0-py2.py3-none-any.whl (56kB) [1G Collecting python-dateutil==2.6.0 (from -r /tmp/build/requirements.txt (line 22)) [1G Downloading python_dateutil-2.6.0-py2.py3-none-any.whl (194kB) [1G Collecting python-memcached==1.58 (from -r /tmp/build/requirements.txt (line 23)) [1G Downloading python-memcached-1.58.tar.gz [1G Collecting python-utils==2.1.0 (from -r /tmp/build/requirements.txt (line 24)) [1G Downloading python_utils-2.1.0-py2.py3-none-any.whl [1G Collecting requests==2.13.0 (from -r /tmp/build/requirements.txt (line 25)) [1G Downloading requests-2.13.0-py2.py3-none-any.whl (584kB) [1G Collecting requests-aws4auth==0.9 (from -r /tmp/build/requirements.txt (line 26)) [1G Downloading requests_aws4auth-0.9-py2.py3-none-any.whl (54kB) [1G Collecting s3transfer==0.1.10 (from -r /tmp/build/requirements.txt (line 27)) [1G Downloading s3transfer-0.1.10-py2.py3-none-any.whl (54kB) [1G Collecting six==1.10.0 (from -r /tmp/build/requirements.txt (line 28)) [1G Downloading six-1.10.0-py2.py3-none-any.whl [1G Collecting slackclient==1.0.6 (from -r /tmp/build/requirements.txt (line 29)) [1G Downloading slackclient-1.0.6.tar.gz [1G Collecting tqdm==4.14.0 (from -r /tmp/build/requirements.txt (line 30)) [1G Downloading tqdm-4.14.0-py2.py3-none-any.whl (46kB) [1G Collecting urllib3==1.21.1 (from -r /tmp/build/requirements.txt (line 31)) [1G Downloading urllib3-1.21.1-py2.py3-none-any.whl (131kB) [1G Collecting websocket-client==0.40.0 (from -r /tmp/build/requirements.txt (line 32)) [1G Downloading websocket_client-0.40.0.tar.gz (196kB) [1G Collecting Werkzeug>=0.7 (from Flask==0.12.1->-r /tmp/build/requirements.txt (line 11)) [1G Downloading Werkzeug-0.12.2-py2.py3-none-any.whl (312kB) [1G Installing collected packages: appdirs, backports.ssl-match-hostname, beautifulsoup4, jmespath, futures, docutils, six, python-dateutil, botocore, s3transfer, boto3, bs4, certifi, click, urllib3, elasticsearch, itsdangerous, Werkzeug, MarkupSafe, Jinja2, Flask, gunicorn, nose, pyparsing, packaging, python-utils, progressbar2, python-memcached, requests, requests-aws4auth, websocket-client, slackclient, tqdm [1G Running setup.py install for backports.ssl-match-hostname: started [1G Running setup.py install for backports.ssl-match-hostname: finished with status 'done' [1G Running setup.py install for bs4: started [1G Running setup.py install for bs4: finished with status 'done' [1G Running setup.py install for itsdangerous: started [1G Running setup.py install for itsdangerous: finished with status 'done' [1G Running setup.py install for MarkupSafe: started [1G Running setup.py install for MarkupSafe: finished with status 'done' [1G Running setup.py install for python-memcached: started [1G Running setup.py install for python-memcached: finished with status 'done' [1G Running setup.py install for websocket-client: started [1G Running setup.py install for websocket-client: finished with status 'done' [1G Running setup.py install for slackclient: started [1G Running setup.py install for slackclient: finished with status 'done' [1G Successfully installed Flask-0.12.1 Jinja2-2.9.6 MarkupSafe-1.0 Werkzeug-0.12.2 appdirs-1.4.3 backports.ssl-match-hostname-3.5.0.1 beautifulsoup4-4.5.3 boto3-1.4.4 botocore-1.5.27 bs4-0.0.1 certifi-2017.4.17 click-6.7 docutils-0.13.1 elasticsearch-5.4.0 futures-3.0.5 gunicorn-19.7.1 itsdangerous-0.24 jmespath-0.9.2 nose-1.3.7 packaging-16.8 progressbar2-3.30.2 pyparsing-2.2.0 python-dateutil-2.6.0 python-memcached-1.58 python-utils-2.1.0 requests-2.13.0 requests-aws4auth-0.9 s3transfer-0.1.10 six-1.10.0 slackclient-1.0.6 tqdm-4.14.0 urllib3-1.21.1 websocket-client-0.40.0 [1G [1G [1G-----> Discovering process types [1G Procfile declares types -> scraper Injecting scraper and running... [Doctors Scraper] [2017-07-18 14:49:37] Started Scraper. [Doctors Scraper] [2017-07-18 14:49:37] Started Scraper. [Doctors Scraper] [2017-07-18 14:49:37] Started Scraper. [Doctors Scraper] [2017-07-18 14:49:37] Started Scraper. 0%| | 0/235 [00:00<?, ?it/s] 0%| | 1/235 [00:08<32:59, 8.46s/it] 1%| | 2/235 [00:16<32:51, 8.46s/it] 1%|1 | 3/235 [00:25<32:44, 8.47s/it] 2%|1 | 4/235 [00:33<32:34, 8.46s/it] 2%|2 | 5/235 [00:42<32:29, 8.47s/it] 3%|2 | 6/235 [00:50<32:18, 8.47s/it] 3%|2 | 7/235 [00:59<32:09, 8.46s/it] 3%|3 | 8/235 [01:07<32:03, 8.48s/it] 4%|3 | 9/235 [01:16<31:55, 8.48s/it] 4%|4 | 10/235 [01:24<31:49, 8.48s/it] 5%|4 | 11/235 [01:33<31:41, 8.49s/it] 5%|5 | 12/235 [01:41<31:30, 8.48s/it] 6%|5 | 13/235 [01:50<31:20, 8.47s/it] 6%|5 | 14/235 [01:58<31:11, 8.47s/it] 6%|6 | 15/235 [02:07<31:02, 8.46s/it] 7%|6 | 16/235 [02:15<31:09, 8.54s/it] 7%|7 | 17/235 [02:24<31:03, 8.55s/it] 8%|7 | 18/235 [02:33<31:03, 8.59s/it] 8%|8 | 19/235 [02:41<30:45, 8.55s/it] 9%|8 | 20/235 [02:49<30:31, 8.52s/it] 9%|8 | 21/235 [02:58<30:18, 8.50s/it] 9%|9 | 22/235 [03:07<30:19, 8.54s/it] 10%|9 | 23/235 [03:15<30:09, 8.53s/it] 10%|# | 24/235 [03:24<29:57, 8.52s/it] 11%|# | 25/235 [03:32<29:50, 8.52s/it] 11%|#1 | 26/235 [03:37<25:29, 7.32s/it] 11%|#1 | 27/235 [03:37<18:15, 5.27s/it] 12%|#1 | 28/235 [03:38<13:18, 3.86s/it] 12%|#2 | 29/235 [03:38<09:44, 2.84s/it] 13%|#2 | 30/235 [03:39<07:19, 2.14s/it] 13%|#3 | 31/235 [03:39<05:35, 1.65s/it] 14%|#3 | 32/235 [03:40<04:20, 1.29s/it] 14%|#4 | 33/235 [03:40<03:40, 1.09s/it] 14%|#4 | 34/235 [03:41<03:06, 1.08it/s] 15%|#4 | 35/235 [03:41<02:50, 1.17it/s] 15%|#5 | 36/235 [03:42<02:27, 1.35it/s] 16%|#5 | 37/235 [03:42<02:09, 1.53it/s] 16%|#6 | 38/235 [03:43<02:02, 1.61it/s] 17%|#6 | 39/235 [03:43<01:51, 1.76it/s] 17%|#7 | 40/235 [03:44<01:50, 1.77it/s] 17%|#7 | 41/235 [03:44<01:42, 1.88it/s] 18%|#7 | 42/235 [03:45<01:38, 1.96it/s] 18%|#8 | 43/235 [03:45<01:35, 2.01it/s] 19%|#8 | 44/235 [03:46<01:47, 1.78it/s] 19%|#9 | 45/235 [03:46<01:40, 1.89it/s] 20%|#9 | 46/235 [03:47<01:35, 1.98it/s] 20%|## | 47/235 [03:47<01:31, 2.05it/s] 20%|## | 48/235 [03:48<01:29, 2.10it/s] 21%|## | 49/235 [03:48<01:27, 2.13it/s] 21%|##1 | 50/235 [03:49<01:26, 2.15it/s] 22%|##1 | 51/235 [03:49<01:25, 2.14it/s] 22%|##2 | 52/235 [03:50<01:24, 2.16it/s] 23%|##2 | 53/235 [03:50<01:23, 2.17it/s] 23%|##2 | 54/235 [03:51<01:23, 2.17it/s] 23%|##3 | 55/235 [03:51<01:22, 2.17it/s] 24%|##3 | 56/235 [03:51<01:21, 2.19it/s] 24%|##4 | 57/235 [03:52<01:20, 2.20it/s] 25%|##4 | 58/235 [03:52<01:20, 2.20it/s] 25%|##5 | 59/235 [03:53<01:19, 2.20it/s] 26%|##5 | 60/235 [03:53<01:19, 2.21it/s] 26%|##5 | 61/235 [03:54<01:19, 2.20it/s] 26%|##6 | 62/235 [03:54<01:18, 2.21it/s] 27%|##6 | 63/235 [03:55<01:21, 2.11it/s] 27%|##7 | 64/235 [03:55<01:24, 2.01it/s] 28%|##7 | 65/235 [03:56<01:23, 2.03it/s] 28%|##8 | 66/235 [03:56<01:21, 2.08it/s] 29%|##8 | 67/235 [03:57<01:19, 2.12it/s] 29%|##8 | 68/235 [03:57<01:18, 2.14it/s] 29%|##9 | 69/235 [03:58<01:16, 2.17it/s] 30%|##9 | 70/235 [03:58<01:15, 2.19it/s] 30%|### | 71/235 [03:58<01:14, 2.20it/s] 31%|### | 72/235 [03:59<01:14, 2.19it/s] 31%|###1 | 73/235 [04:00<01:25, 1.88it/s] 31%|###1 | 74/235 [04:00<01:21, 1.98it/s] 32%|###1 | 75/235 [04:00<01:18, 2.04it/s] 32%|###2 | 76/235 [04:01<01:16, 2.09it/s] 33%|###2 | 77/235 [04:01<01:15, 2.11it/s] 33%|###3 | 78/235 [04:02<01:13, 2.13it/s] 34%|###3 | 79/235 [04:03<01:25, 1.83it/s] 34%|###4 | 80/235 [04:03<01:19, 1.94it/s] 34%|###4 | 81/235 [04:03<01:16, 2.01it/s] 35%|###4 | 82/235 [04:04<01:14, 2.07it/s] 35%|###5 | 83/235 [04:04<01:12, 2.11it/s] 36%|###5 | 84/235 [04:05<01:12, 2.07it/s] 36%|###6 | 85/235 [04:05<01:10, 2.11it/s] 37%|###6 | 86/235 [04:06<01:09, 2.14it/s] 37%|###7 | 87/235 [04:06<01:08, 2.16it/s] 37%|###7 | 88/235 [04:07<01:07, 2.17it/s] 38%|###7 | 89/235 [04:07<01:13, 1.97it/s] 38%|###8 | 90/235 [04:08<01:33, 1.55it/s] 39%|###8 | 91/235 [04:09<01:31, 1.58it/s] 39%|###9 | 92/235 [04:10<01:46, 1.34it/s] 40%|###9 | 93/235 [04:10<01:33, 1.52it/s] 40%|#### | 94/235 [04:11<01:23, 1.68it/s] 40%|#### | 95/235 [04:11<01:17, 1.81it/s] 41%|#### | 96/235 [04:12<01:12, 1.92it/s] 41%|####1 | 97/235 [04:12<01:08, 2.00it/s] 42%|####1 | 98/235 [04:13<01:06, 2.07it/s] 42%|####2 | 99/235 [04:13<01:12, 1.88it/s] 43%|####2 | 100/235 [04:14<01:08, 1.97it/s] 43%|####2 | 101/235 [04:14<01:06, 2.02it/s] 43%|####3 | 102/235 [04:15<01:06, 2.01it/s] 44%|####3 | 103/235 [04:15<01:03, 2.07it/s] 44%|####4 | 104/235 [04:16<01:04, 2.02it/s] 45%|####4 | 105/235 [04:16<01:14, 1.75it/s] 45%|####5 | 106/235 [04:17<01:09, 1.87it/s] 46%|####5 | 107/235 [04:17<01:05, 1.95it/s] 46%|####5 | 108/235 [04:18<01:11, 1.78it/s] 46%|####6 | 109/235 [04:18<01:07, 1.87it/s] 47%|####6 | 110/235 [04:19<01:04, 1.95it/s] 47%|####7 | 111/235 [04:19<01:01, 2.02it/s] 48%|####7 | 112/235 [04:20<00:59, 2.06it/s] 48%|####8 | 113/235 [04:20<00:58, 2.10it/s] 49%|####8 | 114/235 [04:21<00:56, 2.14it/s] 49%|####8 | 115/235 [04:21<00:55, 2.15it/s] 49%|####9 | 116/235 [04:22<00:54, 2.17it/s] 50%|####9 | 117/235 [04:22<00:53, 2.19it/s] 50%|##### | 118/235 [04:23<00:53, 2.19it/s] 51%|##### | 119/235 [04:23<00:53, 2.16it/s] 51%|#####1 | 120/235 [04:23<00:52, 2.17it/s] 51%|#####1 | 121/235 [04:24<00:54, 2.10it/s] 52%|#####1 | 122/235 [04:24<00:53, 2.13it/s] 52%|#####2 | 123/235 [04:25<00:52, 2.12it/s] 53%|#####2 | 124/235 [04:25<00:52, 2.13it/s] 53%|#####3 | 125/235 [04:26<00:52, 2.08it/s] 54%|#####3 | 126/235 [04:26<00:51, 2.12it/s] 54%|#####4 | 127/235 [04:27<00:50, 2.15it/s] 54%|#####4 | 128/235 [04:27<00:49, 2.16it/s] 55%|#####4 | 129/235 [04:28<00:48, 2.17it/s] 55%|#####5 | 130/235 [04:28<00:48, 2.18it/s] 56%|#####5 | 131/235 [04:29<00:47, 2.18it/s] 56%|#####6 | 132/235 [04:29<00:47, 2.18it/s] 57%|#####6 | 133/235 [04:30<00:46, 2.17it/s] 57%|#####7 | 134/235 [04:30<00:46, 2.17it/s] 57%|#####7 | 135/235 [04:30<00:45, 2.18it/s] 58%|#####7 | 136/235 [04:31<00:45, 2.19it/s] 58%|#####8 | 137/235 [04:31<00:44, 2.18it/s] 59%|#####8 | 138/235 [04:32<00:44, 2.17it/s] 59%|#####9 | 139/235 [04:32<00:43, 2.19it/s] 60%|#####9 | 140/235 [04:33<00:43, 2.18it/s] 60%|###### | 141/235 [04:33<00:42, 2.20it/s] 60%|###### | 142/235 [04:34<00:42, 2.21it/s] 61%|###### | 143/235 [04:34<00:42, 2.18it/s] 61%|######1 | 144/235 [04:35<00:41, 2.20it/s] 62%|######1 | 145/235 [04:35<00:43, 2.08it/s] 62%|######2 | 146/235 [04:36<00:52, 1.70it/s] 63%|######2 | 147/235 [04:36<00:48, 1.82it/s] 63%|######2 | 148/235 [04:37<00:45, 1.93it/s] 63%|######3 | 149/235 [04:37<00:45, 1.90it/s] 64%|######3 | 150/235 [04:38<00:42, 1.98it/s] 64%|######4 | 151/235 [04:38<00:44, 1.88it/s] 65%|######4 | 152/235 [04:39<00:42, 1.96it/s] 65%|######5 | 153/235 [04:39<00:43, 1.88it/s] 66%|######5 | 154/235 [04:40<00:41, 1.96it/s] 66%|######5 | 155/235 [04:40<00:41, 1.91it/s] 66%|######6 | 156/235 [04:41<00:40, 1.97it/s] 67%|######6 | 157/235 [04:42<00:43, 1.80it/s] 67%|######7 | 158/235 [04:42<00:40, 1.91it/s] 68%|######7 | 159/235 [04:43<00:38, 2.00it/s] 68%|######8 | 160/235 [04:43<00:36, 2.05it/s] 69%|######8 | 161/235 [04:43<00:35, 2.11it/s] 69%|######8 | 162/235 [04:44<00:34, 2.11it/s] 69%|######9 | 163/235 [04:44<00:33, 2.14it/s] 70%|######9 | 164/235 [04:45<00:32, 2.15it/s] 70%|####### | 165/235 [04:45<00:32, 2.14it/s] 71%|####### | 166/235 [04:46<00:33, 2.06it/s] 71%|#######1 | 167/235 [04:46<00:32, 2.08it/s] 71%|#######1 | 168/235 [04:47<00:31, 2.11it/s] 72%|#######1 | 169/235 [04:47<00:31, 2.12it/s] 72%|#######2 | 170/235 [04:48<00:30, 2.11it/s] 73%|#######2 | 171/235 [04:48<00:30, 2.13it/s] 73%|#######3 | 172/235 [04:49<00:29, 2.15it/s] 74%|#######3 | 173/235 [04:49<00:29, 2.12it/s] 74%|#######4 | 174/235 [04:50<00:29, 2.08it/s] 74%|#######4 | 175/235 [04:50<00:28, 2.10it/s] 75%|#######4 | 176/235 [04:50<00:27, 2.14it/s] 75%|#######5 | 177/235 [04:51<00:26, 2.15it/s] 76%|#######5 | 178/235 [04:51<00:27, 2.11it/s] 76%|#######6 | 179/235 [04:52<00:26, 2.13it/s] 77%|#######6 | 180/235 [04:53<00:28, 1.93it/s] 77%|#######7 | 181/235 [04:53<00:27, 1.99it/s] 77%|#######7 | 182/235 [04:54<00:28, 1.84it/s] 78%|#######7 | 183/235 [04:54<00:26, 1.94it/s] 78%|#######8 | 184/235 [04:55<00:26, 1.94it/s] 79%|#######8 | 185/235 [04:55<00:29, 1.71it/s] 79%|#######9 | 186/235 [04:56<00:29, 1.66it/s] 80%|#######9 | 187/235 [04:56<00:26, 1.79it/s] 80%|######## | 188/235 [04:57<00:26, 1.78it/s] 80%|######## | 189/235 [04:57<00:24, 1.87it/s] 81%|######## | 190/235 [04:58<00:24, 1.84it/s] 81%|########1 | 191/235 [04:59<00:23, 1.90it/s] 82%|########1 | 192/235 [04:59<00:27, 1.57it/s] 82%|########2 | 193/235 [05:00<00:24, 1.69it/s] 83%|########2 | 194/235 [05:01<00:28, 1.45it/s] 83%|########2 | 195/235 [05:01<00:27, 1.47it/s] 83%|########3 | 196/235 [05:02<00:23, 1.64it/s] 84%|########3 | 197/235 [05:03<00:23, 1.61it/s] 84%|########4 | 198/235 [05:03<00:21, 1.75it/s] 85%|########4 | 199/235 [05:04<00:23, 1.53it/s] 85%|########5 | 200/235 [05:04<00:21, 1.65it/s] 86%|########5 | 201/235 [05:05<00:20, 1.69it/s] 86%|########5 | 202/235 [05:05<00:18, 1.83it/s] 86%|########6 | 203/235 [05:06<00:16, 1.92it/s] 87%|########6 | 204/235 [05:06<00:16, 1.89it/s] 87%|########7 | 205/235 [05:07<00:15, 1.97it/s] 88%|########7 | 206/235 [05:07<00:14, 1.96it/s] 88%|########8 | 207/235 [05:08<00:13, 2.03it/s] 89%|########8 | 208/235 [05:08<00:13, 2.06it/s] 89%|########8 | 209/235 [05:09<00:12, 2.10it/s] 89%|########9 | 210/235 [05:09<00:11, 2.14it/s] 90%|########9 | 211/235 [05:10<00:11, 2.16it/s] 90%|######### | 212/235 [05:10<00:10, 2.15it/s] 91%|######### | 213/235 [05:11<00:10, 2.17it/s] 91%|#########1| 214/235 [05:11<00:11, 1.90it/s] 91%|#########1| 215/235 [05:12<00:10, 1.98it/s] 92%|#########1| 216/235 [05:12<00:09, 2.04it/s] 92%|#########2| 217/235 [05:13<00:08, 2.10it/s] 93%|#########2| 218/235 [05:13<00:08, 1.89it/s] 93%|#########3| 219/235 [05:14<00:09, 1.68it/s] 94%|#########3| 220/235 [05:14<00:08, 1.78it/s] 94%|#########4| 221/235 [05:15<00:08, 1.65it/s] 94%|#########4| 222/235 [05:16<00:08, 1.61it/s] 95%|#########4| 223/235 [05:16<00:06, 1.73it/s] 95%|#########5| 224/235 [05:17<00:06, 1.66it/s] 96%|#########5| 225/235 [05:17<00:05, 1.79it/s] 96%|#########6| 226/235 [05:18<00:04, 1.86it/s] 97%|#########6| 227/235 [05:18<00:04, 1.88it/s] 97%|#########7| 228/235 [05:19<00:03, 1.97it/s] 97%|#########7| 229/235 [05:19<00:02, 2.03it/s] 98%|#########7| 230/235 [05:20<00:02, 2.08it/s] 98%|#########8| 231/235 [05:20<00:01, 2.08it/s] 99%|#########8| 232/235 [05:21<00:01, 2.12it/s] 99%|#########9| 233/235 [05:21<00:00, 2.14it/s] 100%|#########9| 234/235 [05:22<00:00, 2.14it/s] 100%|##########| 235/235 [05:22<00:00, 2.15it/s] [2017-07-18 14:55:09] - Scraper completed. 7050 documents retrieved. [2017-07-18 14:55:09] - ERROR - delete_elasticsearch_docs() - DoctorsScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/doctors/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd13d82d0>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/doctors/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd13d82d0>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 14:55:10] - ERROR - upload_data() - DoctorsScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd13d8350>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd13d8350>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 14:55:11] - ERROR - archive_data() - data/doctors.json - An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records. [2017-07-18 14:55:11] - Completed Scraper. [Foreign Doctors Scraper] [2017-07-18 14:55:11] Started Scraper. 0%| | 0/47 [00:00<?, ?it/s] 2%|2 | 1/47 [00:00<00:24, 1.88it/s] 4%|4 | 2/47 [00:01<00:23, 1.92it/s] 6%|6 | 3/47 [00:01<00:22, 1.99it/s] 9%|8 | 4/47 [00:01<00:20, 2.06it/s] 11%|# | 5/47 [00:02<00:20, 2.06it/s] 13%|#2 | 6/47 [00:03<00:22, 1.83it/s] 15%|#4 | 7/47 [00:03<00:25, 1.55it/s] 17%|#7 | 8/47 [00:04<00:22, 1.70it/s] 19%|#9 | 9/47 [00:04<00:20, 1.84it/s] 21%|##1 | 10/47 [00:05<00:19, 1.94it/s] 23%|##3 | 11/47 [00:05<00:18, 1.95it/s] 26%|##5 | 12/47 [00:06<00:17, 2.01it/s] 28%|##7 | 13/47 [00:06<00:16, 2.04it/s] 30%|##9 | 14/47 [00:07<00:15, 2.08it/s] 32%|###1 | 15/47 [00:07<00:15, 2.13it/s] 34%|###4 | 16/47 [00:08<00:14, 2.16it/s] 36%|###6 | 17/47 [00:08<00:13, 2.18it/s] 38%|###8 | 18/47 [00:09<00:13, 2.20it/s] 40%|#### | 19/47 [00:09<00:12, 2.20it/s] 43%|####2 | 20/47 [00:09<00:12, 2.21it/s] 45%|####4 | 21/47 [00:10<00:11, 2.22it/s] 47%|####6 | 22/47 [00:10<00:11, 2.22it/s] 49%|####8 | 23/47 [00:11<00:11, 2.10it/s] 51%|#####1 | 24/47 [00:11<00:10, 2.15it/s] 53%|#####3 | 25/47 [00:12<00:10, 2.18it/s] 55%|#####5 | 26/47 [00:12<00:09, 2.20it/s] 57%|#####7 | 27/47 [00:13<00:09, 2.21it/s] 60%|#####9 | 28/47 [00:13<00:08, 2.21it/s] 62%|######1 | 29/47 [00:14<00:08, 2.21it/s] 64%|######3 | 30/47 [00:14<00:07, 2.22it/s] 66%|######5 | 31/47 [00:14<00:07, 2.21it/s] 68%|######8 | 32/47 [00:15<00:07, 2.11it/s] 70%|####### | 33/47 [00:15<00:06, 2.16it/s] 72%|#######2 | 34/47 [00:16<00:05, 2.18it/s] 74%|#######4 | 35/47 [00:16<00:05, 2.20it/s] 77%|#######6 | 36/47 [00:17<00:04, 2.20it/s] 79%|#######8 | 37/47 [00:17<00:04, 2.13it/s] 81%|######## | 38/47 [00:18<00:04, 2.15it/s] 83%|########2 | 39/47 [00:18<00:03, 2.18it/s] 85%|########5 | 40/47 [00:19<00:03, 2.19it/s] 87%|########7 | 41/47 [00:19<00:02, 2.20it/s] 89%|########9 | 42/47 [00:20<00:02, 2.17it/s] 91%|#########1| 43/47 [00:20<00:01, 2.15it/s] 94%|#########3| 44/47 [00:20<00:01, 2.18it/s] 96%|#########5| 45/47 [00:21<00:00, 2.18it/s] 98%|#########7| 46/47 [00:21<00:00, 2.20it/s] 100%|##########| 47/47 [00:22<00:00, 2.21it/s] [2017-07-18 14:55:34] - Scraper completed. 1404 documents retrieved. [2017-07-18 14:55:34] - ERROR - delete_elasticsearch_docs() - ForeignDoctorsScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/doctors/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd1594690>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/doctors/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd1594690>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 14:55:34] - ERROR - upload_data() - ForeignDoctorsScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd09b6ad0>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd09b6ad0>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 14:55:35] - ERROR - archive_data() - data/foreign_doctors.json - An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records. [2017-07-18 14:55:36] - Completed Scraper. [Clinical Officers Scraper] [2017-07-18 14:55:36] Started Scraper. 0%| | 0/235 [00:00<?, ?it/s] 0%| | 1/235 [00:08<32:59, 8.46s/it] 1%| | 2/235 [00:16<32:51, 8.46s/it] 1%|1 | 3/235 [00:25<32:44, 8.47s/it] 2%|1 | 4/235 [00:33<32:34, 8.46s/it] 2%|2 | 5/235 [00:42<32:29, 8.47s/it] 3%|2 | 6/235 [00:50<32:18, 8.47s/it] 3%|2 | 7/235 [00:59<32:09, 8.46s/it] 3%|3 | 8/235 [01:07<32:03, 8.48s/it] 4%|3 | 9/235 [01:16<31:55, 8.48s/it] 4%|4 | 10/235 [01:24<31:49, 8.48s/it] 5%|4 | 11/235 [01:33<31:41, 8.49s/it] 5%|5 | 12/235 [01:41<31:30, 8.48s/it] 6%|5 | 13/235 [01:50<31:20, 8.47s/it] 6%|5 | 14/235 [01:58<31:11, 8.47s/it] 6%|6 | 15/235 [02:07<31:02, 8.46s/it] 7%|6 | 16/235 [02:15<31:09, 8.54s/it] 7%|7 | 17/235 [02:24<31:03, 8.55s/it] 8%|7 | 18/235 [02:33<31:03, 8.59s/it] 8%|8 | 19/235 [02:41<30:45, 8.55s/it] 9%|8 | 20/235 [02:49<30:31, 8.52s/it] 9%|8 | 21/235 [02:58<30:18, 8.50s/it] 9%|9 | 22/235 [03:07<30:19, 8.54s/it] 10%|9 | 23/235 [03:15<30:09, 8.53s/it] 10%|# | 24/235 [03:24<29:57, 8.52s/it] 11%|# | 25/235 [03:32<29:50, 8.52s/it] 11%|#1 | 26/235 [03:37<25:29, 7.32s/it] 11%|#1 | 27/235 [03:37<18:15, 5.27s/it] 12%|#1 | 28/235 [03:38<13:18, 3.86s/it] 12%|#2 | 29/235 [03:38<09:44, 2.84s/it] 13%|#2 | 30/235 [03:39<07:19, 2.14s/it] 13%|#3 | 31/235 [03:39<05:35, 1.65s/it] 14%|#3 | 32/235 [03:40<04:20, 1.29s/it] 14%|#4 | 33/235 [03:40<03:40, 1.09s/it] 14%|#4 | 34/235 [03:41<03:06, 1.08it/s] 15%|#4 | 35/235 [03:41<02:50, 1.17it/s] 15%|#5 | 36/235 [03:42<02:27, 1.35it/s] 16%|#5 | 37/235 [03:42<02:09, 1.53it/s] 16%|#6 | 38/235 [03:43<02:02, 1.61it/s] 17%|#6 | 39/235 [03:43<01:51, 1.76it/s] 17%|#7 | 40/235 [03:44<01:50, 1.77it/s] 17%|#7 | 41/235 [03:44<01:42, 1.88it/s] 18%|#7 | 42/235 [03:45<01:38, 1.96it/s] 18%|#8 | 43/235 [03:45<01:35, 2.01it/s] 19%|#8 | 44/235 [03:46<01:47, 1.78it/s] 19%|#9 | 45/235 [03:46<01:40, 1.89it/s] 20%|#9 | 46/235 [03:47<01:35, 1.98it/s] 20%|## | 47/235 [03:47<01:31, 2.05it/s] 20%|## | 48/235 [03:48<01:29, 2.10it/s] 21%|## | 49/235 [03:48<01:27, 2.13it/s] 21%|##1 | 50/235 [03:49<01:26, 2.15it/s] 22%|##1 | 51/235 [03:49<01:25, 2.14it/s] 22%|##2 | 52/235 [03:50<01:24, 2.16it/s] 23%|##2 | 53/235 [03:50<01:23, 2.17it/s] 23%|##2 | 54/235 [03:51<01:23, 2.17it/s] 23%|##3 | 55/235 [03:51<01:22, 2.17it/s] 24%|##3 | 56/235 [03:51<01:21, 2.19it/s] 24%|##4 | 57/235 [03:52<01:20, 2.20it/s] 25%|##4 | 58/235 [03:52<01:20, 2.20it/s] 25%|##5 | 59/235 [03:53<01:19, 2.20it/s] 26%|##5 | 60/235 [03:53<01:19, 2.21it/s] 26%|##5 | 61/235 [03:54<01:19, 2.20it/s] 26%|##6 | 62/235 [03:54<01:18, 2.21it/s] 27%|##6 | 63/235 [03:55<01:21, 2.11it/s] 27%|##7 | 64/235 [03:55<01:24, 2.01it/s] 28%|##7 | 65/235 [03:56<01:23, 2.03it/s] 28%|##8 | 66/235 [03:56<01:21, 2.08it/s] 29%|##8 | 67/235 [03:57<01:19, 2.12it/s] 29%|##8 | 68/235 [03:57<01:18, 2.14it/s] 29%|##9 | 69/235 [03:58<01:16, 2.17it/s] 30%|##9 | 70/235 [03:58<01:15, 2.19it/s] 30%|### | 71/235 [03:58<01:14, 2.20it/s] 31%|### | 72/235 [03:59<01:14, 2.19it/s] 31%|###1 | 73/235 [04:00<01:25, 1.88it/s] 31%|###1 | 74/235 [04:00<01:21, 1.98it/s] 32%|###1 | 75/235 [04:00<01:18, 2.04it/s] 32%|###2 | 76/235 [04:01<01:16, 2.09it/s] 33%|###2 | 77/235 [04:01<01:15, 2.11it/s] 33%|###3 | 78/235 [04:02<01:13, 2.13it/s] 34%|###3 | 79/235 [04:03<01:25, 1.83it/s] 34%|###4 | 80/235 [04:03<01:19, 1.94it/s] 34%|###4 | 81/235 [04:03<01:16, 2.01it/s] 35%|###4 | 82/235 [04:04<01:14, 2.07it/s] 35%|###5 | 83/235 [04:04<01:12, 2.11it/s] 36%|###5 | 84/235 [04:05<01:12, 2.07it/s] 36%|###6 | 85/235 [04:05<01:10, 2.11it/s] 37%|###6 | 86/235 [04:06<01:09, 2.14it/s] 37%|###7 | 87/235 [04:06<01:08, 2.16it/s] 37%|###7 | 88/235 [04:07<01:07, 2.17it/s] 38%|###7 | 89/235 [04:07<01:13, 1.97it/s] 38%|###8 | 90/235 [04:08<01:33, 1.55it/s] 39%|###8 | 91/235 [04:09<01:31, 1.58it/s] 39%|###9 | 92/235 [04:10<01:46, 1.34it/s] 40%|###9 | 93/235 [04:10<01:33, 1.52it/s] 40%|#### | 94/235 [04:11<01:23, 1.68it/s] 40%|#### | 95/235 [04:11<01:17, 1.81it/s] 41%|#### | 96/235 [04:12<01:12, 1.92it/s] 41%|####1 | 97/235 [04:12<01:08, 2.00it/s] 42%|####1 | 98/235 [04:13<01:06, 2.07it/s] 42%|####2 | 99/235 [04:13<01:12, 1.88it/s] 43%|####2 | 100/235 [04:14<01:08, 1.97it/s] 43%|####2 | 101/235 [04:14<01:06, 2.02it/s] 43%|####3 | 102/235 [04:15<01:06, 2.01it/s] 44%|####3 | 103/235 [04:15<01:03, 2.07it/s] 44%|####4 | 104/235 [04:16<01:04, 2.02it/s] 45%|####4 | 105/235 [04:16<01:14, 1.75it/s] 45%|####5 | 106/235 [04:17<01:09, 1.87it/s] 46%|####5 | 107/235 [04:17<01:05, 1.95it/s] 46%|####5 | 108/235 [04:18<01:11, 1.78it/s] 46%|####6 | 109/235 [04:18<01:07, 1.87it/s] 47%|####6 | 110/235 [04:19<01:04, 1.95it/s] 47%|####7 | 111/235 [04:19<01:01, 2.02it/s] 48%|####7 | 112/235 [04:20<00:59, 2.06it/s] 48%|####8 | 113/235 [04:20<00:58, 2.10it/s] 49%|####8 | 114/235 [04:21<00:56, 2.14it/s] 49%|####8 | 115/235 [04:21<00:55, 2.15it/s] 49%|####9 | 116/235 [04:22<00:54, 2.17it/s] 50%|####9 | 117/235 [04:22<00:53, 2.19it/s] 50%|##### | 118/235 [04:23<00:53, 2.19it/s] 51%|##### | 119/235 [04:23<00:53, 2.16it/s] 51%|#####1 | 120/235 [04:23<00:52, 2.17it/s] 51%|#####1 | 121/235 [04:24<00:54, 2.10it/s] 52%|#####1 | 122/235 [04:24<00:53, 2.13it/s] 52%|#####2 | 123/235 [04:25<00:52, 2.12it/s] 53%|#####2 | 124/235 [04:25<00:52, 2.13it/s] 53%|#####3 | 125/235 [04:26<00:52, 2.08it/s] 54%|#####3 | 126/235 [04:26<00:51, 2.12it/s] 54%|#####4 | 127/235 [04:27<00:50, 2.15it/s] 54%|#####4 | 128/235 [04:27<00:49, 2.16it/s] 55%|#####4 | 129/235 [04:28<00:48, 2.17it/s] 55%|#####5 | 130/235 [04:28<00:48, 2.18it/s] 56%|#####5 | 131/235 [04:29<00:47, 2.18it/s] 56%|#####6 | 132/235 [04:29<00:47, 2.18it/s] 57%|#####6 | 133/235 [04:30<00:46, 2.17it/s] 57%|#####7 | 134/235 [04:30<00:46, 2.17it/s] 57%|#####7 | 135/235 [04:30<00:45, 2.18it/s] 58%|#####7 | 136/235 [04:31<00:45, 2.19it/s] 58%|#####8 | 137/235 [04:31<00:44, 2.18it/s] 59%|#####8 | 138/235 [04:32<00:44, 2.17it/s] 59%|#####9 | 139/235 [04:32<00:43, 2.19it/s] 60%|#####9 | 140/235 [04:33<00:43, 2.18it/s] 60%|###### | 141/235 [04:33<00:42, 2.20it/s] 60%|###### | 142/235 [04:34<00:42, 2.21it/s] 61%|###### | 143/235 [04:34<00:42, 2.18it/s] 61%|######1 | 144/235 [04:35<00:41, 2.20it/s] 62%|######1 | 145/235 [04:35<00:43, 2.08it/s] 62%|######2 | 146/235 [04:36<00:52, 1.70it/s] 63%|######2 | 147/235 [04:36<00:48, 1.82it/s] 63%|######2 | 148/235 [04:37<00:45, 1.93it/s] 63%|######3 | 149/235 [04:37<00:45, 1.90it/s] 64%|######3 | 150/235 [04:38<00:42, 1.98it/s] 64%|######4 | 151/235 [04:38<00:44, 1.88it/s] 65%|######4 | 152/235 [04:39<00:42, 1.96it/s] 65%|######5 | 153/235 [04:39<00:43, 1.88it/s] 66%|######5 | 154/235 [04:40<00:41, 1.96it/s] 66%|######5 | 155/235 [04:40<00:41, 1.91it/s] 66%|######6 | 156/235 [04:41<00:40, 1.97it/s] 67%|######6 | 157/235 [04:42<00:43, 1.80it/s] 67%|######7 | 158/235 [04:42<00:40, 1.91it/s] 68%|######7 | 159/235 [04:43<00:38, 2.00it/s] 68%|######8 | 160/235 [04:43<00:36, 2.05it/s] 69%|######8 | 161/235 [04:43<00:35, 2.11it/s] 69%|######8 | 162/235 [04:44<00:34, 2.11it/s] 69%|######9 | 163/235 [04:44<00:33, 2.14it/s] 70%|######9 | 164/235 [04:45<00:32, 2.15it/s] 70%|####### | 165/235 [04:45<00:32, 2.14it/s] 71%|####### | 166/235 [04:46<00:33, 2.06it/s] 71%|#######1 | 167/235 [04:46<00:32, 2.08it/s] 71%|#######1 | 168/235 [04:47<00:31, 2.11it/s] 72%|#######1 | 169/235 [04:47<00:31, 2.12it/s] 72%|#######2 | 170/235 [04:48<00:30, 2.11it/s] 73%|#######2 | 171/235 [04:48<00:30, 2.13it/s] 73%|#######3 | 172/235 [04:49<00:29, 2.15it/s] 74%|#######3 | 173/235 [04:49<00:29, 2.12it/s] 74%|#######4 | 174/235 [04:50<00:29, 2.08it/s] 74%|#######4 | 175/235 [04:50<00:28, 2.10it/s] 75%|#######4 | 176/235 [04:50<00:27, 2.14it/s] 75%|#######5 | 177/235 [04:51<00:26, 2.15it/s] 76%|#######5 | 178/235 [04:51<00:27, 2.11it/s] 76%|#######6 | 179/235 [04:52<00:26, 2.13it/s] 77%|#######6 | 180/235 [04:53<00:28, 1.93it/s] 77%|#######7 | 181/235 [04:53<00:27, 1.99it/s] 77%|#######7 | 182/235 [04:54<00:28, 1.84it/s] 78%|#######7 | 183/235 [04:54<00:26, 1.94it/s] 78%|#######8 | 184/235 [04:55<00:26, 1.94it/s] 79%|#######8 | 185/235 [04:55<00:29, 1.71it/s] 79%|#######9 | 186/235 [04:56<00:29, 1.66it/s] 80%|#######9 | 187/235 [04:56<00:26, 1.79it/s] 80%|######## | 188/235 [04:57<00:26, 1.78it/s] 80%|######## | 189/235 [04:57<00:24, 1.87it/s] 81%|######## | 190/235 [04:58<00:24, 1.84it/s] 81%|########1 | 191/235 [04:59<00:23, 1.90it/s] 82%|########1 | 192/235 [04:59<00:27, 1.57it/s] 82%|########2 | 193/235 [05:00<00:24, 1.69it/s] 83%|########2 | 194/235 [05:01<00:28, 1.45it/s] 83%|########2 | 195/235 [05:01<00:27, 1.47it/s] 83%|########3 | 196/235 [05:02<00:23, 1.64it/s] 84%|########3 | 197/235 [05:03<00:23, 1.61it/s] 84%|########4 | 198/235 [05:03<00:21, 1.75it/s] 85%|########4 | 199/235 [05:04<00:23, 1.53it/s] 85%|########5 | 200/235 [05:04<00:21, 1.65it/s] 86%|########5 | 201/235 [05:05<00:20, 1.69it/s] 86%|########5 | 202/235 [05:05<00:18, 1.83it/s] 86%|########6 | 203/235 [05:06<00:16, 1.92it/s] 87%|########6 | 204/235 [05:06<00:16, 1.89it/s] 87%|########7 | 205/235 [05:07<00:15, 1.97it/s] 88%|########7 | 206/235 [05:07<00:14, 1.96it/s] 88%|########8 | 207/235 [05:08<00:13, 2.03it/s] 89%|########8 | 208/235 [05:08<00:13, 2.06it/s] 89%|########8 | 209/235 [05:09<00:12, 2.10it/s] 89%|########9 | 210/235 [05:09<00:11, 2.14it/s] 90%|########9 | 211/235 [05:10<00:11, 2.16it/s] 90%|######### | 212/235 [05:10<00:10, 2.15it/s] 91%|######### | 213/235 [05:11<00:10, 2.17it/s] 91%|#########1| 214/235 [05:11<00:11, 1.90it/s] 91%|#########1| 215/235 [05:12<00:10, 1.98it/s] 92%|#########1| 216/235 [05:12<00:09, 2.04it/s] 92%|#########2| 217/235 [05:13<00:08, 2.10it/s] 93%|#########2| 218/235 [05:13<00:08, 1.89it/s] 93%|#########3| 219/235 [05:14<00:09, 1.68it/s] 94%|#########3| 220/235 [05:14<00:08, 1.78it/s] 94%|#########4| 221/235 [05:15<00:08, 1.65it/s] 94%|#########4| 222/235 [05:16<00:08, 1.61it/s] 95%|#########4| 223/235 [05:16<00:06, 1.73it/s] 95%|#########5| 224/235 [05:17<00:06, 1.66it/s] 96%|#########5| 225/235 [05:17<00:05, 1.79it/s] 96%|#########6| 226/235 [05:18<00:04, 1.86it/s] 97%|#########6| 227/235 [05:18<00:04, 1.88it/s] 97%|#########7| 228/235 [05:19<00:03, 1.97it/s] 97%|#########7| 229/235 [05:19<00:02, 2.03it/s] 98%|#########7| 230/235 [05:20<00:02, 2.08it/s] 98%|#########8| 231/235 [05:20<00:01, 2.08it/s] 99%|#########8| 232/235 [05:21<00:01, 2.12it/s] 99%|#########9| 233/235 [05:21<00:00, 2.14it/s] 100%|#########9| 234/235 [05:22<00:00, 2.14it/s] 100%|##########| 235/235 [05:22<00:00, 2.15it/s] [2017-07-18 14:55:09] - Scraper completed. 7050 documents retrieved. [2017-07-18 14:55:09] - ERROR - delete_elasticsearch_docs() - DoctorsScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/doctors/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd13d82d0>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/doctors/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd13d82d0>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 14:55:10] - ERROR - upload_data() - DoctorsScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd13d8350>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd13d8350>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 14:55:11] - ERROR - archive_data() - data/doctors.json - An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records. [2017-07-18 14:55:11] - Completed Scraper. [Foreign Doctors Scraper] [2017-07-18 14:55:11] Started Scraper. 0%| | 0/47 [00:00<?, ?it/s] 2%|2 | 1/47 [00:00<00:24, 1.88it/s] 4%|4 | 2/47 [00:01<00:23, 1.92it/s] 6%|6 | 3/47 [00:01<00:22, 1.99it/s] 9%|8 | 4/47 [00:01<00:20, 2.06it/s] 11%|# | 5/47 [00:02<00:20, 2.06it/s] 13%|#2 | 6/47 [00:03<00:22, 1.83it/s] 15%|#4 | 7/47 [00:03<00:25, 1.55it/s] 17%|#7 | 8/47 [00:04<00:22, 1.70it/s] 19%|#9 | 9/47 [00:04<00:20, 1.84it/s] 21%|##1 | 10/47 [00:05<00:19, 1.94it/s] 23%|##3 | 11/47 [00:05<00:18, 1.95it/s] 26%|##5 | 12/47 [00:06<00:17, 2.01it/s] 28%|##7 | 13/47 [00:06<00:16, 2.04it/s] 30%|##9 | 14/47 [00:07<00:15, 2.08it/s] 32%|###1 | 15/47 [00:07<00:15, 2.13it/s] 34%|###4 | 16/47 [00:08<00:14, 2.16it/s] 36%|###6 | 17/47 [00:08<00:13, 2.18it/s] 38%|###8 | 18/47 [00:09<00:13, 2.20it/s] 40%|#### | 19/47 [00:09<00:12, 2.20it/s] 43%|####2 | 20/47 [00:09<00:12, 2.21it/s] 45%|####4 | 21/47 [00:10<00:11, 2.22it/s] 47%|####6 | 22/47 [00:10<00:11, 2.22it/s] 49%|####8 | 23/47 [00:11<00:11, 2.10it/s] 51%|#####1 | 24/47 [00:11<00:10, 2.15it/s] 53%|#####3 | 25/47 [00:12<00:10, 2.18it/s] 55%|#####5 | 26/47 [00:12<00:09, 2.20it/s] 57%|#####7 | 27/47 [00:13<00:09, 2.21it/s] 60%|#####9 | 28/47 [00:13<00:08, 2.21it/s] 62%|######1 | 29/47 [00:14<00:08, 2.21it/s] 64%|######3 | 30/47 [00:14<00:07, 2.22it/s] 66%|######5 | 31/47 [00:14<00:07, 2.21it/s] 68%|######8 | 32/47 [00:15<00:07, 2.11it/s] 70%|####### | 33/47 [00:15<00:06, 2.16it/s] 72%|#######2 | 34/47 [00:16<00:05, 2.18it/s] 74%|#######4 | 35/47 [00:16<00:05, 2.20it/s] 77%|#######6 | 36/47 [00:17<00:04, 2.20it/s] 79%|#######8 | 37/47 [00:17<00:04, 2.13it/s] 81%|######## | 38/47 [00:18<00:04, 2.15it/s] 83%|########2 | 39/47 [00:18<00:03, 2.18it/s] 85%|########5 | 40/47 [00:19<00:03, 2.19it/s] 87%|########7 | 41/47 [00:19<00:02, 2.20it/s] 89%|########9 | 42/47 [00:20<00:02, 2.17it/s] 91%|#########1| 43/47 [00:20<00:01, 2.15it/s] 94%|#########3| 44/47 [00:20<00:01, 2.18it/s] 96%|#########5| 45/47 [00:21<00:00, 2.18it/s] 98%|#########7| 46/47 [00:21<00:00, 2.20it/s] 100%|##########| 47/47 [00:22<00:00, 2.21it/s] [2017-07-18 14:55:34] - Scraper completed. 1404 documents retrieved. [2017-07-18 14:55:34] - ERROR - delete_elasticsearch_docs() - ForeignDoctorsScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/doctors/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd1594690>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/doctors/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd1594690>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 14:55:34] - ERROR - upload_data() - ForeignDoctorsScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd09b6ad0>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd09b6ad0>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 14:55:35] - ERROR - archive_data() - data/foreign_doctors.json - An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records. [2017-07-18 14:55:36] - Completed Scraper. [Clinical Officers Scraper] [2017-07-18 14:55:36] Started Scraper. 0%| | 0/235 [00:00<?, ?it/s] 0%| | 1/235 [00:08<32:59, 8.46s/it] 1%| | 2/235 [00:16<32:51, 8.46s/it] 1%|1 | 3/235 [00:25<32:44, 8.47s/it] 2%|1 | 4/235 [00:33<32:34, 8.46s/it] 2%|2 | 5/235 [00:42<32:29, 8.47s/it] 3%|2 | 6/235 [00:50<32:18, 8.47s/it] 3%|2 | 7/235 [00:59<32:09, 8.46s/it] 3%|3 | 8/235 [01:07<32:03, 8.48s/it] 4%|3 | 9/235 [01:16<31:55, 8.48s/it] 4%|4 | 10/235 [01:24<31:49, 8.48s/it] 5%|4 | 11/235 [01:33<31:41, 8.49s/it] 5%|5 | 12/235 [01:41<31:30, 8.48s/it] 6%|5 | 13/235 [01:50<31:20, 8.47s/it] 6%|5 | 14/235 [01:58<31:11, 8.47s/it] 6%|6 | 15/235 [02:07<31:02, 8.46s/it] 7%|6 | 16/235 [02:15<31:09, 8.54s/it] 7%|7 | 17/235 [02:24<31:03, 8.55s/it] 8%|7 | 18/235 [02:33<31:03, 8.59s/it] 8%|8 | 19/235 [02:41<30:45, 8.55s/it] 9%|8 | 20/235 [02:49<30:31, 8.52s/it] 9%|8 | 21/235 [02:58<30:18, 8.50s/it] 9%|9 | 22/235 [03:07<30:19, 8.54s/it] 10%|9 | 23/235 [03:15<30:09, 8.53s/it] 10%|# | 24/235 [03:24<29:57, 8.52s/it] 11%|# | 25/235 [03:32<29:50, 8.52s/it] 11%|#1 | 26/235 [03:37<25:29, 7.32s/it] 11%|#1 | 27/235 [03:37<18:15, 5.27s/it] 12%|#1 | 28/235 [03:38<13:18, 3.86s/it] 12%|#2 | 29/235 [03:38<09:44, 2.84s/it] 13%|#2 | 30/235 [03:39<07:19, 2.14s/it] 13%|#3 | 31/235 [03:39<05:35, 1.65s/it] 14%|#3 | 32/235 [03:40<04:20, 1.29s/it] 14%|#4 | 33/235 [03:40<03:40, 1.09s/it] 14%|#4 | 34/235 [03:41<03:06, 1.08it/s] 15%|#4 | 35/235 [03:41<02:50, 1.17it/s] 15%|#5 | 36/235 [03:42<02:27, 1.35it/s] 16%|#5 | 37/235 [03:42<02:09, 1.53it/s] 16%|#6 | 38/235 [03:43<02:02, 1.61it/s] 17%|#6 | 39/235 [03:43<01:51, 1.76it/s] 17%|#7 | 40/235 [03:44<01:50, 1.77it/s] 17%|#7 | 41/235 [03:44<01:42, 1.88it/s] 18%|#7 | 42/235 [03:45<01:38, 1.96it/s] 18%|#8 | 43/235 [03:45<01:35, 2.01it/s] 19%|#8 | 44/235 [03:46<01:47, 1.78it/s] 19%|#9 | 45/235 [03:46<01:40, 1.89it/s] 20%|#9 | 46/235 [03:47<01:35, 1.98it/s] 20%|## | 47/235 [03:47<01:31, 2.05it/s] 20%|## | 48/235 [03:48<01:29, 2.10it/s] 21%|## | 49/235 [03:48<01:27, 2.13it/s] 21%|##1 | 50/235 [03:49<01:26, 2.15it/s] 22%|##1 | 51/235 [03:49<01:25, 2.14it/s] 22%|##2 | 52/235 [03:50<01:24, 2.16it/s] 23%|##2 | 53/235 [03:50<01:23, 2.17it/s] 23%|##2 | 54/235 [03:51<01:23, 2.17it/s] 23%|##3 | 55/235 [03:51<01:22, 2.17it/s] 24%|##3 | 56/235 [03:51<01:21, 2.19it/s] 24%|##4 | 57/235 [03:52<01:20, 2.20it/s] 25%|##4 | 58/235 [03:52<01:20, 2.20it/s] 25%|##5 | 59/235 [03:53<01:19, 2.20it/s] 26%|##5 | 60/235 [03:53<01:19, 2.21it/s] 26%|##5 | 61/235 [03:54<01:19, 2.20it/s] 26%|##6 | 62/235 [03:54<01:18, 2.21it/s] 27%|##6 | 63/235 [03:55<01:21, 2.11it/s] 27%|##7 | 64/235 [03:55<01:24, 2.01it/s] 28%|##7 | 65/235 [03:56<01:23, 2.03it/s] 28%|##8 | 66/235 [03:56<01:21, 2.08it/s] 29%|##8 | 67/235 [03:57<01:19, 2.12it/s] 29%|##8 | 68/235 [03:57<01:18, 2.14it/s] 29%|##9 | 69/235 [03:58<01:16, 2.17it/s] 30%|##9 | 70/235 [03:58<01:15, 2.19it/s] 30%|### | 71/235 [03:58<01:14, 2.20it/s] 31%|### | 72/235 [03:59<01:14, 2.19it/s] 31%|###1 | 73/235 [04:00<01:25, 1.88it/s] 31%|###1 | 74/235 [04:00<01:21, 1.98it/s] 32%|###1 | 75/235 [04:00<01:18, 2.04it/s] 32%|###2 | 76/235 [04:01<01:16, 2.09it/s] 33%|###2 | 77/235 [04:01<01:15, 2.11it/s] 33%|###3 | 78/235 [04:02<01:13, 2.13it/s] 34%|###3 | 79/235 [04:03<01:25, 1.83it/s] 34%|###4 | 80/235 [04:03<01:19, 1.94it/s] 34%|###4 | 81/235 [04:03<01:16, 2.01it/s] 35%|###4 | 82/235 [04:04<01:14, 2.07it/s] 35%|###5 | 83/235 [04:04<01:12, 2.11it/s] 36%|###5 | 84/235 [04:05<01:12, 2.07it/s] 36%|###6 | 85/235 [04:05<01:10, 2.11it/s] 37%|###6 | 86/235 [04:06<01:09, 2.14it/s] 37%|###7 | 87/235 [04:06<01:08, 2.16it/s] 37%|###7 | 88/235 [04:07<01:07, 2.17it/s] 38%|###7 | 89/235 [04:07<01:13, 1.97it/s] 38%|###8 | 90/235 [04:08<01:33, 1.55it/s] 39%|###8 | 91/235 [04:09<01:31, 1.58it/s] 39%|###9 | 92/235 [04:10<01:46, 1.34it/s] 40%|###9 | 93/235 [04:10<01:33, 1.52it/s] 40%|#### | 94/235 [04:11<01:23, 1.68it/s] 40%|#### | 95/235 [04:11<01:17, 1.81it/s] 41%|#### | 96/235 [04:12<01:12, 1.92it/s] 41%|####1 | 97/235 [04:12<01:08, 2.00it/s] 42%|####1 | 98/235 [04:13<01:06, 2.07it/s] 42%|####2 | 99/235 [04:13<01:12, 1.88it/s] 43%|####2 | 100/235 [04:14<01:08, 1.97it/s] 43%|####2 | 101/235 [04:14<01:06, 2.02it/s] 43%|####3 | 102/235 [04:15<01:06, 2.01it/s] 44%|####3 | 103/235 [04:15<01:03, 2.07it/s] 44%|####4 | 104/235 [04:16<01:04, 2.02it/s] 45%|####4 | 105/235 [04:16<01:14, 1.75it/s] 45%|####5 | 106/235 [04:17<01:09, 1.87it/s] 46%|####5 | 107/235 [04:17<01:05, 1.95it/s] 46%|####5 | 108/235 [04:18<01:11, 1.78it/s] 46%|####6 | 109/235 [04:18<01:07, 1.87it/s] 47%|####6 | 110/235 [04:19<01:04, 1.95it/s] 47%|####7 | 111/235 [04:19<01:01, 2.02it/s] 48%|####7 | 112/235 [04:20<00:59, 2.06it/s] 48%|####8 | 113/235 [04:20<00:58, 2.10it/s] 49%|####8 | 114/235 [04:21<00:56, 2.14it/s] 49%|####8 | 115/235 [04:21<00:55, 2.15it/s] 49%|####9 | 116/235 [04:22<00:54, 2.17it/s] 50%|####9 | 117/235 [04:22<00:53, 2.19it/s] 50%|##### | 118/235 [04:23<00:53, 2.19it/s] 51%|##### | 119/235 [04:23<00:53, 2.16it/s] 51%|#####1 | 120/235 [04:23<00:52, 2.17it/s] 51%|#####1 | 121/235 [04:24<00:54, 2.10it/s] 52%|#####1 | 122/235 [04:24<00:53, 2.13it/s] 52%|#####2 | 123/235 [04:25<00:52, 2.12it/s] 53%|#####2 | 124/235 [04:25<00:52, 2.13it/s] 53%|#####3 | 125/235 [04:26<00:52, 2.08it/s] 54%|#####3 | 126/235 [04:26<00:51, 2.12it/s] 54%|#####4 | 127/235 [04:27<00:50, 2.15it/s] 54%|#####4 | 128/235 [04:27<00:49, 2.16it/s] 55%|#####4 | 129/235 [04:28<00:48, 2.17it/s] 55%|#####5 | 130/235 [04:28<00:48, 2.18it/s] 56%|#####5 | 131/235 [04:29<00:47, 2.18it/s] 56%|#####6 | 132/235 [04:29<00:47, 2.18it/s] 57%|#####6 | 133/235 [04:30<00:46, 2.17it/s] 57%|#####7 | 134/235 [04:30<00:46, 2.17it/s] 57%|#####7 | 135/235 [04:30<00:45, 2.18it/s] 58%|#####7 | 136/235 [04:31<00:45, 2.19it/s] 58%|#####8 | 137/235 [04:31<00:44, 2.18it/s] 59%|#####8 | 138/235 [04:32<00:44, 2.17it/s] 59%|#####9 | 139/235 [04:32<00:43, 2.19it/s] 60%|#####9 | 140/235 [04:33<00:43, 2.18it/s] 60%|###### | 141/235 [04:33<00:42, 2.20it/s] 60%|###### | 142/235 [04:34<00:42, 2.21it/s] 61%|###### | 143/235 [04:34<00:42, 2.18it/s] 61%|######1 | 144/235 [04:35<00:41, 2.20it/s] 62%|######1 | 145/235 [04:35<00:43, 2.08it/s] 62%|######2 | 146/235 [04:36<00:52, 1.70it/s] 63%|######2 | 147/235 [04:36<00:48, 1.82it/s] 63%|######2 | 148/235 [04:37<00:45, 1.93it/s] 63%|######3 | 149/235 [04:37<00:45, 1.90it/s] 64%|######3 | 150/235 [04:38<00:42, 1.98it/s] 64%|######4 | 151/235 [04:38<00:44, 1.88it/s] 65%|######4 | 152/235 [04:39<00:42, 1.96it/s] 65%|######5 | 153/235 [04:39<00:43, 1.88it/s] 66%|######5 | 154/235 [04:40<00:41, 1.96it/s] 66%|######5 | 155/235 [04:40<00:41, 1.91it/s] 66%|######6 | 156/235 [04:41<00:40, 1.97it/s] 67%|######6 | 157/235 [04:42<00:43, 1.80it/s] 67%|######7 | 158/235 [04:42<00:40, 1.91it/s] 68%|######7 | 159/235 [04:43<00:38, 2.00it/s] 68%|######8 | 160/235 [04:43<00:36, 2.05it/s] 69%|######8 | 161/235 [04:43<00:35, 2.11it/s] 69%|######8 | 162/235 [04:44<00:34, 2.11it/s] 69%|######9 | 163/235 [04:44<00:33, 2.14it/s] 70%|######9 | 164/235 [04:45<00:32, 2.15it/s] 70%|####### | 165/235 [04:45<00:32, 2.14it/s] 71%|####### | 166/235 [04:46<00:33, 2.06it/s] 71%|#######1 | 167/235 [04:46<00:32, 2.08it/s] 71%|#######1 | 168/235 [04:47<00:31, 2.11it/s] 72%|#######1 | 169/235 [04:47<00:31, 2.12it/s] 72%|#######2 | 170/235 [04:48<00:30, 2.11it/s] 73%|#######2 | 171/235 [04:48<00:30, 2.13it/s] 73%|#######3 | 172/235 [04:49<00:29, 2.15it/s] 74%|#######3 | 173/235 [04:49<00:29, 2.12it/s] 74%|#######4 | 174/235 [04:50<00:29, 2.08it/s] 74%|#######4 | 175/235 [04:50<00:28, 2.10it/s] 75%|#######4 | 176/235 [04:50<00:27, 2.14it/s] 75%|#######5 | 177/235 [04:51<00:26, 2.15it/s] 76%|#######5 | 178/235 [04:51<00:27, 2.11it/s] 76%|#######6 | 179/235 [04:52<00:26, 2.13it/s] 77%|#######6 | 180/235 [04:53<00:28, 1.93it/s] 77%|#######7 | 181/235 [04:53<00:27, 1.99it/s] 77%|#######7 | 182/235 [04:54<00:28, 1.84it/s] 78%|#######7 | 183/235 [04:54<00:26, 1.94it/s] 78%|#######8 | 184/235 [04:55<00:26, 1.94it/s] 79%|#######8 | 185/235 [04:55<00:29, 1.71it/s] 79%|#######9 | 186/235 [04:56<00:29, 1.66it/s] 80%|#######9 | 187/235 [04:56<00:26, 1.79it/s] 80%|######## | 188/235 [04:57<00:26, 1.78it/s] 80%|######## | 189/235 [04:57<00:24, 1.87it/s] 81%|######## | 190/235 [04:58<00:24, 1.84it/s] 81%|########1 | 191/235 [04:59<00:23, 1.90it/s] 82%|########1 | 192/235 [04:59<00:27, 1.57it/s] 82%|########2 | 193/235 [05:00<00:24, 1.69it/s] 83%|########2 | 194/235 [05:01<00:28, 1.45it/s] 83%|########2 | 195/235 [05:01<00:27, 1.47it/s] 83%|########3 | 196/235 [05:02<00:23, 1.64it/s] 84%|########3 | 197/235 [05:03<00:23, 1.61it/s] 84%|########4 | 198/235 [05:03<00:21, 1.75it/s] 85%|########4 | 199/235 [05:04<00:23, 1.53it/s] 85%|########5 | 200/235 [05:04<00:21, 1.65it/s] 86%|########5 | 201/235 [05:05<00:20, 1.69it/s] 86%|########5 | 202/235 [05:05<00:18, 1.83it/s] 86%|########6 | 203/235 [05:06<00:16, 1.92it/s] 87%|########6 | 204/235 [05:06<00:16, 1.89it/s] 87%|########7 | 205/235 [05:07<00:15, 1.97it/s] 88%|########7 | 206/235 [05:07<00:14, 1.96it/s] 88%|########8 | 207/235 [05:08<00:13, 2.03it/s] 89%|########8 | 208/235 [05:08<00:13, 2.06it/s] 89%|########8 | 209/235 [05:09<00:12, 2.10it/s] 89%|########9 | 210/235 [05:09<00:11, 2.14it/s] 90%|########9 | 211/235 [05:10<00:11, 2.16it/s] 90%|######### | 212/235 [05:10<00:10, 2.15it/s] 91%|######### | 213/235 [05:11<00:10, 2.17it/s] 91%|#########1| 214/235 [05:11<00:11, 1.90it/s] 91%|#########1| 215/235 [05:12<00:10, 1.98it/s] 92%|#########1| 216/235 [05:12<00:09, 2.04it/s] 92%|#########2| 217/235 [05:13<00:08, 2.10it/s] 93%|#########2| 218/235 [05:13<00:08, 1.89it/s] 93%|#########3| 219/235 [05:14<00:09, 1.68it/s] 94%|#########3| 220/235 [05:14<00:08, 1.78it/s] 94%|#########4| 221/235 [05:15<00:08, 1.65it/s] 94%|#########4| 222/235 [05:16<00:08, 1.61it/s] 95%|#########4| 223/235 [05:16<00:06, 1.73it/s] 95%|#########5| 224/235 [05:17<00:06, 1.66it/s] 96%|#########5| 225/235 [05:17<00:05, 1.79it/s] 96%|#########6| 226/235 [05:18<00:04, 1.86it/s] 97%|#########6| 227/235 [05:18<00:04, 1.88it/s] 97%|#########7| 228/235 [05:19<00:03, 1.97it/s] 97%|#########7| 229/235 [05:19<00:02, 2.03it/s] 98%|#########7| 230/235 [05:20<00:02, 2.08it/s] 98%|#########8| 231/235 [05:20<00:01, 2.08it/s] 99%|#########8| 232/235 [05:21<00:01, 2.12it/s] 99%|#########9| 233/235 [05:21<00:00, 2.14it/s] 100%|#########9| 234/235 [05:22<00:00, 2.14it/s] 100%|##########| 235/235 [05:22<00:00, 2.15it/s] [2017-07-18 14:55:09] - Scraper completed. 7050 documents retrieved. [2017-07-18 14:55:09] - ERROR - delete_elasticsearch_docs() - DoctorsScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/doctors/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd13d82d0>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/doctors/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd13d82d0>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 14:55:10] - ERROR - upload_data() - DoctorsScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd13d8350>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd13d8350>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 14:55:11] - ERROR - archive_data() - data/doctors.json - An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records. [2017-07-18 14:55:11] - Completed Scraper. [Foreign Doctors Scraper] [2017-07-18 14:55:11] Started Scraper. 0%| | 0/47 [00:00<?, ?it/s] 2%|2 | 1/47 [00:00<00:24, 1.88it/s] 4%|4 | 2/47 [00:01<00:23, 1.92it/s] 6%|6 | 3/47 [00:01<00:22, 1.99it/s] 9%|8 | 4/47 [00:01<00:20, 2.06it/s] 11%|# | 5/47 [00:02<00:20, 2.06it/s] 13%|#2 | 6/47 [00:03<00:22, 1.83it/s] 15%|#4 | 7/47 [00:03<00:25, 1.55it/s] 17%|#7 | 8/47 [00:04<00:22, 1.70it/s] 19%|#9 | 9/47 [00:04<00:20, 1.84it/s] 21%|##1 | 10/47 [00:05<00:19, 1.94it/s] 23%|##3 | 11/47 [00:05<00:18, 1.95it/s] 26%|##5 | 12/47 [00:06<00:17, 2.01it/s] 28%|##7 | 13/47 [00:06<00:16, 2.04it/s] 30%|##9 | 14/47 [00:07<00:15, 2.08it/s] 32%|###1 | 15/47 [00:07<00:15, 2.13it/s] 34%|###4 | 16/47 [00:08<00:14, 2.16it/s] 36%|###6 | 17/47 [00:08<00:13, 2.18it/s] 38%|###8 | 18/47 [00:09<00:13, 2.20it/s] 40%|#### | 19/47 [00:09<00:12, 2.20it/s] 43%|####2 | 20/47 [00:09<00:12, 2.21it/s] 45%|####4 | 21/47 [00:10<00:11, 2.22it/s] 47%|####6 | 22/47 [00:10<00:11, 2.22it/s] 49%|####8 | 23/47 [00:11<00:11, 2.10it/s] 51%|#####1 | 24/47 [00:11<00:10, 2.15it/s] 53%|#####3 | 25/47 [00:12<00:10, 2.18it/s] 55%|#####5 | 26/47 [00:12<00:09, 2.20it/s] 57%|#####7 | 27/47 [00:13<00:09, 2.21it/s] 60%|#####9 | 28/47 [00:13<00:08, 2.21it/s] 62%|######1 | 29/47 [00:14<00:08, 2.21it/s] 64%|######3 | 30/47 [00:14<00:07, 2.22it/s] 66%|######5 | 31/47 [00:14<00:07, 2.21it/s] 68%|######8 | 32/47 [00:15<00:07, 2.11it/s] 70%|####### | 33/47 [00:15<00:06, 2.16it/s] 72%|#######2 | 34/47 [00:16<00:05, 2.18it/s] 74%|#######4 | 35/47 [00:16<00:05, 2.20it/s] 77%|#######6 | 36/47 [00:17<00:04, 2.20it/s] 79%|#######8 | 37/47 [00:17<00:04, 2.13it/s] 81%|######## | 38/47 [00:18<00:04, 2.15it/s] 83%|########2 | 39/47 [00:18<00:03, 2.18it/s] 85%|########5 | 40/47 [00:19<00:03, 2.19it/s] 87%|########7 | 41/47 [00:19<00:02, 2.20it/s] 89%|########9 | 42/47 [00:20<00:02, 2.17it/s] 91%|#########1| 43/47 [00:20<00:01, 2.15it/s] 94%|#########3| 44/47 [00:20<00:01, 2.18it/s] 96%|#########5| 45/47 [00:21<00:00, 2.18it/s] 98%|#########7| 46/47 [00:21<00:00, 2.20it/s] 100%|##########| 47/47 [00:22<00:00, 2.21it/s] [2017-07-18 14:55:34] - Scraper completed. 1404 documents retrieved. [2017-07-18 14:55:34] - ERROR - delete_elasticsearch_docs() - ForeignDoctorsScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/doctors/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd1594690>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/doctors/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd1594690>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 14:55:34] - ERROR - upload_data() - ForeignDoctorsScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd09b6ad0>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd09b6ad0>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 14:55:35] - ERROR - archive_data() - data/foreign_doctors.json - An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records. [2017-07-18 14:55:36] - Completed Scraper. [Clinical Officers Scraper] [2017-07-18 14:55:36] Started Scraper. 0%| | 0/235 [00:00<?, ?it/s] 0%| | 1/235 [00:08<32:59, 8.46s/it] 1%| | 2/235 [00:16<32:51, 8.46s/it] 1%|1 | 3/235 [00:25<32:44, 8.47s/it] 2%|1 | 4/235 [00:33<32:34, 8.46s/it] 2%|2 | 5/235 [00:42<32:29, 8.47s/it] 3%|2 | 6/235 [00:50<32:18, 8.47s/it] 3%|2 | 7/235 [00:59<32:09, 8.46s/it] 3%|3 | 8/235 [01:07<32:03, 8.48s/it] 4%|3 | 9/235 [01:16<31:55, 8.48s/it] 4%|4 | 10/235 [01:24<31:49, 8.48s/it] 5%|4 | 11/235 [01:33<31:41, 8.49s/it] 5%|5 | 12/235 [01:41<31:30, 8.48s/it] 6%|5 | 13/235 [01:50<31:20, 8.47s/it] 6%|5 | 14/235 [01:58<31:11, 8.47s/it] 6%|6 | 15/235 [02:07<31:02, 8.46s/it] 7%|6 | 16/235 [02:15<31:09, 8.54s/it] 7%|7 | 17/235 [02:24<31:03, 8.55s/it] 8%|7 | 18/235 [02:33<31:03, 8.59s/it] 8%|8 | 19/235 [02:41<30:45, 8.55s/it] 9%|8 | 20/235 [02:49<30:31, 8.52s/it] 9%|8 | 21/235 [02:58<30:18, 8.50s/it] 9%|9 | 22/235 [03:07<30:19, 8.54s/it] 10%|9 | 23/235 [03:15<30:09, 8.53s/it] 10%|# | 24/235 [03:24<29:57, 8.52s/it] 11%|# | 25/235 [03:32<29:50, 8.52s/it] 11%|#1 | 26/235 [03:37<25:29, 7.32s/it] 11%|#1 | 27/235 [03:37<18:15, 5.27s/it] 12%|#1 | 28/235 [03:38<13:18, 3.86s/it] 12%|#2 | 29/235 [03:38<09:44, 2.84s/it] 13%|#2 | 30/235 [03:39<07:19, 2.14s/it] 13%|#3 | 31/235 [03:39<05:35, 1.65s/it] 14%|#3 | 32/235 [03:40<04:20, 1.29s/it] 14%|#4 | 33/235 [03:40<03:40, 1.09s/it] 14%|#4 | 34/235 [03:41<03:06, 1.08it/s] 15%|#4 | 35/235 [03:41<02:50, 1.17it/s] 15%|#5 | 36/235 [03:42<02:27, 1.35it/s] 16%|#5 | 37/235 [03:42<02:09, 1.53it/s] 16%|#6 | 38/235 [03:43<02:02, 1.61it/s] 17%|#6 | 39/235 [03:43<01:51, 1.76it/s] 17%|#7 | 40/235 [03:44<01:50, 1.77it/s] 17%|#7 | 41/235 [03:44<01:42, 1.88it/s] 18%|#7 | 42/235 [03:45<01:38, 1.96it/s] 18%|#8 | 43/235 [03:45<01:35, 2.01it/s] 19%|#8 | 44/235 [03:46<01:47, 1.78it/s] 19%|#9 | 45/235 [03:46<01:40, 1.89it/s] 20%|#9 | 46/235 [03:47<01:35, 1.98it/s] 20%|## | 47/235 [03:47<01:31, 2.05it/s] 20%|## | 48/235 [03:48<01:29, 2.10it/s] 21%|## | 49/235 [03:48<01:27, 2.13it/s] 21%|##1 | 50/235 [03:49<01:26, 2.15it/s] 22%|##1 | 51/235 [03:49<01:25, 2.14it/s] 22%|##2 | 52/235 [03:50<01:24, 2.16it/s] 23%|##2 | 53/235 [03:50<01:23, 2.17it/s] 23%|##2 | 54/235 [03:51<01:23, 2.17it/s] 23%|##3 | 55/235 [03:51<01:22, 2.17it/s] 24%|##3 | 56/235 [03:51<01:21, 2.19it/s] 24%|##4 | 57/235 [03:52<01:20, 2.20it/s] 25%|##4 | 58/235 [03:52<01:20, 2.20it/s] 25%|##5 | 59/235 [03:53<01:19, 2.20it/s] 26%|##5 | 60/235 [03:53<01:19, 2.21it/s] 26%|##5 | 61/235 [03:54<01:19, 2.20it/s] 26%|##6 | 62/235 [03:54<01:18, 2.21it/s] 27%|##6 | 63/235 [03:55<01:21, 2.11it/s] 27%|##7 | 64/235 [03:55<01:24, 2.01it/s] 28%|##7 | 65/235 [03:56<01:23, 2.03it/s] 28%|##8 | 66/235 [03:56<01:21, 2.08it/s] 29%|##8 | 67/235 [03:57<01:19, 2.12it/s] 29%|##8 | 68/235 [03:57<01:18, 2.14it/s] 29%|##9 | 69/235 [03:58<01:16, 2.17it/s] 30%|##9 | 70/235 [03:58<01:15, 2.19it/s] 30%|### | 71/235 [03:58<01:14, 2.20it/s] 31%|### | 72/235 [03:59<01:14, 2.19it/s] 31%|###1 | 73/235 [04:00<01:25, 1.88it/s] 31%|###1 | 74/235 [04:00<01:21, 1.98it/s] 32%|###1 | 75/235 [04:00<01:18, 2.04it/s] 32%|###2 | 76/235 [04:01<01:16, 2.09it/s] 33%|###2 | 77/235 [04:01<01:15, 2.11it/s] 33%|###3 | 78/235 [04:02<01:13, 2.13it/s] 34%|###3 | 79/235 [04:03<01:25, 1.83it/s] 34%|###4 | 80/235 [04:03<01:19, 1.94it/s] 34%|###4 | 81/235 [04:03<01:16, 2.01it/s] 35%|###4 | 82/235 [04:04<01:14, 2.07it/s] 35%|###5 | 83/235 [04:04<01:12, 2.11it/s] 36%|###5 | 84/235 [04:05<01:12, 2.07it/s] 36%|###6 | 85/235 [04:05<01:10, 2.11it/s] 37%|###6 | 86/235 [04:06<01:09, 2.14it/s] 37%|###7 | 87/235 [04:06<01:08, 2.16it/s] 37%|###7 | 88/235 [04:07<01:07, 2.17it/s] 38%|###7 | 89/235 [04:07<01:13, 1.97it/s] 38%|###8 | 90/235 [04:08<01:33, 1.55it/s] 39%|###8 | 91/235 [04:09<01:31, 1.58it/s] 39%|###9 | 92/235 [04:10<01:46, 1.34it/s] 40%|###9 | 93/235 [04:10<01:33, 1.52it/s] 40%|#### | 94/235 [04:11<01:23, 1.68it/s] 40%|#### | 95/235 [04:11<01:17, 1.81it/s] 41%|#### | 96/235 [04:12<01:12, 1.92it/s] 41%|####1 | 97/235 [04:12<01:08, 2.00it/s] 42%|####1 | 98/235 [04:13<01:06, 2.07it/s] 42%|####2 | 99/235 [04:13<01:12, 1.88it/s] 43%|####2 | 100/235 [04:14<01:08, 1.97it/s] 43%|####2 | 101/235 [04:14<01:06, 2.02it/s] 43%|####3 | 102/235 [04:15<01:06, 2.01it/s] 44%|####3 | 103/235 [04:15<01:03, 2.07it/s] 44%|####4 | 104/235 [04:16<01:04, 2.02it/s] 45%|####4 | 105/235 [04:16<01:14, 1.75it/s] 45%|####5 | 106/235 [04:17<01:09, 1.87it/s] 46%|####5 | 107/235 [04:17<01:05, 1.95it/s] 46%|####5 | 108/235 [04:18<01:11, 1.78it/s] 46%|####6 | 109/235 [04:18<01:07, 1.87it/s] 47%|####6 | 110/235 [04:19<01:04, 1.95it/s] 47%|####7 | 111/235 [04:19<01:01, 2.02it/s] 48%|####7 | 112/235 [04:20<00:59, 2.06it/s] 48%|####8 | 113/235 [04:20<00:58, 2.10it/s] 49%|####8 | 114/235 [04:21<00:56, 2.14it/s] 49%|####8 | 115/235 [04:21<00:55, 2.15it/s] 49%|####9 | 116/235 [04:22<00:54, 2.17it/s] 50%|####9 | 117/235 [04:22<00:53, 2.19it/s] 50%|##### | 118/235 [04:23<00:53, 2.19it/s] 51%|##### | 119/235 [04:23<00:53, 2.16it/s] 51%|#####1 | 120/235 [04:23<00:52, 2.17it/s] 51%|#####1 | 121/235 [04:24<00:54, 2.10it/s] 52%|#####1 | 122/235 [04:24<00:53, 2.13it/s] 52%|#####2 | 123/235 [04:25<00:52, 2.12it/s] 53%|#####2 | 124/235 [04:25<00:52, 2.13it/s] 53%|#####3 | 125/235 [04:26<00:52, 2.08it/s] 54%|#####3 | 126/235 [04:26<00:51, 2.12it/s] 54%|#####4 | 127/235 [04:27<00:50, 2.15it/s] 54%|#####4 | 128/235 [04:27<00:49, 2.16it/s] 55%|#####4 | 129/235 [04:28<00:48, 2.17it/s] 55%|#####5 | 130/235 [04:28<00:48, 2.18it/s] 56%|#####5 | 131/235 [04:29<00:47, 2.18it/s] 56%|#####6 | 132/235 [04:29<00:47, 2.18it/s] 57%|#####6 | 133/235 [04:30<00:46, 2.17it/s] 57%|#####7 | 134/235 [04:30<00:46, 2.17it/s] 57%|#####7 | 135/235 [04:30<00:45, 2.18it/s] 58%|#####7 | 136/235 [04:31<00:45, 2.19it/s] 58%|#####8 | 137/235 [04:31<00:44, 2.18it/s] 59%|#####8 | 138/235 [04:32<00:44, 2.17it/s] 59%|#####9 | 139/235 [04:32<00:43, 2.19it/s] 60%|#####9 | 140/235 [04:33<00:43, 2.18it/s] 60%|###### | 141/235 [04:33<00:42, 2.20it/s] 60%|###### | 142/235 [04:34<00:42, 2.21it/s] 61%|###### | 143/235 [04:34<00:42, 2.18it/s] 61%|######1 | 144/235 [04:35<00:41, 2.20it/s] 62%|######1 | 145/235 [04:35<00:43, 2.08it/s] 62%|######2 | 146/235 [04:36<00:52, 1.70it/s] 63%|######2 | 147/235 [04:36<00:48, 1.82it/s] 63%|######2 | 148/235 [04:37<00:45, 1.93it/s] 63%|######3 | 149/235 [04:37<00:45, 1.90it/s] 64%|######3 | 150/235 [04:38<00:42, 1.98it/s] 64%|######4 | 151/235 [04:38<00:44, 1.88it/s] 65%|######4 | 152/235 [04:39<00:42, 1.96it/s] 65%|######5 | 153/235 [04:39<00:43, 1.88it/s] 66%|######5 | 154/235 [04:40<00:41, 1.96it/s] 66%|######5 | 155/235 [04:40<00:41, 1.91it/s] 66%|######6 | 156/235 [04:41<00:40, 1.97it/s] 67%|######6 | 157/235 [04:42<00:43, 1.80it/s] 67%|######7 | 158/235 [04:42<00:40, 1.91it/s] 68%|######7 | 159/235 [04:43<00:38, 2.00it/s] 68%|######8 | 160/235 [04:43<00:36, 2.05it/s] 69%|######8 | 161/235 [04:43<00:35, 2.11it/s] 69%|######8 | 162/235 [04:44<00:34, 2.11it/s] 69%|######9 | 163/235 [04:44<00:33, 2.14it/s] 70%|######9 | 164/235 [04:45<00:32, 2.15it/s] 70%|####### | 165/235 [04:45<00:32, 2.14it/s] 71%|####### | 166/235 [04:46<00:33, 2.06it/s] 71%|#######1 | 167/235 [04:46<00:32, 2.08it/s] 71%|#######1 | 168/235 [04:47<00:31, 2.11it/s] 72%|#######1 | 169/235 [04:47<00:31, 2.12it/s] 72%|#######2 | 170/235 [04:48<00:30, 2.11it/s] 73%|#######2 | 171/235 [04:48<00:30, 2.13it/s] 73%|#######3 | 172/235 [04:49<00:29, 2.15it/s] 74%|#######3 | 173/235 [04:49<00:29, 2.12it/s] 74%|#######4 | 174/235 [04:50<00:29, 2.08it/s] 74%|#######4 | 175/235 [04:50<00:28, 2.10it/s] 75%|#######4 | 176/235 [04:50<00:27, 2.14it/s] 75%|#######5 | 177/235 [04:51<00:26, 2.15it/s] 76%|#######5 | 178/235 [04:51<00:27, 2.11it/s] 76%|#######6 | 179/235 [04:52<00:26, 2.13it/s] 77%|#######6 | 180/235 [04:53<00:28, 1.93it/s] 77%|#######7 | 181/235 [04:53<00:27, 1.99it/s] 77%|#######7 | 182/235 [04:54<00:28, 1.84it/s] 78%|#######7 | 183/235 [04:54<00:26, 1.94it/s] 78%|#######8 | 184/235 [04:55<00:26, 1.94it/s] 79%|#######8 | 185/235 [04:55<00:29, 1.71it/s] 79%|#######9 | 186/235 [04:56<00:29, 1.66it/s] 80%|#######9 | 187/235 [04:56<00:26, 1.79it/s] 80%|######## | 188/235 [04:57<00:26, 1.78it/s] 80%|######## | 189/235 [04:57<00:24, 1.87it/s] 81%|######## | 190/235 [04:58<00:24, 1.84it/s] 81%|########1 | 191/235 [04:59<00:23, 1.90it/s] 82%|########1 | 192/235 [04:59<00:27, 1.57it/s] 82%|########2 | 193/235 [05:00<00:24, 1.69it/s] 83%|########2 | 194/235 [05:01<00:28, 1.45it/s] 83%|########2 | 195/235 [05:01<00:27, 1.47it/s] 83%|########3 | 196/235 [05:02<00:23, 1.64it/s] 84%|########3 | 197/235 [05:03<00:23, 1.61it/s] 84%|########4 | 198/235 [05:03<00:21, 1.75it/s] 85%|########4 | 199/235 [05:04<00:23, 1.53it/s] 85%|########5 | 200/235 [05:04<00:21, 1.65it/s] 86%|########5 | 201/235 [05:05<00:20, 1.69it/s] 86%|########5 | 202/235 [05:05<00:18, 1.83it/s] 86%|########6 | 203/235 [05:06<00:16, 1.92it/s] 87%|########6 | 204/235 [05:06<00:16, 1.89it/s] 87%|########7 | 205/235 [05:07<00:15, 1.97it/s] 88%|########7 | 206/235 [05:07<00:14, 1.96it/s] 88%|########8 | 207/235 [05:08<00:13, 2.03it/s] 89%|########8 | 208/235 [05:08<00:13, 2.06it/s] 89%|########8 | 209/235 [05:09<00:12, 2.10it/s] 89%|########9 | 210/235 [05:09<00:11, 2.14it/s] 90%|########9 | 211/235 [05:10<00:11, 2.16it/s] 90%|######### | 212/235 [05:10<00:10, 2.15it/s] 91%|######### | 213/235 [05:11<00:10, 2.17it/s] 91%|#########1| 214/235 [05:11<00:11, 1.90it/s] 91%|#########1| 215/235 [05:12<00:10, 1.98it/s] 92%|#########1| 216/235 [05:12<00:09, 2.04it/s] 92%|#########2| 217/235 [05:13<00:08, 2.10it/s] 93%|#########2| 218/235 [05:13<00:08, 1.89it/s] 93%|#########3| 219/235 [05:14<00:09, 1.68it/s] 94%|#########3| 220/235 [05:14<00:08, 1.78it/s] 94%|#########4| 221/235 [05:15<00:08, 1.65it/s] 94%|#########4| 222/235 [05:16<00:08, 1.61it/s] 95%|#########4| 223/235 [05:16<00:06, 1.73it/s] 95%|#########5| 224/235 [05:17<00:06, 1.66it/s] 96%|#########5| 225/235 [05:17<00:05, 1.79it/s] 96%|#########6| 226/235 [05:18<00:04, 1.86it/s] 97%|#########6| 227/235 [05:18<00:04, 1.88it/s] 97%|#########7| 228/235 [05:19<00:03, 1.97it/s] 97%|#########7| 229/235 [05:19<00:02, 2.03it/s] 98%|#########7| 230/235 [05:20<00:02, 2.08it/s] 98%|#########8| 231/235 [05:20<00:01, 2.08it/s] 99%|#########8| 232/235 [05:21<00:01, 2.12it/s] 99%|#########9| 233/235 [05:21<00:00, 2.14it/s] 100%|#########9| 234/235 [05:22<00:00, 2.14it/s] 100%|##########| 235/235 [05:22<00:00, 2.15it/s] [2017-07-18 14:55:09] - Scraper completed. 7050 documents retrieved. [2017-07-18 14:55:09] - ERROR - delete_elasticsearch_docs() - DoctorsScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/doctors/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd13d82d0>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/doctors/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd13d82d0>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 14:55:10] - ERROR - upload_data() - DoctorsScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd13d8350>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd13d8350>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 14:55:11] - ERROR - archive_data() - data/doctors.json - An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records. [2017-07-18 14:55:11] - Completed Scraper. [Foreign Doctors Scraper] [2017-07-18 14:55:11] Started Scraper. 0%| | 0/47 [00:00<?, ?it/s] 2%|2 | 1/47 [00:00<00:24, 1.88it/s] 4%|4 | 2/47 [00:01<00:23, 1.92it/s] 6%|6 | 3/47 [00:01<00:22, 1.99it/s] 9%|8 | 4/47 [00:01<00:20, 2.06it/s] 11%|# | 5/47 [00:02<00:20, 2.06it/s] 13%|#2 | 6/47 [00:03<00:22, 1.83it/s] 15%|#4 | 7/47 [00:03<00:25, 1.55it/s] 17%|#7 | 8/47 [00:04<00:22, 1.70it/s] 19%|#9 | 9/47 [00:04<00:20, 1.84it/s] 21%|##1 | 10/47 [00:05<00:19, 1.94it/s] 23%|##3 | 11/47 [00:05<00:18, 1.95it/s] 26%|##5 | 12/47 [00:06<00:17, 2.01it/s] 28%|##7 | 13/47 [00:06<00:16, 2.04it/s] 30%|##9 | 14/47 [00:07<00:15, 2.08it/s] 32%|###1 | 15/47 [00:07<00:15, 2.13it/s] 34%|###4 | 16/47 [00:08<00:14, 2.16it/s] 36%|###6 | 17/47 [00:08<00:13, 2.18it/s] 38%|###8 | 18/47 [00:09<00:13, 2.20it/s] 40%|#### | 19/47 [00:09<00:12, 2.20it/s] 43%|####2 | 20/47 [00:09<00:12, 2.21it/s] 45%|####4 | 21/47 [00:10<00:11, 2.22it/s] 47%|####6 | 22/47 [00:10<00:11, 2.22it/s] 49%|####8 | 23/47 [00:11<00:11, 2.10it/s] 51%|#####1 | 24/47 [00:11<00:10, 2.15it/s] 53%|#####3 | 25/47 [00:12<00:10, 2.18it/s] 55%|#####5 | 26/47 [00:12<00:09, 2.20it/s] 57%|#####7 | 27/47 [00:13<00:09, 2.21it/s] 60%|#####9 | 28/47 [00:13<00:08, 2.21it/s] 62%|######1 | 29/47 [00:14<00:08, 2.21it/s] 64%|######3 | 30/47 [00:14<00:07, 2.22it/s] 66%|######5 | 31/47 [00:14<00:07, 2.21it/s] 68%|######8 | 32/47 [00:15<00:07, 2.11it/s] 70%|####### | 33/47 [00:15<00:06, 2.16it/s] 72%|#######2 | 34/47 [00:16<00:05, 2.18it/s] 74%|#######4 | 35/47 [00:16<00:05, 2.20it/s] 77%|#######6 | 36/47 [00:17<00:04, 2.20it/s] 79%|#######8 | 37/47 [00:17<00:04, 2.13it/s] 81%|######## | 38/47 [00:18<00:04, 2.15it/s] 83%|########2 | 39/47 [00:18<00:03, 2.18it/s] 85%|########5 | 40/47 [00:19<00:03, 2.19it/s] 87%|########7 | 41/47 [00:19<00:02, 2.20it/s] 89%|########9 | 42/47 [00:20<00:02, 2.17it/s] 91%|#########1| 43/47 [00:20<00:01, 2.15it/s] 94%|#########3| 44/47 [00:20<00:01, 2.18it/s] 96%|#########5| 45/47 [00:21<00:00, 2.18it/s] 98%|#########7| 46/47 [00:21<00:00, 2.20it/s] 100%|##########| 47/47 [00:22<00:00, 2.21it/s] [2017-07-18 14:55:34] - Scraper completed. 1404 documents retrieved. [2017-07-18 14:55:34] - ERROR - delete_elasticsearch_docs() - ForeignDoctorsScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/doctors/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd1594690>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/doctors/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd1594690>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 14:55:34] - ERROR - upload_data() - ForeignDoctorsScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd09b6ad0>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd09b6ad0>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 14:55:35] - ERROR - archive_data() - data/foreign_doctors.json - An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records. [2017-07-18 14:55:36] - Completed Scraper. [Clinical Officers Scraper] [2017-07-18 14:55:36] Started Scraper. 0%| | 0/417 [00:00<?, ?it/s] 0%| | 1/417 [00:00<05:24, 1.28it/s] 0%| | 2/417 [00:01<05:37, 1.23it/s] 1%| | 3/417 [00:02<05:47, 1.19it/s] 1%| | 4/417 [00:03<05:51, 1.18it/s] 1%|1 | 5/417 [00:04<06:07, 1.12it/s] 1%|1 | 6/417 [00:05<06:03, 1.13it/s] 2%|1 | 7/417 [00:06<05:42, 1.20it/s] 2%|1 | 8/417 [00:07<06:40, 1.02it/s] 2%|2 | 9/417 [00:08<07:02, 1.03s/it] 2%|2 | 10/417 [00:09<06:39, 1.02it/s] 3%|2 | 11/417 [00:10<06:20, 1.07it/s] 3%|2 | 12/417 [00:10<05:54, 1.14it/s] 3%|3 | 13/417 [00:11<05:31, 1.22it/s] 3%|3 | 14/417 [00:12<05:27, 1.23it/s] 4%|3 | 15/417 [00:13<05:29, 1.22it/s] 4%|3 | 16/417 [00:14<05:38, 1.19it/s] 4%|4 | 17/417 [00:14<05:29, 1.22it/s] 4%|4 | 18/417 [00:16<06:15, 1.06it/s] 5%|4 | 19/417 [00:16<05:56, 1.12it/s] 5%|4 | 20/417 [00:17<05:34, 1.19it/s] 5%|5 | 21/417 [00:18<05:45, 1.15it/s] 5%|5 | 22/417 [00:19<05:36, 1.18it/s] 6%|5 | 23/417 [00:20<05:24, 1.21it/s] 6%|5 | 24/417 [00:20<05:25, 1.21it/s] 6%|5 | 25/417 [00:21<05:14, 1.25it/s] 6%|6 | 26/417 [00:22<05:16, 1.23it/s] 6%|6 | 27/417 [00:23<05:07, 1.27it/s] 7%|6 | 28/417 [00:24<05:12, 1.24it/s] 7%|6 | 29/417 [00:25<05:37, 1.15it/s] 7%|7 | 30/417 [00:26<05:46, 1.12it/s] 7%|7 | 31/417 [00:27<05:50, 1.10it/s] 8%|7 | 32/417 [00:28<05:55, 1.08it/s] 8%|7 | 33/417 [00:28<05:43, 1.12it/s] 8%|8 | 34/417 [00:29<05:29, 1.16it/s] 8%|8 | 35/417 [00:30<05:26, 1.17it/s] 9%|8 | 36/417 [00:31<06:11, 1.02it/s] 9%|8 | 37/417 [00:32<05:49, 1.09it/s] 9%|9 | 38/417 [00:33<05:33, 1.14it/s] 9%|9 | 39/417 [00:34<05:18, 1.19it/s] 10%|9 | 40/417 [00:34<04:56, 1.27it/s] 10%|9 | 41/417 [00:35<04:58, 1.26it/s] 10%|# | 42/417 [00:36<04:57, 1.26it/s] 10%|# | 43/417 [00:37<04:57, 1.26it/s] 11%|# | 44/417 [00:37<05:06, 1.22it/s] 11%|# | 45/417 [00:38<04:52, 1.27it/s] 11%|#1 | 46/417 [00:39<04:49, 1.28it/s] 11%|#1 | 47/417 [00:40<04:55, 1.25it/s] 12%|#1 | 48/417 [00:41<04:57, 1.24it/s] 12%|#1 | 49/417 [00:41<04:45, 1.29it/s] 12%|#1 | 50/417 [00:42<04:37, 1.32it/s] 12%|#2 | 51/417 [00:43<04:28, 1.36it/s] 12%|#2 | 52/417 [00:44<04:50, 1.26it/s] 13%|#2 | 53/417 [00:46<06:57, 1.15s/it] 13%|#2 | 54/417 [00:46<06:14, 1.03s/it] 13%|#3 | 55/417 [00:47<05:33, 1.08it/s] 13%|#3 | 56/417 [00:48<05:28, 1.10it/s] 14%|#3 | 57/417 [00:49<05:13, 1.15it/s] 14%|#3 | 58/417 [00:49<04:55, 1.21it/s] 14%|#4 | 59/417 [00:50<04:50, 1.23it/s] 14%|#4 | 60/417 [00:51<05:04, 1.17it/s] 15%|#4 | 61/417 [00:52<04:52, 1.22it/s] 15%|#4 | 62/417 [00:53<04:45, 1.24it/s] 15%|#5 | 63/417 [00:53<04:29, 1.31it/s] 15%|#5 | 64/417 [00:54<04:27, 1.32it/s] 16%|#5 | 65/417 [00:55<04:43, 1.24it/s] 16%|#5 | 66/417 [00:56<04:41, 1.25it/s] 16%|#6 | 67/417 [00:57<04:45, 1.23it/s] 16%|#6 | 68/417 [00:57<04:29, 1.30it/s] 17%|#6 | 69/417 [00:58<04:45, 1.22it/s] 17%|#6 | 70/417 [00:59<04:41, 1.23it/s] 17%|#7 | 71/417 [01:00<04:35, 1.26it/s] 17%|#7 | 72/417 [01:01<04:51, 1.18it/s] 18%|#7 | 73/417 [01:02<05:12, 1.10it/s] 18%|#7 | 74/417 [01:03<04:47, 1.19it/s] 18%|#7 | 75/417 [01:03<04:48, 1.19it/s] 18%|#8 | 76/417 [01:04<04:42, 1.21it/s] 18%|#8 | 77/417 [01:05<04:34, 1.24it/s] 19%|#8 | 78/417 [01:06<04:55, 1.15it/s] 19%|#8 | 79/417 [01:07<04:35, 1.23it/s] 19%|#9 | 80/417 [01:07<04:31, 1.24it/s] 19%|#9 | 81/417 [01:08<04:23, 1.28it/s] 20%|#9 | 82/417 [01:09<04:23, 1.27it/s] 20%|#9 | 83/417 [01:10<04:23, 1.27it/s] 20%|## | 84/417 [01:10<04:18, 1.29it/s] 20%|## | 85/417 [01:11<04:15, 1.30it/s] 21%|## | 86/417 [01:12<04:19, 1.27it/s] 21%|## | 87/417 [01:13<04:16, 1.29it/s] 21%|##1 | 88/417 [01:14<04:31, 1.21it/s] 21%|##1 | 89/417 [01:15<04:39, 1.18it/s] 22%|##1 | 90/417 [01:16<04:44, 1.15it/s] 22%|##1 | 91/417 [01:16<04:28, 1.21it/s] 22%|##2 | 92/417 [01:17<04:16, 1.27it/s] 22%|##2 | 93/417 [01:18<04:13, 1.28it/s] 23%|##2 | 94/417 [01:18<04:06, 1.31it/s] 23%|##2 | 95/417 [01:19<04:31, 1.19it/s] 23%|##3 | 96/417 [01:20<04:26, 1.21it/s] 23%|##3 | 97/417 [01:21<04:27, 1.20it/s] 24%|##3 | 98/417 [01:22<04:17, 1.24it/s] 24%|##3 | 99/417 [01:23<04:28, 1.18it/s] 24%|##3 | 100/417 [01:24<04:23, 1.20it/s] 24%|##4 | 101/417 [01:24<04:21, 1.21it/s] 24%|##4 | 102/417 [01:26<04:51, 1.08it/s] 25%|##4 | 103/417 [01:26<04:30, 1.16it/s] 25%|##4 | 104/417 [01:27<04:33, 1.14it/s] 25%|##5 | 105/417 [01:28<04:28, 1.16it/s] 25%|##5 | 106/417 [01:29<04:28, 1.16it/s] 26%|##5 | 107/417 [01:30<04:13, 1.22it/s] 26%|##5 | 108/417 [01:30<04:05, 1.26it/s] 26%|##6 | 109/417 [01:31<04:01, 1.28it/s] 26%|##6 | 110/417 [01:32<04:02, 1.27it/s] 27%|##6 | 111/417 [01:33<04:21, 1.17it/s] 27%|##6 | 112/417 [01:34<04:43, 1.08it/s] 27%|##7 | 113/417 [01:35<04:19, 1.17it/s] 27%|##7 | 114/417 [01:35<04:06, 1.23it/s] 28%|##7 | 115/417 [01:36<03:55, 1.28it/s] 28%|##7 | 116/417 [01:37<03:47, 1.32it/s] 28%|##8 | 117/417 [01:38<03:54, 1.28it/s] 28%|##8 | 118/417 [01:38<03:44, 1.33it/s] 29%|##8 | 119/417 [01:39<03:45, 1.32it/s] 29%|##8 | 120/417 [01:40<03:37, 1.36it/s] 29%|##9 | 121/417 [01:40<03:33, 1.39it/s] 29%|##9 | 122/417 [01:41<03:26, 1.43it/s] 29%|##9 | 123/417 [01:42<03:30, 1.40it/s] 30%|##9 | 124/417 [01:43<03:39, 1.33it/s] 30%|##9 | 125/417 [01:43<03:42, 1.31it/s] 30%|### | 126/417 [01:44<03:40, 1.32it/s] 30%|### | 127/417 [01:45<03:53, 1.24it/s] 31%|### | 128/417 [01:46<03:45, 1.28it/s] 31%|### | 129/417 [01:47<03:52, 1.24it/s] 31%|###1 | 130/417 [01:47<03:43, 1.28it/s] 31%|###1 | 131/417 [01:48<03:53, 1.22it/s] 32%|###1 | 132/417 [01:49<03:48, 1.25it/s] 32%|###1 | 133/417 [01:50<03:49, 1.24it/s] 32%|###2 | 134/417 [01:51<03:35, 1.31it/s] 32%|###2 | 135/417 [01:51<03:25, 1.38it/s] 33%|###2 | 136/417 [01:52<03:48, 1.23it/s] 33%|###2 | 137/417 [01:53<03:49, 1.22it/s] 33%|###3 | 138/417 [01:54<03:49, 1.21it/s] 33%|###3 | 139/417 [01:55<03:50, 1.20it/s] 34%|###3 | 140/417 [01:55<03:33, 1.30it/s] 34%|###3 | 141/417 [01:56<03:35, 1.28it/s] 34%|###4 | 142/417 [01:57<03:43, 1.23it/s] 34%|###4 | 143/417 [01:58<03:43, 1.22it/s] 35%|###4 | 144/417 [01:59<03:55, 1.16it/s] 35%|###4 | 145/417 [02:00<03:47, 1.19it/s] 35%|###5 | 146/417 [02:01<03:47, 1.19it/s] 35%|###5 | 147/417 [02:01<03:41, 1.22it/s] 35%|###5 | 148/417 [02:02<03:34, 1.26it/s] 36%|###5 | 149/417 [02:03<03:39, 1.22it/s] 36%|###5 | 150/417 [02:04<03:33, 1.25it/s] 36%|###6 | 151/417 [02:04<03:23, 1.31it/s] 36%|###6 | 152/417 [02:05<03:25, 1.29it/s] 37%|###6 | 153/417 [02:06<03:22, 1.31it/s] 37%|###6 | 154/417 [02:07<03:18, 1.32it/s] 37%|###7 | 155/417 [02:07<03:07, 1.39it/s] 37%|###7 | 156/417 [02:08<03:41, 1.18it/s] 38%|###7 | 157/417 [02:10<04:20, 1.00s/it] 38%|###7 | 158/417 [02:10<03:53, 1.11it/s] 38%|###8 | 159/417 [02:11<03:41, 1.17it/s] 38%|###8 | 160/417 [02:12<03:40, 1.17it/s] 39%|###8 | 161/417 [02:13<03:38, 1.17it/s] 39%|###8 | 162/417 [02:14<03:35, 1.18it/s] 39%|###9 | 163/417 [02:14<03:25, 1.24it/s] 39%|###9 | 164/417 [02:15<03:19, 1.27it/s] 40%|###9 | 165/417 [02:16<03:26, 1.22it/s] 40%|###9 | 166/417 [02:17<03:26, 1.21it/s] 40%|#### | 167/417 [02:18<03:31, 1.18it/s] 40%|#### | 168/417 [02:19<03:26, 1.20it/s] 41%|#### | 169/417 [02:19<03:27, 1.19it/s] 41%|#### | 170/417 [02:20<03:26, 1.20it/s] 41%|####1 | 171/417 [02:21<03:21, 1.22it/s] 41%|####1 | 172/417 [02:22<03:10, 1.29it/s] 41%|####1 | 173/417 [02:22<03:00, 1.35it/s] 42%|####1 | 174/417 [02:23<02:59, 1.35it/s] 42%|####1 | 175/417 [02:24<03:06, 1.30it/s] 42%|####2 | 176/417 [02:25<03:13, 1.25it/s] 42%|####2 | 177/417 [02:26<03:12, 1.24it/s] 43%|####2 | 178/417 [02:26<03:03, 1.30it/s] 43%|####2 | 179/417 [02:27<03:04, 1.29it/s] 43%|####3 | 180/417 [02:28<03:05, 1.28it/s] 43%|####3 | 181/417 [02:29<02:59, 1.32it/s] 44%|####3 | 182/417 [02:29<02:58, 1.32it/s] 44%|####3 | 183/417 [02:30<02:53, 1.35it/s] 44%|####4 | 184/417 [02:31<03:03, 1.27it/s] 44%|####4 | 185/417 [02:32<03:20, 1.16it/s] 45%|####4 | 186/417 [02:33<03:16, 1.18it/s] 45%|####4 | 187/417 [02:34<03:17, 1.17it/s] 45%|####5 | 188/417 [02:35<03:12, 1.19it/s] 45%|####5 | 189/417 [02:35<03:10, 1.19it/s] 46%|####5 | 190/417 [02:36<03:03, 1.23it/s] 46%|####5 | 191/417 [02:37<03:05, 1.22it/s] 46%|####6 | 192/417 [02:38<03:05, 1.21it/s] 46%|####6 | 193/417 [02:39<03:02, 1.22it/s] 47%|####6 | 194/417 [02:39<02:58, 1.25it/s] 47%|####6 | 195/417 [02:40<03:02, 1.22it/s] 47%|####7 | 196/417 [02:41<03:07, 1.18it/s] 47%|####7 | 197/417 [02:42<02:56, 1.24it/s] 47%|####7 | 198/417 [02:43<02:54, 1.26it/s] 48%|####7 | 199/417 [02:43<02:50, 1.28it/s] 48%|####7 | 200/417 [02:44<02:48, 1.29it/s] 48%|####8 | 201/417 [02:45<02:54, 1.24it/s] 48%|####8 | 202/417 [02:46<02:56, 1.22it/s] 49%|####8 | 203/417 [02:47<03:07, 1.14it/s] 49%|####8 | 204/417 [02:48<03:13, 1.10it/s] 49%|####9 | 205/417 [02:49<03:24, 1.04it/s] 49%|####9 | 206/417 [02:50<03:42, 1.05s/it] 50%|####9 | 207/417 [02:51<03:30, 1.00s/it] 50%|####9 | 208/417 [02:52<03:13, 1.08it/s] 50%|##### | 209/417 [02:53<03:03, 1.13it/s] 50%|##### | 210/417 [02:53<02:52, 1.20it/s] 51%|##### | 211/417 [02:54<02:59, 1.15it/s] 51%|##### | 212/417 [02:55<03:05, 1.11it/s] 51%|#####1 | 213/417 [02:56<03:00, 1.13it/s] 51%|#####1 | 214/417 [02:57<03:08, 1.08it/s] 52%|#####1 | 215/417 [02:58<02:55, 1.15it/s] 52%|#####1 | 216/417 [02:59<03:03, 1.10it/s] 52%|#####2 | 217/417 [03:00<02:54, 1.15it/s] 52%|#####2 | 218/417 [03:00<02:49, 1.17it/s] 53%|#####2 | 219/417 [03:01<02:47, 1.18it/s] 53%|#####2 | 220/417 [03:02<02:45, 1.19it/s] 53%|#####2 | 221/417 [03:03<02:49, 1.16it/s] 53%|#####3 | 222/417 [03:04<02:43, 1.19it/s] 53%|#####3 | 223/417 [03:05<02:34, 1.25it/s] 54%|#####3 | 224/417 [03:05<02:33, 1.26it/s] 54%|#####3 | 225/417 [03:06<02:41, 1.19it/s] 54%|#####4 | 226/417 [03:07<02:39, 1.20it/s] 54%|#####4 | 227/417 [03:08<02:42, 1.17it/s] 55%|#####4 | 228/417 [03:09<02:35, 1.21it/s] 55%|#####4 | 229/417 [03:09<02:25, 1.29it/s] 55%|#####5 | 230/417 [03:10<02:22, 1.31it/s] 55%|#####5 | 231/417 [03:11<02:21, 1.31it/s] 56%|#####5 | 232/417 [03:12<02:22, 1.30it/s] 56%|#####5 | 233/417 [03:13<02:29, 1.23it/s] 56%|#####6 | 234/417 [03:13<02:27, 1.24it/s] 56%|#####6 | 235/417 [03:14<02:26, 1.24it/s] 57%|#####6 | 236/417 [03:15<02:28, 1.22it/s] 57%|#####6 | 237/417 [03:16<02:32, 1.18it/s] 57%|#####7 | 238/417 [03:17<02:27, 1.21it/s] 57%|#####7 | 239/417 [03:18<02:33, 1.16it/s] 58%|#####7 | 240/417 [03:18<02:22, 1.24it/s] 58%|#####7 | 241/417 [03:19<02:21, 1.24it/s] 58%|#####8 | 242/417 [03:20<02:22, 1.23it/s] 58%|#####8 | 243/417 [03:21<02:25, 1.20it/s] 59%|#####8 | 244/417 [03:22<02:23, 1.21it/s] 59%|#####8 | 245/417 [03:23<02:48, 1.02it/s] 59%|#####8 | 246/417 [03:24<02:33, 1.12it/s] 59%|#####9 | 247/417 [03:24<02:22, 1.19it/s] 59%|#####9 | 248/417 [03:25<02:20, 1.20it/s] 60%|#####9 | 249/417 [03:26<02:12, 1.27it/s] 60%|#####9 | 250/417 [03:27<02:10, 1.28it/s] 60%|###### | 251/417 [03:27<02:03, 1.34it/s] 60%|###### | 252/417 [03:28<02:05, 1.31it/s] 61%|###### | 253/417 [03:29<02:01, 1.35it/s] 61%|###### | 254/417 [03:30<01:57, 1.38it/s] 61%|######1 | 255/417 [03:30<01:54, 1.41it/s] 61%|######1 | 256/417 [03:31<01:55, 1.40it/s] 62%|######1 | 257/417 [03:32<01:57, 1.36it/s] 62%|######1 | 258/417 [03:32<01:55, 1.37it/s] 62%|######2 | 259/417 [03:33<01:59, 1.32it/s] 62%|######2 | 260/417 [03:34<02:06, 1.24it/s] 63%|######2 | 261/417 [03:35<02:04, 1.25it/s] 63%|######2 | 262/417 [03:36<02:02, 1.27it/s] 63%|######3 | 263/417 [03:36<01:59, 1.29it/s] 63%|######3 | 264/417 [03:37<01:57, 1.30it/s] 64%|######3 | 265/417 [03:38<02:00, 1.26it/s] 64%|######3 | 266/417 [03:39<02:06, 1.19it/s] 64%|######4 | 267/417 [03:40<02:05, 1.20it/s] 64%|######4 | 268/417 [03:41<02:15, 1.10it/s] 65%|######4 | 269/417 [03:42<02:04, 1.19it/s] 65%|######4 | 270/417 [03:42<01:59, 1.23it/s] 65%|######4 | 271/417 [03:43<01:57, 1.24it/s] 65%|######5 | 272/417 [03:44<01:54, 1.27it/s] 65%|######5 | 273/417 [03:45<01:50, 1.30it/s] 66%|######5 | 274/417 [03:46<01:53, 1.26it/s] 66%|######5 | 275/417 [03:46<01:50, 1.29it/s] 66%|######6 | 276/417 [03:47<01:45, 1.33it/s] 66%|######6 | 277/417 [03:48<01:42, 1.36it/s] 67%|######6 | 278/417 [03:48<01:41, 1.37it/s] 67%|######6 | 279/417 [03:49<01:43, 1.33it/s] 67%|######7 | 280/417 [03:50<01:43, 1.33it/s] 67%|######7 | 281/417 [03:51<01:46, 1.28it/s] 68%|######7 | 282/417 [03:52<01:47, 1.25it/s] 68%|######7 | 283/417 [03:52<01:46, 1.26it/s] 68%|######8 | 284/417 [03:53<01:44, 1.28it/s] 68%|######8 | 285/417 [03:54<01:45, 1.26it/s] 69%|######8 | 286/417 [03:55<01:43, 1.26it/s] 69%|######8 | 287/417 [03:55<01:38, 1.32it/s] 69%|######9 | 288/417 [03:56<01:40, 1.28it/s] 69%|######9 | 289/417 [03:57<01:41, 1.27it/s] 70%|######9 | 290/417 [03:58<01:40, 1.27it/s] 70%|######9 | 291/417 [03:59<01:39, 1.27it/s] 70%|####### | 292/417 [04:00<01:42, 1.23it/s] 70%|####### | 293/417 [04:00<01:35, 1.30it/s] 71%|####### | 294/417 [04:01<01:30, 1.36it/s] 71%|####### | 295/417 [04:02<01:28, 1.38it/s] 71%|####### | 296/417 [04:02<01:29, 1.36it/s] 71%|#######1 | 297/417 [04:03<01:30, 1.32it/s] 71%|#######1 | 298/417 [04:04<01:39, 1.20it/s] 72%|#######1 | 299/417 [04:05<01:39, 1.18it/s] 72%|#######1 | 300/417 [04:06<01:42, 1.14it/s] 72%|#######2 | 301/417 [04:07<01:39, 1.17it/s] 72%|#######2 | 302/417 [04:07<01:34, 1.21it/s] 73%|#######2 | 303/417 [04:08<01:32, 1.24it/s] 73%|#######2 | 304/417 [04:09<01:34, 1.20it/s] 73%|#######3 | 305/417 [04:10<01:37, 1.15it/s] 73%|#######3 | 306/417 [04:11<01:36, 1.15it/s] 74%|#######3 | 307/417 [04:12<01:40, 1.10it/s] 74%|#######3 | 308/417 [04:13<01:40, 1.08it/s] 74%|#######4 | 309/417 [04:14<01:37, 1.11it/s] 74%|#######4 | 310/417 [04:15<01:34, 1.13it/s] 75%|#######4 | 311/417 [04:15<01:31, 1.16it/s] 75%|#######4 | 312/417 [04:16<01:28, 1.19it/s] 75%|#######5 | 313/417 [04:17<01:24, 1.23it/s] 75%|#######5 | 314/417 [04:18<01:19, 1.30it/s] 76%|#######5 | 315/417 [04:18<01:15, 1.34it/s] 76%|#######5 | 316/417 [04:19<01:15, 1.33it/s] 76%|#######6 | 317/417 [04:20<01:14, 1.34it/s] 76%|#######6 | 318/417 [04:21<01:14, 1.33it/s] 76%|#######6 | 319/417 [04:21<01:15, 1.30it/s] 77%|#######6 | 320/417 [04:22<01:12, 1.33it/s] 77%|#######6 | 321/417 [04:23<01:10, 1.37it/s] 77%|#######7 | 322/417 [04:24<01:12, 1.30it/s] 77%|#######7 | 323/417 [04:25<01:28, 1.06it/s] 78%|#######7 | 324/417 [04:28<02:18, 1.49s/it] 78%|#######7 | 325/417 [04:29<02:04, 1.35s/it] 78%|#######8 | 326/417 [04:30<01:54, 1.26s/it] 78%|#######8 | 327/417 [04:31<01:46, 1.19s/it] 79%|#######8 | 328/417 [04:32<01:36, 1.09s/it] 79%|#######8 | 329/417 [04:32<01:26, 1.02it/s] 79%|#######9 | 330/417 [04:33<01:19, 1.09it/s] 79%|#######9 | 331/417 [04:34<01:13, 1.17it/s] 80%|#######9 | 332/417 [04:35<01:08, 1.25it/s] 80%|#######9 | 333/417 [04:35<01:04, 1.29it/s] 80%|######## | 334/417 [04:36<01:03, 1.30it/s] 80%|######## | 335/417 [04:37<01:03, 1.28it/s] 81%|######## | 336/417 [04:38<01:08, 1.17it/s] 81%|######## | 337/417 [04:39<01:05, 1.22it/s] 81%|########1 | 338/417 [04:39<01:02, 1.27it/s] 81%|########1 | 339/417 [04:40<01:04, 1.20it/s] 82%|########1 | 340/417 [04:41<01:08, 1.13it/s] 82%|########1 | 341/417 [04:43<01:15, 1.01it/s] 82%|########2 | 342/417 [04:43<01:10, 1.06it/s] 82%|########2 | 343/417 [04:44<01:06, 1.11it/s] 82%|########2 | 344/417 [04:45<01:03, 1.14it/s] 83%|########2 | 345/417 [04:46<01:06, 1.08it/s] 83%|########2 | 346/417 [04:47<01:00, 1.17it/s] 83%|########3 | 347/417 [04:47<00:57, 1.21it/s] 83%|########3 | 348/417 [04:48<00:53, 1.28it/s] 84%|########3 | 349/417 [04:49<00:52, 1.29it/s] 84%|########3 | 350/417 [04:50<00:50, 1.33it/s] 84%|########4 | 351/417 [04:50<00:50, 1.31it/s] 84%|########4 | 352/417 [04:51<00:50, 1.28it/s] 85%|########4 | 353/417 [04:52<00:51, 1.25it/s] 85%|########4 | 354/417 [04:53<00:49, 1.28it/s] 85%|########5 | 355/417 [04:53<00:46, 1.33it/s] 85%|########5 | 356/417 [04:54<00:46, 1.31it/s] 86%|########5 | 357/417 [04:55<00:46, 1.29it/s] 86%|########5 | 358/417 [04:56<00:51, 1.14it/s] 86%|########6 | 359/417 [04:57<00:50, 1.14it/s] 86%|########6 | 360/417 [04:58<00:47, 1.20it/s] 87%|########6 | 361/417 [04:59<00:46, 1.21it/s] 87%|########6 | 362/417 [04:59<00:45, 1.20it/s] 87%|########7 | 363/417 [05:00<00:45, 1.20it/s] 87%|########7 | 364/417 [05:01<00:44, 1.19it/s] 88%|########7 | 365/417 [05:02<00:43, 1.19it/s] 88%|########7 | 366/417 [05:03<00:43, 1.18it/s] 88%|########8 | 367/417 [05:04<00:40, 1.24it/s] 88%|########8 | 368/417 [05:05<00:41, 1.17it/s] 88%|########8 | 369/417 [05:06<00:43, 1.11it/s] 89%|########8 | 370/417 [05:07<00:43, 1.08it/s] 89%|########8 | 371/417 [05:07<00:41, 1.11it/s] 89%|########9 | 372/417 [05:08<00:41, 1.10it/s] 89%|########9 | 373/417 [05:09<00:39, 1.10it/s] 90%|########9 | 374/417 [05:10<00:42, 1.02it/s] 90%|########9 | 375/417 [05:11<00:39, 1.06it/s] 90%|######### | 376/417 [05:12<00:37, 1.08it/s] 90%|######### | 377/417 [05:13<00:36, 1.11it/s] 91%|######### | 378/417 [05:14<00:37, 1.03it/s] 91%|######### | 379/417 [05:16<00:44, 1.16s/it] 91%|#########1| 380/417 [05:17<00:41, 1.11s/it] 91%|#########1| 381/417 [05:17<00:36, 1.01s/it] 92%|#########1| 382/417 [05:18<00:34, 1.02it/s] 92%|#########1| 383/417 [05:19<00:31, 1.06it/s] 92%|#########2| 384/417 [05:20<00:29, 1.11it/s] 92%|#########2| 385/417 [05:21<00:27, 1.14it/s] 93%|#########2| 386/417 [05:22<00:26, 1.17it/s] 93%|#########2| 387/417 [05:22<00:24, 1.21it/s] 93%|#########3| 388/417 [05:23<00:25, 1.15it/s] 93%|#########3| 389/417 [05:25<00:26, 1.05it/s] 94%|#########3| 390/417 [05:25<00:25, 1.06it/s] 94%|#########3| 391/417 [05:26<00:24, 1.05it/s] 94%|#########4| 392/417 [05:27<00:24, 1.02it/s] 94%|#########4| 393/417 [05:28<00:21, 1.10it/s] 94%|#########4| 394/417 [05:29<00:22, 1.00it/s] 95%|#########4| 395/417 [05:30<00:21, 1.05it/s] 95%|#########4| 396/417 [05:31<00:20, 1.01it/s] 95%|#########5| 397/417 [05:32<00:19, 1.02it/s] 95%|#########5| 398/417 [05:33<00:17, 1.07it/s] 96%|#########5| 399/417 [05:34<00:17, 1.03it/s] 96%|#########5| 400/417 [05:35<00:16, 1.02it/s] 96%|#########6| 401/417 [05:36<00:15, 1.02it/s] 96%|#########6| 402/417 [05:37<00:14, 1.00it/s] 97%|#########6| 403/417 [05:38<00:14, 1.03s/it] 97%|#########6| 404/417 [05:39<00:13, 1.05s/it] 97%|#########7| 405/417 [05:40<00:11, 1.00it/s] 97%|#########7| 406/417 [05:41<00:10, 1.01it/s] 98%|#########7| 407/417 [05:42<00:09, 1.03it/s] 98%|#########7| 408/417 [05:43<00:08, 1.02it/s] 98%|#########8| 409/417 [05:44<00:07, 1.06it/s] 98%|#########8| 410/417 [05:45<00:06, 1.08it/s] 99%|#########8| 411/417 [05:46<00:05, 1.05it/s] 99%|#########8| 412/417 [05:47<00:04, 1.08it/s] 99%|#########9| 413/417 [05:48<00:03, 1.05it/s] 99%|#########9| 414/417 [05:49<00:02, 1.01it/s] 100%|#########9| 415/417 [05:50<00:02, 1.01s/it] 100%|#########9| 416/417 [05:51<00:00, 1.02it/s] 100%|##########| 417/417 [05:52<00:00, 1.02s/it] [2017-07-18 15:01:29] - Scraper completed. 12497 documents retrieved. [2017-07-18 15:01:29] - ERROR - delete_elasticsearch_docs() - ClinicalOfficersScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/clinical-officers/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1ccac4e110>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/clinical-officers/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1ccac4e110>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 15:01:31] - ERROR - upload_data() - ClinicalOfficersScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1ccac4e290>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1ccac4e290>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 15:01:33] - ERROR - archive_data() - data/clinical_officers.json - An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records. [2017-07-18 15:01:34] - Completed Scraper. [Health Facilities Scraper] [2017-07-18 15:01:38] - Started Scraper. [2017-07-18 15:01:48] - ERROR - delete_elasticsearch_docs() - HealthFacilitiesScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/health-facilities/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1ccb23fc90>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/health-facilities/_delete_by_query (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1ccb23fc90>: Failed to establish a new connection: [Errno -2] Name or service not known',))) [2017-07-18 15:01:49] - ERROR - upload_data() - HealthFacilitiesScraper - ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd0789d50>: Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-cfa-htools-fnqfgsmzlye2kdtijchxm5wbcu.eu-west-1.es.amazonaws.com/:443/healthtools-ke/_bulk?refresh=true (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cd0789d50>: Failed to establish a new connection: [Errno -2] Name or service not known',))) {2017-07-18 15:01:50} - Scraper completed. 8902 records retrieved. [2017-07-18 15:01:51] - ERROR - archive_data() - data/health_facilities.json - An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records. {2017-07-18 15:01:52} - Completed Scraper.

Statistics

Average successful run time: 6 minutes

Total run time: about 2 hours

Total cpu time used: 8 minutes

Total disk space used: 370 KB

History

  • Manually ran revision 681cd4c6 and completed successfully .
    nothing changed in the database
  • Manually ran revision f6be2fa0 and failed .
    nothing changed in the database
  • Manually ran revision f6be2fa0 and completed successfully .
    nothing changed in the database
  • Manually ran revision 357e6f1e and completed successfully .
    nothing changed in the database
  • Manually ran revision 6950d828 and completed successfully .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

healthtools_ke