#Tomahawk Charts
Contains the scrapers/crawlers and webserivce that powers Tomahawk Player charts.
Refer to docs/scrapers to add/manage scrapers
##Basic Environment Setup Read more about virtualenv @readthedocs.org
##Setup a virtualenv to separate your env's (valid for OSX, perhaps others to): $ easy_install pip $ pip install virtualenv $ pip install virtualenvwrapper
##Update your .bash_profile: #virtualenv export WORKON_HOME=~/.virtualenvs source /usr/local/bin/virtualenvwrapper.sh export PIP_VIRTUALENV_BASE=$WORKON_HOME export PIP_RESPECT_VIRTUALENV=true
$ source .bash_profile
##Create virtualenv: $ mkvirtualenv --no-site-packages scraper
##Switch to virtualenv $ workon scraper
##Install dependencies Compiling C modules on OSX using Xcode 5.1 we need to set some flags to not default to error on unknown args: see stackoverflow
(scraper)$ export CFLAGS=-Qunused-arguments
(scraper)$ export CPPFLAGS=-Qunused-arguments
(scraper)$ pip install -r requirements.txt
If you fail to build requirement on osx, check if xcode command tools is installed:
$ xcode-select --install
##Create a spider Creating spiders should be as straight forward as
scrapy genspider -t tomahawk MySpider MySpider.com
##Deploy Create egg file of the project
###Start scrapyd
(scraper)$ scrapyd
###Deploy
(scraper)$ scrapyd-deploy
scrapyd-deploy default -p tomahawk
Packing version r47-master
Deploying to project "tomahawk" in http://localhost:6800/addversion.json
Server response (200):
{"status": "ok", "project": "tomahawk", "version": "r47-master", "spiders": 9, "node_name": "spider-server"}
###Schedule a scrape
(scraper)$ curl http://localhost:6800/schedule.json -d project=tomahawk -d spider=Billboard
###References: http://virtualenvwrapper.readthedocs.org/en/latest/ http://www.jontourage.com/2011/02/09/virtualenv-pip-basics/ http://blog.sidmitra.com/manage-multiple-projects-better-with-virtuale