Python: Web scraper script on Linux Debian

Install basic scraper Lxml

sudo apt-get install python-lxml


Install QT library for getting into JS rendered web content.

sudo apt-get install python-qt4


Then simple example using Python 2.7 to take pollution value of one city

import sys
from PyQt4.QtGui import *
from PyQt4.QtCore import *
from PyQt4.QtWebKit import *
from lxml import html
class Render(QWebPage):
def __init__(self, url): = QApplication(sys.argv)
def _loadFinished(self, result):
self.frame = self.mainFrame()
url = ''
r = Render(url)
result = r.frame.toHtml()
formatted_result = str(result.toAscii())
tree = html.fromstring(formatted_result)
waw = tree.xpath('//*[@id="xatzcaqv"]/text()')
print waw

After rendering it shoud give you result like

This is just one simple way to make own statistics using scraped data for MySQL, LibreOffice Calc charts and so on.