The name crul comes from a mashup of the word "crawl" (as in web crawling), and "curl", one of our favorite tools for interacting with the web. crul was built from the desire to be able to transform the web (open, saas, api and dark) into a dynamically explorable data set in real time.

The crul query language allows you to transform web pages and API requests into a shapeable data set, with built in concepts of expansion into new links, and a processing language to filter, transform and export your data set to a growing collection of common destinations.

In this tutorial, the focus will be on one of the best frameworks for web crawling called Scrapy. You will learn the basics of Scrapy and how to create your first web crawler or spider. Furthermore, the tutorial gives a demonstration of extracting and storing the scraped data.

Scrapy, a web framework written in Python that is used to crawl through a website and to extract data in an efficient manner.

Newspaper is a Python3 library for News, full-text, and article metadata extraction.
Documentation : http://newspaper.readthedocs.org