Making Sense of Data



29 Sep 2016

The world moves on data! A decade ago, 10 MB was a lot of data. Now, hard disks are 10 TB! Studies suggest that we shall soon be generating more data than existing scales of TB, PB (Petabyte), EB (Exabyte), ZB (Zettabyte) or YB (Yottabyte) can even measure. Can you imagine the how large Google’s hard drives must be?
So, what are we doing with this “Big Data”? We are analysing, researching and building Business Intelligence to deliver better experiences. But for this we need meaningful data - information that humans can understand. If it’s not in an Excel file that can be filtered and sorted, the entire data is useless and provides us with no meaningful information. Enter Web Scraping and data mining.


What is Data Scraping?
Data Scraping is a way of collecting information from existing websites. Most websites give data freely, the act of extracting this data using a bot and structuring it to make sense is the essence of data scraping. It is also known as web scraping, data extraction, and web harvesting


How is Data Scraping done?
Usually, data from websites can only be viewed using a web browser. This data can be manually copy pasted into files, but the process is tedious and will take months to complete. Instead, the same results can be achieved by writing a simple program that combs these websites for data and saves them automatically in the required format on your hard drive or database.


What are the uses of Data Scraping in Business?
For business owners, data related to their business is of utmost importance. It has been shown that use of relevant data in the right way can help your business grow by over 30%. Some use cases include

  • Price comparison sites deploying bots to auto-fetch prices and product descriptions for affiliate commission.
  • Market research companies scraping data from forums and social media
  • Generating leads for business from trade directories, social media and/or competitor websites
  • Getting a list of professionals from professional listing websites like LinkedIn

Leads represent the first stage of the sales process and for most small businesses, finding new leads is one of the biggest challenge. If you target a very specific niche, web scraping can be a successful approach.


Fun Fact: We got our first few clients through Data Scraping