#+TITLE: Instant Data Scraper Utils
#+AUTHOR: Lord Devi

* Instant Data Scraper Utils
These are the tools I use to sort through data that has been scraped using the chrome plugin "Instant Data Scraper".

That data needs to be sorted, deduplicated, normalized, have everything in the correct rows, have invalid data removed, etc.

These tools aim to make the data more useful for me.

* Config Files
For each industry scraped, there are various filters and lists that these tools use to perform their task.

These files are stored in "conf.d" off of the project root.  But for each dataset, these will copied into the ".data" folder which gets created in each industry folder.

It is from that location that the filters get applied

** gbp-matching-services.txt
When someone searches for businesses in Google Maps, many businesses will show as providing multiple services.  This is what that is.

When a scrape is done using Instant Data Scraper for businesses, it will provide a single service in this column, which is the service Google thinks most matches your search.

This is why this is called "Matching Services".

For best results, this list is grown and maintained for each industry that is being targeted.  It can get quite lengthy.  But any service that Google provides us, for busineses which show up for the search term we are interested in, should have that service added to this list.

This list gets used for many things.

** gbp-business-categories.txt
This is similar to the matching services list, except that it is more reliable and accurate.  Each business in Google, will have a primary business category assigned to it.  Which makes it easy for us to track and organize things.

That is what this is.

Many of these categories may sound similar to entries in the "gbp-matching-services.txt" file. A "Pressure Washing Company" in this text file may have a service of "Pressure Watching Service".  Pretty similar.  But different.

** bad-matching-services.txt
This  is a list of entries that have shown up in the "GBP Matching Service" column in the past, which we decided to blacklist.  These terms will be removed from the final results, and used to sort and organize data.

For each industry, we want to maintain a file such as this.  Although a single "bad-matching-services.txt" could be used from one industry to another, care needs to be taken that we are not accidentally excluding terms and services that we might actually be interested in.

** unknown-blacklist.txt