instant-data-scraper-utils/bin/process-stage-1.md

Prompt 1

- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts. - [Project Root]: "/home/ld/mgk-scrapes" - [Current Dataset]: "[Project Root]/current-data/" - [Data Directory]: "[Current Dataset]/.data/" - [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".") - [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.) - [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state. - [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under. - [Stage 1]: "[Data Directory]/stage-1/" - [Stage 2]: "[Data Directory]/stage-2/" - [Stage 3]: "[Data Directory]/stage-3/" - [Stage 4]: "[Data Directory]/stage-4/" - [Stage 5]: "[Data Directory]/stage-5/" - [Stage 6]: "[Data Directory]/stage-6/" - [Binaries]: "[Project Root]/bin/" - [Stage 1 Binaries]: "[Binaries]/stage-1/" - [Stage 2 Binaries]: "[Binaries]/stage-2/" - [Stage 3 Binaries]: "[Binaries]/stage-3/" - [Stage 4 Binaries]: "[Binaries]/stage-4/" - [Stage 5 Binaries]: "[Binaries]/stage-5/" - [Stage 6 Binaries]: "[Binaries]/stage-6/" - [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt" - [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt" - [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt" - [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt" - [Rule 1]: All scripts need to be able to be ran from any directory. - [Rule 2]: All scripts need output what they are doing, as they are doing it. - [Rule 3]: All tasks should use halo to report successes and failures. - [Rule 4]: The python module "tqdm" can be used to report progress when appropriate. - [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for. - [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python". - [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ". - [Rule 8]: Scripts should generally have robust error checking. - [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.

Please be sure to reference project details above when factoring code, or answering questions.

I wish to modify the "process-stage-1" script:

I need this script to run the following scripts in order:

[Project Root]/bin/stage-1/prepare-stage-1.py [Project Root]/bin/delete-empty-counties.py stage-1 [Project Root]/bin/column-count-correction.py stage-1 [Project Root]/bin/stage-1/remove-obviously-bad-columns.py [Project Root]/bin/remove-empty-columns.py stage-1 [Project Root]/bin/stage-1/delete-malformed-csvs.py [Project Root]/bin/column-count-correction.py stage-1

Prompt 2

- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts. - [Project Root]: "/home/ld/mgk-scrapes" - [Current Dataset]: "[Project Root]/current-data/" - [Data Directory]: "[Current Dataset]/.data/" - [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".") - [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.) - [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state. - [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under. - [Stage 1]: "[Data Directory]/stage-1/" - [Stage 2]: "[Data Directory]/stage-2/" - [Stage 3]: "[Data Directory]/stage-3/" - [Stage 4]: "[Data Directory]/stage-4/" - [Stage 5]: "[Data Directory]/stage-5/" - [Stage 6]: "[Data Directory]/stage-6/" - [Binaries]: "[Project Root]/bin/" - [Stage 1 Binaries]: "[Binaries]/stage-1/" - [Stage 2 Binaries]: "[Binaries]/stage-2/" - [Stage 3 Binaries]: "[Binaries]/stage-3/" - [Stage 4 Binaries]: "[Binaries]/stage-4/" - [Stage 5 Binaries]: "[Binaries]/stage-5/" - [Stage 6 Binaries]: "[Binaries]/stage-6/" - [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt" - [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt" - [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt" - [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt" - [Rule 1]: All scripts need to be able to be ran from any directory. - [Rule 2]: All scripts need output what they are doing, as they are doing it. - [Rule 3]: All tasks should use halo to report successes and failures. - [Rule 4]: The python module "tqdm" can be used to report progress when appropriate. - [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for. - [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python". - [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ". - [Rule 8]: Scripts should generally have robust error checking. - [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.

Please be sure to reference project details above when factoring code, or answering questions.

I wish to modify the "process-stage-1" script:

I need this script to run the following scripts in the order in which they are listed below:

[Project Root]/bin/stage-1/prepare-stage-1.py [Project Root]/bin/delete-empty-counties.py stage-1 [Project Root]/bin/column-count-correction.py stage-1 [Project Root]/bin/stage-1/remove-obviously-bad-columns.py [Project Root]/bin/remove-empty-columns.py stage-1 [Project Root]/bin/stage-1/delete-malformed-csvs.py [Project Root]/bin/delete-empty-counties.py stage-1 [Project Root]/bin/column-count-correction.py stage-1

Prompt 3

It looks to me like the script is trying to verify the existence of Stage 2, and is deleting it, and then creating it.

I need this to be done, but not in the process-stage-2 script.

I need this task to be performed in the 'prepare-stage-2' script.

Please modify the process-stage-2 script to offload the stage 2 creation task to the prepare-stage-2 script.

After I get the process-stage-2 I will afterwards ask for the prepare-stage-2.

Prompt 4

- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts. - [Project Root]: "/home/ld/mgk-scrapes" - [Current Dataset]: "[Project Root]/current-data/" - [Data Directory]: "[Current Dataset]/.data/" - [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".") - [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.) - [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state. - [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under. - [Stage 1]: "[Data Directory]/stage-1/" - [Stage 2]: "[Data Directory]/stage-2/" - [Stage 3]: "[Data Directory]/stage-3/" - [Stage 4]: "[Data Directory]/stage-4/" - [Stage 5]: "[Data Directory]/stage-5/" - [Stage 6]: "[Data Directory]/stage-6/" - [Binaries]: "[Project Root]/bin/" - [Stage 1 Binaries]: "[Binaries]/stage-1/" - [Stage 2 Binaries]: "[Binaries]/stage-2/" - [Stage 3 Binaries]: "[Binaries]/stage-3/" - [Stage 4 Binaries]: "[Binaries]/stage-4/" - [Stage 5 Binaries]: "[Binaries]/stage-5/" - [Stage 6 Binaries]: "[Binaries]/stage-6/" - [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt" - [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt" - [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt" - [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt" - [Rule 1]: All scripts need to be able to be ran from any directory. - [Rule 2]: All scripts need output what they are doing, as they are doing it. - [Rule 3]: All tasks should use halo to report successes and failures. - [Rule 4]: The python module "tqdm" can be used to report progress when appropriate. - [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for. - [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python". - [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ". - [Rule 8]: Scripts should generally have robust error checking. - [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.

I wish to adjust the 'process-stage-1' script:

Change 1: I wish to remove the 'column-count-correction.py' script from being used in process-stage-1. Completely remove references to that script, we do not need it anymore.