mgk/instant-data-scraper-utils
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity

Initial commit.

Browse source
This commit is contained in:
Lord_Devi 2024-07-16 18:08:36 -04:00
commit 1056a09945
130 changed files with 20551 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.gitignore vendored Normal file
View file

@ -0,0 +1,2 @@
current-data
instant-data-scraper-data

9
README.org Normal file
View file

@ -0,0 +1,9 @@
#+TITLE: Instant Data Scraper Utils
#+AUTHOR: Lord Devi
* Instant Data Scraper Utils
These are the tools I use to sort through data that has been scraped using the chrome plugin "Instant Data Scraper".
That data needs to be sorted, deduplicated, normalized, have everything in the correct rows, have invalid data removed, etc.
These tools aim to make the data more useful for me.

317
bin/delete-empty-counties.md Normal file
View file

@ -0,0 +1,317 @@
# Prompt 1
<project details>
- Description: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- Project Root: "/home/ld/mgk-scrapes"
- Current Dataset: "<Project Root>/current-data/"
- Data Directory: "<Current Dataset>/.data/"
- Parent Data: "<Current Dataset>/"(All directories that do not begin with a period ".")
- Stage Directories: "<Data Directory>/stage-#" (Where # is a number.)
- Stage 1: "<Data Directory>/stage-1/"
- Stage 2: "<Data Directory>/stage-2/"
- Stage 3: "<Data Directory>/stage-3/"
- Stage 4: "<Data Directory>/stage-4/"
- Stage 5: "<Data Directory>/stage-5/"
- Stage 6: "<Data Directory>/stage-6/"
- Binaries: "<Project Root>/bin/"
- Stage 1 Binaries: "<Binaries>/stage-1/"
- Stage 2 Binaries: "<Binaries>/stage-2/"
- Stage 3 Binaries: "<Binaries>/stage-3/"
- Stage 4 Binaries: "<Binaries>/stage-4/"
- Stage 5 Binaries: "<Binaries>/stage-5/"
- Stage 6 Binaries: "<Binaries>/stage-6/"
- Rule 1: All scripts need to be able to be ran from any directory.
- Rule 2: All scripts need output what they are doing, as they are doing it.
- Rule 3: All tasks should use halo to report successes and failures.
- Rule 4: The python module "tqdm" can be used to report progress when appropriate.
- Rule 5: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- Rule 6: Every script should begin with a hashbang "#!/usr/bin/env python".
- Rule 7: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- Rule 8: Scripts should generally have robust error checking.
</Project Details>
I need help creating a new script. See <Project Details> above for metadata and information regarding our project setup, configuration, purpose, and general rules to follow.
This script should be called "delete-empty-counties".
This script should iterate through the sub directories in <Stage 1> looking for empty sub-directories and then delete them.
<Stage 1> consists of <State Directories> and <County Directories> beneath the <State Directories>.
Sometimes these <County Directories> are empty of CSV files. When that is the case, we want to delete those directories.
The script should report what it is doing.
It should report "Empty counties found:" and then on the next line, provide a comma separated list of all <County Directories> we found that have no CSV files which we are going to delete.
It should then report that it is deleting the directories, and then delete them.
At the end, report a final tally of all directories deleted.
This script will reside in the <Binaries> directory, and needs to be able to be ran against different stages.
So it will need a command line option provided to it, such as these examples:
<example command line>
delete-empty-counties.py stage-1
delete-empty-counties.py stage-2
</example command line>
# Prompt 2
I got this error:
<error>
Error: The directory stage-1 does not exist.
</error>
Please ensure that the <Project Details> have been referenced to setup the correct variables at the start of the script so that the script knows where to find the <Stage Directories>.
# Prompt 3
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to modify the "delete-empty-columns" script:
<script>
#!/usr/bin/env python
# Script Name: delete-empty-counties
import os
import sys
import shutil
import halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
def get_stage_directory(stage):
"""Get the path for the specified stage directory."""
return os.path.join(DATA_DIRECTORY, stage)
def find_empty_counties(stage_directory):
"""Find all empty county directories in the given stage directory."""
empty_counties = []
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
# Check if the county directory contains any CSV files
csv_files = [f for f in os.listdir(county_path) if f.endswith('.csv')]
if not csv_files:
empty_counties.append(county_path)
return empty_counties
def delete_directories(directories):
"""Delete the given directories."""
for directory in directories:
shutil.rmtree(directory)
def main():
if len(sys.argv) != 2:
print("Usage: delete-empty-counties.py <stage-directory>")
sys.exit(1)
stage = sys.argv[1]
stage_directory = get_stage_directory(stage)
if not os.path.exists(stage_directory):
print(f"Error: The directory {stage_directory} does not exist.")
sys.exit(1)
print(f"Scanning for empty counties in {stage_directory}...")
spinner = halo.Halo(text='Scanning', spinner='dots')
spinner.start()
empty_counties = find_empty_counties(stage_directory)
spinner.succeed("Scan complete.")
if empty_counties:
print("Empty counties found:")
print(", ".join(empty_counties))
print("Deleting empty counties...")
spinner.start()
delete_directories(empty_counties)
spinner.succeed("Deletion complete.")
print(f"Total directories deleted: {len(empty_counties)}")
else:
print("No empty counties found.")
if __name__ == "__main__":
main()
</script>
This script should use pyfiglet to output the name of the script as it runs.
# Prompt 4
Can the pyfiglet portion of this script be made to strip hyphens from the script name before using pyfiglet to display the name of the script?
# Prompt 5
Ok I would like the hypens actually replaced with spaces, not stripped or removed.
# Prompt 6
Additionally, after the hyphens have been replaced with spaces, I would like the words that are left to be capitalized before printing the pyfiglet.
# Prompt 7
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
I wish to modify the 'delete-empty-counties' script:
<delete-empty-counties script>
#!/usr/bin/env python
# Script Name: delete-empty-counties
import os
import sys
import shutil
import halo
from pyfiglet import Figlet
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
def get_stage_directory(stage):
"""Get the path for the specified stage directory."""
return os.path.join(DATA_DIRECTORY, stage)
def find_empty_counties(stage_directory):
"""Find all empty county directories in the given stage directory."""
empty_counties = []
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
# Check if the county directory contains any CSV files
csv_files = [f for f in os.listdir(county_path) if f.endswith('.csv')]
if not csv_files:
empty_counties.append(county_path)
return empty_counties
def delete_directories(directories):
"""Delete the given directories."""
for directory in directories:
shutil.rmtree(directory)
def main():
figlet = Figlet(font='slant')
script_name = "delete-empty-counties".replace("-", " ").title()
print(figlet.renderText(script_name))
if len(sys.argv) != 2:
print("Usage: delete-empty-counties.py <stage-directory>")
sys.exit(1)
stage = sys.argv[1]
stage_directory = get_stage_directory(stage)
if not os.path.exists(stage_directory):
print(f"Error: The directory {stage_directory} does not exist.")
sys.exit(1)
print(f"Scanning for empty counties in {stage_directory}...")
spinner = halo.Halo(text='Scanning', spinner='dots')
spinner.start()
empty_counties = find_empty_counties(stage_directory)
spinner.succeed("Scan complete.")
if empty_counties:
print("Empty counties found:")
print(", ".join(empty_counties))
print("Deleting empty counties...")
spinner.start()
delete_directories(empty_counties)
spinner.succeed("Deletion complete.")
print(f"Total directories deleted: {len(empty_counties)}")
else:
print("No empty counties found.")
if __name__ == "__main__":
main()
</delete-empty-counties script>
Change 1: Right now it it deletes empty county directories, but I would like it to also delete empty state directories in the stage folder being processed.

107
bin/delete-empty-counties.py Executable file
View file

@ -0,0 +1,107 @@
#!/usr/bin/env python
# Script Name: delete-empty-counties
import os
import sys
import shutil
import halo
from pyfiglet import Figlet
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
def get_stage_directory(stage):
"""Get the path for the specified stage directory."""
return os.path.join(DATA_DIRECTORY, stage)
def find_empty_counties(stage_directory):
"""Find all empty county directories in the given stage directory."""
empty_counties = []
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
# Check if the county directory contains any CSV files
csv_files = [f for f in os.listdir(county_path) if f.endswith('.csv')]
if not csv_files:
empty_counties.append(county_path)
return empty_counties
def find_empty_states(stage_directory):
"""Find all empty state directories in the given stage directory."""
empty_states = []
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
county_dirs = [d for d in os.listdir(state_path) if os.path.isdir(os.path.join(state_path, d))]
if not county_dirs:
empty_states.append(state_path)
else:
# Check if all counties are empty
if all(os.path.isdir(os.path.join(state_path, county_dir)) and
not any(f.endswith('.csv') for f in os.listdir(os.path.join(state_path, county_dir)))
for county_dir in county_dirs):
empty_states.append(state_path)
return empty_states
def delete_directories(directories):
"""Delete the given directories."""
for directory in directories:
shutil.rmtree(directory)
def main():
figlet = Figlet(font='slant')
script_name = "delete-empty-counties".replace("-", " ").title()
print(figlet.renderText(script_name))
if len(sys.argv) != 2:
print("Usage: delete-empty-counties.py <stage-directory>")
sys.exit(1)
stage = sys.argv[1]
stage_directory = get_stage_directory(stage)
if not os.path.exists(stage_directory):
print(f"Error: The directory {stage_directory} does not exist.")
sys.exit(1)
print(f"Scanning for empty counties and states in {stage_directory}...")
spinner = halo.Halo(text='Scanning', spinner='dots')
spinner.start()
empty_counties = find_empty_counties(stage_directory)
empty_states = find_empty_states(stage_directory)
spinner.succeed("Scan complete.")
if empty_counties:
print("Empty counties found:")
print(", ".join(empty_counties))
print("Deleting empty counties...")
spinner.start()
delete_directories(empty_counties)
spinner.succeed("County deletion complete.")
print(f"Total county directories deleted: {len(empty_counties)}")
else:
print("No empty counties found.")
if empty_states:
print("Empty states found:")
print(", ".join(empty_states))
print("Deleting empty states...")
spinner.start()
delete_directories(empty_states)
spinner.succeed("State deletion complete.")
print(f"Total state directories deleted: {len(empty_states)}")
else:
print("No empty states found.")
if __name__ == "__main__":
main()

27
bin/fetch-census-columns.py Executable file
View file

@ -0,0 +1,27 @@
#!/usr/bin/env python
import requests
import csv
import io
# Send a GET request to the US Census Bureau dataset URL
url = "https://www2.census.gov/programs-surveys/popest/datasets/2010-2019/cities/totals/sub-est2019_all.csv"
response = requests.get(url)
# Check if the request was successful
if response.status_code == 200:
# Read the CSV content from the response
csv_content = io.StringIO(response.text)
# Create a CSV reader object
reader = csv.reader(csv_content)
# Get the column names from the first row
column_names = next(reader)
# Print the column names
print("Column names:")
for column_name in column_names:
print(column_name)
else:
print("Failed to fetch the dataset.")

32
bin/fetch-census-examples.py Executable file
View file

@ -0,0 +1,32 @@
#!/usr/bin/env python
import requests
import csv
import io
# Send a GET request to the US Census Bureau dataset URL
url = "https://www2.census.gov/programs-surveys/popest/datasets/2010-2019/cities/totals/sub-est2019_all.csv"
response = requests.get(url)
# Check if the request was successful
if response.status_code == 200:
# Read the CSV content from the response
csv_content = io.StringIO(response.text)
# Create a CSV reader object
reader = csv.DictReader(csv_content)
# Fetch a small set of example data (e.g., first 5 rows)
example_data = []
for _ in range(5):
row = next(reader)
example_data.append(row)
# Print the example data for each column
for column_name in reader.fieldnames:
print(f"Column: {column_name}")
for row in example_data:
print(f" {row[column_name]}")
print()
else:
print("Failed to fetch the dataset.")

65
bin/find-bad-review-data.py Executable file
View file

@ -0,0 +1,65 @@
#!/usr/bin/env python
import os
import csv
import re
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
# Regular expression to match the pattern "-dd"
pattern = re.compile(r'^-\d{2}$')
def check_csv_for_pattern(file_path):
"""Check if the CSV file contains a column with mostly strings that match the pattern."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return False
column_counts = [0] * len(rows[0])
for row in rows:
for idx, cell in enumerate(row):
if pattern.match(cell):
column_counts[idx] += 1
# Consider a column with mostly matching patterns if more than 50% of its cells match
threshold = len(rows) / 2
return any(count > threshold for count in column_counts)
def find_csv_files_with_pattern(stage_directory):
"""Find all CSV files in the stage directory that contain columns with the specified pattern."""
matching_files = []
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
if check_csv_for_pattern(file_path):
matching_files.append(file_path)
return matching_files
def main():
matching_files = find_csv_files_with_pattern(STAGE_2_DIRECTORY)
if matching_files:
print("CSV files containing columns with '-dd' patterns:")
for file in matching_files:
print(file)
else:
print("No CSV files with the '-dd' pattern found.")
if __name__ == "__main__":
main()

56
bin/process-all.md Normal file
View file

@ -0,0 +1,56 @@
# Prompt 1
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output llwhat they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
The Project Details above contain locations for files, data, scripts, and rules to follow when creating project scripts and toolkit.
I would like a new script created, which would reside in [Binaries]. This script should be called 'process-all'.
And it should basically just call 6 other scripts in order.
Here are the scripts it should call:
<scripts to call>
process-stage-1.py
process-stage-2.py
process-stage-3.py
process-stage-4.py
process-stage-5.py
process-stage-6.py
</scripts to call>

45
bin/process-all.py Executable file
View file

@ -0,0 +1,45 @@
#!/usr/bin/env python
# Script Name: process-all
import os
import subprocess
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
BIN_DIR = os.path.join(PROJECT_ROOT, 'bin')
def run_scripts():
"""Run the specified scripts in order."""
scripts = [
os.path.join(BIN_DIR, "process-stage-1.py"),
os.path.join(BIN_DIR, "process-stage-2.py"),
os.path.join(BIN_DIR, "process-stage-3.py"),
os.path.join(BIN_DIR, "process-stage-4.py"),
os.path.join(BIN_DIR, "process-stage-5.py"),
os.path.join(BIN_DIR, "process-stage-6.py")
]
for script in scripts:
try:
subprocess.run(["python", script], check=True)
except subprocess.CalledProcessError as e:
print(f"Error: {e}")
break
def main():
figlet = Figlet(font='slant')
script_name = "process-all".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Running all stage processing scripts...\n")
run_scripts()
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")
if __name__ == '__main__':
main()

328
bin/process-stage-1.md Normal file
View file

@ -0,0 +1,328 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to modify the "process-stage-1" script:
<script>
#!/usr/bin/env python
import os
import shutil
import subprocess
# Set the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
current_data_dir = os.path.join(project_root, 'current-data')
stage_dir = os.path.join(current_data_dir, '.data', 'stage-1')
bin_stage_1_dir = os.path.join(project_root, 'bin', 'stage-1')
def prepare_stage_1():
# Check if the stage directory exists
if os.path.exists(stage_dir):
# Remove the existing stage directory
shutil.rmtree(stage_dir)
# Create the stage directory
os.makedirs(stage_dir)
# Copy all state directories to the stage directory
for state_dir in os.listdir(current_data_dir):
state_path = os.path.join(current_data_dir, state_dir)
if os.path.isdir(state_path) and not state_dir.startswith('.'):
stage_state_path = os.path.join(stage_dir, state_dir)
shutil.copytree(state_path, stage_state_path)
def run_scripts():
scripts = [
"prepare-stage-1.py",
"delete-empty-counties.py",
"delete-malformed-csvs.py",
"process-csv-files.py"
]
for script in scripts:
subprocess.run(["python", os.path.join(bin_stage_1_dir, script)], check=True)
if __name__ == '__main__':
prepare_stage_1()
run_scripts()
</script>
I need this script to run the following scripts in order:
<scripts to run>
[Project Root]/bin/stage-1/prepare-stage-1.py
[Project Root]/bin/delete-empty-counties.py stage-1
[Project Root]/bin/column-count-correction.py stage-1
[Project Root]/bin/stage-1/remove-obviously-bad-columns.py
[Project Root]/bin/remove-empty-columns.py stage-1
[Project Root]/bin/stage-1/delete-malformed-csvs.py
[Project Root]/bin/column-count-correction.py stage-1
</scripts to run>
# Prompt 2
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to modify the "process-stage-1" script:
<script>
#!/usr/bin/env python
# Script Name: process-stage-1
import os
import shutil
import subprocess
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATA_DIR = os.path.join(PROJECT_ROOT, 'current-data')
STAGE_1_DIR = os.path.join(CURRENT_DATA_DIR, '.data', 'stage-1')
BIN_STAGE_1_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-1')
def prepare_stage_1():
"""Prepare the stage-1 directory by creating it and copying state directories into it."""
# Check if the stage directory exists
if os.path.exists(STAGE_1_DIR):
# Remove the existing stage directory
shutil.rmtree(STAGE_1_DIR)
# Create the stage directory
os.makedirs(STAGE_1_DIR)
# Copy all state directories to the stage directory
for state_dir in os.listdir(CURRENT_DATA_DIR):
state_path = os.path.join(CURRENT_DATA_DIR, state_dir)
if os.path.isdir(state_path) and not state_dir.startswith('.'):
stage_state_path = os.path.join(STAGE_1_DIR, state_dir)
shutil.copytree(state_path, stage_state_path)
def run_scripts():
"""Run the specified scripts in order."""
scripts = [
os.path.join(BIN_STAGE_1_DIR, "prepare-stage-1.py"),
os.path.join(PROJECT_ROOT, "bin", "delete-empty-counties.py"),
os.path.join(PROJECT_ROOT, "bin", "column-count-correction.py"),
os.path.join(BIN_STAGE_1_DIR, "remove-obviously-bad-columns.py"),
os.path.join(PROJECT_ROOT, "bin", "remove-empty-columns.py"),
os.path.join(BIN_STAGE_1_DIR, "delete-malformed-csvs.py"),
os.path.join(PROJECT_ROOT, "bin", "column-count-correction.py")
]
# Run scripts that do not require arguments
subprocess.run(["python", scripts[0]], check=True)
subprocess.run(["python", scripts[3]], check=True)
subprocess.run(["python", scripts[5]], check=True)
# Run scripts that require the 'stage-1' argument
subprocess.run(["python", scripts[1], "stage-1"], check=True)
subprocess.run(["python", scripts[2], "stage-1"], check=True)
subprocess.run(["python", scripts[4], "stage-1"], check=True)
subprocess.run(["python", scripts[6], "stage-1"], check=True)
if __name__ == '__main__':
prepare_stage_1()
run_scripts()
</script>
I need this script to run the following scripts in the order in which they are listed below:
<scripts to run>
[Project Root]/bin/stage-1/prepare-stage-1.py
[Project Root]/bin/delete-empty-counties.py stage-1
[Project Root]/bin/column-count-correction.py stage-1
[Project Root]/bin/stage-1/remove-obviously-bad-columns.py
[Project Root]/bin/remove-empty-columns.py stage-1
[Project Root]/bin/stage-1/delete-malformed-csvs.py
[Project Root]/bin/delete-empty-counties.py stage-1
[Project Root]/bin/column-count-correction.py stage-1
</scripts to run>
# Prompt 3
It looks to me like the script is trying to verify the existence of Stage 2, and is deleting it, and then creating it.
I need this to be done, but not in the process-stage-2 script.
I need this task to be performed in the 'prepare-stage-2' script.
Please modify the process-stage-2 script to offload the stage 2 creation task to the prepare-stage-2 script.
After I get the process-stage-2 I will afterwards ask for the prepare-stage-2.
# Prompt 4
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
I wish to adjust the 'process-stage-1' script:
<script>
#!/usr/bin/env python
# Script Name: process-stage-1
import os
import shutil
import subprocess
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATA_DIR = os.path.join(PROJECT_ROOT, 'current-data')
STAGE_1_DIR = os.path.join(CURRENT_DATA_DIR, '.data', 'stage-1')
BIN_STAGE_1_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-1')
def prepare_stage_1():
"""Prepare the stage-1 directory by creating it and copying state directories into it."""
# Check if the stage directory exists
if os.path.exists(STAGE_1_DIR):
# Remove the existing stage directory
shutil.rmtree(STAGE_1_DIR)
# Create the stage directory
os.makedirs(STAGE_1_DIR)
# Copy all state directories to the stage directory
for state_dir in os.listdir(CURRENT_DATA_DIR):
state_path = os.path.join(CURRENT_DATA_DIR, state_dir)
if os.path.isdir(state_path) and not state_dir.startswith('.'):
stage_state_path = os.path.join(STAGE_1_DIR, state_dir)
shutil.copytree(state_path, stage_state_path)
def run_scripts():
"""Run the specified scripts in order."""
scripts = [
os.path.join(BIN_STAGE_1_DIR, "prepare-stage-1.py"),
os.path.join(PROJECT_ROOT, "bin", "delete-empty-counties.py"),
os.path.join(PROJECT_ROOT, "bin", "column-count-correction.py"),
os.path.join(BIN_STAGE_1_DIR, "remove-obviously-bad-columns.py"),
os.path.join(PROJECT_ROOT, "bin", "remove-empty-columns.py"),
os.path.join(BIN_STAGE_1_DIR, "delete-malformed-csvs.py"),
os.path.join(PROJECT_ROOT, "bin", "delete-empty-counties.py"),
os.path.join(PROJECT_ROOT, "bin", "column-count-correction.py")
]
# Run scripts that do not require arguments
subprocess.run(["python", scripts[0]], check=True)
subprocess.run(["python", scripts[3]], check=True)
subprocess.run(["python", scripts[5]], check=True)
# Run scripts that require the 'stage-1' argument
subprocess.run(["python", scripts[1], "stage-1"], check=True)
subprocess.run(["python", scripts[2], "stage-1"], check=True)
subprocess.run(["python", scripts[4], "stage-1"], check=True)
subprocess.run(["python", scripts[6], "stage-1"], check=True)
subprocess.run(["python", scripts[7], "stage-1"], check=True)
if __name__ == '__main__':
prepare_stage_1()
run_scripts()
</script>
Change 1: I wish to remove the 'column-count-correction.py' script from being used in process-stage-1. Completely remove references to that script, we do not need it anymore.

51
bin/process-stage-1.py Executable file
View file

@ -0,0 +1,51 @@
#!/usr/bin/env python
# Script Name: process-stage-1
import os
import shutil
import subprocess
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATA_DIR = os.path.join(PROJECT_ROOT, 'current-data')
STAGE_1_DIR = os.path.join(CURRENT_DATA_DIR, '.data', 'stage-1')
BIN_STAGE_1_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-1')
def run_scripts():
"""Run the specified scripts in order."""
scripts = [
os.path.join(BIN_STAGE_1_DIR, "prepare-stage-1.py"),
os.path.join(PROJECT_ROOT, "bin", "delete-empty-counties.py"),
os.path.join(BIN_STAGE_1_DIR, "remove-obviously-bad-columns.py"),
os.path.join(PROJECT_ROOT, "bin", "remove-empty-columns.py"),
os.path.join(BIN_STAGE_1_DIR, "delete-malformed-csvs.py"),
os.path.join(PROJECT_ROOT, "bin", "delete-empty-counties.py"),
os.path.join(BIN_STAGE_1_DIR, "md5-the-scrapes.py") # Add the new script here
]
subprocess.run(["python", scripts[0]], check=True)
subprocess.run(["python", scripts[2]], check=True)
subprocess.run(["python", scripts[4]], check=True)
subprocess.run(["python", scripts[1], "stage-1"], check=True)
subprocess.run(["python", scripts[3], "stage-1"], check=True)
subprocess.run(["python", scripts[5], "stage-1"], check=True)
subprocess.run(["python", scripts[6]], check=True) # Run the new script here
def main():
figlet = Figlet(font='slant')
script_name = "process-stage-1".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Running scripts for Stage 1...\n")
run_scripts()
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")
if __name__ == '__main__':
main()

360
bin/process-stage-2.md Normal file
View file

@ -0,0 +1,360 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I have a script called 'process-stage-2' which is similar to the 'process-stage-1' script we worked on earlier.
I need this process-stage-2 script to be refactored to be in direct accordance with the <Project Details> above.
Here is the process-stage-2 script:
<process-stage-2>
#!/usr/bin/env python
import os
import shutil
import subprocess
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage1_dir = os.path.join(project_root, '.data', 'stage-1')
stage2_dir = os.path.join(project_root, '.data', 'stage-2')
def check_and_prepare_stage2():
# Check if the stage-2 directory exists
if os.path.exists(stage2_dir):
# Remove the existing stage-2 directory
shutil.rmtree(stage2_dir)
# Copy stage-1 directory to stage-2
shutil.copytree(stage1_dir, stage2_dir)
def run_scripts():
scripts = [
"remove-utm.py",
"sanitize-review-count.py",
"sanitize-phone-data.py",
"sanitize-gbp-business-website.py",
("remove-empty-columns.py", "stage-2")
]
for script in scripts:
if isinstance(script, tuple):
script_name, arg = script
subprocess.run(["python", script_name, arg], check=True)
else:
subprocess.run(["python", script], check=True)
if __name__ == '__main__':
check_and_prepare_stage2()
run_scripts()
</process-stage-2>
I need this script to be refactored to be more like our 'process-stage-1' script in terms of style and reporting.
Here ist he process-stage-1 script I am referencing:
<process-stage-1>
#!/usr/bin/env python
# Script Name: process-stage-1
import os
import shutil
import subprocess
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATA_DIR = os.path.join(PROJECT_ROOT, 'current-data')
STAGE_1_DIR = os.path.join(CURRENT_DATA_DIR, '.data', 'stage-1')
BIN_STAGE_1_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-1')
def prepare_stage_1():
"""Prepare the stage-1 directory by creating it and copying state directories into it."""
# Check if the stage directory exists
if os.path.exists(STAGE_1_DIR):
# Remove the existing stage directory
shutil.rmtree(STAGE_1_DIR)
# Create the stage directory
os.makedirs(STAGE_1_DIR)
# Copy all state directories to the stage directory
for state_dir in os.listdir(CURRENT_DATA_DIR):
state_path = os.path.join(CURRENT_DATA_DIR, state_dir)
if os.path.isdir(state_path) and not state_dir.startswith('.'):
stage_state_path = os.path.join(STAGE_1_DIR, state_dir)
shutil.copytree(state_path, stage_state_path)
def run_scripts():
"""Run the specified scripts in order."""
scripts = [
os.path.join(BIN_STAGE_1_DIR, "prepare-stage-1.py"),
os.path.join(PROJECT_ROOT, "bin", "delete-empty-counties.py"),
os.path.join(PROJECT_ROOT, "bin", "column-count-correction.py"),
os.path.join(BIN_STAGE_1_DIR, "remove-obviously-bad-columns.py"),
os.path.join(PROJECT_ROOT, "bin", "remove-empty-columns.py"),
os.path.join(BIN_STAGE_1_DIR, "delete-malformed-csvs.py"),
os.path.join(PROJECT_ROOT, "bin", "delete-empty-counties.py"),
os.path.join(PROJECT_ROOT, "bin", "column-count-correction.py")
]
# Run scripts that do not require arguments
subprocess.run(["python", scripts[0]], check=True)
subprocess.run(["python", scripts[3]], check=True)
subprocess.run(["python", scripts[5]], check=True)
# Run scripts that require the 'stage-1' argument
subprocess.run(["python", scripts[1], "stage-1"], check=True)
subprocess.run(["python", scripts[2], "stage-1"], check=True)
subprocess.run(["python", scripts[4], "stage-1"], check=True)
subprocess.run(["python", scripts[6], "stage-1"], check=True)
subprocess.run(["python", scripts[7], "stage-1"], check=True)
if __name__ == '__main__':
prepare_stage_1()
run_scripts()
</process-stage-1>
Currently the process-stage-2 script is running the following scripts (and paremeters):
<current stage-2 scripts>
remove-utm.py
sanitize-review-count.py
sanitize-phone-data.py
sanitize-gbp-business-website.py
remove-empty-columns.py stage-2
</current stage-2 scripts>
This list of will need slight adjustment.
The remove-utm.py, sanitize-review-count.py, sanitize-phone-data.py, and sanitize-gbp-business-website.py scripts will be located in [Stage 2 Binaries] (as per <Project Details).
While the remove-empty-columns.py script is located in the standard [Binaries] location.
Lastly, we need to add a script to this list called 'prepare-stage-2'. This script is intended to fullfill the part of the 'process-stage-2' script which is normally responsible for ensuring a new copy of Stage 2 in place before the script continues.
For an idea of how this works, see the script above <process-stage-1> for how it calls its own version of 'prepare-stage-2', called 'prepare-stage-1'.
Here is the prepare-stage-2 script below to give you a reference point:
<prepare-stage-3>
#!/usr/bin/env python
# Script Name: prepare-stage-1
import os
import shutil
import halo
from tqdm import tqdm
from pyfiglet import Figlet
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_1_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-1")
PARENT_DATA_DIRECTORY = CURRENT_DATASET
def create_stage_1_directory():
"""Create the stage-1 directory, replacing it if it already exists."""
if os.path.exists(STAGE_1_DIRECTORY):
print("Existing Stage 1 found, deleting and replacing...")
shutil.rmtree(STAGE_1_DIRECTORY)
os.makedirs(STAGE_1_DIRECTORY)
print("Stage 1 directory created.")
def find_parent_data():
"""Find all parent data directories."""
parent_data_dirs = [
d for d in os.listdir(PARENT_DATA_DIRECTORY)
if os.path.isdir(os.path.join(PARENT_DATA_DIRECTORY, d)) and not d.startswith('.')
]
return parent_data_dirs
def copy_data_to_stage_1(parent_data_dirs):
"""Copy data from parent data directories to stage-1."""
for directory in parent_data_dirs:
src_dir = os.path.join(PARENT_DATA_DIRECTORY, directory)
dest_dir = os.path.join(STAGE_1_DIRECTORY, directory)
shutil.copytree(src_dir, dest_dir)
def verify_stage_1_data(parent_data_dirs):
"""Verify that the data in stage-1 matches the parent data."""
for directory in parent_data_dirs:
src_dir = os.path.join(PARENT_DATA_DIRECTORY, directory)
dest_dir = os.path.join(STAGE_1_DIRECTORY, directory)
for root, dirs, files in os.walk(src_dir):
for file in files:
src_file = os.path.join(root, file)
dest_file = src_file.replace(PARENT_DATA_DIRECTORY, STAGE_1_DIRECTORY)
if not os.path.exists(dest_file):
return False
return True
def main():
figlet = Figlet(font='slant')
script_name = "prepare-stage-1".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Preparing Stage 1...")
create_stage_1_directory()
parent_data_dirs = find_parent_data()
if parent_data_dirs:
print("Found data for the following states:")
print(", ".join(parent_data_dirs))
print("Copying data to Stage 1...")
spinner = halo.Halo(text='Copying data', spinner='dots')
spinner.start()
copy_data_to_stage_1(parent_data_dirs)
spinner.succeed("Data copied.")
print("Verifying Stage 1 data...")
is_valid = verify_stage_1_data(parent_data_dirs)
if is_valid:
print("Data verification successful. Stage 1 data is valid.")
else:
print("Data verification failed. Stage 1 data is not valid.")
else:
print("No parent data found.")
if __name__ == "__main__":
main()
</prepare-stage-3>
Now please refactor the 'process-stage-2' script in the ways perscribed.
# Prompt 2
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
I need to adjust the 'process-stage-2' script.
The text "Running scripts" does not need a spinnder, and it should have a new line after it.
It is causing the initial output of each script that is ran to appear on the same line as the "Running scripts" text and spinner. Which does not look good.
<process-stage-2 script>
#!/usr/bin/env python
# Script Name: process-stage-2
import os
import subprocess
from pyfiglet import Figlet
import halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
BIN_STAGE_2_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-2')
BIN_DIR = os.path.join(PROJECT_ROOT, 'bin')
def run_scripts():
"""Run the specified scripts in order."""
scripts = [
os.path.join(BIN_STAGE_2_DIR, "prepare-stage-2.py"),
os.path.join(BIN_STAGE_2_DIR, "remove-utm.py"),
os.path.join(BIN_STAGE_2_DIR, "sanitize-review-count.py"),
os.path.join(BIN_STAGE_2_DIR, "sanitize-phone-data.py"),
os.path.join(BIN_STAGE_2_DIR, "sanitize-gbp-business-website.py"),
os.path.join(BIN_DIR, "remove-empty-columns.py")
]
# Run scripts that do not require arguments
for script in scripts[:-1]:
subprocess.run(["python", script], check=True)
# Run script that requires the 'stage-2' argument
subprocess.run(["python", scripts[-1], "stage-2"], check=True)
def main():
figlet = Figlet(font='slant')
script_name = "process-stage-2".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Running scripts for Stage 2...")
spinner = halo.Halo(text='Running scripts', spinner='dots')
spinner.start()
run_scripts()
spinner.succeed("All scripts completed.")
if __name__ == '__main__':
main()
</process-stage-2 script>

46
bin/process-stage-2.py Executable file
View file

@ -0,0 +1,46 @@
#!/usr/bin/env python
# Script Name: process-stage-2
import os
import subprocess
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
BIN_STAGE_2_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-2')
BIN_DIR = os.path.join(PROJECT_ROOT, 'bin')
def run_scripts():
"""Run the specified scripts in order."""
scripts = [
os.path.join(BIN_STAGE_2_DIR, "prepare-stage-2.py"),
os.path.join(BIN_STAGE_2_DIR, "remove-utm.py"),
os.path.join(BIN_STAGE_2_DIR, "sanitize-review-count.py"),
os.path.join(BIN_STAGE_2_DIR, "sanitize-phone-data.py"),
os.path.join(BIN_STAGE_2_DIR, "sanitize-gbp-business-website.py"),
os.path.join(BIN_DIR, "remove-empty-columns.py")
]
# Run scripts that do not require arguments
for script in scripts[:-1]:
subprocess.run(["python", script], check=True)
# Run script that requires the 'stage-2' argument
subprocess.run(["python", scripts[-1], "stage-2"], check=True)
def main():
figlet = Figlet(font='slant')
script_name = "process-stage-2".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Running scripts for Stage 2...\n")
run_scripts()
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")
if __name__ == '__main__':
main()

478
bin/process-stage-3.md Normal file
View file

@ -0,0 +1,478 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please refactor the process-stage-3 script below to be in accordance with the <Project Details> above.
This script will not need to check for the existence of Stage 3, or worry about creating it. This task is handled elsewhere now.
Please create the process-stage-3 in the same style as the process-stage-2 script below:
<process-stage-2 script>
#!/usr/bin/env python
# Script Name: process-stage-2
import os
import subprocess
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
BIN_STAGE_2_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-2')
BIN_DIR = os.path.join(PROJECT_ROOT, 'bin')
def run_scripts():
"""Run the specified scripts in order."""
scripts = [
os.path.join(BIN_STAGE_2_DIR, "prepare-stage-2.py"),
os.path.join(BIN_STAGE_2_DIR, "remove-utm.py"),
os.path.join(BIN_STAGE_2_DIR, "sanitize-review-count.py"),
os.path.join(BIN_STAGE_2_DIR, "sanitize-phone-data.py"),
os.path.join(BIN_STAGE_2_DIR, "sanitize-gbp-business-website.py"),
os.path.join(BIN_DIR, "remove-empty-columns.py")
]
# Run scripts that do not require arguments
for script in scripts[:-1]:
subprocess.run(["python", script], check=True)
# Run script that requires the 'stage-2' argument
subprocess.run(["python", scripts[-1], "stage-2"], check=True)
def main():
figlet = Figlet(font='slant')
script_name = "process-stage-2".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Running scripts for Stage 2...\n")
run_scripts()
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")
if __name__ == '__main__':
main()
</process-stage-2 script>
Here is the current version of the process-stage-3 script which needs refactoring:
<process-stage-3 script>
#!/usr/bin/env python
import os
import shutil
import subprocess
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
data_dir = os.path.join(project_root, '.data')
stage2_dir = os.path.join(data_dir, 'stage-2')
stage3_dir = os.path.join(data_dir, 'stage-3')
def check_and_prepare_stage3():
# Check if the stage-3 directory exists
if os.path.exists(stage3_dir):
# Remove the existing stage-3 directory
shutil.rmtree(stage3_dir)
# Copy stage-2 directory to stage-3
shutil.copytree(stage2_dir, stage3_dir)
def run_scripts():
scripts = [
"column-search_review-count.py",
"column-search_review-rating.py",
"column-search_gbp-location.py",
"column-search_yib.py",
"column-search_gbp-business-category.py",
"column-search_gbp-matching-services.py",
"column-search_business-name.py",
("remove-empty-columns.py", "stage-3"),
"column-search_unknowns.py"
]
for script in scripts:
if isinstance(script, tuple):
script_name, arg = script
subprocess.run(["python", script_name, arg], check=True)
else:
subprocess.run(["python", script], check=True)
# Call report-unknowns.py with the "stage-3" option
subprocess.run(["python", "report-unknowns.py", "stage-3"], check=True)
if __name__ == '__main__':
check_and_prepare_stage3()
run_scripts()
</process-stage-3 script>
Item 1: The process-stage-3 script will need to call a scrip named "prepare-stage-3" script as the first script it calls.
Item 2: The following scripts which process-stage-3 calls are located in [Stage 3 Binaries]:
<stage 3 scripts>
column-search_review-count.py
column-search_review-rating.py
column-search_gbp-location.py
column-search_yib.py
column-search_gbp-business-category.py
column-search_gbp-matching-services.py
column-search_business-name.py
column-search_unknowns.py
</stage 3 scripts>
The following script is located in [Binaries].
<binary scripts>
remove-empty-columns.py
</binary scripts>
# Prompt 2
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
I ran into a problem from the 'process-stage-3' script:
<process-stage-3 script>
#!/usr/bin/env python
# Script Name: process-stage-3
import os
import subprocess
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
BIN_STAGE_3_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-3')
BIN_DIR = os.path.join(PROJECT_ROOT, 'bin')
def run_scripts():
"""Run the specified scripts in order."""
scripts = [
os.path.join(BIN_STAGE_3_DIR, "prepare-stage-3.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_review-count.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_review-rating.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_gbp-location.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_yib.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_gbp-business-category.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_gbp-matching-services.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_business-name.py"),
os.path.join(BIN_DIR, "remove-empty-columns.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_unknowns.py"),
os.path.join(BIN_DIR, "report-unknowns.py")
]
# Run scripts that do not require arguments
for script in scripts[:-2]:
subprocess.run(["python", script], check=True)
# Run script that requires the 'stage-3' argument
subprocess.run(["python", scripts[-2], "stage-3"], check=True)
subprocess.run(["python", scripts[-1], "stage-3"], check=True)
def main():
figlet = Figlet(font='slant')
script_name = "process-stage-3".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Running scripts for Stage 3...\n")
run_scripts()
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")
if __name__ == '__main__':
main()
</process-stage-3 script>
When it called the 'remove-empty-columns' script:
<remove-empty-columns script>
#!/usr/bin/env python
# Script Name: remove-empty-columns
import os
import csv
import sys
from pyfiglet import Figlet
import halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
def print_help_message():
"""Prints the help message for script usage."""
print("Usage: remove-empty-columns.py <stage-#>")
print("Example: remove-empty-columns.py stage-1")
print("The provided stage directory must exist within the .data directory of the project root.")
def delete_empty_columns(file_path):
"""Deletes empty columns from a CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return 0
headers = rows[0]
data_rows = rows[1:]
# Ensure each row has the same length as the headers
for row in data_rows:
while len(row) < len(headers):
row.append('')
# Identify non-empty columns
non_empty_columns = [i for i in range(len(headers)) if any(row[i].strip() for row in data_rows)]
empty_column_count = len(headers) - len(non_empty_columns)
# Create a new list to store the modified rows without empty columns
modified_rows = [[headers[i] for i in non_empty_columns]]
for row in data_rows:
modified_rows.append([row[i] for i in non_empty_columns])
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
return empty_column_count
def remove_empty_columns(stage_directory):
"""Remove empty columns from all CSV files in the given stage directory."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
print(f"Searching for empty columns to delete in {state_dir}...")
spinner = halo.Halo(spinner='dots', color='green')
spinner.start()
total_removed_columns = 0
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
total_removed_columns += delete_empty_columns(file_path)
spinner.succeed(f"Removed columns from {state_dir}. Columns removed: {total_removed_columns}")
def main():
figlet = Figlet(font='slant')
script_name = "remove-empty-columns".replace("-", " ").title()
print(figlet.renderText(script_name))
if len(sys.argv) != 2:
print("Error: Incorrect number of arguments.")
print_help_message()
sys.exit(1)
stage_arg = sys.argv[1]
if not stage_arg.startswith('stage-'):
print("Error: Invalid stage directory format.")
print_help_message()
sys.exit(1)
stage_dir = os.path.join(DATA_DIRECTORY, stage_arg)
if not os.path.exists(stage_dir):
print(f"Error: The directory '{stage_dir}' does not exist.")
sys.exit(1)
print(f"Removing empty columns in {stage_arg}...")
remove_empty_columns(stage_dir)
final_spinner = halo.Halo(spinner='dots', color='green')
final_spinner.succeed("Finished removing empty columns.")
if __name__ == "__main__":
main()
</remove-empty-columns script>
The problem was an error I got:
<error>
Error: Incorrect number of arguments.
Usage: remove-empty-columns.py <stage-#>
Example: remove-empty-columns.py stage-1
The provided stage directory must exist within the .data directory of the project root.
Traceback (most recent call last):
File "/home/ld/mgk-scrapes/bin/./process-stage-3.py", line 51, in <module>
main()
File "/home/ld/mgk-scrapes/bin/./process-stage-3.py", line 44, in main
run_scripts()
File "/home/ld/mgk-scrapes/bin/./process-stage-3.py", line 32, in run_scripts
subprocess.run(["python", script], check=True)
File "/usr/lib64/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '/home/ld/mgk-scrapes/bin/remove-empty-columns.py']' returned non-zero exit status 1.
</error>
If I run 'remove-empty-columns' manually such as :
<remove-empty-columns manual test>
./remove-empty-columns stage3
</remove-empty-columns manual test>
It runs correctly.
# Prompt 3
I got the following error:
<error>
Error: Incorrect number of arguments.
Usage: remove-empty-columns.py <stage-#>
Example: remove-empty-columns.py stage-1
The provided stage directory must exist within the .data directory of the project root.
Traceback (most recent call last):
File "/home/ld/mgk-scrapes/bin/./process-stage-3.py", line 52, in <module>
main()
File "/home/ld/mgk-scrapes/bin/./process-stage-3.py", line 45, in main
run_scripts()
File "/home/ld/mgk-scrapes/bin/./process-stage-3.py", line 32, in run_scripts
subprocess.run(["python", script], check=True)
File "/usr/lib64/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '/home/ld/mgk-scrapes/bin/remove-empty-columns.py']' returned non-zero exit status 1.
</error>
Are you sure the script is trying to run the command "remove-empty-columns.py stage-3"?
Here is the working copy of the script:
<script>
#!/usr/bin/env python
# Script Name: process-stage-3
import os
import subprocess
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
BIN_STAGE_3_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-3')
BIN_DIR = os.path.join(PROJECT_ROOT, 'bin')
def run_scripts():
"""Run the specified scripts in order."""
scripts = [
os.path.join(BIN_STAGE_3_DIR, "prepare-stage-3.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_review-count.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_review-rating.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_gbp-location.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_yib.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_gbp-business-category.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_gbp-matching-services.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_business-name.py"),
os.path.join(BIN_DIR, "remove-empty-columns.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_unknowns.py"),
os.path.join(BIN_DIR, "report-unknowns.py")
]
# Run scripts that do not require arguments
for script in scripts[:-2]:
subprocess.run(["python", script], check=True)
# Run scripts that require the 'stage-3' argument
subprocess.run(["python", scripts[-3], "stage-3"], check=True)
subprocess.run(["python", scripts[-2], "stage-3"], check=True)
subprocess.run(["python", scripts[-1], "stage-3"], check=True)
def main():
figlet = Figlet(font='slant')
script_name = "process-stage-3".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Running scripts for Stage 3...\n")
run_scripts()
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")
if __name__ == '__main__':
main()
</script>

53
bin/process-stage-3.py Executable file
View file

@ -0,0 +1,53 @@
#!/usr/bin/env python
# Script Name: process-stage-3
import os
import subprocess
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
BIN_STAGE_3_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-3')
BIN_DIR = os.path.join(PROJECT_ROOT, 'bin')
def run_scripts():
"""Run the specified scripts in order."""
scripts = [
os.path.join(BIN_STAGE_3_DIR, "prepare-stage-3.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_review-count.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_review-rating.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_gbp-location.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_yib.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_gbp-business-category.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_gbp-matching-services.py"),
os.path.join(BIN_STAGE_3_DIR, "column-search_business-name.py"),
(os.path.join(BIN_DIR, "remove-empty-columns.py"), "stage-3"),
os.path.join(BIN_STAGE_3_DIR, "column-search_unknowns.py"),
os.path.join(BIN_DIR, "report-unknowns.py")
]
# Run scripts that do not require arguments
for script in scripts[:-3]:
subprocess.run(["python", script], check=True)
# Run scripts that require the 'stage-3' argument
subprocess.run(["python", scripts[-3][0], scripts[-3][1]], check=True)
subprocess.run(["python", scripts[-2], "stage-3"], check=True)
subprocess.run(["python", scripts[-1], "stage-3"], check=True)
def main():
figlet = Figlet(font='slant')
script_name = "process-stage-3".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Running scripts for Stage 3...\n")
run_scripts()
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")
if __name__ == '__main__':
main()

175
bin/process-stage-4.md Normal file
View file

@ -0,0 +1,175 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please refactor the process-stage-4 script below to be in accordance with the <Project Details> above.
This script will not need to check for the existence of Stage 4, or worry about creating it. This task is handled elsewhere now. Remove any function that checks for Stage 4, creates it or recreates it.
Please create the process-stage-4 in the same style as the process-stage-2 script below:
<process-stage-2 script>
#!/usr/bin/env python
# Script Name: process-stage-2
import os
import subprocess
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
BIN_STAGE_2_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-2')
BIN_DIR = os.path.join(PROJECT_ROOT, 'bin')
def run_scripts():
"""Run the specified scripts in order."""
scripts = [
os.path.join(BIN_STAGE_2_DIR, "prepare-stage-2.py"),
os.path.join(BIN_STAGE_2_DIR, "remove-utm.py"),
os.path.join(BIN_STAGE_2_DIR, "sanitize-review-count.py"),
os.path.join(BIN_STAGE_2_DIR, "sanitize-phone-data.py"),
os.path.join(BIN_STAGE_2_DIR, "sanitize-gbp-business-website.py"),
os.path.join(BIN_DIR, "remove-empty-columns.py")
]
# Run scripts that do not require arguments
for script in scripts[:-1]:
subprocess.run(["python", script], check=True)
# Run script that requires the 'stage-2' argument
subprocess.run(["python", scripts[-1], "stage-2"], check=True)
def main():
figlet = Figlet(font='slant')
script_name = "process-stage-2".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Running scripts for Stage 2...\n")
run_scripts()
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")
if __name__ == '__main__':
main()
</process-stage-2 script>
Here is the current version of the process-stage-4 script which needs refactoring:
<process-stage-4 script>
#!/usr/bin/env python
import os
import shutil
import subprocess
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
data_dir = os.path.join(project_root, '.data')
stage2_dir = os.path.join(data_dir, 'stage-2')
stage3_dir = os.path.join(data_dir, 'stage-3')
def check_and_prepare_stage3():
# Check if the stage-3 directory exists
if os.path.exists(stage3_dir):
# Remove the existing stage-3 directory
shutil.rmtree(stage3_dir)
# Copy stage-2 directory to stage-3
shutil.copytree(stage2_dir, stage3_dir)
def run_scripts():
scripts = [
"column-search_review-count.py",
"column-search_review-rating.py",
"column-search_gbp-location.py",
"column-search_yib.py",
"column-search_gbp-business-category.py",
"column-search_gbp-matching-services.py",
"column-search_business-name.py",
("remove-empty-columns.py", "stage-3"),
"column-search_unknowns.py"
]
for script in scripts:
if isinstance(script, tuple):
script_name, arg = script
subprocess.run(["python", script_name, arg], check=True)
else:
subprocess.run(["python", script], check=True)
# Call report-unknowns.py with the "stage-3" option
subprocess.run(["python", "report-unknowns.py", "stage-3"], check=True)
if __name__ == '__main__':
check_and_prepare_stage3()
run_scripts()
</process-stage-4 script>
Item 1: The process-stage-4 script will need to call a script named "prepare-stage-4" script as the first script it calls. This script will need to be ran first by process-stage-4, and will be located in [Stage 4 Binaries]
Item 2: The following scripts which process-stage-4 calls are located in [Stage 4 Binaries]:
<stage 4 scripts>
reshift-terms.py
strip-neg-terms-from-unknown-cols.py
strip-neg-patterns-from-unknown-cols.py
delete-sponsored-listings.py
</stage 4 scripts>
The following script is located in [Binaries].
<binary scripts>
remove-empty-columns.py
report-unknowns.py
</binary scripts>
The scripts need to be ran in the following order:
<order of execution for scripts>
reshift-terms.py
strip-neg-terms-from-unknown-cols.py
strip-neg-patterns-from-unknown-cols.py
delete-sponsored-listings.py
remove-empty-columns.py
report-unknowns.py
</order of execution for scripts>

48
bin/process-stage-4.py Executable file
View file

@ -0,0 +1,48 @@
#!/usr/bin/env python
# Script Name: process-stage-4
import os
import subprocess
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
BIN_STAGE_4_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-4')
BIN_DIR = os.path.join(PROJECT_ROOT, 'bin')
def run_scripts():
"""Run the specified scripts in order."""
scripts = [
os.path.join(BIN_STAGE_4_DIR, "prepare-stage-4.py"),
os.path.join(BIN_STAGE_4_DIR, "reshift-terms.py"),
os.path.join(BIN_STAGE_4_DIR, "strip-neg-terms-from-unknown-cols.py"),
os.path.join(BIN_STAGE_4_DIR, "strip-neg-patterns-from-unknown-cols.py"),
os.path.join(BIN_STAGE_4_DIR, "delete-sponsored-listings.py"),
os.path.join(BIN_DIR, "remove-empty-columns.py"),
os.path.join(BIN_DIR, "report-unknowns.py")
]
# Run scripts that do not require arguments
for script in scripts[:-2]:
subprocess.run(["python", script], check=True)
# Run scripts that require the 'stage-4' argument
subprocess.run(["python", scripts[-2], "stage-4"], check=True)
subprocess.run(["python", scripts[-1], "stage-4"], check=True)
def main():
figlet = Figlet(font='slant')
script_name = "process-stage-4".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Running scripts for Stage 4...\n")
run_scripts()
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")
if __name__ == '__main__':
main()

116
bin/process-stage-5.md Normal file
View file

@ -0,0 +1,116 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please refactor the process-stage-5 script below to be in accordance with the <Project Details> above.
This script will not need to check for the existence of Stage 5, or worry about creating it. This task is handled elsewhere now. Remove any function that checks for Stage 5, creates it or recreates it.
Please create the process-stage-5 script in the same style as the process-stage-2 script below:
<process-stage-2 script>
#!/usr/bin/env python
# Script Name: process-stage-2
import os
import subprocess
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
BIN_STAGE_2_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-2')
BIN_DIR = os.path.join(PROJECT_ROOT, 'bin')
def run_scripts():
"""Run the specified scripts in order."""
scripts = [
os.path.join(BIN_STAGE_2_DIR, "prepare-stage-2.py"),
os.path.join(BIN_STAGE_2_DIR, "remove-utm.py"),
os.path.join(BIN_STAGE_2_DIR, "sanitize-review-count.py"),
os.path.join(BIN_STAGE_2_DIR, "sanitize-phone-data.py"),
os.path.join(BIN_STAGE_2_DIR, "sanitize-gbp-business-website.py"),
os.path.join(BIN_DIR, "remove-empty-columns.py")
]
# Run scripts that do not require arguments
for script in scripts[:-1]:
subprocess.run(["python", script], check=True)
# Run script that requires the 'stage-2' argument
subprocess.run(["python", scripts[-1], "stage-2"], check=True)
def main():
figlet = Figlet(font='slant')
script_name = "process-stage-2".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Running scripts for Stage 2...\n")
run_scripts()
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")
if __name__ == '__main__':
main()
</process-stage-2 script>
Here is the current version of the process-stage-5 script which needs refactoring:
<process-stage-5 script>
</process-stage-5 script>
Item 1: The process-stage-5 script will need to call a script named "prepare-stage-5" script as the first script it calls. This script will need to be ran first by process-stage-5, and will be located in [Stage 5 Binaries]
Item 2: The following scripts which process-stage-5 calls are located in [Stage 5 Binaries]:
<stage 5 scripts>
standardize-location-data.py
retrieve-root-domain.py
strip-brackets-from-review-count.py
</stage 5 scripts>
The scripts need to be ran in the following order:
<order of execution for scripts>
standardize-location-data.py
retrieve-root-domain.py
strip-brackets-from-review-count.py
</order of execution for scripts>

41
bin/process-stage-5.py Executable file
View file

@ -0,0 +1,41 @@
#!/usr/bin/env python
# Script Name: process-stage-5
import os
import subprocess
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
BIN_STAGE_5_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-5')
BIN_DIR = os.path.join(PROJECT_ROOT, 'bin')
def run_scripts():
"""Run the specified scripts in order."""
scripts = [
os.path.join(BIN_STAGE_5_DIR, "prepare-stage-5.py"),
os.path.join(BIN_STAGE_5_DIR, "standardize-location-data.py"),
os.path.join(BIN_STAGE_5_DIR, "retrieve-root-domain.py"),
os.path.join(BIN_STAGE_5_DIR, "strip-brackets-from-review-count.py"),
]
# Run scripts that do not require arguments
for script in scripts:
subprocess.run(["python", script], check=True)
def main():
figlet = Figlet(font='slant')
script_name = "process-stage-5".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Running scripts for Stage 5...\n")
run_scripts()
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")
if __name__ == '__main__':
main()

1026
bin/process-stage-6.md Normal file
View file

File diff suppressed because it is too large Load diff

63
bin/process-stage-6.py Executable file
View file

@ -0,0 +1,63 @@
#!/usr/bin/env python
# Script Name: process-stage-6
import os
import subprocess
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
BIN_STAGE_6_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-6')
def run_scripts():
"""Run the specified scripts in order."""
scripts = [
os.path.join(BIN_STAGE_6_DIR, "prepare-stage-6.py"),
os.path.join(BIN_STAGE_6_DIR, "pre-sort.py"),
os.path.join(BIN_STAGE_6_DIR, "trim-invalid-rows-from-merger.py"),
os.path.join(BIN_STAGE_6_DIR, "delete-bad-matching-services.py"),
os.path.join(BIN_STAGE_6_DIR, "standardize-capital-letters.py"),
os.path.join(BIN_STAGE_6_DIR, "initial-deduplication.py"),
os.path.join(BIN_STAGE_6_DIR, "business-name-strip-structures.py"),
os.path.join(BIN_STAGE_6_DIR, "business-names-with-locations.py"),
os.path.join(BIN_STAGE_6_DIR, "uuid-business-1-by-gbp-business-phone.py"),
os.path.join(BIN_STAGE_6_DIR, "uuid-business-2-by-root-domain.py"),
os.path.join(BIN_STAGE_6_DIR, "uuid-business-3-by-business-name.py"),
os.path.join(BIN_STAGE_6_DIR, "re-order-columns.py"),
os.path.join(BIN_STAGE_6_DIR, "fill-in-missing-data.py"),
os.path.join(BIN_STAGE_6_DIR, "uuid-business-4-single-location-entries.py"),
os.path.join(BIN_STAGE_6_DIR, "standardize-yib.py"),
os.path.join(BIN_STAGE_6_DIR, "uniqify-pass-1.py"),
os.path.join(BIN_STAGE_6_DIR, "uniqify-pass-2.py"),
os.path.join(BIN_STAGE_6_DIR, "services-gbp-reputation-management.py"),
os.path.join(BIN_STAGE_6_DIR, "services-gbp-optimization.py"),
os.path.join(BIN_STAGE_6_DIR, "services-needs-website.py")
]
# Run scripts that do not require arguments
for script in scripts:
spinner = Halo(text=f'Running {os.path.basename(script)}', spinner='dots')
spinner.start()
try:
subprocess.run(["python", script], check=True)
spinner.succeed(f'Successfully ran {os.path.basename(script)}')
except subprocess.CalledProcessError as e:
spinner.fail(f"Error: {e}")
break
def main():
figlet = Figlet(font='slant')
script_name = "process-stage-6".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Running scripts for Stage 6...\n")
run_scripts()
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")
if __name__ == '__main__':
main()

477
bin/remove-empty-columns.md Normal file
View file

@ -0,0 +1,477 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I have the following script which you made earlier named "remove-empty-columns".
Item 1: I would like you to refactor it according to the <Project Details> above.
Item 2: This script should be able to be ran from anywhere. It will need to be able to be called upon different Stage's using a command line option like "remove-empty-columns.py stage-1".
Item 3: I would also like this script to use a spinner to report working being done rather than a progress bar. It should report to the user state by state, and not county by county as it does not.
item 4: This script should use pyfiglet to repor the script name as it runs as the first output.
Here is the script:
<script>
#!/usr/bin/env python
import os
import csv
import sys
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
data_dir = os.path.join(project_root, '.data')
def print_help_message():
print("Usage: script.py <stage-#>")
print("Example: script.py stage-2")
print("The provided stage directory must exist within the .data directory of the project root.")
def delete_empty_columns(file_path):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
headers = rows[0]
data_rows = rows[1:]
# Ensure each row has the same length as the headers
for row in data_rows:
while len(row) < len(headers):
row.append('')
# Identify non-empty columns
non_empty_columns = [i for i in range(len(headers)) if any(row[i].strip() for row in data_rows)]
# Create a new list to store the modified rows without empty columns
modified_rows = [[headers[i] for i in non_empty_columns]]
for row in data_rows:
modified_rows.append([row[i] for i in non_empty_columns])
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def remove_empty_columns(stage_dir):
# Process for deleting empty columns
for state_dir in os.listdir(stage_dir):
state_path = os.path.join(stage_dir, state_dir)
if os.path.isdir(state_path):
# Print the state name in big print using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText(state_dir))
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Deleting empty columns in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
delete_empty_columns(file_path)
progress_bar.update(1)
if __name__ == '__main__':
if len(sys.argv) != 2:
print("Error: Incorrect number of arguments.")
print_help_message()
sys.exit(1)
stage_arg = sys.argv[1]
if not stage_arg.startswith('stage-'):
print("Error: Invalid stage directory format.")
print_help_message()
sys.exit(1)
stage_dir = os.path.join(data_dir, stage_arg)
if not os.path.exists(stage_dir):
print(f"Error: The directory '{stage_dir}' does not exist.")
sys.exit(1)
remove_empty_columns(stage_dir)
</script>
# Prompt 2
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I have the following script which you made earlier named "remove-empty-columns".
Item 1: I would like you to refactor it according to the <Project Details> above.
Item 2: This script should be able to be ran from anywhere. It will need to be able to be called upon different Stage's using a command line option like "remove-empty-columns.py stage-1".
Item 3: I would also like this script to use a spinner to report working being done rather than a progress bar. It should report to the user state by state, and not county by county as it does not.
item 4: This script should use pyfiglet to repor the script name as it runs as the first output.
Here is the script:
<script>
#!/usr/bin/env python
import os
import csv
import sys
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
data_dir = os.path.join(project_root, '.data')
def print_help_message():
print("Usage: script.py <stage-#>")
print("Example: script.py stage-2")
print("The provided stage directory must exist within the .data directory of the project root.")
def delete_empty_columns(file_path):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
headers = rows[0]
data_rows = rows[1:]
# Ensure each row has the same length as the headers
for row in data_rows:
while len(row) < len(headers):
row.append('')
# Identify non-empty columns
non_empty_columns = [i for i in range(len(headers)) if any(row[i].strip() for row in data_rows)]
# Create a new list to store the modified rows without empty columns
modified_rows = [[headers[i] for i in non_empty_columns]]
for row in data_rows:
modified_rows.append([row[i] for i in non_empty_columns])
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def remove_empty_columns(stage_dir):
# Process for deleting empty columns
for state_dir in os.listdir(stage_dir):
state_path = os.path.join(stage_dir, state_dir)
if os.path.isdir(state_path):
# Print the state name in big print using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText(state_dir))
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Deleting empty columns in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
delete_empty_columns(file_path)
progress_bar.update(1)
if __name__ == '__main__':
if len(sys.argv) != 2:
print("Error: Incorrect number of arguments.")
print_help_message()
sys.exit(1)
stage_arg = sys.argv[1]
if not stage_arg.startswith('stage-'):
print("Error: Invalid stage directory format.")
print_help_message()
sys.exit(1)
stage_dir = os.path.join(data_dir, stage_arg)
if not os.path.exists(stage_dir):
print(f"Error: The directory '{stage_dir}' does not exist.")
sys.exit(1)
remove_empty_columns(stage_dir)
</script>
I would like to modify this script:
Item 1: It should tally up the deleted columns for each state, and report on that tally.
When reporting as:
<old reporting>
Finsihed processing <state>.
</old reporting>
I want it instead to say:
<new reporting>
Removed columns from <state>. Columns removed: <tally>
</new reporting>
Where <tally> is the tally of columns removed from that state.
Item 2: The first line of output besides the pyfiglet header is a line that reads:
<output line 1>
Correcting column counts in /home/ld/mgk-scrapes/current-data/.data/stage-1...
</output line 1>
This should report as:
<new output line 1>
Removing empty columns in <stage>
</new output line 1>
Where <stage> is the stage being operated on.
Item 3: I wish to modify this reporting line too:
<begin processing>
Correcting CSV files in <state>
</begin processing>
I would like the above line to be modified to the following instead:
<new processing message>
Searching for empty columns to delete in <state>
</new processing message>
# Prompt 3
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to modify the "remove-empty-columns" script:
<script>
#!/usr/bin/env python
# Script Name: remove-empty-columns
import os
import csv
import sys
from pyfiglet import Figlet
import halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
def print_help_message():
"""Prints the help message for script usage."""
print("Usage: remove-empty-columns.py <stage-#>")
print("Example: remove-empty-columns.py stage-1")
print("The provided stage directory must exist within the .data directory of the project root.")
def delete_empty_columns(file_path):
"""Deletes empty columns from a CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return 0
headers = rows[0]
data_rows = rows[1:]
# Ensure each row has the same length as the headers
for row in data_rows:
while len(row) < len(headers):
row.append('')
# Identify non-empty columns
non_empty_columns = [i for i in range(len(headers)) if any(row[i].strip() for row in data_rows)]
empty_column_count = len(headers) - len(non_empty_columns)
# Create a new list to store the modified rows without empty columns
modified_rows = [[headers[i] for i in non_empty_columns]]
for row in data_rows:
modified_rows.append([row[i] for i in non_empty_columns])
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
return empty_column_count
def remove_empty_columns(stage_directory):
"""Remove empty columns from all CSV files in the given stage directory."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
print(f"Searching for empty columns to delete in {state_dir}...")
spinner = halo.Halo(spinner='dots', color='green')
spinner.start()
total_removed_columns = 0
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
total_removed_columns += delete_empty_columns(file_path)
spinner.succeed(f"Removed columns from {state_dir}. Columns removed: {total_removed_columns}")
def main():
figlet = Figlet(font='slant')
print(figlet.renderText('remove-empty-columns'))
if len(sys.argv) != 2:
print("Error: Incorrect number of arguments.")
print_help_message()
sys.exit(1)
stage_arg = sys.argv[1]
if not stage_arg.startswith('stage-'):
print("Error: Invalid stage directory format.")
print_help_message()
sys.exit(1)
stage_dir = os.path.join(DATA_DIRECTORY, stage_arg)
if not os.path.exists(stage_dir):
print(f"Error: The directory '{stage_dir}' does not exist.")
sys.exit(1)
print(f"Removing empty columns in {stage_arg}...")
remove_empty_columns(stage_dir)
final_spinner = halo.Halo(spinner='dots', color='green')
final_spinner.succeed("Finished removing empty columns.")
if __name__ == "__main__":
main()
</script>
I would like this scripts pyfiglet function to be modified.
Before displaying the pyfiglet banner as the first output, I would like the string that is output to be modified first.
The hyphens in the script name should be replaced with spaces. Then the words that are left behind in the script name should be capitalized. Then we can print that version of the script name with pyfiglet.

103
bin/remove-empty-columns.py Executable file
View file

@ -0,0 +1,103 @@
#!/usr/bin/env python
# Script Name: remove-empty-columns
import os
import csv
import sys
from pyfiglet import Figlet
import halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
def print_help_message():
"""Prints the help message for script usage."""
print("Usage: remove-empty-columns.py <stage-#>")
print("Example: remove-empty-columns.py stage-1")
print("The provided stage directory must exist within the .data directory of the project root.")
def delete_empty_columns(file_path):
"""Deletes empty columns from a CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return 0
headers = rows[0]
data_rows = rows[1:]
# Ensure each row has the same length as the headers
for row in data_rows:
while len(row) < len(headers):
row.append('')
# Identify non-empty columns
non_empty_columns = [i for i in range(len(headers)) if any(row[i].strip() for row in data_rows)]
empty_column_count = len(headers) - len(non_empty_columns)
# Create a new list to store the modified rows without empty columns
modified_rows = [[headers[i] for i in non_empty_columns]]
for row in data_rows:
modified_rows.append([row[i] for i in non_empty_columns])
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
return empty_column_count
def remove_empty_columns(stage_directory):
"""Remove empty columns from all CSV files in the given stage directory."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
print(f"Searching for empty columns to delete in {state_dir}...")
spinner = halo.Halo(spinner='dots', color='green')
spinner.start()
total_removed_columns = 0
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
total_removed_columns += delete_empty_columns(file_path)
spinner.succeed(f"Removed columns from {state_dir}. Columns removed: {total_removed_columns}")
def main():
figlet = Figlet(font='slant')
script_name = "remove-empty-columns".replace("-", " ").title()
print(figlet.renderText(script_name))
if len(sys.argv) != 2:
print("Error: Incorrect number of arguments.")
print_help_message()
sys.exit(1)
stage_arg = sys.argv[1]
if not stage_arg.startswith('stage-'):
print("Error: Invalid stage directory format.")
print_help_message()
sys.exit(1)
stage_dir = os.path.join(DATA_DIRECTORY, stage_arg)
if not os.path.exists(stage_dir):
print(f"Error: The directory '{stage_dir}' does not exist.")
sys.exit(1)
print(f"Removing empty columns in {stage_arg}...")
remove_empty_columns(stage_dir)
final_spinner = halo.Halo(spinner='dots', color='green')
final_spinner.succeed("Finished removing empty columns.")
if __name__ == "__main__":
main()

224
bin/report-unknowns.md Normal file
View file

@ -0,0 +1,224 @@
# Prompt 1
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'report-unknowns' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<report-unknowns script>
#!/usr/bin/env python
import os
import csv
import sys
from tqdm import tqdm
from pyfiglet import Figlet
def process_csv_file(file_path):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
headers = next(reader)
# Count the number of unknown columns
unknown_columns = [header for header in headers if header.startswith('unknown-')]
return len(unknown_columns)
def report_unknowns(stage_dir):
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage_dir_path = os.path.join(project_root, '.data', stage_dir)
unknown_columns_report_file = os.path.join(stage_dir_path, 'unknown-columns.csv')
# Print the report title using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText('Unknown Columns Report'))
# Clear the report file before writing
with open(unknown_columns_report_file, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(['csv', 'unknowns'])
total_unknown_columns = 0
# Process each CSV file in the specified stage directory
for state_dir in os.listdir(stage_dir_path):
state_path = os.path.join(stage_dir_path, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Reporting unknown columns in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
unknown_count = process_csv_file(file_path)
if unknown_count > 0:
total_unknown_columns += unknown_count
with open(unknown_columns_report_file, 'a', newline='') as report_file:
writer = csv.writer(report_file)
writer.writerow([file_path, unknown_count])
progress_bar.update(1)
# Print the total unknown columns found
print(f"Total Unknown Columns Found: {total_unknown_columns:,}")
if __name__ == '__main__':
if len(sys.argv) != 2:
print("Usage: ./report-unknown.py <stage-directory>")
sys.exit(1)
stage_directory = sys.argv[1]
report_unknowns(stage_directory)
</report-unknowns script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>
# Prompt 2
I wish to modify the "report-unknowns" script further.
I do not want it output progress bars for every county.
Just the output we find in the remove-utm script is fine, but reworded to be accurate for this script.
# Prompt 3
The Total Tally is not correct.
If you add up the tally of each state the number is much larger than the number reported for the Total Unknown Columns.
Please examine the output below to see what I mean:
<bad tally>
✔ Finished processing Alabama. Unknown columns found: 0
✔ Finished processing Connecticut. Unknown columns found: 0
✔ Finished processing Delaware. Unknown columns found: 0
✔ Finished processing Florida. Unknown columns found: 2
✔ Finished processing Georgia. Unknown columns found: 6
✔ Finished processing Maine. Unknown columns found: 7
✔ Finished processing Maryland. Unknown columns found: 7
✔ Finished processing New Hampshire. Unknown columns found: 7
✔ Finished processing North Carolina. Unknown columns found: 8
✔ Finished processing Pennsylvania. Unknown columns found: 8
✔ Finished processing Rhode Island. Unknown columns found: 8
✔ Finished processing South Carolina. Unknown columns found: 9
✔ Finished processing Vermont. Unknown columns found: 9
✔ Finished processing Virginia. Unknown columns found: 11
✔ Finished processing West Virginia. Unknown columns found: 11
✔ Total Unknown Columns Found: 11
</bad tally>

74
bin/report-unknowns.py Executable file
View file

@ -0,0 +1,74 @@
#!/usr/bin/env python
# Script Name: report-unknowns
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
def process_csv_file(file_path):
"""Read the CSV file and count the number of unknown columns."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
headers = next(reader)
# Count the number of unknown columns
unknown_columns = [header for header in headers if header.startswith('unknown-')]
return len(unknown_columns)
def report_unknowns(stage_dir):
"""Generate a report of unknown columns in the specified stage directory."""
stage_dir_path = os.path.join(DATA_DIRECTORY, stage_dir)
unknown_columns_report_file = os.path.join(stage_dir_path, 'unknown-columns.csv')
# Print the report title using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText('Unknown Columns Report'))
# Clear the report file before writing
with open(unknown_columns_report_file, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(['csv', 'unknowns'])
total_unknown_columns = 0
# Process each CSV file in the specified stage directory
for state_dir in os.listdir(stage_dir_path):
state_path = os.path.join(stage_dir_path, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
state_unknown_columns = 0
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
unknown_count = process_csv_file(file_path)
if unknown_count > 0:
state_unknown_columns += unknown_count
with open(unknown_columns_report_file, 'a', newline='') as report_file:
writer = csv.writer(report_file)
writer.writerow([file_path, unknown_count])
total_unknown_columns += state_unknown_columns
spinner.succeed(f'Finished processing {state_dir}. Unknown columns found: {state_unknown_columns}')
# Print the total unknown columns found
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed(f"Total Unknown Columns Found: {total_unknown_columns:,}")
if __name__ == '__main__':
if len(sys.argv) != 2:
print("Usage: ./report-unknowns.py <stage-directory>")
sys.exit(1)
stage_directory = sys.argv[1]
report_unknowns(stage_directory)

110
bin/reshift-terms.py Executable file
View file

@ -0,0 +1,110 @@
#!/usr/bin/env python
# reshift-terms
import os
import csv
import re
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage4_dir = os.path.join(project_root, '.data', 'stage-4')
data_dir = os.path.join(project_root, '.data')
# Read the list of terms for GBP Business Category and GBP Matching Service from the text files
gbp_business_categories_file = os.path.join(data_dir, 'gbp-business-categories.txt')
gbp_matching_services_file = os.path.join(data_dir, 'gbp-matching-services.txt')
# Helper function to read terms from a file
def read_terms(file_path):
if not os.path.isfile(file_path) or os.path.getsize(file_path) == 0:
print(f"Error: The file {file_path} is missing or empty.")
exit(1)
with open(file_path, 'r') as file:
terms = [line.strip().lower() for line in file if line.strip()]
if not terms:
print(f"Error: The file {file_path} does not contain any valid terms.")
exit(1)
return terms
gbp_business_categories = read_terms(gbp_business_categories_file)
gbp_matching_services = read_terms(gbp_matching_services_file)
def process_csv_file(file_path):
# Regular expressions for different data types
location_pattern = re.compile(r'^[\w\s]+, [A-Z]{2}(, United States)?$', re.IGNORECASE)
yib_pattern = re.compile(r'^\d+\+ years in business$', re.IGNORECASE)
review_rating_pattern = re.compile(r'^[1-5]\.\d$', re.IGNORECASE)
review_count_pattern = re.compile(r'^\(\d+\)$', re.IGNORECASE)
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.DictReader(file)
rows = list(reader)
if not rows:
return
headers = reader.fieldnames
# Iterate over each row
for row in rows:
for header in headers:
if header.startswith('unknown-'):
cell = row[header].strip().lower()
if location_pattern.match(cell):
if not row['GBP Location']:
row['GBP Location'] = row[header]
row[header] = ''
elif yib_pattern.match(cell):
if not row['YiB']:
row['YiB'] = row[header]
row[header] = ''
elif review_rating_pattern.match(cell):
if not row['GBP Review Rating']:
row['GBP Review Rating'] = row[header]
row[header] = ''
elif review_count_pattern.match(cell):
if not row['GBP Review Count']:
row['GBP Review Count'] = row[header]
row[header] = ''
elif cell in gbp_business_categories:
if not row['GBP Business Category']:
row['GBP Business Category'] = row[header]
row[header] = ''
elif cell in gbp_matching_services:
if not row['GBP Matching Service']:
row['GBP Matching Service'] = row[header]
row[header] = ''
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.DictWriter(file, fieldnames=headers)
writer.writeheader()
writer.writerows(rows)
def reshift_terms():
# Print the task name using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText('Reshift Terms'))
# Process each CSV file in the stage 4 directory
for state_dir in os.listdir(stage4_dir):
state_path = os.path.join(stage4_dir, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Shifting Terms in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
progress_bar.update(1)
if __name__ == '__main__':
reshift_terms()

40
bin/scrape-census.py Executable file
View file

@ -0,0 +1,40 @@
#!/usr/bin/env python
import requests
import csv
import io
# Send a GET request to the US Census Bureau dataset URL
url = "https://www2.census.gov/programs-surveys/popest/datasets/2010-2019/cities/totals/sub-est2019_all.csv"
response = requests.get(url)
# Check if the request was successful
if response.status_code == 200:
# Read the CSV content from the response
csv_content = io.StringIO(response.text)
# Create a CSV reader object
reader = csv.DictReader(csv_content)
# Open the output CSV file in write mode
output_file = "cities_data.csv"
with open(output_file, "w", newline="", encoding="utf-8") as output:
fieldnames = ["city", "state", "population"]
writer = csv.DictWriter(output, fieldnames=fieldnames)
# Write the header
writer.writeheader()
# Iterate over each row in the input CSV
for row in reader:
# Extract the required fields
city = row["NAME"]
state = row["STNAME"]
population = row["POPESTIMATE2019"]
# Write the data to the output CSV file
writer.writerow({"city": city, "state": state, "population": population})
print("Data extraction completed. Output saved as", output_file)
else:
print("Failed to fetch the dataset.")

51
bin/scrape-simplemaps.py Executable file
View file

@ -0,0 +1,51 @@
#!/usr/bin/env python
import requests
import csv
# Send a GET request to the SimpleMaps US Cities Database URL
url = "https://simplemaps.com/static/data/us-cities/1.73/basic/simplemaps_uscities_basicv1.73.zip"
response = requests.get(url)
# Check if the request was successful
if response.status_code == 200:
# Save the ZIP file
with open("simplemaps_uscities_basic.zip", "wb") as file:
file.write(response.content)
print("ZIP file downloaded successfully.")
else:
print("Failed to download the ZIP file.")
# Extract the CSV file from the ZIP (assuming you have already downloaded and saved the ZIP file)
import zipfile
with zipfile.ZipFile("simplemaps_uscities_basic.zip", "r") as zip_ref:
zip_ref.extractall(".")
# Read the extracted CSV file
csv_file = "uscities.csv"
output_file = "cities_data.csv"
with open(csv_file, "r", encoding="utf-8") as file:
reader = csv.DictReader(file)
# Open the output CSV file in write mode
with open(output_file, "w", newline="", encoding="utf-8") as output:
fieldnames = ["city", "state_id", "county_fips", "population"]
writer = csv.DictWriter(output, fieldnames=fieldnames)
# Write the header
writer.writeheader()
# Iterate over each row in the input CSV
for row in reader:
# Extract the required fields
city = row["city"]
state_id = row["state_id"]
county_fips = row["county_fips"]
population = row["population"]
# Write the data to the output CSV file
writer.writerow({"city": city, "state_id": state_id, "county_fips": county_fips, "population": population})
print("Data extraction completed. Output saved as", output_file)

427
bin/stage-1/delete-malformed-csvs.md Normal file
View file

@ -0,0 +1,427 @@
# Prompt 1
Please explain the following script:
<script>
#!/usr/bin/env python
import os
import csv
from tqdm import tqdm
# Set the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', '..'))
current_data_dir = os.path.join(project_root, 'current-data')
stage_dir = os.path.join(current_data_dir, '.data', 'stage-1')
def delete_malformed_csvs():
# Process each state directory in the stage directory
for state_dir in os.listdir(stage_dir):
state_path = os.path.join(stage_dir, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Deleting malformed CSV files in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
with open(file_path, 'r') as f:
reader = csv.reader(f)
rows = list(reader)
num_columns = len(rows[0])
# Check if the file has 12 or more columns
if num_columns >= 12:
os.remove(file_path)
progress_bar.update(1)
if __name__ == '__main__':
delete_malformed_csvs()
</script>
# Prompt 2
Please recreate the script for me. But refactor it according to the project details below:
<project details>
- Description: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- Project Root: "/home/ld/mgk-scrapes"
- Current Dataset: "<Project Root>/current-data/"
- Data Directory: "<Current Dataset>/.data/"
- Parent Data: "<Current Dataset>/"(All directories that do not begin with a period ".")
- Stage Directories: "<Data Directory>/stage-#" (Where # is a number.)
- Stage 1: "<Data Directory>/stage-1/"
- Stage 2: "<Data Directory>/stage-2/"
- Stage 3: "<Data Directory>/stage-3/"
- Stage 4: "<Data Directory>/stage-4/"
- Stage 5: "<Data Directory>/stage-5/"
- Stage 6: "<Data Directory>/stage-6/"
- Binaries: "<Project Root>/bin/"
- Stage 1 Binaries: "<Binaries>/stage-1/"
- Stage 2 Binaries: "<Binaries>/stage-2/"
- Stage 3 Binaries: "<Binaries>/stage-3/"
- Stage 4 Binaries: "<Binaries>/stage-4/"
- Stage 5 Binaries: "<Binaries>/stage-5/"
- Stage 6 Binaries: "<Binaries>/stage-6/"
- Rule 1: All scripts need to be able to be ran from any directory.
- Rule 2: All scripts need output what they are doing, as they are doing it.
- Rule 3: All tasks should use halo to report successes and failures.
- Rule 4: The python module "tqdm" can be used to report progress when appropriate.
- Rule 5: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- Rule 6: Every script should begin with a hashbang "#!/usr/bin/env python".
- Rule 7: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- Rule 8: Scripts should generally have robust error checking.
</Project Details>
This script is named "delete-malformed-csvs". It will reside in <Stage 1 Binaries>.
# Prompt 3
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I have the following script which you made earlier named "delete-malformed-csvs".
<script>
#!/usr/bin/env python
# Script Name: delete-malformed-csvs
import os
import csv
import shutil
from tqdm import tqdm
import halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_1_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-1")
def get_stage_directory(stage):
"""Construct the path to the specified stage directory."""
return os.path.join(DATA_DIRECTORY, stage)
def find_malformed_csvs(stage_directory):
"""Find all malformed CSV files in the given stage directory."""
malformed_csvs = []
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
with open(file_path, 'r') as f:
reader = csv.reader(f)
try:
rows = list(reader)
if len(rows) == 0 or len(rows[0]) < 12:
malformed_csvs.append(file_path)
except Exception as e:
malformed_csvs.append(file_path)
return malformed_csvs
def delete_files(files):
"""Delete the specified files."""
for file in files:
os.remove(file)
def main():
if len(sys.argv) != 2:
print("Usage: delete-malformed-csvs.py <stage-directory>")
sys.exit(1)
stage = sys.argv[1]
stage_directory = get_stage_directory(stage)
if not os.path.exists(stage_directory):
print(f"Error: The directory {stage_directory} does not exist.")
sys.exit(1)
print(f"Scanning for malformed CSV files in {stage_directory}...")
spinner = halo.Halo(text='Scanning', spinner='dots')
spinner.start()
malformed_csvs = find_malformed_csvs(stage_directory)
spinner.succeed("Scan complete.")
if malformed_csvs:
print("Malformed CSV files found:")
print(", ".join(malformed_csvs))
print("Deleting malformed CSV files...")
spinner.start()
delete_files(malformed_csvs)
spinner.succeed("Deletion complete.")
print(f"Total files deleted: {len(malformed_csvs)}")
else:
print("No malformed CSV files found.")
if __name__ == "__main__":
main()
</script>
This script is exiting with an error when I run it. The error is:
<error>
Traceback (most recent call last):
File "/home/ld/mgk-scrapes/bin/stage-1/./delete-malformed-csvs.py", line 79, in <module>
main()
File "/home/ld/mgk-scrapes/bin/stage-1/./delete-malformed-csvs.py", line 48, in main
if len(sys.argv) != 2:
^^^
NameError: name 'sys' is not defined. Did you forget to import 'sys'?
</error>
Can you please fix it?
# Prompt 4
I wish to make a modification to this script.
Change 1: This script is asking for a stage directory to operate on. This script does not need that functionality. It can simply operate on Stage 1.
# Prompt 5
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to modify the "delete-malformed-csvs" script:
<script>
#!/usr/bin/env python
# Script Name: delete-malformed-csvs
import os
import csv
import shutil
import sys
from tqdm import tqdm
import halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_1_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-1")
def find_malformed_csvs(stage_directory):
"""Find all malformed CSV files in the given stage directory."""
malformed_csvs = []
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
with open(file_path, 'r') as f:
reader = csv.reader(f)
try:
rows = list(reader)
if len(rows) == 0 or len(rows[0]) < 12:
malformed_csvs.append(file_path)
except Exception as e:
malformed_csvs.append(file_path)
return malformed_csvs
def delete_files(files):
"""Delete the specified files."""
for file in files:
os.remove(file)
def main():
stage_directory = STAGE_1_DIRECTORY
if not os.path.exists(stage_directory):
print(f"Error: The directory {stage_directory} does not exist.")
sys.exit(1)
print(f"Scanning for malformed CSV files in {stage_directory}...")
spinner = halo.Halo(text='Scanning', spinner='dots')
spinner.start()
malformed_csvs = find_malformed_csvs(stage_directory)
spinner.succeed("Scan complete.")
if malformed_csvs:
print("Malformed CSV files found:")
print(", ".join(malformed_csvs))
print("Deleting malformed CSV files...")
spinner.start()
delete_files(malformed_csvs)
spinner.succeed("Deletion complete.")
print(f"Total files deleted: {len(malformed_csvs)}")
else:
print("No malformed CSV files found.")
if __name__ == "__main__":
main()
</script>
I would like this scripts pyfiglet function to be modified.
Before displaying the pyfiglet banner as the first output, I would like the string that is output to be modified first.
The hyphens in the script name should be replaced with spaces. Then the words that are left behind in the script name should be capitalized. Then we can print that version of the script name with pyfiglet.
# Prompt 6
Please summarize the following script to me:
<script>
#!/usr/bin/env python
# Script Name: delete-malformed-csvs
import os
import csv
import shutil
import sys
import halo
from pyfiglet import Figlet
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_1_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-1")
def find_malformed_csvs(stage_directory):
"""Find all malformed CSV files in the given stage directory."""
malformed_csvs = []
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
with open(file_path, 'r') as f:
reader = csv.reader(f)
try:
rows = list(reader)
if len(rows) == 0 or len(rows[0]) < 12:
malformed_csvs.append(file_path)
except Exception as e:
malformed_csvs.append(file_path)
return malformed_csvs
def delete_files(files):
"""Delete the specified files."""
for file in files:
os.remove(file)
def main():
figlet = Figlet(font='slant')
script_name = "delete-malformed-csvs".replace("-", " ").title()
print(figlet.renderText(script_name))
stage_directory = STAGE_1_DIRECTORY
if not os.path.exists(stage_directory):
print(f"Error: The directory {stage_directory} does not exist.")
sys.exit(1)
print(f"Scanning for malformed CSV files in {stage_directory}...")
spinner = halo.Halo(text='Scanning', spinner='dots')
spinner.start()
malformed_csvs = find_malformed_csvs(stage_directory)
spinner.succeed("Scan complete.")
if malformed_csvs:
print("Malformed CSV files found:")
print(", ".join(malformed_csvs))
print("Deleting malformed CSV files...")
spinner.start()
delete_files(malformed_csvs)
spinner.succeed("Deletion complete.")
print(f"Total files deleted: {len(malformed_csvs)}")
else:
print("No malformed CSV files found.")
if __name__ == "__main__":
main()
</script>
# Prompt 6
Ok the logic behind what makes a CSV file is reversed. A CSV file is malformed if it has more than 12 columns, not fewer than. Please adjust the script accordingly.

76
bin/stage-1/delete-malformed-csvs.py Executable file
View file

@ -0,0 +1,76 @@
#!/usr/bin/env python
# Script Name: delete-malformed-csvs
import os
import csv
import shutil
import sys
import halo
from pyfiglet import Figlet
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_1_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-1")
def find_malformed_csvs(stage_directory):
"""Find all malformed CSV files in the given stage directory."""
malformed_csvs = []
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
with open(file_path, 'r') as f:
reader = csv.reader(f)
try:
rows = list(reader)
if len(rows) == 0 or len(rows[0]) > 12:
malformed_csvs.append(file_path)
except Exception as e:
malformed_csvs.append(file_path)
return malformed_csvs
def delete_files(files):
"""Delete the specified files."""
for file in files:
os.remove(file)
def main():
figlet = Figlet(font='slant')
script_name = "delete-malformed-csvs".replace("-", " ").title()
print(figlet.renderText(script_name))
stage_directory = STAGE_1_DIRECTORY
if not os.path.exists(stage_directory):
print(f"Error: The directory {stage_directory} does not exist.")
sys.exit(1)
print(f"Scanning for malformed CSV files in {stage_directory}...")
spinner = halo.Halo(text='Scanning', spinner='dots')
spinner.start()
malformed_csvs = find_malformed_csvs(stage_directory)
spinner.succeed("Scan complete.")
if malformed_csvs:
print("Malformed CSV files found:")
print(", ".join(malformed_csvs))
print("Deleting malformed CSV files...")
spinner.start()
delete_files(malformed_csvs)
spinner.succeed("Deletion complete.")
print(f"Total files deleted: {len(malformed_csvs)}")
else:
print("No malformed CSV files found.")
if __name__ == "__main__":
main()

116
bin/stage-1/md5-the-scrapes.py Executable file
View file

@ -0,0 +1,116 @@
#!/usr/bin/env python
# Script Name: md5-the-scrapes
import os
import hashlib
import pandas as pd
import pyfiglet
from halo import Halo
from tqdm import tqdm
from multiprocessing import Pool, cpu_count
# Function to calculate MD5 hash
def calculate_md5(row, exclude_columns):
md5_hash = hashlib.md5()
for column in row.index:
if column not in exclude_columns:
md5_hash.update(str(row[column]).encode('utf-8'))
return md5_hash.hexdigest()
# Function to determine if a column should be excluded based on patterns
def should_exclude(column_data):
# Ensure the column data is treated as strings
column_data = column_data.astype(str)
# Exclude column if it contains 5-star ratings (e.g., "4.1", "5.0")
if column_data.str.contains(r'^[1-5]\.\d$').any():
return True
# Exclude column if it contains review counts (e.g., "(10)", "(1,234)", "-10")
if column_data.str.contains(r'^\(\d{1,3}(?:,\d{3})*\)$').any() or column_data.str.contains(r'^-\d+$').any():
return True
# Exclude column if it contains years in business (e.g., "5+ years in business")
if column_data.str.contains(r'^\d+\+ years in business$').any():
return True
return False
# Function to check if a column contains terms from a given list
def contains_terms(column_data, terms):
column_data = column_data.astype(str)
match_count = column_data.apply(lambda x: any(term in x for term in terms)).sum()
return match_count / len(column_data) > 0.5 # Consider column matching if more than 50% cells match terms
# Function to process a single state directory
def process_state(state_dir_info):
state_dir, stage_1_directory, gbp_matching_services, gbp_business_categories = state_dir_info
state_path = os.path.join(stage_1_directory, state_dir)
spinner = Halo(text=f'Processing state: {state_dir}', spinner='dots')
spinner.start()
try:
for root, dirs, files in os.walk(state_path):
for file in files:
if file.endswith(".csv"):
file_path = os.path.join(root, file)
df = pd.read_csv(file_path)
# Identify columns to exclude
exclude_columns = []
# Identify GBP Business Categories column
gbp_business_col = None
for column in df.columns:
if contains_terms(df[column], gbp_business_categories):
gbp_business_col = column
break
# Identify GBP Matching Services column
for column in df.columns:
if column != gbp_business_col and contains_terms(df[column], gbp_matching_services):
exclude_columns.append(column)
break
# Identify other columns to exclude based on patterns
for column in df.columns:
if should_exclude(df[column]):
exclude_columns.append(column)
# Calculate MD5 for each row and add to new column "MD5 for Scrape"
df['MD5 for Scrape'] = df.apply(lambda row: calculate_md5(row, exclude_columns), axis=1)
# Save the modified CSV
df.to_csv(file_path, index=False)
spinner.succeed(f'Processing of state {state_dir} completed successfully.')
except Exception as e:
spinner.fail(f'Error processing state {state_dir}: {e}')
def main():
# Print script name using pyfiglet
script_name = "md5-the-scrapes".replace("-", " ").title()
print(pyfiglet.figlet_format(script_name))
# Define the directory structure
project_root = "/home/ld/mgk-scrapes"
stage_1_directory = os.path.join(project_root, "current-data", ".data", "stage-1")
gbp_matching_services_path = os.path.join(project_root, "current-data", ".data", "gbp-matching-services.txt")
gbp_business_categories_path = os.path.join(project_root, "current-data", ".data", "gbp-business-categories.txt")
# Load GBP Matching Services and Business Categories lists
with open(gbp_matching_services_path, 'r') as file:
gbp_matching_services = file.read().splitlines()
with open(gbp_business_categories_path, 'r') as file:
gbp_business_categories = file.read().splitlines()
# Get the list of state directories
state_dirs = [d for d in os.listdir(stage_1_directory) if os.path.isdir(os.path.join(stage_1_directory, d))]
# Prepare arguments for parallel processing
state_dir_info_list = [(state_dir, stage_1_directory, gbp_matching_services, gbp_business_categories) for state_dir in state_dirs]
# Process each state directory in parallel using multiprocessing
with Pool(cpu_count()) as pool:
pool.map(process_state, state_dir_info_list)
if __name__ == "__main__":
main()

325
bin/stage-1/prepare-stage-1.md Normal file
View file

@ -0,0 +1,325 @@
# First Prompt
<project details>
- Description: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- Project Root: "/home/ld/mgk-scrapes"
- Current Dataset: "<Project Root>/current-data/"
- Data Directory: "<Current Dataset>/.data/"
- Parent Data: "<Current Dataset>/"(All directories that do not begin with a period ".")
- Stage Directories: "<Data Directory>/stage-#" (Where # is a number.)
- Stage 1: "<Data Directory>/stage-1/"
- Stage 2: "<Data Directory>/stage-2/"
- Stage 3: "<Data Directory>/stage-3/"
- Stage 4: "<Data Directory>/stage-4/"
- Stage 5: "<Data Directory>/stage-5/"
- Stage 6: "<Data Directory>/stage-6/"
- Binaries: "<Project Root>/bin/"
- Stage 1 Binaries: "<Binaries>/stage-1/"
- Stage 2 Binaries: "<Binaries>/stage-2/"
- Stage 3 Binaries: "<Binaries>/stage-3/"
- Stage 4 Binaries: "<Binaries>/stage-4/"
- Stage 5 Binaries: "<Binaries>/stage-5/"
- Stage 6 Binaries: "<Binaries>/stage-6/"
- Rule 1: All scripts need to be able to be ran from any directory.
- Rule 2: All scripts need output what they are doing, as they are doing it.
- Rule 3: All tasks should use halo to report successes and failures.
- Rule 4: The python module "tqdm" can be used to report progress when appropriate.
- Rule 5: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- Rule 6: Every script should begin with a hashbang "#!/usr/bin/env python".
- Rule 7: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- Rule 8: Scripts should generally have robust error checking.
</Project Details>
I need help creating a new script. See <Project Details> above for metadata and information regarding our project setup, configuration, purpose, and general rules to follow.
This script should be called "prepare-stage-1".
It should begin by telling us it is preparing <Stage 1> for us.
Then it should check for the existence of Stage 1.
If Stage 1 is found to already exist, then the script should display a message that an existing Stage 1 was found, and is going to be delete and replaced.
Then the script should delete Stage 1, and replace it.
If Stage 1 was not found to already exist, the script can go straigh to creating Stage 1 for us.
Stage 1 is created by creating the "stage-1" directory in the <Data Directory>.
Once the directory for Stage 1 has been created, the script needs to check for the <Parent Data>.
This part of the script should report: "Found data for the following states:".
Then on the line after that message, it should provide a comma seperated list of all the directories found in the <Current Dataset>. (Those directories, we call the <Parent Data>).
After this, the script should then provide a message telling the user that the data is being copied, and present a spinner to the user as the data is being copied.
After the data has been copied, report to the user that data was copied.
Then the script should compare the data in <Stage 1> with the <Parent Data> to make sure all the CSV files and sub directories were copied over correctly. The data in <Stage 1> should match the <Parent Data>.
As this check is being performed, the script should report "Verifying Stage 1 data."
Once complete, the script should report whether or not the data was valid.
# Prompt 2
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to modify the "prepare-stage-1" script.
<prepare-stage-1 script>
#!/usr/bin/env python
# Script Name: prepare-stage-1
import os
import shutil
import halo
from tqdm import tqdm
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_1_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-1")
PARENT_DATA_DIRECTORY = CURRENT_DATASET
def create_stage_1_directory():
"""Create the stage-1 directory, replacing it if it already exists."""
if os.path.exists(STAGE_1_DIRECTORY):
print("Existing Stage 1 found, deleting and replacing...")
shutil.rmtree(STAGE_1_DIRECTORY)
os.makedirs(STAGE_1_DIRECTORY)
print("Stage 1 directory created.")
def find_parent_data():
"""Find all parent data directories."""
parent_data_dirs = [
d for d in os.listdir(PARENT_DATA_DIRECTORY)
if os.path.isdir(os.path.join(PARENT_DATA_DIRECTORY, d)) and not d.startswith('.')
]
return parent_data_dirs
def copy_data_to_stage_1(parent_data_dirs):
"""Copy data from parent data directories to stage-1."""
for directory in parent_data_dirs:
src_dir = os.path.join(PARENT_DATA_DIRECTORY, directory)
dest_dir = os.path.join(STAGE_1_DIRECTORY, directory)
shutil.copytree(src_dir, dest_dir)
def verify_stage_1_data(parent_data_dirs):
"""Verify that the data in stage-1 matches the parent data."""
for directory in parent_data_dirs:
src_dir = os.path.join(PARENT_DATA_DIRECTORY, directory)
dest_dir = os.path.join(STAGE_1_DIRECTORY, directory)
for root, dirs, files in os.walk(src_dir):
for file in files:
src_file = os.path.join(root, file)
dest_file = src_file.replace(PARENT_DATA_DIRECTORY, STAGE_1_DIRECTORY)
if not os.path.exists(dest_file):
return False
return True
def main():
print("Preparing Stage 1...")
create_stage_1_directory()
parent_data_dirs = find_parent_data()
if parent_data_dirs:
print("Found data for the following states:")
print(", ".join(parent_data_dirs))
print("Copying data to Stage 1...")
spinner = halo.Halo(text='Copying data', spinner='dots')
spinner.start()
copy_data_to_stage_1(parent_data_dirs)
spinner.succeed("Data copied.")
print("Verifying Stage 1 data...")
is_valid = verify_stage_1_data(parent_data_dirs)
if is_valid:
print("Data verification successful. Stage 1 data is valid.")
else:
print("Data verification failed. Stage 1 data is not valid.")
else:
print("No parent data found.")
if __name__ == "__main__":
main()
</prepare-stage-1 script>
This script should use pyfiglet to output the name of the script as it runs.
# Prompt 3
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to modify the "prepare-stage-1" script:
<script>
#!/usr/bin/env python
# Script Name: prepare-stage-1
import os
import shutil
import halo
from tqdm import tqdm
from pyfiglet import Figlet
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_1_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-1")
PARENT_DATA_DIRECTORY = CURRENT_DATASET
def create_stage_1_directory():
"""Create the stage-1 directory, replacing it if it already exists."""
if os.path.exists(STAGE_1_DIRECTORY):
print("Existing Stage 1 found, deleting and replacing...")
shutil.rmtree(STAGE_1_DIRECTORY)
os.makedirs(STAGE_1_DIRECTORY)
print("Stage 1 directory created.")
def find_parent_data():
"""Find all parent data directories."""
parent_data_dirs = [
d for d in os.listdir(PARENT_DATA_DIRECTORY)
if os.path.isdir(os.path.join(PARENT_DATA_DIRECTORY, d)) and not d.startswith('.')
]
return parent_data_dirs
def copy_data_to_stage_1(parent_data_dirs):
"""Copy data from parent data directories to stage-1."""
for directory in parent_data_dirs:
src_dir = os.path.join(PARENT_DATA_DIRECTORY, directory)
dest_dir = os.path.join(STAGE_1_DIRECTORY, directory)
shutil.copytree(src_dir, dest_dir)
def verify_stage_1_data(parent_data_dirs):
"""Verify that the data in stage-1 matches the parent data."""
for directory in parent_data_dirs:
src_dir = os.path.join(PARENT_DATA_DIRECTORY, directory)
dest_dir = os.path.join(STAGE_1_DIRECTORY, directory)
for root, dirs, files in os.walk(src_dir):
for file in files:
src_file = os.path.join(root, file)
dest_file = src_file.replace(PARENT_DATA_DIRECTORY, STAGE_1_DIRECTORY)
if not os.path.exists(dest_file):
return False
return True
def main():
figlet = Figlet(font='slant')
print(figlet.renderText('prepare-stage-1'))
print("Preparing Stage 1...")
create_stage_1_directory()
parent_data_dirs = find_parent_data()
if parent_data_dirs:
print("Found data for the following states:")
print(", ".join(parent_data_dirs))
print("Copying data to Stage 1...")
spinner = halo.Halo(text='Copying data', spinner='dots')
spinner.start()
copy_data_to_stage_1(parent_data_dirs)
spinner.succeed("Data copied.")
print("Verifying Stage 1 data...")
is_valid = verify_stage_1_data(parent_data_dirs)
if is_valid:
print("Data verification successful. Stage 1 data is valid.")
else:
print("Data verification failed. Stage 1 data is not valid.")
else:
print("No parent data found.")
if __name__ == "__main__":
main()
</script>
I would like this scripts pyfiglet function to be modified.
Before displaying the pyfiglet banner as the first output, I would like the string that is output to be modified first.
The hyphens in the script name should be replaced with spaces. Then the words that are left behind in the script name should be capitalized. Then we can print that version of the script name with pyfiglet.

87
bin/stage-1/prepare-stage-1.py Executable file
View file

@ -0,0 +1,87 @@
#!/usr/bin/env python
# Script Name: prepare-stage-1
import os
import shutil
import halo
from tqdm import tqdm
from pyfiglet import Figlet
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_1_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-1")
PARENT_DATA_DIRECTORY = CURRENT_DATASET
def create_stage_1_directory():
"""Create the stage-1 directory, replacing it if it already exists."""
if os.path.exists(STAGE_1_DIRECTORY):
print("Existing Stage 1 found, deleting and replacing...")
shutil.rmtree(STAGE_1_DIRECTORY)
os.makedirs(STAGE_1_DIRECTORY)
print("Stage 1 directory created.")
def find_parent_data():
"""Find all parent data directories."""
parent_data_dirs = [
d for d in os.listdir(PARENT_DATA_DIRECTORY)
if os.path.isdir(os.path.join(PARENT_DATA_DIRECTORY, d)) and not d.startswith('.')
]
return parent_data_dirs
def copy_data_to_stage_1(parent_data_dirs):
"""Copy data from parent data directories to stage-1."""
for directory in parent_data_dirs:
src_dir = os.path.join(PARENT_DATA_DIRECTORY, directory)
dest_dir = os.path.join(STAGE_1_DIRECTORY, directory)
shutil.copytree(src_dir, dest_dir)
def verify_stage_1_data(parent_data_dirs):
"""Verify that the data in stage-1 matches the parent data."""
for directory in parent_data_dirs:
src_dir = os.path.join(PARENT_DATA_DIRECTORY, directory)
dest_dir = os.path.join(STAGE_1_DIRECTORY, directory)
for root, dirs, files in os.walk(src_dir):
for file in files:
src_file = os.path.join(root, file)
dest_file = src_file.replace(PARENT_DATA_DIRECTORY, STAGE_1_DIRECTORY)
if not os.path.exists(dest_file):
return False
return True
def main():
figlet = Figlet(font='slant')
script_name = "prepare-stage-1".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Preparing Stage 1...")
create_stage_1_directory()
parent_data_dirs = find_parent_data()
if parent_data_dirs:
print("Found data for the following states:")
print(", ".join(parent_data_dirs))
print("Copying data to Stage 1...")
spinner = halo.Halo(text='Copying data', spinner='dots')
spinner.start()
copy_data_to_stage_1(parent_data_dirs)
spinner.succeed("Data copied.")
print("Verifying Stage 1 data...")
is_valid = verify_stage_1_data(parent_data_dirs)
if is_valid:
print("Data verification successful. Stage 1 data is valid.")
else:
print("Data verification failed. Stage 1 data is not valid.")
else:
print("No parent data found.")
if __name__ == "__main__":
main()

573
bin/stage-1/remove-obviously-bad-columns.md Normal file
View file

@ -0,0 +1,573 @@
# First Prompt
Please explain the following script:
<script>
#!/usr/bin/env python
import os
import csv
from tqdm import tqdm
from pyfiglet import Figlet
# Set the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', '..'))
current_data_dir = os.path.join(project_root, 'current-data')
stage_dir = os.path.join(current_data_dir, '.data', 'stage-1')
def process_csv_file(file_path):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
# Create a list to track columns to be deleted
num_columns = len(rows[0])
columns_to_delete = [False] * num_columns
# Apply the rules to mark columns for deletion
for col in range(num_columns - 1, -1, -1):
for row in rows:
cell_value = row[col]
if "googleusercontent" in cell_value or "googleapis" in cell_value:
columns_to_delete[col] = True
elif "www.google.com/maps/vt/data" in cell_value:
columns_to_delete[col] = True
elif " Opens " in cell_value or "geocode" in cell_value or "gstatic.com" in cell_value:
columns_to_delete[col] = True
elif cell_value.startswith("tel:+"):
columns_to_delete[col] = True
elif cell_value.startswith("⋅ "):
columns_to_delete[col] = True
elif cell_value == "Open":
columns_to_delete[col] = True
elif cell_value == "Quote":
columns_to_delete[col] = True
elif cell_value == "Share":
columns_to_delete[col] = True
elif cell_value in ["Closed", "Closes soon", "Open 24 hours", "Directions", "Booking", "Website",
"https://www.google.com/#", "Call", "Provides:", "Online estimates",
"Onsite services", "No ratings or reviews"]:
columns_to_delete[col] = True
elif cell_value.startswith('"'):
columns_to_delete[col] = True
if columns_to_delete[col]:
break
# Create a new list to store the modified rows
modified_rows = []
# Delete the marked columns
for row in rows:
modified_row = [cell for idx, cell in enumerate(row) if not columns_to_delete[idx]]
modified_rows.append(modified_row)
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files():
# Process each state directory in the stage directory
for state_dir in os.listdir(stage_dir):
state_path = os.path.join(stage_dir, state_dir)
if os.path.isdir(state_path):
# Print the state name in big print using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText(state_dir))
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Processing CSV files in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
progress_bar.update(1)
if __name__ == '__main__':
process_csv_files()
</script>
# Second Prompt
<project details>
- Description: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- Project Root: "/home/ld/mgk-scrapes"
- Current Dataset: "<Project Root>/current-data/"
- Data Directory: "<Current Dataset>/.data/"
- Parent Data: "<Current Dataset>/"(All directories that do not begin with a period ".")
- Stage Directories: "<Data Directory>/stage-#" (Where # is a number.)
- Stage 1: "<Data Directory>/stage-1/"
- Stage 2: "<Data Directory>/stage-2/"
- Stage 3: "<Data Directory>/stage-3/"
- Stage 4: "<Data Directory>/stage-4/"
- Stage 5: "<Data Directory>/stage-5/"
- Stage 6: "<Data Directory>/stage-6/"
- Binaries: "<Project Root>/bin/"
- Stage 1 Binaries: "<Binaries>/stage-1/"
- Stage 2 Binaries: "<Binaries>/stage-2/"
- Stage 3 Binaries: "<Binaries>/stage-3/"
- Stage 4 Binaries: "<Binaries>/stage-4/"
- Stage 5 Binaries: "<Binaries>/stage-5/"
- Stage 6 Binaries: "<Binaries>/stage-6/"
- Rule 1: All scripts need to be able to be ran from any directory.
- Rule 2: All scripts need output what they are doing, as they are doing it.
- Rule 3: All tasks should use halo to report successes and failures.
- Rule 4: The python module "tqdm" can be used to report progress when appropriate.
- Rule 5: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- Rule 6: Every script should begin with a hashbang "#!/usr/bin/env python".
- Rule 7: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- Rule 8: Scripts should generally have robust error checking.
</Project Details>
Ok please refactor that script to match the <Project Details> above.
This script is named "prepare-csv-files", and will reside the <Stage 1 Binaries> directory.
# Prompt 3
That script should not ask for a stage directory. It should simply operate on <Stage 1>.
# Prompt 4
I wish to make some changes to this script:
Change 1: I do not want it to report for every county is looping through, instead just display which state is being processed, and use a spinner beside it instead of a progress bar.
Change 2: The new name of this script is "remove-obviously-bad-columns".
Change 3: The Reporting line should not say "Processing CSV files in <state>, <county>". It should say "Removing obviously bad columns in <state>." and then have the progress spinner there.
Change 4: We do not need to use pyfiglet to display the state name when processing a state.
# Prompt 5
I wish to make some changes to this script:
Change 1: Instead of starting it with "Preparing CSV files in <Stage 1>..." I want it to say "Removing obviously bad columns in <Stage 1>."
Change 2: The second line the script displays is:
<second line of output>
⠙ ProcessingRemoving obviously bad columns in Alabama...".
</second line of output>
That looks ugly. The ProcessingRemoving.
Just get rid of the "⠙ Processing" portion of the reporting.
Change 3: At the end of the script, instead of saying "CSV file preparation complete." say "Obviously bad columns removed.".
Change 4: Use pyfiglet to display the name of the script as the first output from the script.
# Prompt 6
I wish to make some changes to this script:
Change 1: Remove the spinner before the first "Removing obviously bad columns in <state>. Also remove the spinner before the halo checkmark before "Finished processing <state>" after processing the first state.
Here is the output I want to fix with this change:
<bad output>
Removing obviously bad columns in /home/ld/mgk-scrapes/current-data/.data/stage-1...
⠙ Removing obviously bad columns in Alabama...
⠦ ✔ Finished processing Alabama.
</bad output>
To look correct, this output should resemble:
<good output>
Removing obviously bad columns in /home/ld/mgk-scrapes/current-data/.data/stage-1...
Removing obviously bad columns in Alabama...
✔ Finished processing Alabama.
</good output>
# Prompt 7
There are some changes I wish to make to this script:
Change 1: The halo checkmark denoting success of a completed task is no longer green. I would like the halo icons to be colored like they were before.
# Prompt 8
There are some changes I wish to make to this script:
Change 1: There are 2 checkmarks appearing before every "Finished processing <state>." line. There should only be 1. One of the checkmarks is green as desired, and the other checkmark is uncolored, as not desired.
Here is an example of the output I wish to fix:
<example output>
Removing obviously bad columns in /home/ld/mgk-scrapes/current-data/.data/stage-1...
Removing obviously bad columns in Alabama...
✔ ✔ Finished processing Alabama.
Removing obviously bad columns in Connecticut...
✔ ✔ Finished processing Connecticut.
Removing obviously bad columns in Delaware...
✔ ✔ Finished processing Delaware.
Removing obviously bad columns in Florida...
✔ ✔ Finished processing Florida.
Removing obviously bad columns in Georgia...
✔ ✔ Finished processing Georgia.
Removing obviously bad columns in Maine...
✔ ✔ Finished processing Maine.
</example output>
# Prompt 9
There are some changes I wish to make to this script:
Change 1: The final line of the script "Obviously bad columns removed." has an uncolored checkmark from halo. It should be colored green like the other checkmarks.
# Prompt 10
There are some changes I wish to make to this script:
Change 1: The final line of the script "Obviously bad columns removed." has two checkmarks in it. I only want one halo checkmark. One of the checkmarks is colored green, which is desired. The other checkmark is uncolored, which is undesired. Please remove the undesired checkmark.
Here is the undesired output:
<undesired output>
✔ Finished processing West Virginia.
✔ ✔ Obviously bad columns removed.
</undesired output>
Here is what it should look like instead:
<desired output>
✔ Finished processing West Virginia.
✔ Obviously bad columns removed.
</desired output>
# Prompt 11
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to modify the "remove-obviously-bad-columns" script:
<script>
#!/usr/bin/env python
# Script Name: remove-obviously-bad-columns
import os
import csv
import sys
from tqdm import tqdm
from pyfiglet import Figlet
import halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_1_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-1")
def process_csv_file(file_path):
"""
Process a CSV file to remove columns containing specific unwanted patterns.
Parameters:
file_path (str): The path to the CSV file to be processed.
"""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
num_columns = len(rows[0])
columns_to_delete = [False] * num_columns
for col in range(num_columns - 1, -1, -1):
for row in rows:
cell_value = row[col]
if (
"googleusercontent" in cell_value or
"googleapis" in cell_value or
"www.google.com/maps/vt/data" in cell_value or
" Opens " in cell_value or
"geocode" in cell_value or
"gstatic.com" in cell_value or
cell_value.startswith("tel:+") or
cell_value.startswith("⋅ ") or
cell_value in ["Open", "Quote", "Share", "Closed", "Closes soon", "Open 24 hours",
"Directions", "Booking", "Website", "https://www.google.com/#", "Call",
"Provides:", "Online estimates", "Onsite services", "No ratings or reviews"] or
cell_value.startswith('"')
):
columns_to_delete[col] = True
break
modified_rows = []
for row in rows:
modified_row = [cell for idx, cell in enumerate(row) if not columns_to_delete[idx]]
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""
Process all CSV files in the given stage directory to remove unwanted columns.
Parameters:
stage_directory (str): The path to the stage directory containing the CSV files to be processed.
"""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
print(f"Removing obviously bad columns in {state_dir}...")
spinner = halo.Halo(spinner='dots', color='green')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
spinner.succeed(f"Finished processing {state_dir}.")
def main():
figlet = Figlet(font='slant')
print(figlet.renderText('remove-obviously-bad-columns'))
stage_directory = STAGE_1_DIRECTORY
if not os.path.exists(stage_directory):
print(f"Error: The directory {stage_directory} does not exist.")
sys.exit(1)
print(f"Removing obviously bad columns in {stage_directory}...")
process_csv_files(stage_directory)
# Final success message with a single colored checkmark
final_spinner = halo.Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("Obviously bad columns removed.")
if __name__ == "__main__":
main()
</script>
I would like this scripts pyfiglet function to be modified.
Before displaying the pyfiglet banner as the first output, I would like the string that is output to be modified first.
The hyphens in the script name should be replaced with spaces. Then the words that are left behind in the script name should be capitalized. Then we can print that version of the script name with pyfiglet.
# Prompt 12
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to modify the "remove-obviously-bad-columns" script:
<script>
#!/usr/bin/env python
# Script Name: remove-obviously-bad-columns
import os
import csv
import sys
from pyfiglet import Figlet
import halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_1_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-1")
def process_csv_file(file_path):
"""
Process a CSV file to remove columns containing specific unwanted patterns.
Parameters:
file_path (str): The path to the CSV file to be processed.
"""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
num_columns = len(rows[0])
columns_to_delete = [False] * num_columns
for col in range(num_columns - 1, -1, -1):
for row in rows:
cell_value = row[col]
if (
"googleusercontent" in cell_value or
"googleapis" in cell_value or
"www.google.com/maps/vt/data" in cell_value or
" Opens " in cell_value or
"geocode" in cell_value or
"gstatic.com" in cell_value or
cell_value.startswith("tel:+") or
cell_value.startswith("⋅ ") or
cell_value in ["Open", "Quote", "Share", "Closed", "Closes soon", "Open 24 hours",
"Directions", "Booking", "Website", "https://www.google.com/#", "Call",
"Provides:", "Online estimates", "Onsite services", "No ratings or reviews"] or
cell_value.startswith('"')
):
columns_to_delete[col] = True
break
modified_rows = []
for row in rows:
modified_row = [cell for idx, cell in enumerate(row) if not columns_to_delete[idx]]
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""
Process all CSV files in the given stage directory to remove unwanted columns.
Parameters:
stage_directory (str): The path to the stage directory containing the CSV files to be processed.
"""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
print(f"Removing obviously bad columns in {state_dir}...")
spinner = halo.Halo(spinner='dots', color='green')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
spinner.succeed(f"Finished processing {state_dir}.")
def main():
figlet = Figlet(font='slant')
script_name = "remove-obviously-bad-columns".replace("-", " ").title()
print(figlet.renderText(script_name))
stage_directory = STAGE_1_DIRECTORY
if not os.path.exists(stage_directory):
print(f"Error: The directory {stage_directory} does not exist.")
sys.exit(1)
print(f"Removing obviously bad columns in {stage_directory}...")
process_csv_files(stage_directory)
# Final success message with a single colored checkmark
final_spinner = halo.Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("Obviously bad columns removed.")
if __name__ == "__main__":
main()
</script>
Change 1: I would like this script to add a tallying function to it. For each line that reports such as:
<current report line>
Finished processing [State].
</current report line>
I want it to instead report as such:
<new report line>
Removed bad columns in [State]. Bad columns found: [Tally]
</new report line>
Where [Tally] is the total tally of the number of bad columns which were removed from all of the CSV files in that state.
At the very end of the script, it should provide a total tally:
<total tally>
Total number of obviously bad columns deleted: [Total Tally]
</total tally>
Where [Total Tally] is the total number of all bad columns which were deleted.

117
bin/stage-1/remove-obviously-bad-columns.py Executable file
View file

@ -0,0 +1,117 @@
#!/usr/bin/env python
# Script Name: remove-obviously-bad-columns
import os
import csv
import sys
from pyfiglet import Figlet
import halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_1_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-1")
def process_csv_file(file_path):
"""
Process a CSV file to remove columns containing specific unwanted patterns.
Parameters:
file_path (str): The path to the CSV file to be processed.
Returns:
int: The number of columns removed from the CSV file.
"""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return 0
num_columns = len(rows[0])
columns_to_delete = [False] * num_columns
for col in range(num_columns - 1, -1, -1):
for row in rows:
cell_value = row[col]
if (
"googleusercontent" in cell_value or
"googleapis" in cell_value or
"www.google.com/maps/vt/data" in cell_value or
" Opens " in cell_value or
"geocode" in cell_value or
"gstatic.com" in cell_value or
cell_value.startswith("tel:+") or
cell_value.startswith("⋅ ") or
cell_value in ["Open", "Quote", "Share", "Closed", "Closes soon", "Open 24 hours",
"Directions", "Booking", "Website", "https://www.google.com/#", "Call",
"Provides:", "Online estimates", "Onsite services", "No ratings or reviews"] or
cell_value.startswith('"')
):
columns_to_delete[col] = True
break
modified_rows = []
for row in rows:
modified_row = [cell for idx, cell in enumerate(row) if not columns_to_delete[idx]]
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
return sum(columns_to_delete)
def process_csv_files(stage_directory):
"""
Process all CSV files in the given stage directory to remove unwanted columns.
Parameters:
stage_directory (str): The path to the stage directory containing the CSV files to be processed.
"""
total_bad_columns = 0
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
print(f"Removing obviously bad columns in {state_dir}...")
spinner = halo.Halo(spinner='dots', color='green')
spinner.start()
state_bad_columns = 0
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
state_bad_columns += process_csv_file(file_path)
total_bad_columns += state_bad_columns
spinner.succeed(f"Removed bad columns in {state_dir}. Bad columns found: {state_bad_columns}")
return total_bad_columns
def main():
figlet = Figlet(font='slant')
script_name = "remove-obviously-bad-columns".replace("-", " ").title()
print(figlet.renderText(script_name))
stage_directory = STAGE_1_DIRECTORY
if not os.path.exists(stage_directory):
print(f"Error: The directory {stage_directory} does not exist.")
sys.exit(1)
print(f"Removing obviously bad columns in {stage_directory}...")
total_bad_columns = process_csv_files(stage_directory)
# Final success message with a single colored checkmark
final_spinner = halo.Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed(f"Total number of obviously bad columns deleted: {total_bad_columns}")
if __name__ == "__main__":
main()

160
bin/stage-2/prepare-stage-2.md Normal file
View file

@ -0,0 +1,160 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
Now please create the prepare-stage-2 script.
Base the 'prepare-stage-2' script on the 'prepare-stage-1' script below:
<prepare-stage-1>
#!/usr/bin/env python
# Script Name: prepare-stage-1
import os
import shutil
import halo
from tqdm import tqdm
from pyfiglet import Figlet
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_1_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-1")
PARENT_DATA_DIRECTORY = CURRENT_DATASET
def create_stage_1_directory():
"""Create the stage-1 directory, replacing it if it already exists."""
if os.path.exists(STAGE_1_DIRECTORY):
print("Existing Stage 1 found, deleting and replacing...")
shutil.rmtree(STAGE_1_DIRECTORY)
os.makedirs(STAGE_1_DIRECTORY)
print("Stage 1 directory created.")
def find_parent_data():
"""Find all parent data directories."""
parent_data_dirs = [
d for d in os.listdir(PARENT_DATA_DIRECTORY)
if os.path.isdir(os.path.join(PARENT_DATA_DIRECTORY, d)) and not d.startswith('.')
]
return parent_data_dirs
def copy_data_to_stage_1(parent_data_dirs):
"""Copy data from parent data directories to stage-1."""
for directory in parent_data_dirs:
src_dir = os.path.join(PARENT_DATA_DIRECTORY, directory)
dest_dir = os.path.join(STAGE_1_DIRECTORY, directory)
shutil.copytree(src_dir, dest_dir)
def verify_stage_1_data(parent_data_dirs):
"""Verify that the data in stage-1 matches the parent data."""
for directory in parent_data_dirs:
src_dir = os.path.join(PARENT_DATA_DIRECTORY, directory)
dest_dir = os.path.join(STAGE_1_DIRECTORY, directory)
for root, dirs, files in os.walk(src_dir):
for file in files:
src_file = os.path.join(root, file)
dest_file = src_file.replace(PARENT_DATA_DIRECTORY, STAGE_1_DIRECTORY)
if not os.path.exists(dest_file):
return False
return True
def main():
figlet = Figlet(font='slant')
script_name = "prepare-stage-1".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Preparing Stage 1...")
create_stage_1_directory()
parent_data_dirs = find_parent_data()
if parent_data_dirs:
print("Found data for the following states:")
print(", ".join(parent_data_dirs))
print("Copying data to Stage 1...")
spinner = halo.Halo(text='Copying data', spinner='dots')
spinner.start()
copy_data_to_stage_1(parent_data_dirs)
spinner.succeed("Data copied.")
print("Verifying Stage 1 data...")
is_valid = verify_stage_1_data(parent_data_dirs)
if is_valid:
print("Data verification successful. Stage 1 data is valid.")
else:
print("Data verification failed. Stage 1 data is not valid.")
else:
print("No parent data found.")
if __name__ == "__main__":
main()
</prepare-stage-1>
# Prompt 2
The script errored out.
Here is the output:
<output>
Preparing Stage 2...
Stage 2 directory created.
Copying data to Stage 2...
⠙ Copying dataTraceback (most recent call last):
File "/home/ld/mgk-scrapes/bin/stage-2/./prepare-stage-2.py", line 63, in <module>
main()
File "/home/ld/mgk-scrapes/bin/stage-2/./prepare-stage-2.py", line 51, in main
copy_data_to_stage_2()
File "/home/ld/mgk-scrapes/bin/stage-2/./prepare-stage-2.py", line 27, in copy_data_to_stage_2
shutil.copytree(STAGE_1_DIRECTORY, STAGE_2_DIRECTORY)
File "/usr/lib64/python3.12/shutil.py", line 600, in copytree
return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/shutil.py", line 498, in _copytree
os.makedirs(dst, exist_ok=dirs_exist_ok)
File "<frozen os>", line 225, in makedirs
FileExistsError: [Errno 17] File exists: '/home/ld/mgk-scrapes/current-data/.data/stage-2'
</output>

70
bin/stage-2/prepare-stage-2.py Executable file
View file

@ -0,0 +1,70 @@
#!/usr/bin/env python
# Script Name: prepare-stage-2
import os
import shutil
import halo
from tqdm import tqdm
from pyfiglet import Figlet
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_1_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-1")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def create_stage_2_directory():
"""Create the stage-2 directory, replacing it if it already exists."""
if os.path.exists(STAGE_2_DIRECTORY):
print("Existing Stage 2 found, deleting and replacing...")
shutil.rmtree(STAGE_2_DIRECTORY)
os.makedirs(STAGE_2_DIRECTORY)
print("Stage 2 directory created.")
def copy_data_to_stage_2():
"""Copy data from stage-1 to stage-2."""
for item in os.listdir(STAGE_1_DIRECTORY):
s = os.path.join(STAGE_1_DIRECTORY, item)
d = os.path.join(STAGE_2_DIRECTORY, item)
if os.path.isdir(s):
shutil.copytree(s, d)
else:
shutil.copy2(s, d)
def verify_stage_2_data():
"""Verify that the data in stage-2 matches the data in stage-1."""
for root, dirs, files in os.walk(STAGE_1_DIRECTORY):
for file in files:
src_file = os.path.join(root, file)
dest_file = src_file.replace(STAGE_1_DIRECTORY, STAGE_2_DIRECTORY)
if not os.path.exists(dest_file):
return False
return True
def main():
figlet = Figlet(font='slant')
script_name = "prepare-stage-2".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Preparing Stage 2...")
create_stage_2_directory()
print("Copying data to Stage 2...")
spinner = halo.Halo(text='Copying data', spinner='dots')
spinner.start()
copy_data_to_stage_2()
spinner.succeed("Data copied.")
print("Verifying Stage 2 data...")
is_valid = verify_stage_2_data()
if is_valid:
print("Data verification successful. Stage 2 data is valid.")
else:
print("Data verification failed. Stage 2 data is not valid.")
if __name__ == "__main__":
main()

418
bin/stage-2/remove-utm.md Normal file
View file

@ -0,0 +1,418 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I have a script 'remove-utm' which needs refactoring.
Please refactor it to be in accordance with the <Project Details> above.
This script currently reports on a per county basis, and I would prefer it to report on a per state basis.
The only pyfiglet line in this script should be from [Rule 9] in <Project Details>. Get rid of the other pyfiglets.
# Prompt 2
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
The utm codes were not correctly removed.
Please make sure you are removing the utm codes in the same way the old 'remove-utm' script was.
<old remove-utm>
#!/usr/bin/env python
import os
import csv
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage2_dir = os.path.join(project_root, '.data', 'stage-2')
def process_csv_file_remove_utm(file_path):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
# Create a new list to store the modified rows
modified_rows = []
# Iterate over each row
for row in rows:
# Create a new list to store the modified cells
modified_row = []
# Iterate over each cell in the row
for cell in row:
# Check if the cell contains the "?utm" parameter
if "?utm" in cell:
# Remove the "?utm" parameter and everything after it
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def remove_utm_parameters():
# Process for removing "?utm" parameters
for state_dir in os.listdir(stage2_dir):
state_path = os.path.join(stage2_dir, state_dir)
if os.path.isdir(state_path):
# Print the state name in big print using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText(state_dir))
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Trimming UTM tracking from: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
progress_bar.update(1)
if __name__ == '__main__':
remove_utm_parameters()
</old remove-utm>
Here is some sample CSV data how where we can see utm data remaining:
<csv with utm still>
Alford Foundation and Crawl Space Repair,4.7,(272),Concrete contractor,+1 304-948-5855,Concrete Sealant,https://www.goalford.com/?utm_source=google&utm_medium=organic&utm_campaign=gmb-website,60+ years in business,"St Albans, WV, United States"
Basement Authority of West Virginia,4.8,(167),Waterproofing service,+1 304-898-1872,Concrete Leveling,https://www.basementauthorityofwv.com/service-areas/charleston-wv/?utm_source=google&utm_medium=organic&utm_campaign=gmb,20+ years in business,"Scott Depot, WV, United States"
Seal-Tite Basement Waterproofing Co,4.5,(51),Waterproofing service,+1 540-546-4724,Driveway Drains,https://www.sealtitebasement.com/?utm_source=gmb&utm_medium=organic&utm_campaign=gmb-troutville&utm_content=website,50+ years in business,"Troutville, VA, United States"
</csv with utm still>
# Prompt 3
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output llwhat they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
The Project Details above contain locations for files, data, scripts, and rules to follow when creating project scripts and toolkit.
I need to enhance the script 'remove-utm':
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>
It does a fine job right now of what it is supposed to do. We do not want to modify that aspect of the code. It works fine. However, I need to add a new string that it looks for and removes.
Currently it is designed to search for cells that have an URL in them, and that URL contains a string of "?utm" in it. If that string is found in an URL in a cell, that "?utm" and everything after it gets deleted and stripped. This helps remove tracking mechanisms from the URLs in our database essentially.
Another string I found which we need to remove is "?cid". I would like this script to handle "?cid" in the same way it handles the "?utm" string.
Please adjust the remove-utm script accordingly for me.
# Prompt 4
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output llwhat they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
The Project Details above contain locations for files, data, scripts, and rules to follow when creating project scripts and toolkit.
I would like to extend the script 'remove-utm':
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_tracking_params(file_path):
"""Remove UTM and CID parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
elif "?cid" in cell:
modified_cell = cell.split("?cid")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM and CID parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_tracking_params(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing tracking parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("Tracking parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>
In addition to the existing strings being stripped, "?uid", and "?cid", I would like to add: "?cm_mmc".

78
bin/stage-2/remove-utm.py Executable file
View file

@ -0,0 +1,78 @@
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_tracking_params(file_path):
"""Remove UTM, CID, and CMMMC parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
elif "?cid" in cell:
modified_cell = cell.split("?cid")[0]
elif "?cm_mmc" in cell:
modified_cell = cell.split("?cm_mmc")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM, CID, and CMMMC parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_tracking_params(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing tracking parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("Tracking parameters removed.")
if __name__ == "__main__":
main()

196
bin/stage-2/sanitize-gbp-business-website.md Normal file
View file

@ -0,0 +1,196 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'sanitize-gbp-business-website' now:
<sanitize-gbp-business-website script>
#!/usr/bin/env python
import os
import csv
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage2_dir = os.path.join(project_root, '.data', 'stage-2')
def sanitize_website_data(file_path):
new_column_name = 'GBP Business Website'
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
# Get the headers and add the new column name if it does not exist
headers = rows[0]
if new_column_name not in headers:
headers.append(new_column_name)
new_column_index = headers.index(new_column_name)
data_rows = rows[1:]
# Create a new list to store the modified rows
modified_rows = [headers]
# Iterate over each row
for row in data_rows:
new_row = row[:]
# Extend the row if the new column index is out of range
while len(new_row) <= new_column_index:
new_row.append('')
for i, cell in enumerate(row):
if cell.startswith('http://') or cell.startswith('https://'):
new_row[new_column_index] = cell # Assign the website to the new column
new_row[i] = ''
break
modified_rows.append(new_row)
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def sanitize_websites():
# Process for sanitizing website data
for state_dir in os.listdir(stage2_dir):
state_path = os.path.join(stage2_dir, state_dir)
if os.path.isdir(state_path):
# Print the state name in big print using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText(state_dir))
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Sanitizing Website data in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
sanitize_website_data(file_path)
progress_bar.update(1)
if __name__ == '__main__':
sanitize_websites()
<sanitize-gbp-business-website script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>

84
bin/stage-2/sanitize-gbp-business-website.py Executable file
View file

@ -0,0 +1,84 @@
#!/usr/bin/env python
# Script Name: sanitize-gbp-business-website
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def sanitize_website_data(file_path):
"""Sanitize website data in the given CSV file."""
new_column_name = 'GBP Business Website'
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
headers = rows[0]
if new_column_name not in headers:
headers.append(new_column_name)
new_column_index = headers.index(new_column_name)
data_rows = rows[1:]
modified_rows = [headers]
for row in data_rows:
new_row = row[:]
while len(new_row) <= new_column_index:
new_row.append('')
for i, cell in enumerate(row):
if cell.startswith('http://') or cell.startswith('https://'):
new_row[new_column_index] = cell
new_row[i] = ''
break
modified_rows.append(new_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to sanitize website data."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
sanitize_website_data(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "sanitize-gbp-business-website".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Sanitizing website data in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("Website data sanitized.")
if __name__ == "__main__":
main()

207
bin/stage-2/sanitize-phone-data.md Normal file
View file

@ -0,0 +1,207 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'sanitize-phone-data' now:
<sanitize-phone-data script>
#!/usr/bin/env python
import os
import csv
import re
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage2_dir = os.path.join(project_root, '.data', 'stage-2')
def sanitize_phone_data(file_path):
new_column_name = 'GBP Business Phone'
# Read the CSV file
with open(file_path, 'r', newline='') as file:
reader = csv.DictReader(file)
rows = list(reader)
if not rows:
return
# Get the headers and add the new column name if it does not exist
headers = reader.fieldnames
if new_column_name not in headers:
headers.append(new_column_name)
# Regular expressions to match phone numbers
phone_patterns = [
re.compile(r'^\(\d{3}\) \d{3}-\d{4}$'), # (555) 555-5555
re.compile(r'^\+1 \d{3}-\d{3}-\d{4}$'), # +1 555-555-5555
re.compile(r'^\(\d{3}\) \d{3}-\d{4} ext\. \d+$') # (555) 555-5555 ext. 5
]
# Create a new list to store the modified rows
modified_rows = []
# Iterate over each row
for row in rows:
if new_column_name not in row:
row[new_column_name] = ''
for key, cell in row.items():
if any(pattern.match(cell) for pattern in phone_patterns):
row[new_column_name] = cell # Move the phone number to the new column
row[key] = ''
break
modified_rows.append(row)
# Format phone numbers in the new column
for row in modified_rows:
if re.match(r'^\(\d{3}\) \d{3}-\d{4}$', row[new_column_name]):
row[new_column_name] = '+1 ' + row[new_column_name][1:4] + '-' + row[new_column_name][6:9] + '-' + row[new_column_name][10:]
elif re.match(r'^\(\d{3}\) \d{3}-\d{4} ext\. \d+$', row[new_column_name]):
row[new_column_name] = '+1 ' + row[new_column_name][1:4] + '-' + row[new_column_name][6:9] + '-' + row[new_column_name][10:14] + ' ext. ' + row[new_column_name][19:]
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.DictWriter(file, fieldnames=headers)
writer.writeheader()
writer.writerows(modified_rows)
def sanitize_phones():
# Process for sanitizing phone data
for state_dir in os.listdir(stage2_dir):
state_path = os.path.join(stage2_dir, state_dir)
if os.path.isdir(state_path):
# Print the state name in big print using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText(state_dir))
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Sanitizing Phone data in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
sanitize_phone_data(file_path)
progress_bar.update(1)
if __name__ == '__main__':
sanitize_phones()
<sanitize-phone-data script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>

95
bin/stage-2/sanitize-phone-data.py Executable file
View file

@ -0,0 +1,95 @@
#!/usr/bin/env python
# Script Name: sanitize-phone-data
import os
import csv
import re
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def sanitize_phone_data(file_path):
"""Sanitize phone data in the given CSV file."""
new_column_name = 'GBP Business Phone'
with open(file_path, 'r', newline='') as file:
reader = csv.DictReader(file)
rows = list(reader)
if not rows:
return
headers = reader.fieldnames
if new_column_name not in headers:
headers.append(new_column_name)
phone_patterns = [
re.compile(r'^\(\d{3}\) \d{3}-\d{4}$'), # (555) 555-5555
re.compile(r'^\+1 \d{3}-\d{3}-\d{4}$'), # +1 555-555-5555
re.compile(r'^\(\d{3}\) \d{3}-\d{4} ext\. \d+$') # (555) 555-5555 ext. 5
]
modified_rows = []
for row in rows:
if new_column_name not in row:
row[new_column_name] = ''
for key, cell in row.items():
if any(pattern.match(cell) for pattern in phone_patterns):
row[new_column_name] = cell
row[key] = ''
break
modified_rows.append(row)
for row in modified_rows:
if re.match(r'^\(\d{3}\) \d{3}-\d{4}$', row[new_column_name]):
row[new_column_name] = '+1 ' + row[new_column_name][1:4] + '-' + row[new_column_name][6:9] + '-' + row[new_column_name][10:]
elif re.match(r'^\(\d{3}\) \d{3}-\d{4} ext\. \d+$', row[new_column_name]):
row[new_column_name] = '+1 ' + row[new_column_name][1:4] + '-' + row[new_column_name][6:9] + '-' + row[new_column_name][10:14] + ' ext. ' + row[new_column_name][19:]
with open(file_path, 'w', newline='') as file:
writer = csv.DictWriter(file, fieldnames=headers)
writer.writeheader()
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to sanitize phone data."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
sanitize_phone_data(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "sanitize-phone-data".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Sanitizing phone data in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("Phone data sanitized.")
if __name__ == "__main__":
main()

197
bin/stage-2/sanitize-review-count.md Normal file
View file

@ -0,0 +1,197 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'sanitize-review-count' now:
<sanitize-review-count script>
#!/usr/bin/env python
import os
import csv
import re
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage2_dir = os.path.join(project_root, '.data', 'stage-2')
def sanitize_review_count_data(file_path):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
# Get the headers
headers = rows[0]
data_rows = rows[1:]
# Regular expression to match the review count pattern
review_count_pattern = re.compile(r'^-\d{1,3}(?:,\d{3})*$')
# Identify the "GBP Review Count" column
review_count_index = None
for i, header in enumerate(headers):
if any(review_count_pattern.match(row[i]) for row in data_rows):
review_count_index = i
break
if review_count_index is None:
return # No matching column found
# Create a new list to store the modified rows
modified_rows = []
# Iterate over each row (skip header row)
for row in rows:
if review_count_pattern.match(row[review_count_index]):
row[review_count_index] = f"({row[review_count_index][1:]})"
modified_rows.append(row)
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def sanitize_review_counts():
# Process for sanitizing review count data
for state_dir in os.listdir(stage2_dir):
state_path = os.path.join(stage2_dir, state_dir)
if os.path.isdir(state_path):
# Print the state name in big print using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText(state_dir))
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Sanitizing GBP Review Count data in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
sanitize_review_count_data(file_path)
progress_bar.update(1)
if __name__ == '__main__':
sanitize_review_counts()
<sanitize-review-count script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>

84
bin/stage-2/sanitize-review-count.py Executable file
View file

@ -0,0 +1,84 @@
#!/usr/bin/env python
# Script Name: sanitize-review-count
import os
import csv
import re
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def sanitize_review_count_data(file_path):
"""Sanitize review count data in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
headers = rows[0]
data_rows = rows[1:]
review_count_pattern = re.compile(r'^-\d{1,3}(?:,\d{3})*$')
review_count_index = None
for i, header in enumerate(headers):
if any(review_count_pattern.match(row[i]) for row in data_rows):
review_count_index = i
break
if review_count_index is None:
return
modified_rows = []
for row in rows:
if review_count_pattern.match(row[review_count_index]):
row[review_count_index] = f"({row[review_count_index][1:]})"
modified_rows.append(row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to sanitize review count data."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
sanitize_review_count_data(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "sanitize-review-count".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Sanitizing review counts in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("Review counts sanitized.")
if __name__ == "__main__":
main()

178
bin/stage-3/column-search_business-name.md Normal file
View file

@ -0,0 +1,178 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'column-search_business-name' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<column-search_business-name script>
#!/usr/bin/env python
import os
import csv
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage3_dir = os.path.join(project_root, '.data', 'stage-3')
def process_csv_file(file_path):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
headers = rows[0]
data_rows = rows[1:]
# Check if the column has already been processed by other scripts
if 'Business Name' in headers:
return
# Check if the first column is not already renamed
if headers[0] not in ['GBP Business Category', 'GBP Matching Service', 'GBP Review Rating', 'GBP Review Count', 'Business Phone', 'GBP Business Website', 'YiB', 'GBP Location']:
headers[0] = 'Business Name'
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
def search_business_name():
# Print the column name using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText('Business Name'))
# Process each CSV file in the stage 3 directory
for state_dir in os.listdir(stage3_dir):
state_path = os.path.join(stage3_dir, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Searching for 'Business Name' in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
progress_bar.update(1)
if __name__ == '__main__':
search_business_name()
</column-search_business-name script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>

75
bin/stage-3/column-search_business-name.py Executable file
View file

@ -0,0 +1,75 @@
#!/usr/bin/env python
# Script Name: column-search-business-name
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_3_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-3")
def process_csv_file(file_path):
"""Identify and rename the 'Business Name' column in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
headers = rows[0]
data_rows = rows[1:]
# Check if the column has already been processed by other scripts
if 'Business Name' in headers:
return
# Check if the first column is not already renamed
if headers[0] not in ['GBP Business Category', 'GBP Matching Service', 'GBP Review Rating', 'GBP Review Count', 'Business Phone', 'GBP Business Website', 'YiB', 'GBP Location', 'MD5 for Scrape']:
headers[0] = 'Business Name'
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to rename the 'Business Name' column."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "column-search-business-name".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_3_DIRECTORY):
print(f"Error: The directory {STAGE_3_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Renaming 'Business Name' columns in {STAGE_3_DIRECTORY}...")
process_csv_files(STAGE_3_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("Business Name renaming completed.")
if __name__ == "__main__":
main()

215
bin/stage-3/column-search_gbp-business-category.md Normal file
View file

@ -0,0 +1,215 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'column-search_gbp-business-category' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<column-search_gbp-business-category script>
#!/usr/bin/env python
import os
import csv
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage3_dir = os.path.join(project_root, '.data', 'stage-3')
data_dir = os.path.join(project_root, '.data')
# Read the list of terms for GBP Business Category matching from the text file
gbp_business_categories_file = os.path.join(data_dir, 'gbp-business-categories.txt')
# Check if the gbp-business-categories.txt file exists and is non-empty
if not os.path.isfile(gbp_business_categories_file) or os.path.getsize(gbp_business_categories_file) == 0:
print("Error: The gbp-business-categories.txt file is missing or empty.")
exit(1)
with open(gbp_business_categories_file, 'r') as file:
gbp_business_categories = [line.strip().lower() for line in file if line.strip()]
# Check if the gbp_business_categories list is empty
if not gbp_business_categories:
print("Error: The gbp-business-categories.txt file does not contain any valid categories.")
exit(1)
def process_csv_file(file_path):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
headers = rows[0]
data_rows = rows[1:]
# Check if the column has already been processed by other scripts
if 'GBP Business Category' in headers:
return
# Function to count matches
def count_matches(column_data):
matches = sum(1 for cell in column_data if cell.strip().lower() in gbp_business_categories)
return matches
# Find the index of the column to rename
column_index = None
highest_matches = 0
for i, header in enumerate(headers):
if header not in ['GBP Matching Service', 'Business Name', 'GBP Review Rating', 'GBP Review Count', 'GBP Business Phone', 'GBP Business Website', 'YiB', 'GBP Location']:
column_data = [row[i] for row in data_rows if row[i].strip()]
matches = count_matches(column_data)
if matches > highest_matches:
highest_matches = matches
column_index = i
# Rename the column with the highest number of matches
if column_index is not None and highest_matches > 0:
headers[column_index] = 'GBP Business Category'
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
def search_gbp_business_category():
# Print the column name using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText('GBP Business Category'))
# Process each CSV file in the stage 3 directory
for state_dir in os.listdir(stage3_dir):
state_path = os.path.join(stage3_dir, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Searching for 'GBP Business Category' in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
progress_bar.update(1)
if __name__ == '__main__':
search_gbp_business_category()
</column-search_gbp-business-category script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>

108
bin/stage-3/column-search_gbp-business-category.py Executable file
View file

@ -0,0 +1,108 @@
#!/usr/bin/env python
# Script Name: column-search-gbp-business-category
import os
import csv
import sys
from tqdm import tqdm
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_3_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-3")
# Read the list of terms for GBP Business Category matching from the text file
GBP_BUSINESS_CATEGORIES_FILE = os.path.join(DATA_DIRECTORY, 'gbp-business-categories.txt')
# Check if the gbp-business-categories.txt file exists and is non-empty
if not os.path.isfile(GBP_BUSINESS_CATEGORIES_FILE) or os.path.getsize(GBP_BUSINESS_CATEGORIES_FILE) == 0:
print("Error: The gbp-business-categories.txt file is missing or empty.")
sys.exit(1)
with open(GBP_BUSINESS_CATEGORIES_FILE, 'r') as file:
gbp_business_categories = [line.strip().lower() for line in file if line.strip()]
# Check if the gbp_business_categories list is empty
if not gbp_business_categories:
print("Error: The gbp-business-categories.txt file does not contain any valid categories.")
sys.exit(1)
def process_csv_file(file_path):
"""Identify and rename the 'GBP Business Category' column in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
headers = rows[0]
data_rows = rows[1:]
# Check if the column has already been processed by other scripts
if 'GBP Business Category' in headers:
return
def count_matches(column_data):
matches = sum(1 for cell in column_data if cell.strip().lower() in gbp_business_categories)
return matches
# Find the index of the column to rename
column_index = None
highest_matches = 0
for i, header in enumerate(headers):
if header not in ['GBP Matching Service', 'Business Name', 'GBP Review Rating', 'GBP Review Count', 'GBP Business Phone', 'GBP Business Website', 'YiB', 'GBP Location', 'MD5 for Scrape']:
column_data = [row[i] for row in data_rows if row[i].strip()]
matches = count_matches(column_data)
if matches > highest_matches:
highest_matches = matches
column_index = i
# Rename the column with the highest number of matches
if column_index is not None and highest_matches > 0:
headers[column_index] = 'GBP Business Category'
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to search for 'GBP Business Category'."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "column-search-gbp-business-category".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_3_DIRECTORY):
print(f"Error: The directory {STAGE_3_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Searching for 'GBP Business Category' in {STAGE_3_DIRECTORY}...")
process_csv_files(STAGE_3_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("GBP Business Category search completed.")
if __name__ == "__main__":
main()

195
bin/stage-3/column-search_gbp-location.md Normal file
View file

@ -0,0 +1,195 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'column-search_gbp-location' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<column-search_gbp-location script>
#!/usr/bin/env python
import os
import csv
import re
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage3_dir = os.path.join(project_root, '.data', 'stage-3')
def process_csv_file(file_path):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
headers = rows[0]
data_rows = rows[1:]
# Check if the column has already been processed by other scripts
if 'GBP Location' in headers:
return
# Function to count matches
def count_matches(column_data):
matches = sum(1 for cell in column_data if re.match(r'^[\w\s]+, [A-Z]{2}$', cell) or re.match(r'^[\w\s]+, [A-Z]{2}, United States$', cell))
return matches
# Evaluate all columns
column_scores = []
for i, header in enumerate(headers):
if header not in ['GBP Business Category', 'GBP Matching Service', 'Business Name', 'GBP Review Rating', 'GBP Review Count', 'Business Phone', 'GBP Business Website', 'YiB']:
column_data = [row[i] for row in data_rows if row[i]]
matches = count_matches(column_data)
column_scores.append((i, matches))
# Find the column with the highest number of matches
if column_scores:
best_column = max(column_scores, key=lambda x: x[1]) # max by absolute matches
if best_column[1] > 0: # at least one match
headers[best_column[0]] = 'GBP Location'
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
def search_gbp_location():
# Print the column name using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText('GBP Location'))
# Process each CSV file in the stage 3 directory
for state_dir in os.listdir(stage3_dir):
state_path = os.path.join(stage3_dir, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
num_csv_files = len(csv_files)
progress_bar = tqdm(csv_files, desc=f"Searching for 'GBP Location' in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
progress_bar.update(1)
if __name__ == '__main__':
search_gbp_location()
</column-search_gbp-location script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>

90
bin/stage-3/column-search_gbp-location.py Executable file
View file

@ -0,0 +1,90 @@
#!/usr/bin/env python
# Script Name: column-search-gbp-location
import os
import csv
import re
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_3_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-3")
def process_csv_file(file_path):
"""Identify and rename the 'GBP Location' column in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
headers = rows[0]
data_rows = rows[1:]
# Check if the column has already been processed by other scripts
if 'GBP Location' in headers:
return
def count_matches(column_data):
matches = sum(1 for cell in column_data if re.match(r'^[\w\s]+, [A-Z]{2}$', cell) or re.match(r'^[\w\s]+, [A-Z]{2}, United States$', cell))
return matches
# Evaluate all columns
column_scores = []
for i, header in enumerate(headers):
if header not in ['GBP Business Category', 'GBP Matching Service', 'Business Name', 'GBP Review Rating', 'GBP Review Count', 'GBP Business Phone', 'GBP Business Website', 'YiB', 'MD5 for Scrape']:
column_data = [row[i] for row in data_rows if row[i]]
matches = count_matches(column_data)
column_scores.append((i, matches))
# Find the column with the highest number of matches
if column_scores:
best_column = max(column_scores, key=lambda x: x[1]) # max by absolute matches
if best_column[1] > 0: # at least one match
headers[best_column[0]] = 'GBP Location'
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to search for 'GBP Location'."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "column-search-gbp-location".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_3_DIRECTORY):
print(f"Error: The directory {STAGE_3_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Searching for 'GBP Location' in {STAGE_3_DIRECTORY}...")
process_csv_files(STAGE_3_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("GBP Location search completed.")
if __name__ == "__main__":
main()

221
bin/stage-3/column-search_gbp-matching-services.md Normal file
View file

@ -0,0 +1,221 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'column-search_gbp-matching-services' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<column-search_gbp-matching-services script>
#!/usr/bin/env python
import os
import csv
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage3_dir = os.path.join(project_root, '.data', 'stage-3')
data_dir = os.path.join(project_root, '.data')
# Read the list of terms for GBP Matching Service from the text file
gbp_matching_services_file = os.path.join(data_dir, 'gbp-matching-services.txt')
with open(gbp_matching_services_file, 'r') as file:
gbp_matching_services = {line.strip().lower() for line in file}
# List of headers to skip
headers_to_skip = [
'GBP Business Category', 'GBP Matching Service', 'Business Name',
'GBP Review Rating', 'GBP Review Count', 'GBP Business Phone',
'GBP Business Website', 'YiB', 'GBP Location'
]
def process_csv_file(file_path):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
headers = rows[0]
data_rows = rows[1:]
# Check if the column has already been processed by other scripts
if 'GBP Matching Service' in headers:
return
# Function to count matches and non-matches
def count_matches(column_data):
matches = sum(1 for cell in column_data if cell.strip().lower() in gbp_matching_services)
return matches
# Evaluate all columns
column_scores = []
for i, header in enumerate(headers):
if header not in headers_to_skip:
column_data = [row[i] for row in data_rows if row[i].strip()]
matches = count_matches(column_data)
if matches > 0:
column_scores.append((i, matches))
# Find the column with the highest number of matches
if column_scores:
best_column = max(column_scores, key=lambda x: x[1]) # max by number of matches
best_column_index = best_column[0]
# Insert the 'GBP Matching Service' column if it doesn't exist
if 'GBP Matching Service' not in headers:
headers.insert(best_column_index + 1, 'GBP Matching Service')
for row in data_rows:
row.insert(best_column_index + 1, '')
# Move the data from the best column to the 'GBP Matching Service' column
gbp_matching_service_index = headers.index('GBP Matching Service')
for row in data_rows:
row[gbp_matching_service_index] = row[best_column_index]
row[best_column_index] = ''
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
def search_gbp_matching_service():
# Print the column name using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText('GBP Matching Service'))
# Process each CSV file in the stage 3 directory
for state_dir in os.listdir(stage3_dir):
state_path = os.path.join(stage3_dir, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Searching for 'GBP Matching Service' in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
progress_bar.update(1)
if __name__ == '__main__':
search_gbp_matching_service()
</column-search_gbp-matching-services script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>

120
bin/stage-3/column-search_gbp-matching-services.py Executable file
View file

@ -0,0 +1,120 @@
#!/usr/bin/env python
# Script Name: column-search-gbp-matching-services
import os
import csv
import sys
from tqdm import tqdm
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_3_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-3")
# Read the list of terms for GBP Matching Service from the text file
GBP_MATCHING_SERVICES_FILE = os.path.join(DATA_DIRECTORY, 'gbp-matching-services.txt')
# Check if the gbp-matching-services.txt file exists and is non-empty
if not os.path.isfile(GBP_MATCHING_SERVICES_FILE) or os.path.getsize(GBP_MATCHING_SERVICES_FILE) == 0:
print("Error: The gbp-matching-services.txt file is missing or empty.")
sys.exit(1)
with open(GBP_MATCHING_SERVICES_FILE, 'r') as file:
gbp_matching_services = {line.strip().lower() for line in file}
# List of headers to skip
HEADERS_TO_SKIP = [
'GBP Business Category', 'GBP Matching Service', 'Business Name',
'GBP Review Rating', 'GBP Review Count', 'GBP Business Phone',
'GBP Business Website', 'YiB', 'GBP Location', 'MD5 for Scrape'
]
def process_csv_file(file_path):
"""Identify and rename the 'GBP Matching Service' column in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
headers = rows[0]
data_rows = rows[1:]
# Check if the column has already been processed by other scripts
if 'GBP Matching Service' in headers:
return
def count_matches(column_data):
matches = sum(1 for cell in column_data if cell.strip().lower() in gbp_matching_services)
return matches
# Evaluate all columns
column_scores = []
for i, header in enumerate(headers):
if header not in HEADERS_TO_SKIP:
column_data = [row[i] for row in data_rows if row[i].strip()]
matches = count_matches(column_data)
if matches > 0:
column_scores.append((i, matches))
# Find the column with the highest number of matches
if column_scores:
best_column = max(column_scores, key=lambda x: x[1]) # max by number of matches
best_column_index = best_column[0]
# Insert the 'GBP Matching Service' column if it doesn't exist
if 'GBP Matching Service' not in headers:
headers.insert(best_column_index + 1, 'GBP Matching Service')
for row in data_rows:
row.insert(best_column_index + 1, '')
# Move the data from the best column to the 'GBP Matching Service' column
gbp_matching_service_index = headers.index('GBP Matching Service')
for row in data_rows:
row[gbp_matching_service_index] = row[best_column_index]
row[best_column_index] = ''
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to search for 'GBP Matching Service'."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "column-search-gbp-matching-services".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_3_DIRECTORY):
print(f"Error: The directory {STAGE_3_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Searching for 'GBP Matching Service' in {STAGE_3_DIRECTORY}...")
process_csv_files(STAGE_3_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("GBP Matching Service search completed.")
if __name__ == "__main__":
main()

197
bin/stage-3/column-search_review-count.md Normal file
View file

@ -0,0 +1,197 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'column-search_review-count' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<column-search_review-count script>
#!/usr/bin/env python
import os
import csv
import re
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage3_dir = os.path.join(project_root, '.data', 'stage-3')
def process_csv_file(file_path):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
headers = rows[0]
data_rows = rows[1:]
# Check if the column has already been processed by other scripts
if 'GBP Review Count' in headers:
return
# Function to count matches and non-matches
def count_matches(column_data):
pattern = re.compile(r'^\(\d+\)$')
phone_pattern = re.compile(r'^\(\d{3}\) \d{3}-\d{4}$')
matches = sum(1 for cell in column_data if pattern.match(cell) and not phone_pattern.match(cell))
return matches
# Evaluate all columns
column_scores = []
for i, header in enumerate(headers):
if header not in ['GBP Business Category', 'GBP Matching Service', 'Business Name', 'GBP Review Rating', 'GBP Business Phone', 'GBP Business Website', 'YiB', 'GBP Location']:
column_data = [row[i] for row in data_rows if row[i]]
matches = count_matches(column_data)
column_scores.append((i, matches))
# Find the column with the highest number of matches
if column_scores:
best_column = max(column_scores, key=lambda x: x[1]) # max by absolute matches
if best_column[1] > 0: # at least one match
headers[best_column[0]] = 'GBP Review Count'
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
def search_review_count():
# Print the column name using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText('GBP Review Count'))
# Process each CSV file in the stage 3 directory
for state_dir in os.listdir(stage3_dir):
state_path = os.path.join(stage3_dir, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
num_csv_files = len(csv_files)
progress_bar = tqdm(csv_files, desc=f"Searching for 'GBP Review Count' in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
progress_bar.update(1)
if __name__ == '__main__':
search_review_count()
</column-search_review-count script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>

92
bin/stage-3/column-search_review-count.py Executable file
View file

@ -0,0 +1,92 @@
#!/usr/bin/env python
# Script Name: column-search-review-count
import os
import csv
import re
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_3_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-3")
def process_csv_file(file_path):
"""Identify and rename the 'GBP Review Count' column in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
headers = rows[0]
data_rows = rows[1:]
# Check if the column has already been processed by other scripts
if 'GBP Review Count' in headers:
return
def count_matches(column_data):
pattern = re.compile(r'^\(\d+\)$')
phone_pattern = re.compile(r'^\(\d{3}\) \d{3}-\d{4}$')
matches = sum(1 for cell in column_data if pattern.match(cell) and not phone_pattern.match(cell))
return matches
# Evaluate all columns
column_scores = []
for i, header in enumerate(headers):
if header not in ['GBP Business Category', 'GBP Matching Service', 'Business Name', 'GBP Review Rating', 'GBP Business Phone', 'GBP Business Website', 'YiB', 'GBP Location', 'MD5 for Scrape']:
column_data = [row[i] for row in data_rows if row[i]]
matches = count_matches(column_data)
column_scores.append((i, matches))
# Find the column with the highest number of matches
if column_scores:
best_column = max(column_scores, key=lambda x: x[1]) # max by absolute matches
if best_column[1] > 0: # at least one match
headers[best_column[0]] = 'GBP Review Count'
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to search for 'GBP Review Count'."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "column-search-review-count".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_3_DIRECTORY):
print(f"Error: The directory {STAGE_3_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Searching for 'GBP Review Count' in {STAGE_3_DIRECTORY}...")
process_csv_files(STAGE_3_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("'GBP Review Count' search completed.")
if __name__ == "__main__":
main()

196
bin/stage-3/column-search_review-rating.md Normal file
View file

@ -0,0 +1,196 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'column-search_review-rating' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<column-search_review-rating script>
#!/usr/bin/env python
import os
import csv
import re
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage3_dir = os.path.join(project_root, '.data', 'stage-3')
def process_csv_file(file_path):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
headers = rows[0]
data_rows = rows[1:]
# Check if the column has already been processed by other scripts
if 'GBP Review Rating' in headers:
return
# Function to count matches and non-matches
def count_matches(column_data):
pattern = re.compile(r'^\d\.\d$')
matches = sum(1 for cell in column_data if pattern.match(cell))
return matches
# Evaluate all columns
column_scores = []
for i, header in enumerate(headers):
if header not in ['GBP Business Category', 'GBP Matching Service', 'Business Name', 'GBP Review Count', 'GBP Business Phone', 'GBP Business Website', 'YiB', 'GBP Location']:
column_data = [row[i] for row in data_rows if row[i]]
matches = count_matches(column_data)
column_scores.append((i, matches))
# Find the column with the highest number of matches
if column_scores:
best_column = max(column_scores, key=lambda x: x[1]) # max by absolute matches
if best_column[1] > 0: # at least one match
headers[best_column[0]] = 'GBP Review Rating'
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
def search_review_rating():
# Print the column name using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText('GBP Review Rating'))
# Process each CSV file in the stage 3 directory
for state_dir in os.listdir(stage3_dir):
state_path = os.path.join(stage3_dir, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
num_csv_files = len(csv_files)
progress_bar = tqdm(csv_files, desc=f"Searching for 'GBP Review Rating' in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
progress_bar.update(1)
if __name__ == '__main__':
search_review_rating()
</column-search_review-rating script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>

91
bin/stage-3/column-search_review-rating.py Executable file
View file

@ -0,0 +1,91 @@
#!/usr/bin/env python
# Script Name: column-search-review-rating
import os
import csv
import re
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_3_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-3")
def process_csv_file(file_path):
"""Identify and rename the 'GBP Review Rating' column in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
headers = rows[0]
data_rows = rows[1:]
# Check if the column has already been processed by other scripts
if 'GBP Review Rating' in headers:
return
def count_matches(column_data):
pattern = re.compile(r'^\d\.\d$')
matches = sum(1 for cell in column_data if pattern.match(cell))
return matches
# Evaluate all columns
column_scores = []
for i, header in enumerate(headers):
if header not in ['GBP Business Category', 'GBP Matching Service', 'Business Name', 'GBP Review Count', 'GBP Business Phone', 'GBP Business Website', 'YiB', 'GBP Location', 'MD5 for Scrape']:
column_data = [row[i] for row in data_rows if row[i]]
matches = count_matches(column_data)
column_scores.append((i, matches))
# Find the column with the highest number of matches
if column_scores:
best_column = max(column_scores, key=lambda x: x[1]) # max by absolute matches
if best_column[1] > 0: # at least one match
headers[best_column[0]] = 'GBP Review Rating'
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to search for 'GBP Review Rating'."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "column-search-review-rating".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_3_DIRECTORY):
print(f"Error: The directory {STAGE_3_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Searching for 'GBP Review Rating' in {STAGE_3_DIRECTORY}...")
process_csv_files(STAGE_3_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("'GBP Review Rating' search completed.")
if __name__ == "__main__":
main()

182
bin/stage-3/column-search_unknowns.md Normal file
View file

@ -0,0 +1,182 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'column-search_unknowns' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<column-search_unknowns script>
#!/usr/bin/env python
import os
import csv
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage3_dir = os.path.join(project_root, '.data', 'stage-3')
def process_csv_file(file_path):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
headers = rows[0]
data_rows = rows[1:]
# Check for columns with data that don't have a header matching any of the target column headers
target_headers = ['GBP Business Category', 'GBP Matching Service', 'Business Name', 'GBP Review Rating', 'GBP Review Count', 'GBP Business Phone', 'GBP Business Website', 'YiB', 'GBP Location']
unknown_count = 1
for i, header in enumerate(headers):
if header not in target_headers:
# Check if the column has data
has_data = any(row[i] for row in data_rows)
if has_data:
headers[i] = f'unknown-{unknown_count}'
unknown_count += 1
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
def search_unknowns():
# Print the column name using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText('Unknown Columns'))
# Process each CSV file in the stage 3 directory
for state_dir in os.listdir(stage3_dir):
state_path = os.path.join(stage3_dir, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
num_csv_files = len(csv_files)
progress_bar = tqdm(csv_files, desc=f"Searching for unknown columns in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
progress_bar.update(1)
if __name__ == '__main__':
search_unknowns()
</column-search_unknowns script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>

83
bin/stage-3/column-search_unknowns.py Executable file
View file

@ -0,0 +1,83 @@
#!/usr/bin/env python
# Script Name: column-search-unknowns
import os
import csv
import sys
from tqdm import tqdm
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_3_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-3")
def process_csv_file(file_path):
"""Identify and rename columns with unknown data in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
headers = rows[0]
data_rows = rows[1:]
# Check for columns with data that don't have a header matching any of the target column headers
target_headers = [
'GBP Business Category', 'GBP Matching Service', 'Business Name',
'GBP Review Rating', 'GBP Review Count', 'GBP Business Phone',
'GBP Business Website', 'YiB', 'GBP Location', 'MD5 for Scrape'
]
unknown_count = 1
for i, header in enumerate(headers):
if header not in target_headers:
# Check if the column has data
has_data = any(row[i] for row in data_rows)
if has_data:
headers[i] = f'unknown-{unknown_count}'
unknown_count += 1
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to identify unknown columns."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "column-search-unknowns".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_3_DIRECTORY):
print(f"Error: The directory {STAGE_3_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Identifying unknown columns in {STAGE_3_DIRECTORY}...")
process_csv_files(STAGE_3_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("Unknown columns identification completed.")
if __name__ == "__main__":
main()

251
bin/stage-3/column-search_yib.md Normal file
View file

@ -0,0 +1,251 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'column-search_yib' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<column-search_yib script>
#!/usr/bin/env python
import os
import csv
import re
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage3_dir = os.path.join(project_root, '.data', 'stage-3')
def process_csv_file(file_path):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
headers = rows[0]
data_rows = rows[1:]
# Check if the column has already been processed by other scripts
if 'YiB' in headers:
return
# Function to count matches
def count_matches(column_data):
pattern = re.compile(r'^\d+\+ years in business$')
matches = sum(1 for cell in column_data if pattern.match(cell))
return matches
# Evaluate all columns
column_scores = []
for i, header in enumerate(headers):
if header not in ['GBP Business Category', 'GBP Matching Service', 'Business Name', 'GBP Review Rating', 'GBP Review Count', 'GBP Business Phone', 'GBP Business Website', 'GBP Location']:
column_data = [row[i] for row in data_rows if row[i]]
matches = count_matches(column_data)
column_scores.append((i, matches))
# Find the column with the highest number of matches
if column_scores:
best_column = max(column_scores, key=lambda x: x[1]) # max by absolute matches
if best_column[1] > 0: # at least one match
headers[best_column[0]] = 'YiB'
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
def search_yib():
# Print the column name using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText('YiB'))
# Process each CSV file in the stage 3 directory
for state_dir in os.listdir(stage3_dir):
state_path = os.path.join(stage3_dir, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
num_csv_files = len(csv_files)
progress_bar = tqdm(csv_files, desc=f"Searching for 'YiB' in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
progress_bar.update(1)
if __name__ == '__main__':
search_yib()
</column-search_yib script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>
# Prompt 2
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
The script was not correctly refactored according to the <Project Details>
The script errored out. It would not have errored out if the script was referring to the correct locations for data as described in the <Project Details>
Here is the error:
<error>
Traceback (most recent call last):
File "/home/ld/mgk-scrapes/bin/stage-3/./column-search_yib.py", line 74, in <module>
search_yib()
File "/home/ld/mgk-scrapes/bin/stage-3/./column-search_yib.py", line 58, in search_yib
for state_dir in os.listdir(stage3_dir):
^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/ld/mgk-scrapes/bin/Concrete Sealing Company/.data/stage-3'
</error>

91
bin/stage-3/column-search_yib.py Executable file
View file

@ -0,0 +1,91 @@
#!/usr/bin/env python
# Script Name: column-search-yib
import os
import csv
import re
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_3_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-3")
def process_csv_file(file_path):
"""Identify and rename the 'YiB' column in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
headers = rows[0]
data_rows = rows[1:]
# Check if the column has already been processed by other scripts
if 'YiB' in headers:
return
def count_matches(column_data):
pattern = re.compile(r'^\d+\+ years in business$')
matches = sum(1 for cell in column_data if pattern.match(cell))
return matches
# Evaluate all columns
column_scores = []
for i, header in enumerate(headers):
if header not in ['GBP Business Category', 'GBP Matching Service', 'Business Name', 'GBP Review Rating', 'GBP Review Count', 'GBP Business Phone', 'GBP Business Website', 'GBP Location', 'MD5 for Scrape']:
column_data = [row[i] for row in data_rows if row[i]]
matches = count_matches(column_data)
column_scores.append((i, matches))
# Find the column with the highest number of matches
if column_scores:
best_column = max(column_scores, key=lambda x: x[1]) # max by absolute matches
if best_column[1] > 0: # at least one match
headers[best_column[0]] = 'YiB'
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to search for 'YiB'."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "column-search-yib".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_3_DIRECTORY):
print(f"Error: The directory {STAGE_3_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Searching for 'YiB' in {STAGE_3_DIRECTORY}...")
process_csv_files(STAGE_3_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("'YiB' search completed.")
if __name__ == "__main__":
main()

116
bin/stage-3/prepare-stage-3.md Normal file
View file

@ -0,0 +1,116 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
Now please create a prepare-stage-3 version of the script.
Base the 'prepare-stage-3' script on the 'prepare-stage-2' script below:
<prepare-stage-2>
#!/usr/bin/env python
# Script Name: prepare-stage-2
import os
import shutil
import halo
from tqdm import tqdm
from pyfiglet import Figlet
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_1_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-1")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def create_stage_2_directory():
"""Create the stage-2 directory, replacing it if it already exists."""
if os.path.exists(STAGE_2_DIRECTORY):
print("Existing Stage 2 found, deleting and replacing...")
shutil.rmtree(STAGE_2_DIRECTORY)
os.makedirs(STAGE_2_DIRECTORY)
print("Stage 2 directory created.")
def copy_data_to_stage_2():
"""Copy data from stage-1 to stage-2."""
for item in os.listdir(STAGE_1_DIRECTORY):
s = os.path.join(STAGE_1_DIRECTORY, item)
d = os.path.join(STAGE_2_DIRECTORY, item)
if os.path.isdir(s):
shutil.copytree(s, d)
else:
shutil.copy2(s, d)
def verify_stage_2_data():
"""Verify that the data in stage-2 matches the data in stage-1."""
for root, dirs, files in os.walk(STAGE_1_DIRECTORY):
for file in files:
src_file = os.path.join(root, file)
dest_file = src_file.replace(STAGE_1_DIRECTORY, STAGE_2_DIRECTORY)
if not os.path.exists(dest_file):
return False
return True
def main():
figlet = Figlet(font='slant')
script_name = "prepare-stage-2".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Preparing Stage 2...")
create_stage_2_directory()
print("Copying data to Stage 2...")
spinner = halo.Halo(text='Copying data', spinner='dots')
spinner.start()
copy_data_to_stage_2()
spinner.succeed("Data copied.")
print("Verifying Stage 2 data...")
is_valid = verify_stage_2_data()
if is_valid:
print("Data verification successful. Stage 2 data is valid.")
else:
print("Data verification failed. Stage 2 data is not valid.")
if __name__ == "__main__":
main()
</prepare-stage-2>

70
bin/stage-3/prepare-stage-3.py Executable file
View file

@ -0,0 +1,70 @@
#!/usr/bin/env python
# Script Name: prepare-stage-3
import os
import shutil
import halo
from tqdm import tqdm
from pyfiglet import Figlet
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
STAGE_3_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-3")
def create_stage_3_directory():
"""Create the stage-3 directory, replacing it if it already exists."""
if os.path.exists(STAGE_3_DIRECTORY):
print("Existing Stage 3 found, deleting and replacing...")
shutil.rmtree(STAGE_3_DIRECTORY)
os.makedirs(STAGE_3_DIRECTORY)
print("Stage 3 directory created.")
def copy_data_to_stage_3():
"""Copy data from stage-2 to stage-3."""
for item in os.listdir(STAGE_2_DIRECTORY):
s = os.path.join(STAGE_2_DIRECTORY, item)
d = os.path.join(STAGE_3_DIRECTORY, item)
if os.path.isdir(s):
shutil.copytree(s, d)
else:
shutil.copy2(s, d)
def verify_stage_3_data():
"""Verify that the data in stage-3 matches the data in stage-2."""
for root, dirs, files in os.walk(STAGE_2_DIRECTORY):
for file in files:
src_file = os.path.join(root, file)
dest_file = src_file.replace(STAGE_2_DIRECTORY, STAGE_3_DIRECTORY)
if not os.path.exists(dest_file):
return False
return True
def main():
figlet = Figlet(font='slant')
script_name = "prepare-stage-3".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Preparing Stage 3...")
create_stage_3_directory()
print("Copying data to Stage 3...")
spinner = halo.Halo(text='Copying data', spinner='dots')
spinner.start()
copy_data_to_stage_3()
spinner.succeed("Data copied.")
print("Verifying Stage 3 data...")
is_valid = verify_stage_3_data()
if is_valid:
print("Data verification successful. Stage 3 data is valid.")
else:
print("Data verification failed. Stage 3 data is not valid.")
if __name__ == "__main__":
main()

181
bin/stage-4/delete-sponsored-listings.md Normal file
View file

@ -0,0 +1,181 @@
# Prompt 1
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'delete-sponsored-listings' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<delete-sponsored-listings script>
#!/usr/bin/env python
import os
import csv
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage4_dir = os.path.join(project_root, '.data', 'stage-4')
def process_csv_file(file_path):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
# Get the headers and data rows
headers = rows[0]
data_rows = rows[1:]
# Check each "unknown-#" column for the criteria
for i, header in enumerate(headers):
if header.startswith("unknown-"):
non_empty_cells = [row[i] for row in data_rows if row[i]]
if all(cell == "Sponsored" for cell in non_empty_cells):
os.remove(file_path)
print(f"Deleted file: {file_path}")
return True # File was deleted
return False # No file was deleted
def strip_negative_patterns():
# Process each CSV file in the stage-4 directory
for state_dir in os.listdir(stage4_dir):
state_path = os.path.join(stage4_dir, state_dir)
if os.path.isdir(state_path):
# Print the state name in big print using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText(state_dir))
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Stripping sponsored listings in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
if process_csv_file(file_path):
progress_bar.update(1)
# Check if the directory is now empty
if not os.listdir(county_path):
os.rmdir(county_path)
print(f"Deleted directory: {county_path}")
if __name__ == '__main__':
strip_negative_patterns()
</delete-sponsored-listings script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>

67
bin/stage-4/delete-sponsored-listings.py Executable file
View file

@ -0,0 +1,67 @@
#!/usr/bin/env python
# Script Name: delete-sponsored-listings
import os
import csv
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_4_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-4")
def process_csv_file(file_path):
"""Process a CSV file to check and delete sponsored listings."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return False
headers = rows[0]
data_rows = rows[1:]
for i, header in enumerate(headers):
if header.startswith("unknown-"):
non_empty_cells = [row[i] for row in data_rows if row[i]]
if all(cell == "Sponsored" for cell in non_empty_cells):
os.remove(file_path)
return True # File was deleted
return False # No file was deleted
def delete_sponsored_listings():
"""Delete sponsored listings in all CSV files in the stage 4 directory."""
figlet = Figlet(font='slant')
script_name = "delete-sponsored-listings".replace("-", " ").title()
print(figlet.renderText(script_name))
total_files_deleted = 0
for state_dir in os.listdir(STAGE_4_DIRECTORY):
state_path = os.path.join(STAGE_4_DIRECTORY, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
state_files_deleted = 0
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
if process_csv_file(file_path):
state_files_deleted += 1
# Check if the directory is now empty
if not os.listdir(county_path):
os.rmdir(county_path)
total_files_deleted += state_files_deleted
spinner.succeed(f'Finished processing {state_dir}. Files deleted: {state_files_deleted}')
print(f"Total Files Deleted: {total_files_deleted}")
if __name__ == "__main__":
delete_sponsored_listings()

116
bin/stage-4/prepare-stage-4.md Normal file
View file

@ -0,0 +1,116 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
Now please create a prepare-stage-4 version of the script.
Base the 'prepare-stage-4' script on the 'prepare-stage-2' script below:
<prepare-stage-2>
#!/usr/bin/env python
# Script Name: prepare-stage-2
import os
import shutil
import halo
from tqdm import tqdm
from pyfiglet import Figlet
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_1_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-1")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def create_stage_2_directory():
"""Create the stage-2 directory, replacing it if it already exists."""
if os.path.exists(STAGE_2_DIRECTORY):
print("Existing Stage 2 found, deleting and replacing...")
shutil.rmtree(STAGE_2_DIRECTORY)
os.makedirs(STAGE_2_DIRECTORY)
print("Stage 2 directory created.")
def copy_data_to_stage_2():
"""Copy data from stage-1 to stage-2."""
for item in os.listdir(STAGE_1_DIRECTORY):
s = os.path.join(STAGE_1_DIRECTORY, item)
d = os.path.join(STAGE_2_DIRECTORY, item)
if os.path.isdir(s):
shutil.copytree(s, d)
else:
shutil.copy2(s, d)
def verify_stage_2_data():
"""Verify that the data in stage-2 matches the data in stage-1."""
for root, dirs, files in os.walk(STAGE_1_DIRECTORY):
for file in files:
src_file = os.path.join(root, file)
dest_file = src_file.replace(STAGE_1_DIRECTORY, STAGE_2_DIRECTORY)
if not os.path.exists(dest_file):
return False
return True
def main():
figlet = Figlet(font='slant')
script_name = "prepare-stage-2".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Preparing Stage 2...")
create_stage_2_directory()
print("Copying data to Stage 2...")
spinner = halo.Halo(text='Copying data', spinner='dots')
spinner.start()
copy_data_to_stage_2()
spinner.succeed("Data copied.")
print("Verifying Stage 2 data...")
is_valid = verify_stage_2_data()
if is_valid:
print("Data verification successful. Stage 2 data is valid.")
else:
print("Data verification failed. Stage 2 data is not valid.")
if __name__ == "__main__":
main()
</prepare-stage-2>

70
bin/stage-4/prepare-stage-4.py Executable file
View file

@ -0,0 +1,70 @@
#!/usr/bin/env python
# Script Name: prepare-stage-4
import os
import shutil
import halo
from tqdm import tqdm
from pyfiglet import Figlet
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_3_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-3")
STAGE_4_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-4")
def create_stage_4_directory():
"""Create the stage-4 directory, replacing it if it already exists."""
if os.path.exists(STAGE_4_DIRECTORY):
print("Existing Stage 4 found, deleting and replacing...")
shutil.rmtree(STAGE_4_DIRECTORY)
os.makedirs(STAGE_4_DIRECTORY)
print("Stage 4 directory created.")
def copy_data_to_stage_4():
"""Copy data from stage-3 to stage-4."""
for item in os.listdir(STAGE_3_DIRECTORY):
s = os.path.join(STAGE_3_DIRECTORY, item)
d = os.path.join(STAGE_4_DIRECTORY, item)
if os.path.isdir(s):
shutil.copytree(s, d)
else:
shutil.copy2(s, d)
def verify_stage_4_data():
"""Verify that the data in stage-4 matches the data in stage-3."""
for root, dirs, files in os.walk(STAGE_3_DIRECTORY):
for file in files:
src_file = os.path.join(root, file)
dest_file = src_file.replace(STAGE_3_DIRECTORY, STAGE_4_DIRECTORY)
if not os.path.exists(dest_file):
return False
return True
def main():
figlet = Figlet(font='slant')
script_name = "prepare-stage-4".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Preparing Stage 4...")
create_stage_4_directory()
print("Copying data to Stage 4...")
spinner = halo.Halo(text='Copying data', spinner='dots')
spinner.start()
copy_data_to_stage_4()
spinner.succeed("Data copied.")
print("Verifying Stage 4 data...")
is_valid = verify_stage_4_data()
if is_valid:
print("Data verification successful. Stage 4 data is valid.")
else:
print("Data verification failed. Stage 4 data is not valid.")
if __name__ == "__main__":
main()

245
bin/stage-4/reshift-terms.md Normal file
View file

@ -0,0 +1,245 @@
# Prompt 1
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'reshift-terms' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<reshift-terms script>
#!/usr/bin/env python
# reshift-terms
import os
import csv
import re
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage4_dir = os.path.join(project_root, '.data', 'stage-4')
data_dir = os.path.join(project_root, '.data')
# Read the list of terms for GBP Business Category and GBP Matching Service from the text files
gbp_business_categories_file = os.path.join(data_dir, 'gbp-business-categories.txt')
gbp_matching_services_file = os.path.join(data_dir, 'gbp-matching-services.txt')
# Helper function to read terms from a file
def read_terms(file_path):
if not os.path.isfile(file_path) or os.path.getsize(file_path) == 0:
print(f"Error: The file {file_path} is missing or empty.")
exit(1)
with open(file_path, 'r') as file:
terms = [line.strip().lower() for line in file if line.strip()]
if not terms:
print(f"Error: The file {file_path} does not contain any valid terms.")
exit(1)
return terms
gbp_business_categories = read_terms(gbp_business_categories_file)
gbp_matching_services = read_terms(gbp_matching_services_file)
def process_csv_file(file_path):
# Regular expressions for different data types
location_pattern = re.compile(r'^[\w\s]+, [A-Z]{2}(, United States)?$', re.IGNORECASE)
yib_pattern = re.compile(r'^\d+\+ years in business$', re.IGNORECASE)
review_rating_pattern = re.compile(r'^[1-5]\.\d$', re.IGNORECASE)
review_count_pattern = re.compile(r'^\(\d+\)$', re.IGNORECASE)
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.DictReader(file)
rows = list(reader)
if not rows:
return
headers = reader.fieldnames
# Iterate over each row
for row in rows:
for header in headers:
if header.startswith('unknown-'):
cell = row[header].strip().lower()
if location_pattern.match(cell):
if not row['GBP Location']:
row['GBP Location'] = row[header]
row[header] = ''
elif yib_pattern.match(cell):
if not row['YiB']:
row['YiB'] = row[header]
row[header] = ''
elif review_rating_pattern.match(cell):
if not row['GBP Review Rating']:
row['GBP Review Rating'] = row[header]
row[header] = ''
elif review_count_pattern.match(cell):
if not row['GBP Review Count']:
row['GBP Review Count'] = row[header]
row[header] = ''
elif cell in gbp_business_categories:
if not row['GBP Business Category']:
row['GBP Business Category'] = row[header]
row[header] = ''
elif cell in gbp_matching_services:
if not row['GBP Matching Service']:
row['GBP Matching Service'] = row[header]
row[header] = ''
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.DictWriter(file, fieldnames=headers)
writer.writeheader()
writer.writerows(rows)
def reshift_terms():
# Print the task name using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText('Reshift Terms'))
# Process each CSV file in the stage 4 directory
for state_dir in os.listdir(stage4_dir):
state_path = os.path.join(stage4_dir, state_dir)
if os.path.isdir(state_path):
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Shifting Terms in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
progress_bar.update(1)
if __name__ == '__main__':
reshift_terms()
</reshift-terms script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>
# Prompt 2
I would like it if this script could include a tally of the terms that were shifted in each state. And then a total tally when complete.
# Prompt 3
It reported that zero terms were moved at all. That seems unlikely to me. To add a little assurance for me to know that nothing indeed was modified at all, can you do some sort of verification between the data before the script ran compared to the data after it was ran? Perhaps creating some kind of checksum before script has been ran, and then after the script is done and comparing the checksum from the before and after snapshot to see if anything was changed.
A final verification step.
# Prompt 4
Ok that was useful. But it was also spammy. The validation at the end I mean. I would like to only report when a change has been detected. We can be silent about files with no changes detected.

138
bin/stage-4/reshift-terms.py Executable file
View file

@ -0,0 +1,138 @@
#!/usr/bin/env python
# Script Name: reshift-terms
import os
import csv
import re
import hashlib
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_4_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-4")
# Helper function to read terms from a file
def read_terms(file_path):
if not os.path.isfile(file_path) or os.path.getsize(file_path) == 0:
print(f"Error: The file {file_path} is missing or empty.")
exit(1)
with open(file_path, 'r') as file:
terms = [line.strip().lower() for line in file if line.strip()]
if not terms:
print(f"Error: The file {file_path} does not contain any valid terms.")
exit(1)
return terms
# Load GBP Business Categories and GBP Matching Services terms
GBP_BUSINESS_CATEGORIES = read_terms(os.path.join(DATA_DIRECTORY, 'gbp-business-categories.txt'))
GBP_MATCHING_SERVICES = read_terms(os.path.join(DATA_DIRECTORY, 'gbp-matching-services.txt'))
# Patterns for different data types
LOCATION_PATTERN = re.compile(r'^[\w\s]+, [A-Z]{2}(, United States)?$', re.IGNORECASE)
YIB_PATTERN = re.compile(r'^\d+\+ years in business$', re.IGNORECASE)
REVIEW_RATING_PATTERN = re.compile(r'^[1-5]\.\d$', re.IGNORECASE)
REVIEW_COUNT_PATTERN = re.compile(r'^\(\d+\)$', re.IGNORECASE)
def calculate_checksum(file_path):
"""Calculate the MD5 checksum of a file."""
hash_md5 = hashlib.md5()
with open(file_path, 'rb') as f:
for chunk in iter(lambda: f.read(4096), b""):
hash_md5.update(chunk)
return hash_md5.hexdigest()
def process_csv_file(file_path):
"""Process a CSV file to reshift terms and return the number of terms shifted."""
with open(file_path, 'r') as file:
reader = csv.DictReader(file)
rows = list(reader)
if not rows:
return 0
headers = reader.fieldnames
terms_shifted = 0
for row in rows:
for header in headers:
if header.startswith('unknown-'):
cell = row[header].strip().lower()
if LOCATION_PATTERN.match(cell):
if not row['GBP Location']:
row['GBP Location'] = row[header]
row[header] = ''
terms_shifted += 1
elif YIB_PATTERN.match(cell):
if not row['YiB']:
row['YiB'] = row[header]
row[header] = ''
terms_shifted += 1
elif REVIEW_RATING_PATTERN.match(cell):
if not row['GBP Review Rating']:
row['GBP Review Rating'] = row[header]
row[header] = ''
terms_shifted += 1
elif REVIEW_COUNT_PATTERN.match(cell):
if not row['GBP Review Count']:
row['GBP Review Count'] = row[header]
row[header] = ''
terms_shifted += 1
elif cell in GBP_BUSINESS_CATEGORIES:
if not row['GBP Business Category']:
row['GBP Business Category'] = row[header]
row[header] = ''
terms_shifted += 1
elif cell in GBP_MATCHING_SERVICES:
if not row['GBP Matching Service']:
row['GBP Matching Service'] = row[header]
row[header] = ''
terms_shifted += 1
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.DictWriter(file, fieldnames=headers)
writer.writeheader()
writer.writerows(rows)
return terms_shifted
def reshift_terms():
"""Reshift terms in all CSV files in the stage 4 directory and tally the results."""
figlet = Figlet(font='slant')
print(figlet.renderText('Reshift Terms'))
total_terms_shifted = 0
before_checksums = {}
after_checksums = {}
for state_dir in os.listdir(STAGE_4_DIRECTORY):
state_path = os.path.join(STAGE_4_DIRECTORY, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
state_terms_shifted = 0
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
before_checksums[file_path] = calculate_checksum(file_path)
state_terms_shifted += process_csv_file(file_path)
after_checksums[file_path] = calculate_checksum(file_path)
total_terms_shifted += state_terms_shifted
spinner.succeed(f'Finished processing {state_dir}. Terms shifted: {state_terms_shifted}')
# Verification step
for file_path in before_checksums:
if before_checksums[file_path] != after_checksums[file_path]:
print(f"Changes detected in: {file_path}")
print(f"Total Terms Shifted: {total_terms_shifted}")
if __name__ == "__main__":
reshift_terms()

185
bin/stage-4/strip-neg-patterns-from-unknown-cols.md Normal file
View file

@ -0,0 +1,185 @@
# Prompt 1
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'strip-neg-patterns-from-unknown-cols' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<strip-neg-patterns-from-unknown-cols script>
#!/usr/bin/env python
import os
import csv
import re
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage4_dir = os.path.join(project_root, '.data', 'stage-4')
# Define the pattern for foreign phone numbers
foreign_phone_pattern = re.compile(r'^\+\d{1,3} \d{3,4} \d{3} \d{3,4}$')
def process_csv_file(file_path):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
# Get the headers and data rows
headers = rows[0]
data_rows = rows[1:]
# Create a new list to store the modified rows
modified_rows = [headers]
# Iterate through the data rows and remove foreign phone numbers from "unknown-#" columns
for row in data_rows:
for i, header in enumerate(headers):
if header.startswith("unknown-") and foreign_phone_pattern.match(row[i]):
row[i] = '' # Clear the cell if it contains a foreign phone number
modified_rows.append(row)
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def strip_negative_patterns():
# Process each CSV file in the stage-4 directory
for state_dir in os.listdir(stage4_dir):
state_path = os.path.join(stage4_dir, state_dir)
if os.path.isdir(state_path):
# Print the state name in big print using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText(state_dir))
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Stripping negative patterns from unknowns in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
process_csv_file(file_path)
progress_bar.update(1)
if __name__ == '__main__':
strip_negative_patterns()
</strip-neg-patterns-from-unknown-cols script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>

73
bin/stage-4/strip-neg-patterns-from-unknown-cols.py Executable file
View file

@ -0,0 +1,73 @@
#!/usr/bin/env python
# Script Name: strip-neg-patterns-from-unknown-cols
import os
import csv
import re
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_4_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-4")
# Define the pattern for foreign phone numbers
foreign_phone_pattern = re.compile(r'^\+\d{1,3} \d{3,4} \d{3} \d{3,4}$')
def process_csv_file(file_path):
"""Process a CSV file to remove negative patterns."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return 0
headers = rows[0]
data_rows = rows[1:]
modified_rows = [headers]
terms_removed = 0
for row in data_rows:
for i, header in enumerate(headers):
if header.startswith("unknown-") and foreign_phone_pattern.match(row[i]):
row[i] = '' # Clear the cell if it contains a foreign phone number
terms_removed += 1
modified_rows.append(row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
return terms_removed
def strip_negative_patterns():
"""Strip negative patterns from all CSV files in the stage 4 directory and tally the results."""
figlet = Figlet(font='slant')
print(figlet.renderText('Strip Negative Patterns'))
total_terms_removed = 0
for state_dir in os.listdir(STAGE_4_DIRECTORY):
state_path = os.path.join(STAGE_4_DIRECTORY, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
state_terms_removed = 0
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
state_terms_removed += process_csv_file(file_path)
total_terms_removed += state_terms_removed
spinner.succeed(f'Finished processing {state_dir}. Terms removed: {state_terms_removed}')
print(f"Total Terms Removed: {total_terms_removed}")
if __name__ == "__main__":
strip_negative_patterns()

318
bin/stage-4/strip-neg-terms-from-unknown-cols.md Normal file
View file

@ -0,0 +1,318 @@
# Prompt 1
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'strip-neg-terms-from-unknown-cols' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<strip-neg-terms-from-unknown-cols script>
#!/usr/bin/env python
import os
import csv
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage4_dir = os.path.join(project_root, '.data', 'stage-4')
data_dir = os.path.join(project_root, '.data')
blacklist_file = os.path.join(data_dir, 'unknown-blacklist.txt')
def process_csv_file(file_path, blacklist):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
# Get the headers and data rows
headers = rows[0]
data_rows = rows[1:]
# Create a new list to store the modified rows
modified_rows = [headers]
# Iterate through the data rows and remove blacklist terms from "unknown-#" columns
for row in data_rows:
for i, header in enumerate(headers):
if header.startswith("unknown-") and row[i] in blacklist:
row[i] = '' # Clear the cell if it contains a blacklist term
modified_rows.append(row)
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def strip_negative_terms():
# Read the blacklist terms
if not os.path.isfile(blacklist_file):
print("Error: The unknown-blacklist.txt file is missing.")
return
with open(blacklist_file, 'r') as file:
blacklist = [line.strip() for line in file if line.strip()]
# Process each CSV file in the stage-4 directory
for state_dir in os.listdir(stage4_dir):
state_path = os.path.join(stage4_dir, state_dir)
if os.path.isdir(state_path):
# Print the state name in big print using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText(state_dir))
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Stripping neg terms from unknown columns in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
process_csv_file(file_path, blacklist)
progress_bar.update(1)
if __name__ == '__main__':
strip_negative_terms()
</strip-neg-terms-from-unknown-cols script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>
# Prompt 2
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
I wish to adjust the 'strip-neg-terms-from-unknown-cols' script:
<script>
#!/usr/bin/env python
# Script Name: strip-neg-terms-from-unknown-cols
import os
import csv
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_4_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-4")
BLACKLIST_FILE = os.path.join(DATA_DIRECTORY, 'unknown-blacklist.txt')
def read_blacklist(file_path):
"""Read the blacklist terms from a file."""
if not os.path.isfile(file_path):
print("Error: The unknown-blacklist.txt file is missing.")
return []
with open(file_path, 'r') as file:
blacklist = [line.strip() for line in file if line.strip()]
return blacklist
def process_csv_file(file_path, blacklist):
"""Process a CSV file to remove blacklist terms from unknown columns."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return 0
headers = rows[0]
data_rows = rows[1:]
modified_rows = [headers]
terms_removed = 0
for row in data_rows:
for i, header in enumerate(headers):
if header.startswith("unknown-") and row[i] in blacklist:
row[i] = '' # Clear the cell if it contains a blacklist term
terms_removed += 1
modified_rows.append(row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
return terms_removed
def strip_negative_terms():
"""Strip negative terms from all CSV files in the stage 4 directory and tally the results."""
figlet = Figlet(font='slant')
print(figlet.renderText('Strip Neg Terms'))
blacklist = read_blacklist(BLACKLIST_FILE)
if not blacklist:
return
total_terms_removed = 0
for state_dir in os.listdir(STAGE_4_DIRECTORY):
state_path = os.path.join(STAGE_4_DIRECTORY, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
state_terms_removed = 0
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
state_terms_removed += process_csv_file(file_path, blacklist)
total_terms_removed += state_terms_removed
spinner.succeed(f'Finished processing {state_dir}. Terms removed: {state_terms_removed}')
print(f"Total Terms Removed: {total_terms_removed}")
if __name__ == "__main__":
strip_negative_terms()
</script>
This script needs to have the matching it performs be done in a case insensitive manner. Capitalization needs to be ignored.

83
bin/stage-4/strip-neg-terms-from-unknown-cols.py Executable file
View file

@ -0,0 +1,83 @@
#!/usr/bin/env python
# Script Name: strip-neg-terms-from-unknown-cols
import os
import csv
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_4_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-4")
BLACKLIST_FILE = os.path.join(DATA_DIRECTORY, 'unknown-blacklist.txt')
def read_blacklist(file_path):
"""Read the blacklist terms from a file."""
if not os.path.isfile(file_path):
print("Error: The unknown-blacklist.txt file is missing.")
return []
with open(file_path, 'r') as file:
blacklist = [line.strip().lower() for line in file if line.strip()]
return blacklist
def process_csv_file(file_path, blacklist):
"""Process a CSV file to remove blacklist terms from unknown columns."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return 0
headers = rows[0]
data_rows = rows[1:]
modified_rows = [headers]
terms_removed = 0
for row in data_rows:
for i, header in enumerate(headers):
if header.startswith("unknown-") and row[i].strip().lower() in blacklist:
row[i] = '' # Clear the cell if it contains a blacklist term
terms_removed += 1
modified_rows.append(row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
return terms_removed
def strip_negative_terms():
"""Strip negative terms from all CSV files in the stage 4 directory and tally the results."""
figlet = Figlet(font='slant')
print(figlet.renderText('Strip Neg Terms'))
blacklist = read_blacklist(BLACKLIST_FILE)
if not blacklist:
return
total_terms_removed = 0
for state_dir in os.listdir(STAGE_4_DIRECTORY):
state_path = os.path.join(STAGE_4_DIRECTORY, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
state_terms_removed = 0
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
state_terms_removed += process_csv_file(file_path, blacklist)
total_terms_removed += state_terms_removed
spinner.succeed(f'Finished processing {state_dir}. Terms removed: {state_terms_removed}')
print(f"Total Terms Removed: {total_terms_removed}")
if __name__ == "__main__":
strip_negative_terms()

115
bin/stage-5/prepare-stage-5.md Normal file
View file

@ -0,0 +1,115 @@
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
Now please create a prepare-stage-5 version of the script.
Base the 'prepare-stage-5' script on the 'prepare-stage-2' script below:
<prepare-stage-2>
#!/usr/bin/env python
# Script Name: prepare-stage-2
import os
import shutil
import halo
from tqdm import tqdm
from pyfiglet import Figlet
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_1_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-1")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def create_stage_2_directory():
"""Create the stage-2 directory, replacing it if it already exists."""
if os.path.exists(STAGE_2_DIRECTORY):
print("Existing Stage 2 found, deleting and replacing...")
shutil.rmtree(STAGE_2_DIRECTORY)
os.makedirs(STAGE_2_DIRECTORY)
print("Stage 2 directory created.")
def copy_data_to_stage_2():
"""Copy data from stage-1 to stage-2."""
for item in os.listdir(STAGE_1_DIRECTORY):
s = os.path.join(STAGE_1_DIRECTORY, item)
d = os.path.join(STAGE_2_DIRECTORY, item)
if os.path.isdir(s):
shutil.copytree(s, d)
else:
shutil.copy2(s, d)
def verify_stage_2_data():
"""Verify that the data in stage-2 matches the data in stage-1."""
for root, dirs, files in os.walk(STAGE_1_DIRECTORY):
for file in files:
src_file = os.path.join(root, file)
dest_file = src_file.replace(STAGE_1_DIRECTORY, STAGE_2_DIRECTORY)
if not os.path.exists(dest_file):
return False
return True
def main():
figlet = Figlet(font='slant')
script_name = "prepare-stage-2".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Preparing Stage 2...")
create_stage_2_directory()
print("Copying data to Stage 2...")
spinner = halo.Halo(text='Copying data', spinner='dots')
spinner.start()
copy_data_to_stage_2()
spinner.succeed("Data copied.")
print("Verifying Stage 2 data...")
is_valid = verify_stage_2_data()
if is_valid:
print("Data verification successful. Stage 2 data is valid.")
else:
print("Data verification failed. Stage 2 data is not valid.")
if __name__ == "__main__":
main()
</prepare-stage-2>

70
bin/stage-5/prepare-stage-5.py Executable file
View file

@ -0,0 +1,70 @@
#!/usr/bin/env python
# Script Name: prepare-stage-5
import os
import shutil
import halo
from tqdm import tqdm
from pyfiglet import Figlet
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_4_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-4")
STAGE_5_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-5")
def create_stage_5_directory():
"""Create the stage-5 directory, replacing it if it already exists."""
if os.path.exists(STAGE_5_DIRECTORY):
print("Existing Stage 5 found, deleting and replacing...")
shutil.rmtree(STAGE_5_DIRECTORY)
os.makedirs(STAGE_5_DIRECTORY)
print("Stage 5 directory created.")
def copy_data_to_stage_5():
"""Copy data from stage-4 to stage-5."""
for item in os.listdir(STAGE_4_DIRECTORY):
s = os.path.join(STAGE_4_DIRECTORY, item)
d = os.path.join(STAGE_5_DIRECTORY, item)
if os.path.isdir(s):
shutil.copytree(s, d)
else:
shutil.copy2(s, d)
def verify_stage_5_data():
"""Verify that the data in stage-5 matches the data in stage-4."""
for root, dirs, files in os.walk(STAGE_4_DIRECTORY):
for file in files:
src_file = os.path.join(root, file)
dest_file = src_file.replace(STAGE_4_DIRECTORY, STAGE_5_DIRECTORY)
if not os.path.exists(dest_file):
return False
return True
def main():
figlet = Figlet(font='slant')
script_name = "prepare-stage-5".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Preparing Stage 5...")
create_stage_5_directory()
print("Copying data to Stage 5...")
spinner = halo.Halo(text='Copying data', spinner='dots')
spinner.start()
copy_data_to_stage_5()
spinner.succeed("Data copied.")
print("Verifying Stage 5 data...")
is_valid = verify_stage_5_data()
if is_valid:
print("Data verification successful. Stage 5 data is valid.")
else:
print("Data verification failed. Stage 5 data is not valid.")
if __name__ == "__main__":
main()

276
bin/stage-5/retrieve-root-domain.md Normal file
View file

@ -0,0 +1,276 @@
# Prompt 1
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'retrieve-root-domain' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<retrieve-root-domain script>
</retrieve-root-domain script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>
# Prompt 2
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
The Project Details above contain locations for files, data, scripts, and rules to follow when creating project scripts and toolkit.
I need to modify the 'retrieve-root-domain' script:
<retrieve-root-domain script>
#!/usr/bin/env python
# Script Name: retrieve-root-domain
import os
import csv
import re
from urllib.parse import urlparse
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_5_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-5")
def extract_root_domain(url):
"""Extract the root domain from a URL."""
parsed_url = urlparse(url)
domain = parsed_url.netloc
if domain.startswith("www."):
domain = domain[4:]
return domain
def process_csv_file(file_path):
"""Process a CSV file to extract root domains."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return 0
headers = rows[0]
data_rows = rows[1:]
# Ensure the necessary columns exist
if 'GBP Business Website' not in headers:
return 0
# Create new column for root domains if it doesn't exist
if 'Root Domain' not in headers:
headers.append('Root Domain')
website_index = headers.index('GBP Business Website')
root_domain_index = headers.index('Root Domain')
changes = 0
for row in data_rows:
while len(row) < len(headers):
row.append('')
website = row[website_index]
# Extract root domain
if website:
root_domain = extract_root_domain(website)
if root_domain != row[root_domain_index]:
row[root_domain_index] = root_domain
changes += 1
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
return changes
def retrieve_root_domain():
"""Retrieve root domains from all CSV files in the stage 5 directory and tally the results."""
figlet = Figlet(font='slant')
print(figlet.renderText('Retrieve Root Domain'))
total_changes = 0
for state_dir in os.listdir(STAGE_5_DIRECTORY):
state_path = os.path.join(STAGE_5_DIRECTORY, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
state_changes = 0
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
state_changes += process_csv_file(file_path)
total_changes += state_changes
spinner.succeed(f'Finished processing {state_dir}. Changes made: {state_changes}')
print(f"Total Changes Made: {total_changes}")
if __name__ == "__main__":
retrieve_root_domain()
</retrieve-root-domain script>
I need the script to blacklist some domains from being added to the "Root Domain" column:
<blacklisted root domains>
facebook.com
instagram.com
twitter.com
x.com
linkedin.com
</blacklisted root domains>

111
bin/stage-5/retrieve-root-domain.py Executable file
View file

@ -0,0 +1,111 @@
#!/usr/bin/env python
# Script Name: retrieve-root-domain
import os
import csv
from urllib.parse import urlparse
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_5_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-5")
# List of blacklisted root domains
BLACKLISTED_DOMAINS = {
"facebook.com",
"instagram.com",
"twitter.com",
"x.com",
"linkedin.com"
}
def extract_root_domain(url):
"""Extract the root domain from a URL."""
parsed_url = urlparse(url)
domain = parsed_url.netloc
if domain.startswith("www."):
domain = domain[4:]
return domain
def is_blacklisted(domain):
"""Check if the domain or its subdomain is blacklisted."""
for blacklisted in BLACKLISTED_DOMAINS:
if domain == blacklisted or domain.endswith("." + blacklisted):
return True
return False
def process_csv_file(file_path):
"""Process a CSV file to extract root domains."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return 0
headers = rows[0]
data_rows = rows[1:]
# Ensure the necessary columns exist
if 'GBP Business Website' not in headers:
return 0
# Create new column for root domains if it doesn't exist
if 'Root Domain' not in headers:
headers.append('Root Domain')
website_index = headers.index('GBP Business Website')
root_domain_index = headers.index('Root Domain')
changes = 0
for row in data_rows:
while len(row) < len(headers):
row.append('')
website = row[website_index]
# Extract root domain
if website:
root_domain = extract_root_domain(website)
if not is_blacklisted(root_domain) and root_domain != row[root_domain_index]:
row[root_domain_index] = root_domain
changes += 1
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
return changes
def retrieve_root_domain():
"""Retrieve root domains from all CSV files in the stage 5 directory and tally the results."""
figlet = Figlet(font='slant')
print(figlet.renderText('Retrieve Root Domain'))
total_changes = 0
for state_dir in os.listdir(STAGE_5_DIRECTORY):
state_path = os.path.join(STAGE_5_DIRECTORY, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
state_changes = 0
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
state_changes += process_csv_file(file_path)
total_changes += state_changes
spinner.succeed(f'Finished processing {state_dir}. Changes made: {state_changes}')
print(f"Total Changes Made: {total_changes}")
if __name__ == "__main__":
retrieve_root_domain()

221
bin/stage-5/standardize-location-data.md Normal file
View file

@ -0,0 +1,221 @@
# Prompt 1
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'standardize-location-data' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<standardize-location-data script>
#!/usr/bin/env python
import os
import csv
from tqdm import tqdm
from pyfiglet import Figlet
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage5_dir = os.path.join(project_root, '.data', 'stage-5')
def process_csv_file_standardize_location(file_path):
# Read the CSV file
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
headers = rows[0]
data_rows = rows[1:]
# Ensure the necessary columns exist
if 'GBP Location' not in headers:
return
# Create new columns for municipality and state
if 'GBP Location Municipality' not in headers:
headers.append('GBP Location Municipality')
if 'GBP Location State' not in headers:
headers.append('GBP Location State')
municipality_index = headers.index('GBP Location Municipality')
state_index = headers.index('GBP Location State')
location_index = headers.index('GBP Location')
for row in data_rows:
while len(row) < len(headers):
row.append('')
location = row[location_index]
# Standardize location data by removing ", United States"
if ', United States' in location:
location = location.replace(', United States', '').strip()
# Split location into municipality and state
if ', ' in location:
parts = location.split(', ')
if len(parts) == 2:
municipality, state = parts
else:
municipality = ', '.join(parts[:-1])
state = parts[-1]
row[municipality_index] = municipality
row[state_index] = state
row[location_index] = ''
else:
row[municipality_index] = ''
row[state_index] = ''
# Check if the GBP Location column is empty and act accordingly
if all(row[location_index] == '' for row in data_rows):
headers.pop(location_index)
for row in data_rows:
row.pop(location_index)
else:
# Rename the GBP Location column to unknown-#
unknown_count = sum(1 for header in headers if header.startswith('unknown-'))
headers[location_index] = f'unknown-{unknown_count + 1}'
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
def standardize_location_data():
# Process for standardizing location data
for state_dir in os.listdir(stage5_dir):
state_path = os.path.join(stage5_dir, state_dir)
if os.path.isdir(state_path):
# Print the state name in big print using Figlet
figlet = Figlet(font='slant')
print(figlet.renderText(state_dir))
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
progress_bar = tqdm(csv_files, desc=f"Standardizing Location data in: {state_dir}, {county_dir}",
unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
for file in progress_bar:
file_path = os.path.join(county_path, file)
process_csv_file_standardize_location(file_path)
progress_bar.update(1)
if __name__ == '__main__':
standardize_location_data()
</standardize-location-data script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>

102
bin/stage-5/standardize-location-data.py Executable file
View file

@ -0,0 +1,102 @@
#!/usr/bin/env python
# Script Name: standardize-location-data
import os
import csv
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_5_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-5")
def process_csv_file_standardize_location(file_path):
"""Process a CSV file to standardize location data."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return 0
headers = rows[0]
data_rows = rows[1:]
# Ensure the necessary columns exist
if 'GBP Location' not in headers:
return 0
# Create new columns for municipality and state if they don't exist
if 'GBP Location Municipality' not in headers:
headers.append('GBP Location Municipality')
if 'GBP Location State' not in headers:
headers.append('GBP Location State')
municipality_index = headers.index('GBP Location Municipality')
state_index = headers.index('GBP Location State')
location_index = headers.index('GBP Location')
changes = 0
for row in data_rows:
while len(row) < len(headers):
row.append('')
location = row[location_index]
# Standardize location data by removing ", United States"
if ', United States' in location:
location = location.replace(', United States', '').strip()
# Split location into municipality and state
if ', ' in location:
parts = location.split(', ')
if len(parts) == 2:
municipality, state = parts
else:
municipality = ', '.join(parts[:-1])
state = parts[-1]
row[municipality_index] = municipality
row[state_index] = state
row[location_index] = ''
changes += 1
else:
row[municipality_index] = ''
row[state_index] = ''
# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
return changes
def standardize_location_data():
"""Standardize location data in all CSV files in the stage 5 directory and tally the results."""
figlet = Figlet(font='slant')
print(figlet.renderText('Standardize Location Data'))
total_changes = 0
for state_dir in os.listdir(STAGE_5_DIRECTORY):
state_path = os.path.join(STAGE_5_DIRECTORY, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
state_changes = 0
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
state_changes += process_csv_file_standardize_location(file_path)
total_changes += state_changes
spinner.succeed(f'Finished processing {state_dir}. Changes made: {state_changes}')
print(f"Total Changes Made: {total_changes}")
if __name__ == "__main__":
standardize_location_data()

290
bin/stage-5/strip-brackets-from-review-count.md Normal file
View file

@ -0,0 +1,290 @@
# Prompt 1
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'strip-brackets-from-review-count' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<strip-brackets-from-review-count script>
</strip-brackets-from-review-count script>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>
# Prompt 2
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output llwhat they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
The Project Details above contain locations for files, data, scripts, and rules to follow when creating project scripts and toolkit.
I ran into a problem with the 'strip-brackets-from-reviews-count' script:
<strip-brackets-from-reviews-count script>
#!/usr/bin/env python
# Script Name: strip-brackets-from-review-count
import os
import csv
import re
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_5_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-5")
def process_csv_file_strip_brackets(file_path):
"""Strip brackets from review count in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return 0
headers = rows[0]
data_rows = rows[1:]
review_count_index = None
if 'GBP Review Count' in headers:
review_count_index = headers.index('GBP Review Count')
else:
return 0
pattern = re.compile(r'^\((\d+)\)$')
changes_made = 0
for row in data_rows:
if review_count_index is not None:
match = pattern.match(row[review_count_index])
if match:
row[review_count_index] = match.group(1)
changes_made += 1
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
return changes_made
def strip_brackets_from_review_count():
"""Strip brackets from review count in all CSV files in the stage 5 directory and tally the results."""
figlet = Figlet(font='slant')
print(figlet.renderText('Strip Brackets'))
total_changes_made = 0
for state_dir in os.listdir(STAGE_5_DIRECTORY):
state_path = os.path.join(STAGE_5_DIRECTORY, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
state_changes_made = 0
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
state_changes_made += process_csv_file_strip_brackets(file_path)
total_changes_made += state_changes_made
spinner.succeed(f'Finished processing {state_dir}. Changes made: {state_changes_made}')
print(f"Total Changes Made: {total_changes_made}")
if __name__ == "__main__":
strip_brackets_from_review_count()
</strip-brackets-from-reviews-count script>
I discovered that not every row is having the brackets properly stripped from the numbers in the "GBP Review Count" column as we intended.
Most entries did have the brackets removed, but some entries kept them.
This may have something to do with the number inside these entries which got missed all seem to have a comma in them as a thousands seperator.
Here is some sample data which contains entries in the "GBP Review Count" column that retain their brackets, which this script is supposed to remove.
<csv with brackets intact>
Business Name,Business UUID,Location UUID,GBP Review Rating,GBP Review Count,YiB,GBP Business Category,GBP Matching Service,GBP Location Municipality,GBP Location State,Location from GBP Business Name,GBP Business Phone,GBP Business Website,Root Domain
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,5620e68e-249c-4913-a4cc-16fe4b06e8cc,4.3,"(1,821)",,Home Improvement Store,Repair,Cary,NC,,+1 919-380-3210,https://www.lowes.com/store/NC-Cary/1835?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-1835-_-na-_-0-_-0&y_source=1_MTE4NDQxMy03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,1c71993f-3f20-44da-966d-12b31fc67f72,4.2,"(1,310)",,Home Improvement Store,Services,Matthews,NC,,+1 704-321-7170,https://www.lowes.com/store/NC-Matthews/1124?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-1124-_-na-_-0-_-0&y_source=1_MTE4MzgyMy03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,711c10bd-a4eb-45f9-ac28-eb1a64b1469c,4.2,"(2,480)",,Home Improvement Store,Repair,High Point,NC,,+1 336-889-8600,https://www.lowes.com/store/NC-High-Point/0459?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-459-_-na-_-0-_-0&y_source=1_MTE4MzE4MC03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,246fec7f-c54b-43e7-85f3-bca0c94fd399,4.1,"(2,044)",,Home Improvement Store,Repair,Winterville,NC,,+1 252-355-5211,https://www.lowes.com/store/NC-Winterville/0598?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-598-_-na-_-0-_-0&y_source=1_MTE4MjkyOC03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,a548b3e7-8135-4e1c-9b88-b8766b814d98,4.2,"(2,333)",,Home Improvement Store,Services,Gastonia,NC,,+1 704-865-6767,https://www.lowes.com/store/NC-Gastonia/0457?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-457-_-na-_-0-_-0&y_source=1_MTE4Mzg0MC03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,2605004b-89db-4227-a63c-fa5ba8e6946d,4.0,"(1,626)",,Home Improvement Store,Repair,Elizabeth City,NC,,+1 252-331-6160,https://www.lowes.com/store/NC-Elizabeth-City/1713?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-1713-_-na-_-0-_-0&y_source=1_MTE4MjkyNy03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,f3bc4f4e-d03e-421b-8bdf-745b7681ce89,4.2,"(1,766)",,Home Improvement Store,Repair,Greensboro,NC,,+1 336-541-1890,https://www.lowes.com/store/NC-Greensboro/2771?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-2771-_-na-_-0-_-0&y_source=1_MTE4MzE3Mi03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,60191dc1-b471-4137-b3ad-8fb5120af969,4.0,"(2,240)",,Home Improvement Store,Repair,Raleigh,NC,,+1 919-850-9300,https://www.lowes.com/store/NC-Raleigh/0444?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-444-_-na-_-0-_-0&y_source=1_MTE4NDQyMi03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,acfa3d36-094a-4eaa-a090-298709aea066,4.1,"(1,887)",,Home Improvement Store,Repair,Lincolnton,NC,,+1 704-748-9335,https://www.lowes.com/store/NC-Lincolnton/0700?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-700-_-na-_-0-_-0&y_source=1_MTE4MzgzNy03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,4c8bfc16-b7f4-45fb-ab50-d6970589599a,4.2,"(1,017)",,Home Improvement Store,Repair,Waxhaw,NC,,+1 704-843-8520,https://www.lowes.com/store/NC-Waxhaw/2638?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-2638-_-na-_-0-_-0&y_source=1_MTE4MzgzOS03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,e5eb1412-0dea-4f71-acec-b94ec8812ad3,4.0,"(1,483)",,Home Improvement Store,Repair,Henderson,NC,,+1 252-436-0050,https://www.lowes.com/store/NC-Henderson/0738?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-738-_-na-_-0-_-0&y_source=1_MTE4MjkyOS03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,93b7282b-a6ed-45fd-8716-753f51824c60,3.8,"(1,897)",,Home Improvement Store,Repair,Hendersonville,NC,,+1 828-696-4900,https://www.lowes.com/store/NC-Hendersonville/0031?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-31-_-na-_-0-_-0&y_source=1_MTE4NDE3Ni03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,6cf64b13-1986-401e-bdcc-7df4529aa2dd,3.8,"(1,119)",,Home Improvement Store,Repair,Sylva,NC,,+1 828-586-1170,https://www.lowes.com/store/NC-Sylva/2257?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-2257-_-na-_-0-_-0&y_source=1_MTE4NDE3My03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,d3909855-229c-4b0f-be0e-1a7cf82c73d4,4.1,"(1,959)",,Home Improvement Store,Repair,Charlotte,NC,,+1 704-597-2000,https://www.lowes.com/store/NC-Charlotte/0408?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-408-_-na-_-0-_-0&y_source=1_MTE4MzgzMS03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,a3d406a0-bba7-4772-81e4-675d65e33dee,4.1,"(2,185)",,Home Improvement Store,Repair,Charlotte,NC,,+1 704-501-4420,https://www.lowes.com/store/NC-Charlotte/2352?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-2352-_-na-_-0-_-0&y_source=1_MTE4MzgyNy03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,88097b9c-15cc-4a7f-a740-01200ac4312b,4.1,"(1,956)",,Home Improvement Store,Repair,Charlotte,NC,,+1 704-335-5021,https://www.lowes.com/store/NC-Charlotte/2348?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-2348-_-na-_-0-_-0&y_source=1_MTE4MzgyNC03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,fbd16ea4-6a69-482a-a08d-2e3682bcb1e0,4.2,"(1,140)",,Home Improvement Store,Companies,Elkin,NC,,+1 336-526-6550,https://www.lowes.com/store/NC-Elkin/1653?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-1653-_-na-_-0-_-0&y_source=1_MTE4MzE3MS03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,43c1ea1e-e582-4b10-a90a-dfe298cc10df,4.0,"(1,367)",,Home Improvement Store,Repair,Wilkesboro,NC,,+1 336-838-1500,https://www.lowes.com/store/NC-Wilkesboro/0701?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-701-_-na-_-0-_-0&y_source=1_MTE4MzE3OC03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,1c62fedf-41ea-4641-800b-81b09ae4aeb2,4.0,"(1,923)",,Home Improvement Store,Concrete,Munhall,PA,,+1 412-461-8002,https://www.lowes.com/store/PA-Munhall/0780?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-780-_-na-_-0-_-0&y_source=1_MTE4MzI5NS03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,c163adb0-bf5a-4dc8-af63-a3d69bd75017,4.2,"(2,145)",,Home Improvement Store,Construction,Reading,PA,,+1 610-939-0100,https://www.lowes.com/store/PA-Reading/0279?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-279-_-na-_-0-_-0&y_source=1_MTE4MzY4OS03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,606eb01a-4ad4-4125-bce1-229e3fca7d09,4.3,"(1,031)",,Home Improvement Store,Repair,Central,SC,,+1 864-722-6001,https://www.lowes.com/store/SC-Central/3071?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-3071-_-na-_-0-_-0&y_source=1_MTE4NDI3Ny03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,6fbe9ee0-f628-4b83-ab0f-e1d93d6fc14e,4.1,"(1,638)",,Home Improvement Store,Repair,Columbia,SC,,+1 803-476-1320,https://www.lowes.com/store/SC-Columbia/3026?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-3026-_-na-_-0-_-0&y_source=1_MTE4NDA2Mi03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,97c81d30-528e-4d32-8cc8-d8cd2146b9c8,4.2,"(1,732)",,Home Improvement Store,Repair,Harrisonburg,VA,,+1 540-433-7660,https://www.lowes.com/store/VA-Harrisonburg/0509?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-509-_-na-_-0-_-0&y_source=1_MTE4MzUzOS03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,c44ff9eb-7746-484c-b63f-a89f4acf05e9,4.2,"(2,266)",,Home Improvement Store,Companies,Jacksonville,FL,,+1 904-855-8088,https://www.lowes.com/store/FL-Jacksonville/0503?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-503-_-na-_-0-_-0&y_source=1_MTE4NDMyMC03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
Lowe's Home Improvement,f0cc368c-0126-46d1-a64b-3ca90b33f528,4180f074-7218-44d3-8b39-a4c847007261,4.0,"(3,050)",,Home Improvement Store,Repair,Norfolk,VA,,+1 757-455-5205,https://www.lowes.com/store/VA-Norfolk/1065?cm_mmc=lod-_-c-_-lcl-_-awr-_-yxt-_-go-_-1065-_-na-_-0-_-0&y_source=1_MTE4Mzk2OC03MTUtbG9jYXRpb24ud2Vic2l0ZQ%3D%3D,lowes.com
</csv with brackets intact>
I would like this script to be adjusted slightly to also remove the thousands separator from entries in the "GBP Review Count" column as well as the brackets.
It may be be simpler for the script if it processes for the thousands separator first. And then after that remove the brackets.
I would also like this script to as a final loop, run a check on the "GBP Review Count" column looking for any variety of ) or ( style brackets. I want the script to double check that they are gone.
If any are found on the verification check, then I'd like the script to attempt to remove them again, but using a different method. Then doing one more verification check. And if it fails that time, exit out of the script with an error.

116
bin/stage-5/strip-brackets-from-review-count.py Executable file
View file

@ -0,0 +1,116 @@
#!/usr/bin/env python
# Script Name: strip-brackets-from-review-count
import os
import csv
import re
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_5_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-5")
def process_csv_file_strip_brackets(file_path):
"""Strip brackets and thousands separator from review count in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return 0
headers = rows[0]
data_rows = rows[1:]
review_count_index = None
if 'GBP Review Count' in headers:
review_count_index = headers.index('GBP Review Count')
else:
return 0
pattern_brackets = re.compile(r'^\((\d{1,3}(?:,\d{3})*)\)$')
changes_made = 0
# First, remove the thousands separators
for row in data_rows:
if review_count_index is not None:
match = pattern_brackets.match(row[review_count_index])
if match:
# Remove thousands separators
review_count_no_commas = match.group(1).replace(',', '')
row[review_count_index] = review_count_no_commas
changes_made += 1
# Second, remove any remaining brackets
for row in data_rows:
if review_count_index is not None:
row[review_count_index] = re.sub(r'[\(\)]', '', row[review_count_index])
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(data_rows)
return changes_made
def verify_no_brackets(file_path):
"""Verify that there are no brackets remaining in the GBP Review Count column."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return True
headers = rows[0]
data_rows = rows[1:]
review_count_index = None
if 'GBP Review Count' in headers:
review_count_index = headers.index('GBP Review Count')
else:
return True
for row in data_rows:
if review_count_index is not None and ( '(' in row[review_count_index] or ')' in row[review_count_index] ):
return False
return True
def strip_brackets_from_review_count():
"""Strip brackets from review count in all CSV files in the stage 5 directory and tally the results."""
figlet = Figlet(font='slant')
print(figlet.renderText('Strip Brackets'))
total_changes_made = 0
for state_dir in os.listdir(STAGE_5_DIRECTORY):
state_path = os.path.join(STAGE_5_DIRECTORY, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
state_changes_made = 0
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
state_changes_made += process_csv_file_strip_brackets(file_path)
if not verify_no_brackets(file_path):
process_csv_file_strip_brackets(file_path)
if not verify_no_brackets(file_path):
spinner.fail(f"Failed to remove brackets from {file_path}")
return
total_changes_made += state_changes_made
spinner.succeed(f'Finished processing {state_dir}. Changes made: {state_changes_made}')
print(f"Total Changes Made: {total_changes_made}")
if __name__ == "__main__":
strip_brackets_from_review_count()

107
bin/stage-6/business-name-strip-structures.md Normal file
View file

@ -0,0 +1,107 @@
# Prompt 1
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
I wish to make a new script which will have as input the csv file: "06-initial-deduplication.csv". And as output, the csv file: "07-business-name-structures-stripped.csv".
This script should be called 'business-name-strip-structures'.
It will reside in [Stage 6 Binaries]
The file needs to strip various acronyms from Business Names in the "Business Name" column.
The script needs to parse the Business Names for Business Names which contain various acronyms, such as "LLC, "Inc", and "Co".
When a matching string is found, that string needs to be removed.
If the matched string is the end of the entire string found in the cell, then be certain to remove trailing spaces, and commas which may have preceeded the matched string as well as the matched string we are removing.
If the matched string has more unmatched strings occurring after it in the cell, then the script needs to be sure there is a single space between the portion of the cell string which came before the matched string, and the portion of the cell string which came after the matched string. Essentially if we are removing the center part of a longer string, we do not want to mash two words together by removing this matched string. We want those two words to maintain readable spacing. (1 space apart)
All these strings we are searching for should only match if they are found on their own as a complete word. They must not match if they are found inside of another word, or mashed up against another word.
Some of these terms may be the last component in the entire cell string, meaning there will be no spaces after the matching string sometimes.
They will never be the start of the entire cell string. They will always be in the middle or at the end.
Most of the time they will have a space before the string, however sometimes the string we are searching for may have punctuation before it instead of a space. Such as a comma or period.
Also, we are not looking for case sensitive matching. We can ignore case when searching for matches.
If a string in the list we are searching to match against has puntuation in it we need to be careful to match against that punctuation as well. Periods after "Co." count for example as something we may want to match against.
Here are the strings we searching for to remove from the Business Name entries:
<business name extras to remove>
llc
llc.
l.l.c.
co
co.
inc
inc.
corp
corp.
</business name extras to remove>
# Prompt 2
The script also needs to check and make sure 07-business-name-structures-stripped.csv doesn't already exist before it begins. If it does exist, it needs to be deleted before the script continues.
# Prompt 3
This is close. It is almost working.
However, some punctuation is being left behind.
Look at the following example outputs:
<example outputs>
Original: ​B & S Paving and Construction, Inc., Cleaned: ​B & S Paving and Construction, .
Original: Zorro Concrete LLC., Cleaned: Zorro Concrete .
Original: Young's Concrete Pumping, Inc., Cleaned: Young's Concrete Pumping, .
</example outputs>
In some instances commas and periods are being left behind.
For example "B & S Paving and Construction, Inc." became: "B & S Paving and Construction, ." but should have become: "B & S Paving and Construction"
And "Zorro Concrete LLC." became: "Zorro Concrete ." but should have became: "Zorro Concrete"
And "Young's Concrete Pumping, Inc." became: "Young's Concrete Pumping, ." but should have became: "Young's Concrete Pumping"
Please adjust the script to check for clean punctuation in the area the matched string was removed from. We want to clean up left over punctuation as a part of this process.

73
bin/stage-6/business-name-strip-structures.py Executable file
View file

@ -0,0 +1,73 @@
#!/usr/bin/env python
# Script Name: business-name-strip-structures
import os
import csv
import re
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
STAGE_6_DIRECTORY = os.path.join(PROJECT_ROOT, 'current-data', '.data', 'stage-6')
INPUT_CSV = os.path.join(STAGE_6_DIRECTORY, '06-initial-deduplication.csv')
OUTPUT_CSV = os.path.join(STAGE_6_DIRECTORY, '07-business-name-structures-stripped.csv')
# List of terms to remove
terms_to_remove = [
r'\bllc\b', r'\bllc\.\b', r'\bl\.l\.c\.\b',
r'\bco\b', r'\bco\.\b', r'\binc\b', r'\binc\.\b',
r'\bcorp\b', r'\bcorp\.\b'
]
def clean_business_name(business_name):
"""Remove specified terms from the business name and clean up punctuation."""
for term in terms_to_remove:
business_name = re.sub(r'\s*[,.]*\s*' + term + r'[,.]*\s*', ' ', business_name, flags=re.IGNORECASE)
business_name = re.sub(r'\s+', ' ', business_name).strip()
business_name = re.sub(r'[,.]\s*$', '', business_name)
return business_name
def strip_structures_from_business_names():
"""Strip specified structures from business names."""
if not os.path.exists(INPUT_CSV):
print(f"Error: The input file {INPUT_CSV} does not exist.")
return
with open(INPUT_CSV, 'r') as infile:
reader = csv.DictReader(infile)
headers = reader.fieldnames
if 'Business Name' not in headers:
print("Error: 'Business Name' column is missing.")
return
rows = list(reader)
with open(OUTPUT_CSV, 'w', newline='') as outfile:
writer = csv.DictWriter(outfile, fieldnames=headers)
writer.writeheader()
for row in rows:
row['Business Name'] = clean_business_name(row.get('Business Name', '').strip())
writer.writerow(row)
def main():
figlet = Figlet(font='slant')
script_name = "business-name-strip-structures".replace("-", " ").title()
print(figlet.renderText(script_name))
if os.path.exists(OUTPUT_CSV):
os.remove(OUTPUT_CSV)
print(f"Existing output file {OUTPUT_CSV} found and deleted.")
spinner = Halo(text='Stripping business name structures', spinner='dots')
spinner.start()
strip_structures_from_business_names()
spinner.succeed('Business name structures stripped.')
print(f"Processed input CSV: {INPUT_CSV}")
print(f"Output CSV: {OUTPUT_CSV}")
if __name__ == '__main__':
main()

99
bin/stage-6/business-names-with-locations.md Normal file
View file

@ -0,0 +1,99 @@
# Prompt 1
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
I wish to create a new script named 'business-names-with-locations', which will have as input the csv file "07-business-name-structures-stripped.csv", and output will have the csv file "08-business-names-with-locations.csv". These csv files will be located in [Stage 6].
Script Overview: The overview for this script is that we are attempting to process the "Business Name" field of the input CSV looking for Business Names which have locations appended to the end of them. Such as the examples in the following example CSV data:
<example csv data>
Business Name,GBP Review Rating,GBP Review Count,GBP Business Category,YiB,GBP Matching Service,GBP Business Phone,GBP Business Website,GBP Location Municipality,GBP Location State,Root Domain
ultimate concrete resurfacing of scotch valley,3.8,12,Concrete Contractor,5+ years in business,,+1 814-931-7100,http://www.concretedesigns4you.com/,Hollidaysburg,PA,concretedesigns4you.com
Yellow Dawg Asphalt of The Triad,5,12,Asphalt Contractor,7+ years in business,Driveway Sealcoating,+1 336-272-5757,https://www.yellowdawgasphalt.com/,Greensboro,NC,yellowdawgasphalt.com
Window Ninjas of Raleigh,5.0,129,Window Cleaning Service,7+ years in business,"Concrete, Brick, And Stone Washing",+1 919-867-6276,https://windowninjas.com/north-carolina/raleigh-durham/,Garner,NC,windowninjas.com
Window Ninjas of Greenville/Spartanburg,4.9,201,Window Cleaning Service,5+ years in business,"Concrete, Brick, And Stone Washing",+1 864-558-7758,https://windowninjas.com/south-carolina/greenville-spartanburg/,Greenville,SC,windowninjas.com
</example csv data>
In the above example data we have the following example Business Names:
<example business names>
ultimate concrete resurfacing of scotch valley
Yellow Dawg Asphalt of The Triad
Window Ninjas of Raleigh
Window Ninjas of Greenville/Spartanburg
</example business names>
This is an example of a Business Name which follows a formula of "BUSINESS-NAME of LOCATION".
I want to focus on the " of " portion of the string here.
I want the first pass that this script takes when parsing the data to be for the string " of ". When we see that string, including spaces, we can assume that what comes before it is the Business Name, and what comes after it is the Location.
Handling these three string components when a " of " match occurs is what I want to deal with right now in this script.
First, the script should create a new column in the destination CSV file named "Location from GBP Business Name".
That "Location from GBP Business Name" column is where we want to move the Location to when we find a " of " match.
After the "Location from GBP Business Name" column is created we can begin processing the "Business Name" field row by row looking for matches against " of ".
When a match with " of " happens, the script should move whatever is to the right of the " of " into the "Location from GBP Business Name" column for that row. This move process should not allow for leading spaces to be added to the "Location from GBP Business Name" column during this move though. We do not want strings that begin with spaces added to this column.
Then the script should remove the " of " portion of the string from the matching cell to clean it up a bit.
Finally, the script should check for punctuation or symbols at the end of the matching cell, such as commas, hyphens, periods, etc, and delete them if found.
If you understand and have no questions you need to ask first, begin writing the python script for me now.
# Prompt 2
I got the following error when running the script:
<error>
⠙ Processing business names with locationsTraceback (most recent call last):
File "/home/ld/mgk-scrapes/bin/stage-6/./business-names-with-locations.py", line 68, in <module>
main()
File "/home/ld/mgk-scrapes/bin/stage-6/./business-names-with-locations.py", line 61, in main
process_business_names()
File "/home/ld/mgk-scrapes/bin/stage-6/./business-names-with-locations.py", line 44, in process_business_names
cleaned_name, location = extract_location(business_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ld/mgk-scrapes/bin/stage-6/./business-names-with-locations.py", line 20, in extract_location
location = parts[1].strip()
~~~~~^^^
IndexError: list index out of range
</error>

70
bin/stage-6/business-names-with-locations.py Executable file
View file

@ -0,0 +1,70 @@
#!/usr/bin/env python
# Script Name: business-names-with-locations
import os
import csv
import re
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
STAGE_6_DIRECTORY = os.path.join(PROJECT_ROOT, 'current-data', '.data', 'stage-6')
INPUT_CSV = os.path.join(STAGE_6_DIRECTORY, '07-business-name-structures-stripped.csv')
OUTPUT_CSV = os.path.join(STAGE_6_DIRECTORY, '08-business-names-with-locations.csv')
def extract_location(business_name):
"""Extract location from business name if it contains ' of '."""
if ' of ' in business_name.lower():
parts = business_name.rsplit(' of ', 1)
if len(parts) == 2: # Ensure the split produced two parts
location = parts[1].strip()
business_name_clean = parts[0].strip()
# Remove trailing punctuation from the cleaned business name
business_name_clean = re.sub(r'[,.!?;:\-]+$', '', business_name_clean).strip()
return business_name_clean, location
return business_name, ''
def process_business_names():
"""Process the business names to extract locations and save the results."""
with open(INPUT_CSV, 'r') as infile:
reader = csv.DictReader(infile)
headers = reader.fieldnames
if 'Business Name' not in headers:
print("Error: 'Business Name' column is missing.")
return
headers.append('Location from GBP Business Name')
rows = list(reader)
with open(OUTPUT_CSV, 'w', newline='') as outfile:
writer = csv.DictWriter(outfile, fieldnames=headers)
writer.writeheader()
for row in rows:
business_name = row.get('Business Name', '').strip()
cleaned_name, location = extract_location(business_name)
row['Business Name'] = cleaned_name
row['Location from GBP Business Name'] = location
writer.writerow(row)
def main():
figlet = Figlet(font='slant')
script_name = "business-names-with-locations".replace("-", " ").title()
print(figlet.renderText(script_name))
# Check if the output file already exists and delete it if it does
if os.path.exists(OUTPUT_CSV):
os.remove(OUTPUT_CSV)
print(f"Existing output file {OUTPUT_CSV} found and deleted.")
spinner = Halo(text='Processing business names with locations', spinner='dots')
spinner.start()
process_business_names()
spinner.succeed('Business names with locations processed.')
print(f"Processed input CSV: {INPUT_CSV}")
print(f"Output CSV: {OUTPUT_CSV}")
if __name__ == '__main__':
main()

169
bin/stage-6/delete-bad-matching-services.md Normal file
View file

@ -0,0 +1,169 @@
# Prompt 1
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'delete-bad-matching-services' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<delete-bad-matching-services>
#!/usr/bin/env python
import os
import csv
from tqdm import tqdm
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage6_dir = os.path.join(project_root, '.data', 'stage-6')
deduplicated_csv_path = os.path.join(stage6_dir, '02-initial-deduplication.csv')
data_dir = os.path.join(project_root, '.data')
bad_services_file_path = os.path.join(data_dir, 'bad-matching-services.txt')
def load_bad_services(file_path):
with open(file_path, 'r') as file:
bad_services = [line.strip().lower() for line in file if line.strip()]
return bad_services
def delete_bad_matching_services():
bad_services = load_bad_services(bad_services_file_path)
bad_services_set = set(bad_services)
cells_cleared = 0
with open(deduplicated_csv_path, 'r') as csv_file:
reader = csv.DictReader(csv_file)
headers = reader.fieldnames
rows = list(reader)
with open(deduplicated_csv_path, 'w', newline='') as csv_file:
writer = csv.DictWriter(csv_file, fieldnames=headers)
writer.writeheader()
for row in tqdm(rows, desc="Processing rows", unit="row"):
matching_service = row.get('GBP Matching Service', '').strip().lower()
if matching_service in bad_services_set:
row['GBP Matching Service'] = ''
cells_cleared += 1
writer.writerow(row)
print(f'Total cells cleared: {cells_cleared}')
if __name__ == '__main__':
delete_bad_matching_services()
</delete-bad-matching-services>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>
# Prompt 2
I forgot to mention that 'delete-bad-matching-services' should use as input, the csv file: "03-trimmed-invalid-rows.csv". And as output, the csv file: "04-deleted-bad-matching-services.csv".

70
bin/stage-6/delete-bad-matching-services.py Executable file
View file

@ -0,0 +1,70 @@
#!/usr/bin/env python
# Script Name: delete-bad-matching-services
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_6_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-6")
INPUT_CSV_PATH = os.path.join(STAGE_6_DIRECTORY, '03-trimmed-invalid-rows.csv')
OUTPUT_CSV_PATH = os.path.join(STAGE_6_DIRECTORY, '04-deleted-bad-matching-services.csv')
BAD_SERVICES_FILE_PATH = os.path.join(DATA_DIRECTORY, 'bad-matching-services.txt')
def load_bad_services(file_path):
"""Load bad services from the specified file."""
if not os.path.exists(file_path):
print(f"Error: The file {file_path} does not exist.")
sys.exit(1)
with open(file_path, 'r') as file:
bad_services = [line.strip().lower() for line in file if line.strip()]
return bad_services
def delete_bad_matching_services():
"""Delete bad matching services from the input CSV and write to the output CSV."""
bad_services = load_bad_services(BAD_SERVICES_FILE_PATH)
bad_services_set = set(bad_services)
cells_cleared = 0
with open(INPUT_CSV_PATH, 'r') as csv_file:
reader = csv.DictReader(csv_file)
headers = reader.fieldnames
rows = list(reader)
with open(OUTPUT_CSV_PATH, 'w', newline='') as csv_file:
writer = csv.DictWriter(csv_file, fieldnames=headers)
writer.writeheader()
for row in rows:
matching_service = row.get('GBP Matching Service', '').strip().lower()
if matching_service in bad_services_set:
row['GBP Matching Service'] = ''
cells_cleared += 1
writer.writerow(row)
return cells_cleared
def main():
figlet = Figlet(font='slant')
script_name = "delete-bad-matching-services".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(INPUT_CSV_PATH):
print(f"Error: The file {INPUT_CSV_PATH} does not exist.")
sys.exit(1)
print(f"Processing {INPUT_CSV_PATH}...")
spinner = Halo(text='Processing data', spinner='dots')
spinner.start()
cells_cleared = delete_bad_matching_services()
spinner.succeed("Processing complete.")
print(f"Total cells cleared: {cells_cleared}")
if __name__ == "__main__":
main()

58
bin/stage-6/fill-in-missing-data.md Normal file
View file

@ -0,0 +1,58 @@
# Prompt 1
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
The Project Details above contain locations for files, data, scripts, and rules to follow when creating project scripts and toolkit.
I wish to create a new script named 'fill-in-missing-data' which will use the csv file "12-re-ordered-columns.csv" as input, and will output to the csv file "13-filled-in-missing-data.csv" inside of [Stage 6].
I would like the script to merge in missing data for each row of match group, by fetching the missing data from other rows in the match group.
The match groups I want to focus on are rows that have the same "Business UUID" entry.
Rows with the same "Business UUID" should have certain bits of information in column.
I want the script to examine each row in each match group for cells that have data missing from certain rows. If data is found to missing in those certain rows, I would like the script to try and find that data in the cells of other match member rows. If that data can be found, it should be copied over to the row with the cell that is missing that data.
The columns I wish to account for are:
<columns to merge with>
YiB
GBP Business Category
GBP Matching Service
</columns to merge with>

83
bin/stage-6/fill-in-missing-data.py Executable file
View file

@ -0,0 +1,83 @@
#!/usr/bin/env python
# Script Name: fill-in-missing-data
import os
import csv
from collections import defaultdict
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
STAGE_6_DIRECTORY = os.path.join(PROJECT_ROOT, 'current-data', '.data', 'stage-6')
INPUT_CSV = os.path.join(STAGE_6_DIRECTORY, '12-re-ordered-columns.csv')
OUTPUT_CSV = os.path.join(STAGE_6_DIRECTORY, '13-filled-in-missing-data.csv')
def fill_in_missing_data():
"""Fill in missing data for each row of match group by fetching data from other rows in the group."""
columns_to_merge = ['YiB', 'GBP Business Category', 'GBP Matching Service']
uuid_to_rows = defaultdict(list)
# Read the input CSV and group rows by Business UUID
with open(INPUT_CSV, 'r') as infile:
reader = csv.DictReader(infile)
headers = reader.fieldnames
if 'Business UUID' not in headers:
print("Error: 'Business UUID' column is missing.")
return
rows = list(reader)
print(f"Total rows read from input file: {len(rows)}")
for row in rows:
business_uuid = row['Business UUID'].strip()
if business_uuid: # Ignore empty UUIDs
uuid_to_rows[business_uuid].append(row)
print(f"Total groups by Business UUID: {len(uuid_to_rows)}")
# Fill in missing data within each group
for uuid, grouped_rows in uuid_to_rows.items():
# Collect data from all rows in the group
merged_data = {column: '' for column in columns_to_merge}
for row in grouped_rows:
for column in columns_to_merge:
if row[column].strip():
merged_data[column] = row[column]
# Fill missing data in each row
for row in grouped_rows:
for column in columns_to_merge:
if not row[column].strip():
row[column] = merged_data[column]
# Write to the output CSV
with open(OUTPUT_CSV, 'w', newline='') as outfile:
writer = csv.DictWriter(outfile, fieldnames=headers)
writer.writeheader()
for row in rows:
writer.writerow(row)
print(f"Total rows written to output file: {len(rows)}")
def main():
figlet = Figlet(font='slant')
script_name = "fill-in-missing-data".replace("-", " ").title()
print(figlet.renderText(script_name))
# Check if the output file already exists and delete it if it does
if os.path.exists(OUTPUT_CSV):
os.remove(OUTPUT_CSV)
print(f"Existing output file {OUTPUT_CSV} found and deleted.")
spinner = Halo(text='Filling in missing data by Business UUID', spinner='dots')
spinner.start()
fill_in_missing_data()
spinner.succeed('Missing data filled successfully.')
print(f"Processed input CSV: {INPUT_CSV}")
print(f"Output CSV: {OUTPUT_CSV}")
if __name__ == '__main__':
main()

126
bin/stage-6/find-unknown-categories-and-services.md Normal file
View file

@ -0,0 +1,126 @@
# Prompt 1
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to refactor the script 'find-unknown-categories-and-services' to be in accordance with the <Project Details> above, and to adjust the reporting style now:
<find-unknown-categories-and-services>
</find-unknown-categories-and-services>
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>
The 'find-unknown-categories-and-services' script should use as input, the csv file: "05-standard-ized-capital-letters.csv".

75
bin/stage-6/find-unknown-categories-and-services.py Executable file
View file

@ -0,0 +1,75 @@
#!/usr/bin/env python
# Script Name: find-unknown-categories-and-services
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_6_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-6")
INPUT_CSV_PATH = os.path.join(STAGE_6_DIRECTORY, '05-standardized-capital-letters.csv')
GBP_CATEGORIES_PATH = os.path.join(DATA_DIRECTORY, 'gbp-business-categories.txt')
GBP_SERVICES_PATH = os.path.join(DATA_DIRECTORY, 'gbp-matching-services.txt')
UNKNOWN_CATEGORIES_PATH = os.path.join(STAGE_6_DIRECTORY, 'unknown-categories.txt')
UNKNOWN_SERVICES_PATH = os.path.join(STAGE_6_DIRECTORY, 'unknown-services.txt')
def load_known_values(file_path):
"""Load known values from a text file into a set."""
with open(file_path, 'r') as file:
return set(line.strip().lower() for line in file if line.strip())
def find_unknowns(input_csv_path, known_categories, known_services):
"""Find unknown categories and services in the input CSV file."""
unknown_categories = set()
unknown_services = set()
with open(input_csv_path, 'r') as csv_file:
reader = csv.DictReader(csv_file)
for row in reader:
category = row.get('GBP Business Category', '').strip().lower()
service = row.get('GBP Matching Service', '').strip().lower()
if category and category not in known_categories:
unknown_categories.add(category)
if service and service not in known_services:
unknown_services.add(service)
return unknown_categories, unknown_services
def save_unknowns(unknowns, file_path):
"""Save unknown values to a text file."""
with open(file_path, 'w') as file:
for item in sorted(unknowns):
file.write(f"{item}\n")
def main():
figlet = Figlet(font='slant')
script_name = "find-unknown-categories-and-services".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(INPUT_CSV_PATH):
print(f"Error: The file {INPUT_CSV_PATH} does not exist.")
sys.exit(1)
print(f"Processing {INPUT_CSV_PATH}...")
spinner = Halo(text='Finding unknown categories and services', spinner='dots')
spinner.start()
known_categories = load_known_values(GBP_CATEGORIES_PATH)
known_services = load_known_values(GBP_SERVICES_PATH)
unknown_categories, unknown_services = find_unknowns(INPUT_CSV_PATH, known_categories, known_services)
save_unknowns(unknown_categories, UNKNOWN_CATEGORIES_PATH)
save_unknowns(unknown_services, UNKNOWN_SERVICES_PATH)
spinner.succeed("Finding unknown categories and services complete.")
print(f"Total unknown categories found: {len(unknown_categories)}")
print(f"Total unknown services found: {len(unknown_services)}")
if __name__ == "__main__":
main()

50
bin/stage-6/initial-deduplication.md Normal file
View file

@ -0,0 +1,50 @@
# Prompt 1
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
I wish to make a new script which will have as input the csv file: "05-standardized-capital-letters.csv". And as output, the csv file: "06-initial-deduplication.csv".
This script should perform a very basic deduplication on the 05 file. Removing any duplicate lines from the csv file.
For this script, we are consider duplicates to be any line that has another line which is exactly the same accross all fields.
When the script completes, it should report a before count for rows of data, and an after count for rows of data. It should also list how many duplicates were removed.
This script should be called 'initial-deduplication'.
It will reside in [Stage 6 Binaries]

84
bin/stage-6/initial-deduplication.py Executable file
View file

@ -0,0 +1,84 @@
#!/usr/bin/env python
# Script Name: initial-deduplication
import os
import csv
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
STAGE_6_DIRECTORY = os.path.join(PROJECT_ROOT, 'current-data', '.data', 'stage-6')
INPUT_CSV = os.path.join(STAGE_6_DIRECTORY, '05-standardized-capital-letters.csv')
OUTPUT_CSV = os.path.join(STAGE_6_DIRECTORY, '06-initial-deduplication.csv')
MD5_CSV = os.path.join(STAGE_6_DIRECTORY, 'aa-existing-md5-for-scrapes.csv')
def extract_unique_md5():
"""Extract unique MD5 entries and write them to a new CSV file."""
with open(INPUT_CSV, 'r') as infile:
reader = csv.DictReader(infile)
md5_set = set()
for row in reader:
md5_value = row.get("MD5 for Scrape", "").strip()
if md5_value:
md5_set.add(md5_value)
with open(MD5_CSV, 'w', newline='') as outfile:
writer = csv.writer(outfile)
writer.writerow(["MD5 for Scrape"])
for md5_value in md5_set:
writer.writerow([md5_value])
def remove_duplicates():
"""Remove duplicate rows from the CSV file."""
with open(INPUT_CSV, 'r') as infile:
reader = csv.DictReader(infile)
headers = reader.fieldnames
rows = list(reader)
total_rows_before = len(rows)
# Use a set to track unique rows
unique_rows = []
seen = set()
for row in rows:
row_tuple = tuple(row.items())
if row_tuple not in seen:
seen.add(row_tuple)
unique_rows.append(row)
total_rows_after = len(unique_rows)
duplicates_removed = total_rows_before - total_rows_after
# Write the unique rows back to a new CSV file
with open(OUTPUT_CSV, 'w', newline='') as outfile:
writer = csv.DictWriter(outfile, fieldnames=headers)
writer.writeheader()
writer.writerows(unique_rows)
return total_rows_before, total_rows_after, duplicates_removed
def main():
figlet = Figlet(font='slant')
script_name = "initial-deduplication".replace("-", " ").title()
print(figlet.renderText(script_name))
# Extract unique MD5 entries
md5_spinner = Halo(text='Extracting unique MD5 entries', spinner='dots')
md5_spinner.start()
extract_unique_md5()
md5_spinner.succeed('Unique MD5 entries extracted.')
# Remove duplicates
spinner = Halo(text='Removing duplicates', spinner='dots')
spinner.start()
total_rows_before, total_rows_after, duplicates_removed = remove_duplicates()
spinner.succeed('Duplicates removed.')
print(f'Total rows before: {total_rows_before}')
print(f'Total rows after: {total_rows_after}')
print(f'Duplicates removed: {duplicates_removed}')
if __name__ == '__main__':
main()

170
bin/stage-6/old-initial-dedup.py Executable file
View file

@ -0,0 +1,170 @@
#!/usr/bin/env python
import os
import csv
import shutil
from tqdm import tqdm
from collections import defaultdict
# Get the project root directory
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'Concrete Sealing Company'))
stage6_dir = os.path.join(project_root, '.data', 'stage-6')
merged_csv_path = os.path.join(stage6_dir, '01-first-merger.csv')
deduplicated_csv_path = os.path.join(stage6_dir, '02-initial-deduplication.csv')
duplicates_csv_path = os.path.join(stage6_dir, 'AA-deduplicated-duplicates.csv')
def copy_merged_to_deduplicated():
if os.path.exists(deduplicated_csv_path):
os.remove(deduplicated_csv_path)
shutil.copy(merged_csv_path, deduplicated_csv_path)
def merge_rows(rows):
merged_row = rows[0].copy()
for row in rows[1:]:
for key in merged_row:
if not merged_row[key] and row[key]:
merged_row[key] = row[key]
return merged_row
def deduplicate_and_merge():
business_dict = defaultdict(list)
unique_rows = []
duplicate_rows = []
total_duplicates_removed = 0
# Read the CSV file
with open(deduplicated_csv_path, 'r') as csv_file:
reader = csv.DictReader(csv_file)
headers = reader.fieldnames
rows = list(reader)
# Use tqdm to display a progress bar
for row in tqdm(rows, desc="Grouping rows by Business Name", unit="row"):
business_name = row.get('Business Name', '').strip()
business_dict[business_name].append(row)
# Process each group of duplicates
for business_name, business_rows in tqdm(business_dict.items(), desc="Processing business groups", unit="group"):
if len(business_rows) == 1:
unique_rows.append(business_rows[0])
continue
# Group by GBP Review Rating and GBP Review Count (location identifier)
location_dict = defaultdict(list)
for row in business_rows:
location_id = (row.get('GBP Review Rating', '').strip(), row.get('GBP Review Count', '').strip())
location_dict[location_id].append(row)
# Process each location group
for location_id, location_rows in location_dict.items():
if len(location_rows) == 1:
unique_rows.append(location_rows[0])
else:
# Merge rows and keep the most complete one
merged_row = merge_rows(location_rows)
unique_rows.append(merged_row)
duplicate_rows.extend(location_rows)
total_duplicates_removed += (len(location_rows) - 1)
# Second pass for deduplication based on phone number within the same municipality
final_unique_rows = []
municipality_dict = defaultdict(list)
for row in unique_rows:
municipality_key = (row['Business Name'], row['GBP Location Municipality'])
municipality_dict[municipality_key].append(row)
for municipality_key, municipality_rows in municipality_dict.items():
phone_dict = defaultdict(list)
for row in municipality_rows:
phone_dict[row['GBP Business Phone']].append(row)
for phone, phone_rows in phone_dict.items():
if len(phone_rows) == 1:
final_unique_rows.append(phone_rows[0])
else:
merged_row = merge_rows(phone_rows)
final_unique_rows.append(merged_row)
duplicate_rows.extend(phone_rows)
total_duplicates_removed += (len(phone_rows) - 1)
# Third pass for merging based on business name and phone number
final_deduplicated_rows = []
business_dict = defaultdict(list)
for row in final_unique_rows:
business_name = row.get('Business Name', '').strip()
business_dict[business_name].append(row)
for business_name, business_rows in tqdm(business_dict.items(), desc="Processing final business groups", unit="group"):
if len(business_rows) == 1:
final_deduplicated_rows.append(business_rows[0])
continue
location_dict = defaultdict(list)
for row in business_rows:
location_key = (row.get('GBP Location Municipality', '').strip(), row.get('GBP Location State', '').strip())
location_dict[location_key].append(row)
for location_key, location_rows in location_dict.items():
phone_dict = defaultdict(list)
for row in location_rows:
phone_dict[row['GBP Business Phone']].append(row)
for phone, phone_rows in phone_dict.items():
if len(phone_rows) == 1:
final_deduplicated_rows.append(phone_rows[0])
else:
merged_row = merge_rows(phone_rows)
final_deduplicated_rows.append(merged_row)
duplicate_rows.extend(phone_rows)
total_duplicates_removed += (len(phone_rows) - 1)
# Additional pass to merge based on business name and phone number
final_pass_rows = []
final_business_dict = defaultdict(list)
for row in final_deduplicated_rows:
business_name = row.get('Business Name', '').strip()
final_business_dict[business_name].append(row)
for business_name, business_rows in tqdm(final_business_dict.items(), desc="Final processing for business groups", unit="group"):
if len(business_rows) == 1:
final_pass_rows.append(business_rows[0])
continue
phone_dict = defaultdict(list)
for row in business_rows:
phone_dict[row['GBP Business Phone']].append(row)
for phone, phone_rows in phone_dict.items():
if len(phone_rows) == 1:
final_pass_rows.append(phone_rows[0])
else:
merged_row = merge_rows(phone_rows)
final_pass_rows.append(merged_row)
duplicate_rows.extend(phone_rows)
total_duplicates_removed += (len(phone_rows) - 1)
# Sort the final deduplicated rows for easier viewing
final_pass_rows.sort(key=lambda x: (x['Business Name'], x['GBP Location Municipality'], x['GBP Location State'], x['Root Domain']))
# Write the final deduplicated rows to the deduplicated CSV file
with open(deduplicated_csv_path, 'w', newline='') as csv_file:
writer = csv.DictWriter(csv_file, fieldnames=headers)
writer.writeheader()
writer.writerows(final_pass_rows)
# Sort the duplicate rows for easier viewing
duplicate_rows.sort(key=lambda x: (x['Business Name'], x['GBP Location Municipality'], x['GBP Location State'], x['Root Domain']))
# Write the duplicate rows to the duplicates CSV file
with open(duplicates_csv_path, 'w', newline='') as csv_file:
writer = csv.DictWriter(csv_file, fieldnames=headers)
writer.writeheader()
writer.writerows(duplicate_rows)
# Print the total number of duplicates removed
print(f'Total duplicates removed: {total_duplicates_removed}')
if __name__ == '__main__':
copy_merged_to_deduplicated()
deduplicate_and_merge()

1640
bin/stage-6/package-lock.json generated Normal file
View file

File diff suppressed because it is too large Load diff

5
bin/stage-6/package.json Normal file
View file

@ -0,0 +1,5 @@
{
"dependencies": {
"lighthouse": "^12.0.0"
}
}

134
bin/stage-6/pre-sort.md Normal file
View file

@ -0,0 +1,134 @@
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
I wish to create a new script called 'pre-sort'. It should use the file 01-first-merger.csv in the Stage 5 data directory as the input. And the output for this script should a file in the same directory named 02-pre-sorted.csv.
The job of the script will be to sort the data in the input csv using the first column as the index, and output the sorted data to the output file.
Respect the headings in the CSV file as well.
Please make this script have the same kind of by-state reporting as the 'remove-utm' script:
<remove-utm script>
#!/usr/bin/env python
# Script Name: remove-utm
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")
def process_csv_file_remove_utm(file_path):
"""Remove UTM parameters from URLs in the given CSV file."""
with open(file_path, 'r') as file:
reader = csv.reader(file)
rows = list(reader)
if not rows:
return
modified_rows = []
for row in rows:
modified_row = []
for cell in row:
if "?utm" in cell:
modified_cell = cell.split("?utm")[0]
else:
modified_cell = cell
modified_row.append(modified_cell)
modified_rows.append(modified_row)
with open(file_path, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(modified_rows)
def process_csv_files(stage_directory):
"""Process all CSV files in the given stage directory to remove UTM parameters from URLs."""
for state_dir in os.listdir(stage_directory):
state_path = os.path.join(stage_directory, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
process_csv_file_remove_utm(file_path)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "remove-utm".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(STAGE_2_DIRECTORY):
print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
sys.exit(1)
print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)
final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")
if __name__ == "__main__":
main()
</remove-utm script>
# Prompt 2
I got the following error while running this script:
<error>
Error: The file /home/ld/mgk-scrapes/current-data/.data/stage-5/01-first-merger.csv does not exist.
</error>
This script needs to operate in the Stage 6 directory, not the Stage 5 directory.
Also, please have this script output a total tally of all data records found in the CSV file after the new file has been created.

51
bin/stage-6/pre-sort.py Executable file
View file

@ -0,0 +1,51 @@
#!/usr/bin/env python
# Script Name: pre-sort
import os
import csv
import sys
from pyfiglet import Figlet
from halo import Halo
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
DATA_DIRECTORY = os.path.join(PROJECT_ROOT, "current-data", ".data")
STAGE_6_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-6")
MERGED_CSV = os.path.join(STAGE_6_DIRECTORY, "01-first-merger.csv")
SORTED_CSV = os.path.join(STAGE_6_DIRECTORY, "02-pre-sorted.csv")
def sort_csv_file(input_file, output_file):
"""Sort the CSV file by the first column and return the number of records."""
with open(input_file, 'r') as file:
reader = csv.reader(file)
headers = next(reader)
sorted_rows = sorted(reader, key=lambda row: row[0])
with open(output_file, 'w', newline='') as file:
writer = csv.writer(file)
writer.writerow(headers)
writer.writerows(sorted_rows)
return len(sorted_rows)
def main():
figlet = Figlet(font='slant')
script_name = "pre-sort".replace("-", " ").title()
print(figlet.renderText(script_name))
if not os.path.exists(MERGED_CSV):
print(f"Error: The file {MERGED_CSV} does not exist.")
sys.exit(1)
print(f"Sorting {MERGED_CSV}...")
spinner = Halo(text='Sorting data', spinner='dots')
spinner.start()
total_records = sort_csv_file(MERGED_CSV, SORTED_CSV)
spinner.succeed("Sorting complete.")
print(f"Sorted data saved to {SORTED_CSV}")
print(f"Total records in sorted file: {total_records}")
if __name__ == "__main__":
main()

197
bin/stage-6/prepare-stage-6.md Normal file
View file

@ -0,0 +1,197 @@
# Prompt 1
<project details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
Please be sure to reference project details above when factoring code, or answering questions.
Now I need a prepare-stage-6 script.
This script is different than the rest of the prepare-stage-# scripts though.
This script needs to check if Stage 6 exists first, and if it exists, delete it, and then re-create the directory. In the same way as the previous prepare-stage-# scripts.
However, instead of directly copying over the directories and files from stage-5, we need to create a new csv file in the Stage 6 directory called "01-first-merger.csv".
This contents of this CSV file will be populated by iterating through every CSV file in Stage 5. The script needs to start by grabbing a heading from the first CSV file encountered, adding that to 01-first-merger.csv, and then appending data to that CSV file after that. Every CSV file in Stage 5 should be appended to 01-first-merger.csv. Minus the heading. We need to be certain that we only ever include a single heading in 01-first-merger.csv. We do not want mulitple headings to appear in the data, corrupting the data set.
Please be sure to name all script output to accurately describe what the script is doing as it does it. ANd use spinners to show us each State being processed. We do not need a new line of reporting for every county. By state is sufficient.
# Prompt 2
<Project Details>
- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts.
- [Project Root]: "/home/ld/mgk-scrapes"
- [Current Dataset]: "[Project Root]/current-data/"
- [Data Directory]: "[Current Dataset]/.data/"
- [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".")
- [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.)
- [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state.
- [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under.
- [Stage 1]: "[Data Directory]/stage-1/"
- [Stage 2]: "[Data Directory]/stage-2/"
- [Stage 3]: "[Data Directory]/stage-3/"
- [Stage 4]: "[Data Directory]/stage-4/"
- [Stage 5]: "[Data Directory]/stage-5/"
- [Stage 6]: "[Data Directory]/stage-6/"
- [Binaries]: "[Project Root]/bin/"
- [Stage 1 Binaries]: "[Binaries]/stage-1/"
- [Stage 2 Binaries]: "[Binaries]/stage-2/"
- [Stage 3 Binaries]: "[Binaries]/stage-3/"
- [Stage 4 Binaries]: "[Binaries]/stage-4/"
- [Stage 5 Binaries]: "[Binaries]/stage-5/"
- [Stage 6 Binaries]: "[Binaries]/stage-6/"
- [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt"
- [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt"
- [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt"
- [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt"
- [Rule 1]: All scripts need to be able to be ran from any directory.
- [Rule 2]: All scripts need output what they are doing, as they are doing it.
- [Rule 3]: All tasks should use halo to report successes and failures.
- [Rule 4]: The python module "tqdm" can be used to report progress when appropriate.
- [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for.
- [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python".
- [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ".
- [Rule 8]: Scripts should generally have robust error checking.
- [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.
</Project Details>
I am having some difficulty with the 'prepare-stage-6' script:
<prepare-stage-6 script>
#!/usr/bin/env python
# Script Name: prepare-stage-6
import os
import shutil
import csv
from halo import Halo
from tqdm import tqdm
from pyfiglet import Figlet
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_5_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-5")
STAGE_6_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-6")
MERGED_CSV = os.path.join(STAGE_6_DIRECTORY, "01-first-merger.csv")
def create_stage_6_directory():
"""Create the stage-6 directory, replacing it if it already exists."""
if os.path.exists(STAGE_6_DIRECTORY):
print("Existing Stage 6 found, deleting and replacing...")
shutil.rmtree(STAGE_6_DIRECTORY)
os.makedirs(STAGE_6_DIRECTORY)
print("Stage 6 directory created.")
def merge_csv_files():
"""Merge all CSV files from stage-5 into a single CSV file in stage-6."""
first_file = True
with open(MERGED_CSV, 'w', newline='') as outfile:
writer = None
for state_dir in os.listdir(STAGE_5_DIRECTORY):
state_path = os.path.join(STAGE_5_DIRECTORY, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
with open(file_path, 'r') as infile:
reader = csv.reader(infile)
header = next(reader)
if first_file:
writer = csv.writer(outfile)
writer.writerow(header)
first_file = False
for row in reader:
writer.writerow(row)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "prepare-stage-6".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Preparing Stage 6...")
create_stage_6_directory()
print("Merging CSV files to Stage 6...")
spinner = Halo(text='Merging data', spinner='dots')
spinner.start()
merge_csv_files()
spinner.succeed("Data merged.")
print("Stage 6 preparation complete.")
if __name__ == "__main__":
main()
</prepare-stage-6 script>
As it right now, when I run that script the resulting 01-first-merger.csv file has a lot of length mismatches between the header column count and data rows.
I am wondering if we can combat this by taking a different approach when merging the data.
What if we simply create the 01-first-merger.csv with the correct headers first:
<Headers>
Business Name
GBP Review Rating
GBP Review Count
GBP Business Category
YiB
GBP Matching Service
GBP Business Phone
GBP Business Website
GBP Location Municipality
GBP Location State
Root Domain
</Headers>.
Then we begin to iterate through the CSV files in the Stage 5 data directory, copying over the data from those CSV files into our 01-first-merger.csv. But we do it in a row by row, cell by cell fashion.
For each data row in the CSV files we iterate through we copy over the Business Name data to the Business Name column, then the GBP Review Rating data from the GBP Review Rating cell in the source CSV to the GBP Review Rating cell in the destination CSV.
And proceed like this until all cells, which data in them, for that given row have been copied over to the correct column in the destination CSV.
I think doing a cell by cell approach like this, rather than just bulk copying all lines over, may result in less errors.
What are your thoughts on this solution to our problem of too many length mismatches?

85
bin/stage-6/prepare-stage-6.py Executable file
View file

@ -0,0 +1,85 @@
#!/usr/bin/env python
# Script Name: prepare-stage-6
import os
import shutil
import csv
from halo import Halo
from tqdm import tqdm
from pyfiglet import Figlet
# Define the paths based on the project details
PROJECT_ROOT = "/home/ld/mgk-scrapes"
CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data")
DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")
STAGE_5_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-5")
STAGE_6_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-6")
MERGED_CSV = os.path.join(STAGE_6_DIRECTORY, "01-first-merger.csv")
HEADERS = [
"Business Name",
"GBP Review Rating",
"GBP Review Count",
"GBP Business Category",
"YiB",
"GBP Matching Service",
"GBP Business Phone",
"GBP Business Website",
"GBP Location Municipality",
"GBP Location State",
"Root Domain",
"MD5 for Scrape" # Added the new column header
]
def create_stage_6_directory():
"""Create the stage-6 directory, replacing it if it already exists."""
if os.path.exists(STAGE_6_DIRECTORY):
print("Existing Stage 6 found, deleting and replacing...")
shutil.rmtree(STAGE_6_DIRECTORY)
os.makedirs(STAGE_6_DIRECTORY)
print("Stage 6 directory created.")
def merge_csv_files():
"""Merge all CSV files from stage-5 into a single CSV file in stage-6."""
with open(MERGED_CSV, 'w', newline='') as outfile:
writer = csv.DictWriter(outfile, fieldnames=HEADERS)
writer.writeheader()
for state_dir in os.listdir(STAGE_5_DIRECTORY):
state_path = os.path.join(STAGE_5_DIRECTORY, state_dir)
if os.path.isdir(state_path):
spinner = Halo(text=f'Processing {state_dir}', spinner='dots')
spinner.start()
for county_dir in os.listdir(state_path):
county_path = os.path.join(state_path, county_dir)
if os.path.isdir(county_path):
csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
for file in csv_files:
file_path = os.path.join(county_path, file)
with open(file_path, 'r') as infile:
reader = csv.DictReader(infile)
for row in reader:
data = {header: row.get(header, '').strip() for header in HEADERS}
writer.writerow(data)
spinner.succeed(f'Finished processing {state_dir}')
def main():
figlet = Figlet(font='slant')
script_name = "prepare-stage-6".replace("-", " ").title()
print(figlet.renderText(script_name))
print("Preparing Stage 6...")
create_stage_6_directory()
print("Merging CSV files to Stage 6...")
spinner = Halo(text='Merging data', spinner='dots')
spinner.start()
merge_csv_files()
spinner.succeed("Data merged.")
print("Stage 6 preparation complete.")
if __name__ == "__main__":
main()

Some files were not shown because too many files have changed in this diff Show more