Prompt 1

- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts. - [Project Root]: "/home/ld/mgk-scrapes" - [Current Dataset]: "[Project Root]/current-data/" - [Data Directory]: "[Current Dataset]/.data/" - [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".") - [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.) - [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state. - [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under. - [Stage 1]: "[Data Directory]/stage-1/" - [Stage 2]: "[Data Directory]/stage-2/" - [Stage 3]: "[Data Directory]/stage-3/" - [Stage 4]: "[Data Directory]/stage-4/" - [Stage 5]: "[Data Directory]/stage-5/" - [Stage 6]: "[Data Directory]/stage-6/" - [Binaries]: "[Project Root]/bin/" - [Stage 1 Binaries]: "[Binaries]/stage-1/" - [Stage 2 Binaries]: "[Binaries]/stage-2/" - [Stage 3 Binaries]: "[Binaries]/stage-3/" - [Stage 4 Binaries]: "[Binaries]/stage-4/" - [Stage 5 Binaries]: "[Binaries]/stage-5/" - [Stage 6 Binaries]: "[Binaries]/stage-6/" - [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt" - [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt" - [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt" - [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt" - [Rule 1]: All scripts need to be able to be ran from any directory. - [Rule 2]: All scripts need output what they are doing, as they are doing it. - [Rule 3]: All tasks should use halo to report successes and failures. - [Rule 4]: The python module "tqdm" can be used to report progress when appropriate. - [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for. - [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python". - [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ". - [Rule 8]: Scripts should generally have robust error checking. - [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.

Please be sure to reference project details above when factoring code, or answering questions.

I wish to refactor the script 'strip-neg-terms-from-unknown-cols' to be in accordance with the above, and to adjust the reporting style now:

#!/usr/bin/env python

import os import csv from tqdm import tqdm from pyfiglet import Figlet

Get the project root directory

project_root = os.path.abspath(os.path.join(os.path.dirname(file), '..', 'Concrete Sealing Company')) stage4_dir = os.path.join(project_root, '.data', 'stage-4') data_dir = os.path.join(project_root, '.data') blacklist_file = os.path.join(data_dir, 'unknown-blacklist.txt')

def process_csv_file(file_path, blacklist): # Read the CSV file with open(file_path, 'r') as file: reader = csv.reader(file) rows = list(reader)

# Get the headers and data rows
headers = rows[0]
data_rows = rows[1:]

# Create a new list to store the modified rows
modified_rows = [headers]

# Iterate through the data rows and remove blacklist terms from "unknown-#" columns
for row in data_rows:
    for i, header in enumerate(headers):
        if header.startswith("unknown-") and row[i] in blacklist:
            row[i] = ''  # Clear the cell if it contains a blacklist term
    modified_rows.append(row)

# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
    writer = csv.writer(file)
    writer.writerows(modified_rows)

def strip_negative_terms(): # Read the blacklist terms if not os.path.isfile(blacklist_file): print("Error: The unknown-blacklist.txt file is missing.") return

with open(blacklist_file, 'r') as file:
    blacklist = [line.strip() for line in file if line.strip()]

# Process each CSV file in the stage-4 directory
for state_dir in os.listdir(stage4_dir):
    state_path = os.path.join(stage4_dir, state_dir)
    if os.path.isdir(state_path):
        # Print the state name in big print using Figlet
        figlet = Figlet(font='slant')
        print(figlet.renderText(state_dir))

        for county_dir in os.listdir(state_path):
            county_path = os.path.join(state_path, county_dir)
            if os.path.isdir(county_path):
                csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
                progress_bar = tqdm(csv_files, desc=f"Stripping neg terms from unknown columns in: {state_dir}, {county_dir}",
                                    unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
                for file in progress_bar:
                    file_path = os.path.join(county_path, file)
                    process_csv_file(file_path, blacklist)
                    progress_bar.update(1)

if name == 'main': strip_negative_terms() </strip-neg-terms-from-unknown-cols script>

Please make this script have the same kind of by-state reporting as the 'remove-utm' script:

#!/usr/bin/env python # Script Name: remove-utm

import os import csv import sys from pyfiglet import Figlet from halo import Halo

Define the paths based on the project details

PROJECT_ROOT = "/home/ld/mgk-scrapes" CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data") DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data") STAGE_2_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-2")

def process_csv_file_remove_utm(file_path): """Remove UTM parameters from URLs in the given CSV file.""" with open(file_path, 'r') as file: reader = csv.reader(file) rows = list(reader)

if not rows:
    return

modified_rows = []
for row in rows:
    modified_row = []
    for cell in row:
        if "?utm" in cell:
            modified_cell = cell.split("?utm")[0]
        else:
            modified_cell = cell
        modified_row.append(modified_cell)
    modified_rows.append(modified_row)

with open(file_path, 'w', newline='') as file:
    writer = csv.writer(file)
    writer.writerows(modified_rows)

def process_csv_files(stage_directory): """Process all CSV files in the given stage directory to remove UTM parameters from URLs.""" for state_dir in os.listdir(stage_directory): state_path = os.path.join(stage_directory, state_dir) if os.path.isdir(state_path): spinner = Halo(text=f'Processing {state_dir}', spinner='dots') spinner.start() for county_dir in os.listdir(state_path): county_path = os.path.join(state_path, county_dir) if os.path.isdir(county_path): csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')] for file in csv_files: file_path = os.path.join(county_path, file) process_csv_file_remove_utm(file_path) spinner.succeed(f'Finished processing {state_dir}')

def main(): figlet = Figlet(font='slant') script_name = "remove-utm".replace("-", " ").title() print(figlet.renderText(script_name))

if not os.path.exists(STAGE_2_DIRECTORY):
    print(f"Error: The directory {STAGE_2_DIRECTORY} does not exist.")
    sys.exit(1)

print(f"Removing UTM parameters in {STAGE_2_DIRECTORY}...")
process_csv_files(STAGE_2_DIRECTORY)

final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("UTM parameters removed.")

if name == "main": main() </remove-utm script>

Prompt 2

I wish to adjust the 'strip-neg-terms-from-unknown-cols' script:

This script needs to have the matching it performs be done in a case insensitive manner. Capitalization needs to be ignored.

14 KiB Raw Blame History

Prompt 1

Get the project root directory

Define the paths based on the project details

Prompt 2

14 KiB

Raw Blame History