Lord_Devi 1056a09945 Initial commit.

2024-07-16 18:08:36 -04:00

19 KiB

Raw Permalink Blame History

Prompt 1

- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts. - [Project Root]: "/home/ld/mgk-scrapes" - [Current Dataset]: "[Project Root]/current-data/" - [Data Directory]: "[Current Dataset]/.data/" - [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".") - [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.) - [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state. - [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under. - [Stage 1]: "[Data Directory]/stage-1/" - [Stage 2]: "[Data Directory]/stage-2/" - [Stage 3]: "[Data Directory]/stage-3/" - [Stage 4]: "[Data Directory]/stage-4/" - [Stage 5]: "[Data Directory]/stage-5/" - [Stage 6]: "[Data Directory]/stage-6/" - [Binaries]: "[Project Root]/bin/" - [Stage 1 Binaries]: "[Binaries]/stage-1/" - [Stage 2 Binaries]: "[Binaries]/stage-2/" - [Stage 3 Binaries]: "[Binaries]/stage-3/" - [Stage 4 Binaries]: "[Binaries]/stage-4/" - [Stage 5 Binaries]: "[Binaries]/stage-5/" - [Stage 6 Binaries]: "[Binaries]/stage-6/" - [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt" - [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt" - [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt" - [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt" - [Rule 1]: All scripts need to be able to be ran from any directory. - [Rule 2]: All scripts need output what they are doing, as they are doing it. - [Rule 3]: All tasks should use halo to report successes and failures. - [Rule 4]: The python module "tqdm" can be used to report progress when appropriate. - [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for. - [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python". - [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ". - [Rule 8]: Scripts should generally have robust error checking. - [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.

Please refactor the process-stage-3 script below to be in accordance with the above.

This script will not need to check for the existence of Stage 3, or worry about creating it. This task is handled elsewhere now.

Please create the process-stage-3 in the same style as the process-stage-2 script below:

#!/usr/bin/env python # Script Name: process-stage-2

import os import subprocess from pyfiglet import Figlet from halo import Halo

Define the paths based on the project details

PROJECT_ROOT = "/home/ld/mgk-scrapes" BIN_STAGE_2_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-2') BIN_DIR = os.path.join(PROJECT_ROOT, 'bin')

def run_scripts(): """Run the specified scripts in order.""" scripts = [ os.path.join(BIN_STAGE_2_DIR, "prepare-stage-2.py"), os.path.join(BIN_STAGE_2_DIR, "remove-utm.py"), os.path.join(BIN_STAGE_2_DIR, "sanitize-review-count.py"), os.path.join(BIN_STAGE_2_DIR, "sanitize-phone-data.py"), os.path.join(BIN_STAGE_2_DIR, "sanitize-gbp-business-website.py"), os.path.join(BIN_DIR, "remove-empty-columns.py") ]

# Run scripts that do not require arguments
for script in scripts[:-1]:
    subprocess.run(["python", script], check=True)

# Run script that requires the 'stage-2' argument
subprocess.run(["python", scripts[-1], "stage-2"], check=True)

def main(): figlet = Figlet(font='slant') script_name = "process-stage-2".replace("-", " ").title() print(figlet.renderText(script_name))

print("Running scripts for Stage 2...\n")
run_scripts()

final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")

if name == 'main': main() </process-stage-2 script>

Here is the current version of the process-stage-3 script which needs refactoring:

#!/usr/bin/env python

import os import shutil import subprocess from tqdm import tqdm from pyfiglet import Figlet

Get the project root directory

project_root = os.path.abspath(os.path.join(os.path.dirname(file), '..', 'Concrete Sealing Company')) data_dir = os.path.join(project_root, '.data') stage2_dir = os.path.join(data_dir, 'stage-2') stage3_dir = os.path.join(data_dir, 'stage-3')

def check_and_prepare_stage3(): # Check if the stage-3 directory exists if os.path.exists(stage3_dir): # Remove the existing stage-3 directory shutil.rmtree(stage3_dir)

# Copy stage-2 directory to stage-3
shutil.copytree(stage2_dir, stage3_dir)

def run_scripts(): scripts = [ "column-search_review-count.py", "column-search_review-rating.py", "column-search_gbp-location.py", "column-search_yib.py", "column-search_gbp-business-category.py", "column-search_gbp-matching-services.py", "column-search_business-name.py", ("remove-empty-columns.py", "stage-3"), "column-search_unknowns.py" ]

for script in scripts:
    if isinstance(script, tuple):
        script_name, arg = script
        subprocess.run(["python", script_name, arg], check=True)
    else:
        subprocess.run(["python", script], check=True)

# Call report-unknowns.py with the "stage-3" option
subprocess.run(["python", "report-unknowns.py", "stage-3"], check=True)

if name == 'main': check_and_prepare_stage3() run_scripts() </process-stage-3 script>

Item 1: The process-stage-3 script will need to call a scrip named "prepare-stage-3" script as the first script it calls.

Item 2: The following scripts which process-stage-3 calls are located in [Stage 3 Binaries]:

<stage 3 scripts> column-search_review-count.py column-search_review-rating.py column-search_gbp-location.py column-search_yib.py column-search_gbp-business-category.py column-search_gbp-matching-services.py column-search_business-name.py column-search_unknowns.py </stage 3 scripts>

The following script is located in [Binaries].

remove-empty-columns.py

Prompt 2

I ran into a problem from the 'process-stage-3' script:

#!/usr/bin/env python # Script Name: process-stage-3

import os import subprocess from pyfiglet import Figlet from halo import Halo

Define the paths based on the project details

PROJECT_ROOT = "/home/ld/mgk-scrapes" BIN_STAGE_3_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-3') BIN_DIR = os.path.join(PROJECT_ROOT, 'bin')

def run_scripts(): """Run the specified scripts in order.""" scripts = [ os.path.join(BIN_STAGE_3_DIR, "prepare-stage-3.py"), os.path.join(BIN_STAGE_3_DIR, "column-search_review-count.py"), os.path.join(BIN_STAGE_3_DIR, "column-search_review-rating.py"), os.path.join(BIN_STAGE_3_DIR, "column-search_gbp-location.py"), os.path.join(BIN_STAGE_3_DIR, "column-search_yib.py"), os.path.join(BIN_STAGE_3_DIR, "column-search_gbp-business-category.py"), os.path.join(BIN_STAGE_3_DIR, "column-search_gbp-matching-services.py"), os.path.join(BIN_STAGE_3_DIR, "column-search_business-name.py"), os.path.join(BIN_DIR, "remove-empty-columns.py"), os.path.join(BIN_STAGE_3_DIR, "column-search_unknowns.py"), os.path.join(BIN_DIR, "report-unknowns.py") ]

# Run scripts that do not require arguments
for script in scripts[:-2]:
    subprocess.run(["python", script], check=True)

# Run script that requires the 'stage-3' argument
subprocess.run(["python", scripts[-2], "stage-3"], check=True)
subprocess.run(["python", scripts[-1], "stage-3"], check=True)

def main(): figlet = Figlet(font='slant') script_name = "process-stage-3".replace("-", " ").title() print(figlet.renderText(script_name))

print("Running scripts for Stage 3...\n")
run_scripts()

final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")

if name == 'main': main() </process-stage-3 script>

When it called the 'remove-empty-columns' script:

#!/usr/bin/env python # Script Name: remove-empty-columns

import os import csv import sys from pyfiglet import Figlet import halo

Define the paths based on the project details

PROJECT_ROOT = "/home/ld/mgk-scrapes" CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data") DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data")

def print_help_message(): """Prints the help message for script usage.""" print("Usage: remove-empty-columns.py <stage-#>") print("Example: remove-empty-columns.py stage-1") print("The provided stage directory must exist within the .data directory of the project root.")

def delete_empty_columns(file_path): """Deletes empty columns from a CSV file.""" with open(file_path, 'r') as file: reader = csv.reader(file) rows = list(reader)

if not rows:
    return 0

headers = rows[0]
data_rows = rows[1:]

# Ensure each row has the same length as the headers
for row in data_rows:
    while len(row) < len(headers):
        row.append('')

# Identify non-empty columns
non_empty_columns = [i for i in range(len(headers)) if any(row[i].strip() for row in data_rows)]
empty_column_count = len(headers) - len(non_empty_columns)

# Create a new list to store the modified rows without empty columns
modified_rows = [[headers[i] for i in non_empty_columns]]
for row in data_rows:
    modified_rows.append([row[i] for i in non_empty_columns])

# Write the modified rows back to the CSV file
with open(file_path, 'w', newline='') as file:
    writer = csv.writer(file)
    writer.writerows(modified_rows)

return empty_column_count

def remove_empty_columns(stage_directory): """Remove empty columns from all CSV files in the given stage directory.""" for state_dir in os.listdir(stage_directory): state_path = os.path.join(stage_directory, state_dir) if os.path.isdir(state_path): print(f"Searching for empty columns to delete in {state_dir}...") spinner = halo.Halo(spinner='dots', color='green') spinner.start()

        total_removed_columns = 0
        for county_dir in os.listdir(state_path):
            county_path = os.path.join(state_path, county_dir)
            if os.path.isdir(county_path):
                csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]
                for file in csv_files:
                    file_path = os.path.join(county_path, file)
                    total_removed_columns += delete_empty_columns(file_path)

        spinner.succeed(f"Removed columns from {state_dir}. Columns removed: {total_removed_columns}")

def main(): figlet = Figlet(font='slant') script_name = "remove-empty-columns".replace("-", " ").title() print(figlet.renderText(script_name))

if len(sys.argv) != 2:
    print("Error: Incorrect number of arguments.")
    print_help_message()
    sys.exit(1)

stage_arg = sys.argv[1]
if not stage_arg.startswith('stage-'):
    print("Error: Invalid stage directory format.")
    print_help_message()
    sys.exit(1)

stage_dir = os.path.join(DATA_DIRECTORY, stage_arg)
if not os.path.exists(stage_dir):
    print(f"Error: The directory '{stage_dir}' does not exist.")
    sys.exit(1)

print(f"Removing empty columns in {stage_arg}...")
remove_empty_columns(stage_dir)

final_spinner = halo.Halo(spinner='dots', color='green')
final_spinner.succeed("Finished removing empty columns.")

if name == "main": main() </remove-empty-columns script>

The problem was an error I got:

Error: Incorrect number of arguments. Usage: remove-empty-columns.py <stage-#> Example: remove-empty-columns.py stage-1 The provided stage directory must exist within the .data directory of the project root. Traceback (most recent call last): File "/home/ld/mgk-scrapes/bin/./process-stage-3.py", line 51, in main() File "/home/ld/mgk-scrapes/bin/./process-stage-3.py", line 44, in main run_scripts() File "/home/ld/mgk-scrapes/bin/./process-stage-3.py", line 32, in run_scripts subprocess.run(["python", script], check=True) File "/usr/lib64/python3.12/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['python', '/home/ld/mgk-scrapes/bin/remove-empty-columns.py']' returned non-zero exit status 1.

If I run 'remove-empty-columns' manually such as :

./remove-empty-columns stage3

It runs correctly.

Prompt 3

I got the following error:

Error: Incorrect number of arguments. Usage: remove-empty-columns.py Example: remove-empty-columns.py stage-1 The provided stage directory must exist within the .data directory of the project root. Traceback (most recent call last): File "/home/ld/mgk-scrapes/bin/./process-stage-3.py", line 52, in main() File "/home/ld/mgk-scrapes/bin/./process-stage-3.py", line 45, in main run_scripts() File "/home/ld/mgk-scrapes/bin/./process-stage-3.py", line 32, in run_scripts subprocess.run(["python", script], check=True) File "/usr/lib64/python3.12/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['python', '/home/ld/mgk-scrapes/bin/remove-empty-columns.py']' returned non-zero exit status 1.

Are you sure the script is trying to run the command "remove-empty-columns.py stage-3"?

Here is the working copy of the script:

19 KiB Raw Permalink Blame History

Prompt 1

Define the paths based on the project details

Get the project root directory

Prompt 2

Define the paths based on the project details

Define the paths based on the project details

Prompt 3

19 KiB

Raw Permalink Blame History