Lord_Devi 1056a09945 Initial commit.

2024-07-16 18:08:36 -04:00

16 KiB

Raw Blame History

Prompt 1

- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts. - [Project Root]: "/home/ld/mgk-scrapes" - [Current Dataset]: "[Project Root]/current-data/" - [Data Directory]: "[Current Dataset]/.data/" - [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".") - [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.) - [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state. - [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under. - [Stage 1]: "[Data Directory]/stage-1/" - [Stage 2]: "[Data Directory]/stage-2/" - [Stage 3]: "[Data Directory]/stage-3/" - [Stage 4]: "[Data Directory]/stage-4/" - [Stage 5]: "[Data Directory]/stage-5/" - [Stage 6]: "[Data Directory]/stage-6/" - [Binaries]: "[Project Root]/bin/" - [Stage 1 Binaries]: "[Binaries]/stage-1/" - [Stage 2 Binaries]: "[Binaries]/stage-2/" - [Stage 3 Binaries]: "[Binaries]/stage-3/" - [Stage 4 Binaries]: "[Binaries]/stage-4/" - [Stage 5 Binaries]: "[Binaries]/stage-5/" - [Stage 6 Binaries]: "[Binaries]/stage-6/" - [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt" - [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt" - [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt" - [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt" - [Rule 1]: All scripts need to be able to be ran from any directory. - [Rule 2]: All scripts need output what they are doing, as they are doing it. - [Rule 3]: All tasks should use halo to report successes and failures. - [Rule 4]: The python module "tqdm" can be used to report progress when appropriate. - [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for. - [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python". - [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ". - [Rule 8]: Scripts should generally have robust error checking. - [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.

Please be sure to reference project details above when factoring code, or answering questions.

I have a script called 'process-stage-2' which is similar to the 'process-stage-1' script we worked on earlier.

I need this process-stage-2 script to be refactored to be in direct accordance with the above.

Here is the process-stage-2 script:

#!/usr/bin/env python

import os import shutil import subprocess from tqdm import tqdm from pyfiglet import Figlet

Get the project root directory

project_root = os.path.abspath(os.path.join(os.path.dirname(file), '..', 'Concrete Sealing Company')) stage1_dir = os.path.join(project_root, '.data', 'stage-1') stage2_dir = os.path.join(project_root, '.data', 'stage-2')

def check_and_prepare_stage2(): # Check if the stage-2 directory exists if os.path.exists(stage2_dir): # Remove the existing stage-2 directory shutil.rmtree(stage2_dir)

# Copy stage-1 directory to stage-2
shutil.copytree(stage1_dir, stage2_dir)

def run_scripts(): scripts = [ "remove-utm.py", "sanitize-review-count.py", "sanitize-phone-data.py", "sanitize-gbp-business-website.py", ("remove-empty-columns.py", "stage-2") ]

for script in scripts:
    if isinstance(script, tuple):
        script_name, arg = script
        subprocess.run(["python", script_name, arg], check=True)
    else:
        subprocess.run(["python", script], check=True)

if name == 'main': check_and_prepare_stage2() run_scripts()

I need this script to be refactored to be more like our 'process-stage-1' script in terms of style and reporting.

Here ist he process-stage-1 script I am referencing:

#!/usr/bin/env python # Script Name: process-stage-1

import os import shutil import subprocess

Define the paths based on the project details

PROJECT_ROOT = "/home/ld/mgk-scrapes" CURRENT_DATA_DIR = os.path.join(PROJECT_ROOT, 'current-data') STAGE_1_DIR = os.path.join(CURRENT_DATA_DIR, '.data', 'stage-1') BIN_STAGE_1_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-1')

def prepare_stage_1(): """Prepare the stage-1 directory by creating it and copying state directories into it.""" # Check if the stage directory exists if os.path.exists(STAGE_1_DIR): # Remove the existing stage directory shutil.rmtree(STAGE_1_DIR)

# Create the stage directory
os.makedirs(STAGE_1_DIR)

# Copy all state directories to the stage directory
for state_dir in os.listdir(CURRENT_DATA_DIR):
    state_path = os.path.join(CURRENT_DATA_DIR, state_dir)
    if os.path.isdir(state_path) and not state_dir.startswith('.'):
        stage_state_path = os.path.join(STAGE_1_DIR, state_dir)
        shutil.copytree(state_path, stage_state_path)

def run_scripts(): """Run the specified scripts in order.""" scripts = [ os.path.join(BIN_STAGE_1_DIR, "prepare-stage-1.py"), os.path.join(PROJECT_ROOT, "bin", "delete-empty-counties.py"), os.path.join(PROJECT_ROOT, "bin", "column-count-correction.py"), os.path.join(BIN_STAGE_1_DIR, "remove-obviously-bad-columns.py"), os.path.join(PROJECT_ROOT, "bin", "remove-empty-columns.py"), os.path.join(BIN_STAGE_1_DIR, "delete-malformed-csvs.py"), os.path.join(PROJECT_ROOT, "bin", "delete-empty-counties.py"), os.path.join(PROJECT_ROOT, "bin", "column-count-correction.py") ]

# Run scripts that do not require arguments
subprocess.run(["python", scripts[0]], check=True)
subprocess.run(["python", scripts[3]], check=True)
subprocess.run(["python", scripts[5]], check=True)

# Run scripts that require the 'stage-1' argument
subprocess.run(["python", scripts[1], "stage-1"], check=True)
subprocess.run(["python", scripts[2], "stage-1"], check=True)
subprocess.run(["python", scripts[4], "stage-1"], check=True)
subprocess.run(["python", scripts[6], "stage-1"], check=True)
subprocess.run(["python", scripts[7], "stage-1"], check=True)

if name == 'main': prepare_stage_1() run_scripts()

Currently the process-stage-2 script is running the following scripts (and paremeters):

remove-utm.py sanitize-review-count.py sanitize-phone-data.py sanitize-gbp-business-website.py remove-empty-columns.py stage-2

This list of will need slight adjustment.

The remove-utm.py, sanitize-review-count.py, sanitize-phone-data.py, and sanitize-gbp-business-website.py scripts will be located in [Stage 2 Binaries] (as per <Project Details).

While the remove-empty-columns.py script is located in the standard [Binaries] location.

Lastly, we need to add a script to this list called 'prepare-stage-2'. This script is intended to fullfill the part of the 'process-stage-2' script which is normally responsible for ensuring a new copy of Stage 2 in place before the script continues.

For an idea of how this works, see the script above for how it calls its own version of 'prepare-stage-2', called 'prepare-stage-1'.

Here is the prepare-stage-2 script below to give you a reference point:

#!/usr/bin/env python # Script Name: prepare-stage-1

import os import shutil import halo from tqdm import tqdm from pyfiglet import Figlet

Define the paths based on the project details

PROJECT_ROOT = "/home/ld/mgk-scrapes" CURRENT_DATASET = os.path.join(PROJECT_ROOT, "current-data") DATA_DIRECTORY = os.path.join(CURRENT_DATASET, ".data") STAGE_1_DIRECTORY = os.path.join(DATA_DIRECTORY, "stage-1") PARENT_DATA_DIRECTORY = CURRENT_DATASET

def create_stage_1_directory(): """Create the stage-1 directory, replacing it if it already exists.""" if os.path.exists(STAGE_1_DIRECTORY): print("Existing Stage 1 found, deleting and replacing...") shutil.rmtree(STAGE_1_DIRECTORY) os.makedirs(STAGE_1_DIRECTORY) print("Stage 1 directory created.")

def find_parent_data(): """Find all parent data directories.""" parent_data_dirs = [ d for d in os.listdir(PARENT_DATA_DIRECTORY) if os.path.isdir(os.path.join(PARENT_DATA_DIRECTORY, d)) and not d.startswith('.') ] return parent_data_dirs

def copy_data_to_stage_1(parent_data_dirs): """Copy data from parent data directories to stage-1.""" for directory in parent_data_dirs: src_dir = os.path.join(PARENT_DATA_DIRECTORY, directory) dest_dir = os.path.join(STAGE_1_DIRECTORY, directory) shutil.copytree(src_dir, dest_dir)

def verify_stage_1_data(parent_data_dirs): """Verify that the data in stage-1 matches the parent data.""" for directory in parent_data_dirs: src_dir = os.path.join(PARENT_DATA_DIRECTORY, directory) dest_dir = os.path.join(STAGE_1_DIRECTORY, directory)

    for root, dirs, files in os.walk(src_dir):
        for file in files:
            src_file = os.path.join(root, file)
            dest_file = src_file.replace(PARENT_DATA_DIRECTORY, STAGE_1_DIRECTORY)
            if not os.path.exists(dest_file):
                return False
return True

def main(): figlet = Figlet(font='slant') script_name = "prepare-stage-1".replace("-", " ").title() print(figlet.renderText(script_name))

print("Preparing Stage 1...")

create_stage_1_directory()

parent_data_dirs = find_parent_data()

if parent_data_dirs:
    print("Found data for the following states:")
    print(", ".join(parent_data_dirs))
    
    print("Copying data to Stage 1...")
    spinner = halo.Halo(text='Copying data', spinner='dots')
    spinner.start()
    copy_data_to_stage_1(parent_data_dirs)
    spinner.succeed("Data copied.")
    
    print("Verifying Stage 1 data...")
    is_valid = verify_stage_1_data(parent_data_dirs)
    
    if is_valid:
        print("Data verification successful. Stage 1 data is valid.")
    else:
        print("Data verification failed. Stage 1 data is not valid.")
else:
    print("No parent data found.")

if name == "main": main()

Now please refactor the 'process-stage-2' script in the ways perscribed.

Prompt 2

I need to adjust the 'process-stage-2' script.

The text "Running scripts" does not need a spinnder, and it should have a new line after it.

It is causing the initial output of each script that is ran to appear on the same line as the "Running scripts" text and spinner. Which does not look good.

#!/usr/bin/env python # Script Name: process-stage-2

import os import subprocess from pyfiglet import Figlet import halo

Define the paths based on the project details

PROJECT_ROOT = "/home/ld/mgk-scrapes" BIN_STAGE_2_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-2') BIN_DIR = os.path.join(PROJECT_ROOT, 'bin')

def run_scripts(): """Run the specified scripts in order.""" scripts = [ os.path.join(BIN_STAGE_2_DIR, "prepare-stage-2.py"), os.path.join(BIN_STAGE_2_DIR, "remove-utm.py"), os.path.join(BIN_STAGE_2_DIR, "sanitize-review-count.py"), os.path.join(BIN_STAGE_2_DIR, "sanitize-phone-data.py"), os.path.join(BIN_STAGE_2_DIR, "sanitize-gbp-business-website.py"), os.path.join(BIN_DIR, "remove-empty-columns.py") ]

# Run scripts that do not require arguments
for script in scripts[:-1]:
    subprocess.run(["python", script], check=True)

# Run script that requires the 'stage-2' argument
subprocess.run(["python", scripts[-1], "stage-2"], check=True)

def main(): figlet = Figlet(font='slant') script_name = "process-stage-2".replace("-", " ").title() print(figlet.renderText(script_name))

print("Running scripts for Stage 2...")
spinner = halo.Halo(text='Running scripts', spinner='dots')
spinner.start()
run_scripts()
spinner.succeed("All scripts completed.")

if name == 'main': main() </process-stage-2 script>

16 KiB Raw Blame History

Prompt 1

Get the project root directory

Define the paths based on the project details

Define the paths based on the project details

Prompt 2

Define the paths based on the project details

16 KiB

Raw Blame History