Lord_Devi 1056a09945 Initial commit.

2024-07-16 18:08:36 -04:00

49 KiB

Raw Blame History

Prompt 1

- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts. - [Project Root]: "/home/ld/mgk-scrapes" - [Current Dataset]: "[Project Root]/current-data/" - [Data Directory]: "[Current Dataset]/.data/" - [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".") - [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.) - [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state. - [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under. - [Stage 1]: "[Data Directory]/stage-1/" - [Stage 2]: "[Data Directory]/stage-2/" - [Stage 3]: "[Data Directory]/stage-3/" - [Stage 4]: "[Data Directory]/stage-4/" - [Stage 5]: "[Data Directory]/stage-5/" - [Stage 6]: "[Data Directory]/stage-6/" - [Binaries]: "[Project Root]/bin/" - [Stage 1 Binaries]: "[Binaries]/stage-1/" - [Stage 2 Binaries]: "[Binaries]/stage-2/" - [Stage 3 Binaries]: "[Binaries]/stage-3/" - [Stage 4 Binaries]: "[Binaries]/stage-4/" - [Stage 5 Binaries]: "[Binaries]/stage-5/" - [Stage 6 Binaries]: "[Binaries]/stage-6/" - [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt" - [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt" - [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt" - [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt" - [Rule 1]: All scripts need to be able to be ran from any directory. - [Rule 2]: All scripts need output what they are doing, as they are doing it. - [Rule 3]: All tasks should use halo to report successes and failures. - [Rule 4]: The python module "tqdm" can be used to report progress when appropriate. - [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for. - [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python". - [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ". - [Rule 8]: Scripts should generally have robust error checking. - [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.

Please refactor the process-stage-6 script below to be in accordance with the above.

This script will not need to check for the existence of Stage 6, or worry about creating it. This task is handled elsewhere now. Remove any function that checks for Stage 6, creates it or recreates it.

Please create the process-stage-6 script in the same style as the process-stage-2 script below:

#!/usr/bin/env python # Script Name: process-stage-2

import os import subprocess from pyfiglet import Figlet from halo import Halo

Define the paths based on the project details

PROJECT_ROOT = "/home/ld/mgk-scrapes" BIN_STAGE_2_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-2') BIN_DIR = os.path.join(PROJECT_ROOT, 'bin')

def run_scripts(): """Run the specified scripts in order.""" scripts = [ os.path.join(BIN_STAGE_2_DIR, "prepare-stage-2.py"), os.path.join(BIN_STAGE_2_DIR, "remove-utm.py"), os.path.join(BIN_STAGE_2_DIR, "sanitize-review-count.py"), os.path.join(BIN_STAGE_2_DIR, "sanitize-phone-data.py"), os.path.join(BIN_STAGE_2_DIR, "sanitize-gbp-business-website.py"), os.path.join(BIN_DIR, "remove-empty-columns.py") ]

# Run scripts that do not require arguments
for script in scripts[:-1]:
    subprocess.run(["python", script], check=True)

# Run script that requires the 'stage-2' argument
subprocess.run(["python", scripts[-1], "stage-2"], check=True)

def main(): figlet = Figlet(font='slant') script_name = "process-stage-2".replace("-", " ").title() print(figlet.renderText(script_name))

print("Running scripts for Stage 2...\n")
run_scripts()

final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")

if name == 'main': main() </process-stage-2 script>

Here is the current version of the process-stage-6 script which needs refactoring:

#!/usr/bin/env python

import os import shutil import csv import subprocess from tqdm import tqdm from pyfiglet import Figlet

Get the project root directory

project_root = os.path.abspath(os.path.join(os.path.dirname(file), '..', 'Concrete Sealing Company')) stage5_dir = os.path.join(project_root, '.data', 'stage-5') stage6_dir = os.path.join(project_root, '.data', 'stage-6')

def prepare_stage6_directory(): # Check if the stage-6 directory exists if os.path.exists(stage6_dir): # Remove the existing stage-6 directory shutil.rmtree(stage6_dir)

# Create the stage-6 directory
os.makedirs(stage6_dir)

def merge_csv_files(): merged_csv_path = os.path.join(stage6_dir, '01-first-merger.csv') headers_written = False

with open(merged_csv_path, 'w', newline='') as merged_csv_file:
    writer = csv.writer(merged_csv_file)

    for state_dir in os.listdir(stage5_dir):
        state_path = os.path.join(stage5_dir, state_dir)
        if os.path.isdir(state_path):
            for county_dir in os.listdir(state_path):
                county_path = os.path.join(state_path, county_dir)
                if os.path.isdir(county_path):
                    csv_files = [file for file in os.listdir(county_path) if file.endswith('.csv')]

                    progress_bar = tqdm(csv_files, desc=f"Merging CSV files from: {state_dir}, {county_dir}",
                                        unit="file", bar_format="{l_bar}{bar:10}| {n_fmt}/{total_fmt}")
                    for file in progress_bar:
                        file_path = os.path.join(county_path, file)
                        with open(file_path, 'r') as csv_file:
                            reader = csv.reader(csv_file)
                            headers = next(reader)

                            if not headers_written:
                                writer.writerow(headers)
                                headers_written = True
                            
                            for row in reader:
                                writer.writerow(row)
                        
                        progress_bar.update(1)

def run_scripts(): # Run trim-invalid-rows-from-merger.py script subprocess.run(["python", "trim-invalid-rows-from-merger.py"], check=True)

# Run initial-deduplication.py script
subprocess.run(["python", "initial-deduplication.py"], check=True)

# Run delete-bad-matching-services.py script
subprocess.run(["python", "delete-bad-matching-services.py"], check=True)

# Run find-unknown-categories-and-services.py script
subprocess.run(["python", "find-unknown-categories-and-services.py"], check=True)

# Run standardize-capital-letters.py script
subprocess.run(["python", "standardize-capital-letters.py"], check=True)

# Run tally-initial-deduplication.py script
subprocess.run(["python", "tally-initial-deduplication.py"], check=True)

if name == 'main': # Prepare stage-6 directory prepare_stage6_directory()

# Merge CSV files from stage-5
merge_csv_files()

# Run the necessary scripts
run_scripts()

</process-stage-6 script>

Item 1: The process-stage-6 script will need to call a script named "prepare-stage-6" script as the first script it calls. This script will need to be ran first by process-stage-6, and will be located in [Stage 6 Binaries]

Item 2: The following scripts which process-stage-6 calls are located in [Stage 6 Binaries]:

<stage 6 scripts> trim-invalid-rows-from-merger.py delete-bad-matching-services.py standardize-capital-letters.py find-unknown-categories-and-services.py initial-deduplication.py </stage 6 scripts>

Prompt 2

I wish to adjust the 'process-stage-6' script:

Change 1: I it to now run the following scripts, and in the order I present them. These will all be located in [Stage 6 Binaries].

prepare-stage-6.py pre-sort.py trim-invalid-rows-from-merger.py delete-bad-matching-services.py standardize-capital-letters.py

Prompt 3

- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts. - [Project Root]: "/home/ld/mgk-scrapes" - [Current Dataset]: "[Project Root]/current-data/" - [Data Directory]: "[Current Dataset]/.data/" - [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".") - [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.) - [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state. - [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under. - [Stage 1]: "[Data Directory]/stage-1/" - [Stage 2]: "[Data Directory]/stage-2/" - [Stage 3]: "[Data Directory]/stage-3/" - [Stage 4]: "[Data Directory]/stage-4/" - [Stage 5]: "[Data Directory]/stage-5/" - [Stage 6]: "[Data Directory]/stage-6/" - [Binaries]: "[Project Root]/bin/" - [Stage 1 Binaries]: "[Binaries]/stage-1/" - [Stage 2 Binaries]: "[Binaries]/stage-2/" - [Stage 3 Binaries]: "[Binaries]/stage-3/" - [Stage 4 Binaries]: "[Binaries]/stage-4/" - [Stage 5 Binaries]: "[Binaries]/stage-5/" - [Stage 6 Binaries]: "[Binaries]/stage-6/" - [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt" - [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt" - [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt" - [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt" - [Rule 1]: All scripts need to be able to be ran from any directory. - [Rule 2]: All scripts need output llwhat they are doing, as they are doing it. - [Rule 3]: All tasks should use halo to report successes and failures. - [Rule 4]: The python module "tqdm" can be used to report progress when appropriate. - [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for. - [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python". - [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ". - [Rule 8]: Scripts should generally have robust error checking. - [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.

The Project Details above contain locations for files, data, scripts, and rules to follow when creating project scripts and toolkit.

Please extend the 'process-stage-6' script to also call the script 'business-name-strip-structures.py' inside of [Stage 6 Binaries]. This new script can be added to the end of the list of existing scripts, meaning run in it last in the current last.

#!/usr/bin/env python # Script Name: process-stage-6

import os import subprocess from pyfiglet import Figlet from halo import Halo

Define the paths based on the project details

PROJECT_ROOT = "/home/ld/mgk-scrapes" BIN_STAGE_6_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-6')

# Run scripts that do not require arguments
for script in scripts:
    subprocess.run(["python", script], check=True)

def main(): figlet = Figlet(font='slant') script_name = "process-stage-6".replace("-", " ").title() print(figlet.renderText(script_name))

print("Running scripts for Stage 6...\n")
run_scripts()

final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")

if name == 'main': main()

</process-stage-6 script>

Prompt 4

- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts. - [Project Root]: "/home/ld/mgk-scrapes" - [Current Dataset]: "[Project Root]/current-data/" - [Data Directory]: "[Current Dataset]/.data/" - [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".") - [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.) - [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state. - [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under. - [Stage 1]: "[Data Directory]/stage-1/" - [Stage 2]: "[Data Directory]/stage-2/" - [Stage 3]: "[Data Directory]/stage-3/" - [Stage 4]: "[Data Directory]/stage-4/" - [Stage 5]: "[Data Directory]/stage-5/" - [Stage 6]: "[Data Directory]/stage-6/" - [Binaries]: "[Project Root]/bin/" - [Stage 1 Binaries]: "[Binaries]/stage-1/" - [Stage 2 Binaries]: "[Binaries]/stage-2/" - [Stage 3 Binaries]: "[Binaries]/stage-3/" - [Stage 4 Binaries]: "[Binaries]/stage-4/" - [Stage 5 Binaries]: "[Binaries]/stage-5/" - [Stage 6 Binaries]: "[Binaries]/stage-6/" - [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt" - [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt" - [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt" - [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt" - [Rule 1]: All scripts need to be able to be ran from any directory. - [Rule 2]: All scripts need output llwhat they are doing, as they are doing it. - [Rule 3]: All tasks should use halo to report successes and failures. - [Rule 4]: The python module "tqdm" can be used to report progress when appropriate. - [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for. - [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python". - [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ". - [Rule 8]: Scripts should generally have robust error checking. - [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.

The Project Details above contain locations for files, data, scripts, and rules to follow when creating project scripts and toolkit.

#!/usr/bin/env python # Script Name: process-stage-6

import os import subprocess from pyfiglet import Figlet from halo import Halo

Define the paths based on the project details

PROJECT_ROOT = "/home/ld/mgk-scrapes" BIN_STAGE_6_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-6')

# Run scripts that do not require arguments
for script in scripts:
    subprocess.run(["python", script], check=True)

def main(): figlet = Figlet(font='slant') script_name = "process-stage-6".replace("-", " ").title() print(figlet.renderText(script_name))

print("Running scripts for Stage 6...\n")
run_scripts()

final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")

if name == 'main': main()

</process-stage-6 script>

I got the following error running the 'process-stage-6' script:

⠙ Stripping business name structuresTraceback (most recent call last): File "/home/ld/mgk-scrapes/bin/stage-6/business-name-strip-structures.py", line 71, in main() File "/home/ld/mgk-scrapes/bin/stage-6/business-name-strip-structures.py", line 64, in main strip_structures_from_business_names() File "/home/ld/mgk-scrapes/bin/stage-6/business-name-strip-structures.py", line 35, in strip_structures_from_business_names with open(INPUT_CSV, 'r') as infile: ^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/home/ld/mgk-scrapes/current-data/.data/stage-6/06-initial-deduplication.csv' Traceback (most recent call last): File "/home/ld/mgk-scrapes/bin/./process-stage-6.py", line 41, in main() File "/home/ld/mgk-scrapes/bin/./process-stage-6.py", line 34, in main run_scripts() File "/home/ld/mgk-scrapes/bin/./process-stage-6.py", line 26, in run_scripts subprocess.run(["python", script], check=True) File "/usr/lib64/python3.12/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['python', '/home/ld/mgk-scrapes/bin/stage-6/business-name-strip-structures.py']' returned non-zero exit status 1.

Which looks like there was an error when running the script 'business-name-strip-structures.py' from the 'process-stage-6.py' script.

It could be because the 'business-name-strip-structures.py' script does not have the paths to the assets it needs to work on set explicitly or correctly.

I will provide to you two more scripts below so you can judge for yourself where the problem seems to be. I will provide the 'process-stage-6' script, as well as the 'business-name-strip-structures' script.

#!/usr/bin/env python # Script Name: process-stage-6

import os import subprocess from pyfiglet import Figlet from halo import Halo

Define the paths based on the project details

PROJECT_ROOT = "/home/ld/mgk-scrapes" BIN_STAGE_6_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-6')

# Run scripts that do not require arguments
for script in scripts:
    subprocess.run(["python", script], check=True)

def main(): figlet = Figlet(font='slant') script_name = "process-stage-6".replace("-", " ").title() print(figlet.renderText(script_name))

print("Running scripts for Stage 6...\n")
run_scripts()

final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")

if name == 'main': main() </process-stage-6 script>

#!/usr/bin/env python # Script Name: business-name-strip-structures

import os import csv import re from pyfiglet import Figlet from halo import Halo

Define the paths based on the project details

PROJECT_ROOT = "/home/ld/mgk-scrapes" STAGE_6_DIRECTORY = os.path.join(PROJECT_ROOT, 'current-data', '.data', 'stage-6') INPUT_CSV = os.path.join(STAGE_6_DIRECTORY, '06-initial-deduplication.csv') OUTPUT_CSV = os.path.join(STAGE_6_DIRECTORY, '07-business-name-structures-stripped.csv')

List of terms to remove

terms_to_remove = [ r'\bllc\b', r'\bllc.\b', r'\bl.l.c.\b', r'\bco\b', r'\bco.\b', r'\binc\b', r'\binc.\b', r'\bcorp\b', r'\bcorp.\b' ]

def clean_business_name(business_name): """Remove specified terms from the business name and clean up punctuation.""" for term in terms_to_remove: # Remove term and handle preceding punctuation and spaces business_name = re.sub(r'\s*[,.]\s' + term + r'[,.]\s', ' ', business_name, flags=re.IGNORECASE) # Remove extra spaces and trailing punctuation business_name = re.sub(r'\s+', ' ', business_name).strip() business_name = re.sub(r'[,.]\s*$', '', business_name) return business_name

def strip_structures_from_business_names(): """Strip specified structures from business names.""" with open(INPUT_CSV, 'r') as infile: reader = csv.DictReader(infile) headers = reader.fieldnames

    if 'Business Name' not in headers:
        print("Error: 'Business Name' column is missing.")
        return
    
    rows = list(reader)

with open(OUTPUT_CSV, 'w', newline='') as outfile:
    writer = csv.DictWriter(outfile, fieldnames=headers)
    writer.writeheader()
    for row in rows:
        row['Business Name'] = clean_business_name(row.get('Business Name', '').strip())
        writer.writerow(row)

def main(): figlet = Figlet(font='slant') script_name = "business-name-strip-structures".replace("-", " ").title() print(figlet.renderText(script_name))

# Check if the output file already exists and delete it if it does
if os.path.exists(OUTPUT_CSV):
    os.remove(OUTPUT_CSV)
    print(f"Existing output file {OUTPUT_CSV} found and deleted.")

spinner = Halo(text='Stripping business name structures', spinner='dots')
spinner.start()
strip_structures_from_business_names()
spinner.succeed('Business name structures stripped.')

print(f"Processed input CSV: {INPUT_CSV}")
print(f"Output CSV: {OUTPUT_CSV}")

if name == 'main': main() </business-name-strip-structures script>

Please do your best to source out the cause of this error, and then provide the what you judge to be a good solution.

Prompt 5

- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts. - [Project Root]: "/home/ld/mgk-scrapes" - [Current Dataset]: "[Project Root]/current-data/" - [Data Directory]: "[Current Dataset]/.data/" - [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".") - [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.) - [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state. - [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under. - [Stage 1]: "[Data Directory]/stage-1/" - [Stage 2]: "[Data Directory]/stage-2/" - [Stage 3]: "[Data Directory]/stage-3/" - [Stage 4]: "[Data Directory]/stage-4/" - [Stage 5]: "[Data Directory]/stage-5/" - [Stage 6]: "[Data Directory]/stage-6/" - [Binaries]: "[Project Root]/bin/" - [Stage 1 Binaries]: "[Binaries]/stage-1/" - [Stage 2 Binaries]: "[Binaries]/stage-2/" - [Stage 3 Binaries]: "[Binaries]/stage-3/" - [Stage 4 Binaries]: "[Binaries]/stage-4/" - [Stage 5 Binaries]: "[Binaries]/stage-5/" - [Stage 6 Binaries]: "[Binaries]/stage-6/" - [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt" - [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt" - [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt" - [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt" - [Rule 1]: All scripts need to be able to be ran from any directory. - [Rule 2]: All scripts need output llwhat they are doing, as they are doing it. - [Rule 3]: All tasks should use halo to report successes and failures. - [Rule 4]: The python module "tqdm" can be used to report progress when appropriate. - [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for. - [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python". - [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ". - [Rule 8]: Scripts should generally have robust error checking. - [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.

The Project Details above contain locations for files, data, scripts, and rules to follow when creating project scripts and toolkit.

Please extend the 'process-stage-6' script to also call the script 'initial-deduplication.py' inside of [Stage 6 Binaries]. This new script can be made to run between the 'standardize-capital-letters' and 'business-name-strip-structures' scripts. initial-deduplication to be the second to last script to be ran in other words.

Here is the current process-stage-6 script:

#!/usr/bin/env python # Script Name: process-stage-6

import os import subprocess from pyfiglet import Figlet from halo import Halo

Define the paths based on the project details

PROJECT_ROOT = "/home/ld/mgk-scrapes" BIN_STAGE_6_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-6')

# Run scripts that do not require arguments
for script in scripts:
    try:
        subprocess.run(["python", script], check=True)
    except subprocess.CalledProcessError as e:
        print(f"Error: {e}")
        break

def main(): figlet = Figlet(font='slant') script_name = "process-stage-6".replace("-", " ").title() print(figlet.renderText(script_name))

print("Running scripts for Stage 6...\n")
run_scripts()

final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")

if name == 'main': main() </process-stage-6 script>

Prompt 6

- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts. - [Project Root]: "/home/ld/mgk-scrapes" - [Current Dataset]: "[Project Root]/current-data/" - [Data Directory]: "[Current Dataset]/.data/" - [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".") - [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.) - [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state. - [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under. - [Stage 1]: "[Data Directory]/stage-1/" - [Stage 2]: "[Data Directory]/stage-2/" - [Stage 3]: "[Data Directory]/stage-3/" - [Stage 4]: "[Data Directory]/stage-4/" - [Stage 5]: "[Data Directory]/stage-5/" - [Stage 6]: "[Data Directory]/stage-6/" - [Binaries]: "[Project Root]/bin/" - [Stage 1 Binaries]: "[Binaries]/stage-1/" - [Stage 2 Binaries]: "[Binaries]/stage-2/" - [Stage 3 Binaries]: "[Binaries]/stage-3/" - [Stage 4 Binaries]: "[Binaries]/stage-4/" - [Stage 5 Binaries]: "[Binaries]/stage-5/" - [Stage 6 Binaries]: "[Binaries]/stage-6/" - [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt" - [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt" - [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt" - [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt" - [Rule 1]: All scripts need to be able to be ran from any directory. - [Rule 2]: All scripts need output llwhat they are doing, as they are doing it. - [Rule 3]: All tasks should use halo to report successes and failures. - [Rule 4]: The python module "tqdm" can be used to report progress when appropriate. - [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for. - [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python". - [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ". - [Rule 8]: Scripts should generally have robust error checking. - [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.

The Project Details above contain locations for files, data, scripts, and rules to follow when creating project scripts and toolkit.

Please extend the 'process-stage-6' to add some new scripts to the list of scripts already being called.

Here is the current process-stage-6 script:

#!/usr/bin/env python # Script Name: process-stage-6

import os import subprocess from pyfiglet import Figlet from halo import Halo

Define the paths based on the project details

PROJECT_ROOT = "/home/ld/mgk-scrapes" BIN_STAGE_6_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-6')

# Run scripts that do not require arguments
for script in scripts:
    try:
        subprocess.run(["python", script], check=True)
    except subprocess.CalledProcessError as e:
        print(f"Error: {e}")
        break

def main(): figlet = Figlet(font='slant') script_name = "process-stage-6".replace("-", " ").title() print(figlet.renderText(script_name))

print("Running scripts for Stage 6...\n")
run_scripts()

final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")

if name == 'main': main() </process-stage-6 script>

After the 'business-name-strip-structures' script is called, I would like to run these additional scripts. Please have them ran in the order in which I provide them:

business-names-with-locations.py uuid-business-1-by-gbp-business-phone.py uuid-business-2-by-root-domain.py uuid-business-3-by-business-name.py

Prompt 7

- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts. - [Project Root]: "/home/ld/mgk-scrapes" - [Current Dataset]: "[Project Root]/current-data/" - [Data Directory]: "[Current Dataset]/.data/" - [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".") - [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.) - [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state. - [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under. - [Stage 1]: "[Data Directory]/stage-1/" - [Stage 2]: "[Data Directory]/stage-2/" - [Stage 3]: "[Data Directory]/stage-3/" - [Stage 4]: "[Data Directory]/stage-4/" - [Stage 5]: "[Data Directory]/stage-5/" - [Stage 6]: "[Data Directory]/stage-6/" - [Binaries]: "[Project Root]/bin/" - [Stage 1 Binaries]: "[Binaries]/stage-1/" - [Stage 2 Binaries]: "[Binaries]/stage-2/" - [Stage 3 Binaries]: "[Binaries]/stage-3/" - [Stage 4 Binaries]: "[Binaries]/stage-4/" - [Stage 5 Binaries]: "[Binaries]/stage-5/" - [Stage 6 Binaries]: "[Binaries]/stage-6/" - [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt" - [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt" - [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt" - [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt" - [Rule 1]: All scripts need to be able to be ran from any directory. - [Rule 2]: All scripts need output llwhat they are doing, as they are doing it. - [Rule 3]: All tasks should use halo to report successes and failures. - [Rule 4]: The python module "tqdm" can be used to report progress when appropriate. - [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for. - [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python". - [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ". - [Rule 8]: Scripts should generally have robust error checking. - [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.

The Project Details above contain locations for files, data, scripts, and rules to follow when creating project scripts and toolkit.

Please extend the 'process-stage-6' to add some new scripts to the list of scripts already being called.

Here is the current process-stage-6 script:

#!/usr/bin/env python # Script Name: process-stage-6

import os import subprocess from pyfiglet import Figlet from halo import Halo

Define the paths based on the project details

PROJECT_ROOT = "/home/ld/mgk-scrapes" BIN_STAGE_6_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-6')

def run_scripts(): """Run the specified scripts in order.""" scripts = [ os.path.join(BIN_STAGE_6_DIR, "prepare-stage-6.py"), os.path.join(BIN_STAGE_6_DIR, "pre-sort.py"), os.path.join(BIN_STAGE_6_DIR, "trim-invalid-rows-from-merger.py"), os.path.join(BIN_STAGE_6_DIR, "delete-bad-matching-services.py"), os.path.join(BIN_STAGE_6_DIR, "standardize-capital-letters.py"), os.path.join(BIN_STAGE_6_DIR, "initial-deduplication.py"), os.path.join(BIN_STAGE_6_DIR, "business-name-strip-structures.py"), os.path.join(BIN_STAGE_6_DIR, "business-names-with-locations.py"), os.path.join(BIN_STAGE_6_DIR, "uuid-business-1-by-gbp-business-phone.py"), os.path.join(BIN_STAGE_6_DIR, "uuid-business-2-by-root-domain.py"), os.path.join(BIN_STAGE_6_DIR, "uuid-business-3-by-business-name.py") ]

# Run scripts that do not require arguments
for script in scripts:
    try:
        subprocess.run(["python", script], check=True)
    except subprocess.CalledProcessError as e:
        print(f"Error: {e}")
        break

def main(): figlet = Figlet(font='slant') script_name = "process-stage-6".replace("-", " ").title() print(figlet.renderText(script_name))

print("Running scripts for Stage 6...\n")
run_scripts()

final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")

if name == 'main': main() </process-stage-6 script>

After the 'uuid-business-3-by-business-name' script is called, I would like to run these additional scripts. Please have them ran in the order in which I provide them:

re-order-columns.py fill-in-missing-data.py uuid-business-4-single-location-entries.py standardize-yib.py uniqify-pass-1.py

Prompt 8

- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts. - [Project Root]: "/home/ld/mgk-scrapes" - [Current Dataset]: "[Project Root]/current-data/" - [Data Directory]: "[Current Dataset]/.data/" - [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".") - [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.) - [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state. - [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under. - [Stage 1]: "[Data Directory]/stage-1/" - [Stage 2]: "[Data Directory]/stage-2/" - [Stage 3]: "[Data Directory]/stage-3/" - [Stage 4]: "[Data Directory]/stage-4/" - [Stage 5]: "[Data Directory]/stage-5/" - [Stage 6]: "[Data Directory]/stage-6/" - [Binaries]: "[Project Root]/bin/" - [Stage 1 Binaries]: "[Binaries]/stage-1/" - [Stage 2 Binaries]: "[Binaries]/stage-2/" - [Stage 3 Binaries]: "[Binaries]/stage-3/" - [Stage 4 Binaries]: "[Binaries]/stage-4/" - [Stage 5 Binaries]: "[Binaries]/stage-5/" - [Stage 6 Binaries]: "[Binaries]/stage-6/" - [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt" - [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt" - [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt" - [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt" - [Rule 1]: All scripts need to be able to be ran from any directory. - [Rule 2]: All scripts need output llwhat they are doing, as they are doing it. - [Rule 3]: All tasks should use halo to report successes and failures. - [Rule 4]: The python module "tqdm" can be used to report progress when appropriate. - [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for. - [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python". - [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ". - [Rule 8]: Scripts should generally have robust error checking. - [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.

The Project Details above contain locations for files, data, scripts, and rules to follow when creating project scripts and toolkit.

Please extend the 'process-stage-6' to add some new scripts to the list of scripts already being called.

Here is the current process-stage-6 script:

#!/usr/bin/env python # Script Name: process-stage-6

import os import subprocess from pyfiglet import Figlet from halo import Halo

Define the paths based on the project details

PROJECT_ROOT = "/home/ld/mgk-scrapes" BIN_STAGE_6_DIR = os.path.join(PROJECT_ROOT, 'bin', 'stage-6')

# Run scripts that do not require arguments
for script in scripts:
    try:
        subprocess.run(["python", script], check=True)
    except subprocess.CalledProcessError as e:
        print(f"Error: {e}")
        break

def main(): figlet = Figlet(font='slant') script_name = "process-stage-6".replace("-", " ").title() print(figlet.renderText(script_name))

print("Running scripts for Stage 6...\n")
run_scripts()

final_spinner = Halo(spinner='dots', color='green')
final_spinner.start()
final_spinner.succeed("All scripts completed.")

if name == 'main': main() </process-stage-6 script>

After the 'uniqify-pass-1' script is called, I would like to run these additional scripts. Please have them ran in the order in which I provide them:

uniqify-pass-2.py

49 KiB Raw Blame History

Prompt 1

Define the paths based on the project details

Get the project root directory

Prompt 2

Prompt 3

Define the paths based on the project details

Prompt 4

Define the paths based on the project details

Define the paths based on the project details

Define the paths based on the project details

List of terms to remove

Prompt 5

Define the paths based on the project details

Prompt 6

Define the paths based on the project details

Prompt 7

Define the paths based on the project details

Prompt 8

Define the paths based on the project details

49 KiB

Raw Blame History