Prompt 1

- [Description]: This project revolves around creating a set of python scripts which we will use to parse CSV data. The CSV data represents Google Business Profile data. It was scraped from Google Maps. Our purpose is to clean this data up and prepare it for use with a toolkit of custom made python scripts. - [Project Root]: "/home/ld/mgk-scrapes" - [Current Dataset]: "[Project Root]/current-data/" - [Data Directory]: "[Current Dataset]/.data/" - [Parent Data]: "[Current Dataset]/"(All directories that do not begin with a period ".") - [Stage Directories]: "[Data Directory]/stage-#" (Where # is a number.) - [State Directories]: The state directories are the directories directly under one of the [Stage Directories]. These directories store data associated with a specific state. - [County Directories]: The coiunty directories are the directories directly under one of the [State Directories]. These directories store data associated with a specific county inside of the state the county directory is under. - [Stage 1]: "[Data Directory]/stage-1/" - [Stage 2]: "[Data Directory]/stage-2/" - [Stage 3]: "[Data Directory]/stage-3/" - [Stage 4]: "[Data Directory]/stage-4/" - [Stage 5]: "[Data Directory]/stage-5/" - [Stage 6]: "[Data Directory]/stage-6/" - [Binaries]: "[Project Root]/bin/" - [Stage 1 Binaries]: "[Binaries]/stage-1/" - [Stage 2 Binaries]: "[Binaries]/stage-2/" - [Stage 3 Binaries]: "[Binaries]/stage-3/" - [Stage 4 Binaries]: "[Binaries]/stage-4/" - [Stage 5 Binaries]: "[Binaries]/stage-5/" - [Stage 6 Binaries]: "[Binaries]/stage-6/" - [Bad Matching Services]: "[Data Directory]/bad-matching-services.txt" - [GBP Business Categories]: "[Data Directory]/gbp-business-categories.txt" - [GBP Matching Services]: "[Data Directory]/gbp-matching-services.txt" - [Unknown Blacklist]: "[Data Directory]/unknown-blacklist.txt" - [Rule 1]: All scripts need to be able to be ran from any directory. - [Rule 2]: All scripts need output llwhat they are doing, as they are doing it. - [Rule 3]: All tasks should use halo to report successes and failures. - [Rule 4]: The python module "tqdm" can be used to report progress when appropriate. - [Rule 5]: Scripts must be heavily commented, describing the purpose of the script and what each code block inside the code is for. - [Rule 6]: Every script should begin with a hashbang "#!/usr/bin/env python". - [Rule 7]: Every script should have a name, and the name of the script should be immediately beneath the hashbang in a comment field prefixed by a string that reads "Script Name: ". - [Rule 8]: Scripts should generally have robust error checking. - [Rule 9]: The first output from any script should be pyfiglet outputting the name of the script. The name that pyfiglet outputs though should be a modified version of the script name. The pyfiglet script name needs to replace the hyphens with spaces in the script name, and capitalize the words in the script name after the hyphens have been replaced.

The Project Details above contain locations for files, data, scripts, and rules to follow when creating project scripts and toolkit.

I would like to make a new script named "services-website-audit". As input this script will use the CSV file "20-services-needs-website.csv", and as output it will use the CSV file "21-website-audit.csv".

The script should first add some new columns to the output CSV file:

Services: Needs SSL Services: SSL Repair Services: Website Repair Performance Score Accessibility Score Best Practices Score SEO Score Services: Website Speed Optimization

After the script has created these columns, it will be responsible for filling in the details of each.

To do this we will be doing some tests on remote websites that we find in the "Root Domain" column of the CSV file.

For every unique "Root Domain" we will perform a few tests, and use the results of those tests to populate the columns we just created.

After we have gathered the data to insert into the new columns, we need to be sure that every duplicate copy of that "Root Domain" also gets the same data inserted into their rows. The CSV file will have many entries that share a "Root Domain", and I need all of those entries to have this data, but we only want to run these tests once.

The "Services: Needs SSL", "Services: SSL Repair" and "Services: Website Repair" column is the first test.

We need to contact the "Root Domain" we are testing, and see if the page loads successfully. It is ok if the page redirects first, but if we detect that the page loaded successfully, we will set "Services: Website Repair" to "No". If however trying to contact the "Root Domain" results in an error, and the page does not load successfully, then we will set the "Services: Website Repair" to "Yes".

During this same test, we need to attempt to connect to the "Root Domain" using HTTPS, and verify that HTTPS works correctly. If the website does not support HTTPS, then we need to set "Services: Needs SSL" to "Yes". Otherwise if the site loads HTTPS just fine, we can set "Services: Needs SSL" to "No".

If the "Root Domain" loads the HTTPS version of the website correctly, then we have one more test to do. And that is to attempt to connect to the website using HTTP instead of HTTPS. A correctly configured website will redirect the HTTP request to an HTTPS version and load the website. However, an incorrectly configured website will allow the HTTP connection to load as well as the HTTPS version.

If our attempt to connect to the website using standard HTTP does not result in the connection getting redirected to an HTTPS version of the website then we can set the "Services: SSL Repair" to "Yes". Otherwise if our HTTP connection results in a redirect to the HTTPS version of the site we can set the "Services: SSL Repair" column for the row to "No".

If we set "Services: Website Repair" to "No" with our prior test, we can proceed to fill in the remaining 5 columns:

<last 5 columns> Performance Score Accessibility Score Best Practices Score SEO Score Services: Website Speed Optimization </last 5 columns>

These columns will be filled in by using the Lighthouse CLI, and running that test against the "Root Domain".

Earlier, I was given this example by you on how the Lighthouse CLI could be used:

import requests

API_KEY = 'YOUR_API_KEY' URL = 'https://example.com'

def get_page_speed_insights(url, api_key): api_url = f'https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url={url}&key={api_key}' response = requests.get(api_url) return response.json()

def analyze_page_speed(data): # Extract relevant metrics and suggestions fcp = data['lighthouseResult']['audits']['first-contentful-paint']['displayValue'] lcp = data['lighthouseResult']['audits']['largest-contentful-paint']['displayValue'] suggestions = data['lighthouseResult']['categories']['performance']['score']

print(f"First Contentful Paint: {fcp}")
print(f"Largest Contentful Paint: {lcp}")
print(f"Performance Score: {suggestions}")

# Determine if optimization is needed based on thresholds
if suggestions < 0.9:
    print("Page needs optimization")
else:
    print("Page is well optimized")

if name == 'main': data = get_page_speed_insights(URL, API_KEY) analyze_page_speed(data)

</example lighthouse script>

I would like something like that done for each entry "Root Domain" we already successfully connected to once.

In that example lighthouse script, we have a threshold test that will print out either "Page needs optimization" or "Page is well optimized".

I would like that same threshold system to be used to deterimine what we put into the "Services: Website Speed Optimization" field. If the page is determined to be "Page is well optimized" then I want to set "Services: Website Speed Optimization" to "No". However if the page is detected as "Page needs optimization", then I want the "Services: Website Speed Optimization" to be set to "Yes".

I would like this script to be threaded, and to attempt to connect to 100 "Root Domains" at a time to do its tests, so I do not have to wait all day for results.

8.1 KiB Raw Blame History

Prompt 1

8.1 KiB

Raw Blame History