3.9 KiB
This is a collection of utilities I use to scrape various kinds of information from domains.
## Step 1: Add Domains Obtain a list of domains you wish to scrape information from, and put it in a file named "input-domains.csv" in the project root.
## Step 2: Normalize Data Here we are going to start off with a very fast and simple sort and deduplication of the provided domains.
bin/normalize-data.pyThis will create an output file named "01-normalized-data.csv".
## Step 3: Alive Check Next we want to know which domains are actually resolvable, and deliver us web content. Which are alive, and which are dead.
bin/alive-check.pyDepending on the size of your domain list, this could take a while.
The alive-check script will create an output file named "02-alive-check.csv".
The output file will record the original domain used for the alive check, if the domain was found to be alive or not, and also include the full url of the website that resulted from connecting to the domain. This will be after any redirects occurred or anything.
## Step 4: Split Alives into Target Files I like to run my scrape on multiple servers, to cover more space in more time.
To that end, I usually run a script called "split-alives" which will search the "02-alive-check.csv" file created earlier for websites which are marked as alive, and then will output them into 3 files named "target1.csv", "target2.csv", and "target3.csv".
I then migrate these csv's onto my poweredges and put them to work using the process-alives script afterwards.
bin/split-alives.py## Step 5: Process Alives for Pages The next step in the process I use here, is to scrape all the pages from all of the domains which were found to be alive.
There is a script named "scrape-pages" which will scrape all of the pages from any domain we tell it to.
However I call this script in parellel using another script named "process-alives". Which when ran, will search trhough the file "targets.csv" for sites that are alive.
The "process-alives" script will use as input a file named "targets.csv" in the project root. I usually set this up from one of the splits I created in step 4 above.
bin/process-alives.pyThis will take quite a while depending on the size of your list.
It is going to create a directory off of the project root named "scrapes". And inside this directory will becreated a sub-directory for every website being processed.
Inside those sub-directories we will find the following csv files getting created:
- email-addresses.csv
- external-links.csv
- meta-info.csv
- pages.csv
- phone-numbers.csv
- social-media-links.csv
That's going to be the main meat and potatoes of our scrape. Pages, external links, phone numbers, emails, and social media.
## Step 6: Extract External Domains This is where we close the loop and scrape all of the "external-links.csv" files for new domains to process. Just like we did with this set.
The extract external domains script will harvest a domain and website url from all of the external-links.csv files into a csv file named "extracted-external-domains.csv". This list will also be sorted and deduplicated.
bin/extract-external-domains.pyBonus: I should probably add this to the script, but to extract only the domains from the output file in this list you can use awk as in this example below:
awk -F',' 'NR > 1 {print $2}' extracted-external-domains.csv >domains.csv## Step 7: Extract Contact Forms This will not do as thorough a job as some tools I've used to scrape contact forms, but for now it does a pretty good job.
This stage looks through all of the "pages.csv" files we scraped earlier for URLs that could be contact us forms.
These forms can be quite useful.
bin/extract-contact-us-forms.py