scraping/skallwar.suckit
1
0
Fork 0
mirror of https://github.com/Skallwar/suckit.git synced 2026-09-10 23:36:20 -04:00
Code Issues Actions Packages Projects Releases Wiki Activity
skallwar.suckit/README.md
marchellodev ef79cc2318 Add external site downloading support 
Introduces --edepth flag for external domain depth
Transforms urls that start with `//` into full https links for more accurate detection of external links
Fixes panic when url starts with `///`
Readme: Added explanation of the `--edepth` flag
--edepth -> --ext-depth
Refactors code to avoid repetition (normalizing urls)
Improved code documentation (url normalization)
README.md upd: --ext-depth
Minor docs changes (--depth)
rustfmt coding style
Updated `--ext_depth` command docs
Small code documentation improvements

Co-authored-by: CohenArthur <cohenarthur.dev@gmail.com>
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
2021-09-23 20:43:54 +02:00

3.3 KiB
Raw Permalink Blame History

Build and test codecov Crates.io Docs Deps License License MSRV

SuckIT

SuckIT allows you to recursively visit and download a website's content to your disk.

SuckIT Logo

Features

  • Vacuums the entirety of a website recursively
  • Uses multithreading
  • Writes the website's content to your disk
  • Enables offline navigation
  • Offers random delays to avoid IP banning
  • Saves application state on CTRL-C for later pickup

Options

Option Behavior
-h, --help Displays help information
-v, --verbose Activate Verbose output
-d, --depth Specify the level of depth to go to when visiting the website. Default is -1 (infinity)
--ext-depth Specify the level of depth to go to when visiting websites that have a different domain name. Default is 0 (ignore external links), -1 is infinity
-j, --jobs Number of threads to use
-o, --output Output directory where the downloaded files are written
-t, --tries Number of times to retry when the downloading of a page fails
-u, --user-agent User agent to be used for sending requests
-i, --include Specify a regex to include pages that match this pattern
-e, --exclude Specify a regex to exclude pages that match this pattern
-a, --auth Provide usernames and passwords for the downloader to use
--dry-run Do everything without saving the files to the disk

Example

A common use case could be the following:

suckit http://books.toscrape.com -j 8 -o /path/to/downloaded/pages/

asciicast

Installation

As of right now, SuckIT does not work on Windows.

To install it, you need to have Rust installed.

  • Check out this link for instructions on how to install Rust.

  • If you just want to install the suckit executable, you can simply run cargo install --git https://github.com/skallwar/suckit

  • Now, run it from anywhere with the suckit command.

Arch Linux

suckit can be installed from available AUR packages using an AUR helper. For example,

yay -S suckit

Want to contribute ? Feel free to open an issue or submit a PR !

License

SuckIT is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0)

See LICENSE-APACHE and LICENSE-MIT for details.