scraping/skallwar.suckit
1
0
Fork 0
mirror of https://github.com/Skallwar/suckit.git synced 2026-09-10 07:16:19 -04:00
Code Issues Actions Packages Projects Releases Wiki Activity
Tool for downloading websites.
183 commits 5 branches 4 tags 1.2 MiB
Rust 87.1%
Python 11%
Nix 1%
Shell 0.9%
Go to file
CohenArthur 1d122621ae
readme: Add cargo install --git instead of cloning (#65) 
* Update README.md

Co-authored-by: Esteban Blanc <estblcsk@gmail.com>

* readme: Add `cargo install --git` instead of cloning

Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
2020-05-05 20:52:14 +02:00
.github/workflows ci: Split ci in two jobs 2020-04-29 18:54:33 +02:00
src Better help for CLI arguments (#56) 2020-05-01 13:48:39 +02:00
tests speed_regression: Change from 5 retries to 10 (#61) 2020-05-05 17:31:02 +02:00
.gitignore gitignore: Ignore tags folder 2020-04-09 11:36:20 +02:00
Cargo.lock scraper: Selective parsing fully working 2020-04-29 11:24:24 +02:00
Cargo.toml scraper: Selective parsing fully working 2020-04-29 11:24:24 +02:00
LICENSE-APACHE Rename LICENSE to LICENSE-APACHE 2020-04-29 23:51:14 +02:00
LICENSE-MIT Create LICENSE-MIT 2020-04-29 23:53:17 +02:00
README.md readme: Add cargo install --git instead of cloning (#65) 2020-05-05 20:52:14 +02:00
suckit_logo.png logo: Uncrop logo 2020-04-29 23:21:45 +02:00

README.md

Build and test

SuckIT

SuckIT allows you to recursively visit and download a website's content to your disk.

SuckIT Logo

Features

  • Vacuums the entirety of a website recursively
  • Uses multithreading
  • Writes the website's content to your disk
  • Enables offline navigation
  • Saves application state on CTRL-C for later pickup
  • Offers random delays to avoid IP banning

Options

Option Behavior
-h, --help Displays help information
-v, --verbose Activate Verbose output
-d, --depth Specify the level of depth to go to when visiting the website
-j, --jobs Number of threads to use
-o, --output Output directory where the downloaded files are written
-t, --tries Number of times to retry when the downloading of a page fails

Example

A common use case could be the following:

suckit http://books.toscrape.com -j 8 -o /path/to/downloaded/pages/

asciicast

Installation

As of right now, SuckIT does not work on Windows.

To install it, you need to have Rust installed.

  • Check out this link for instructions on how to install Rust.

  • If you just want to install the suckit executable, you can simply run cargo install --git https://github.com/skallwar/suckit

  • Now, run it from anywhere with the suckit command.

Want to contribute ? Feel free to open an issue or submit a PR !

License

SuckIT is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0)

See LICENSE-APACHE and LICENSE-MIT for details.