scraping/skallwar.suckit
1
0
Fork 0
mirror of https://github.com/Skallwar/suckit.git synced 2026-09-10 07:16:19 -04:00
Code Issues Actions Packages Projects Releases Wiki Activity
Tool for downloading websites.
224 commits 5 branches 4 tags 1.2 MiB
Rust 87.1%
Python 11%
Nix 1%
Shell 0.9%
Go to file
dependabot[bot] 2c08ae00f4
robo(deps): bump structopt from 0.3.21 to 0.3.22 (#131) 
Bumps [structopt](https://github.com/TeXitoi/structopt) from 0.3.21 to 0.3.22.
- [Release notes](https://github.com/TeXitoi/structopt/releases)
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md)
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.21...v0.3.22)

---
updated-dependencies:
- dependency-name: structopt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-07-05 09:33:37 +02:00
.github Updated dependencies and dependabot (#123) 2021-07-02 09:19:30 +02:00
src Fix scraping on URLs that have fragments (#120) 2021-05-25 21:47:55 +02:00
tests Respect website architecture (#110) 2021-01-10 22:55:37 +01:00
.gitignore Use local webserver for speed testing (#108) 2021-01-06 11:10:38 +01:00
Cargo.lock robo(deps): bump structopt from 0.3.21 to 0.3.22 (#131) 2021-07-05 09:33:37 +02:00
Cargo.toml robo(deps): bump regex from 1.4.6 to 1.5.4 (#128) 2021-07-02 10:53:17 +02:00
LICENSE-APACHE Rename LICENSE to LICENSE-APACHE 2020-04-29 23:51:14 +02:00
LICENSE-MIT Create LICENSE-MIT 2020-04-29 23:53:17 +02:00
README.md Adds badges (#122) 2021-07-02 09:47:44 +02:00
suckit_logo.png logo: Uncrop logo 2020-04-29 23:21:45 +02:00

README.md

Build and test Crates.io Docs Deps License License

SuckIT

SuckIT allows you to recursively visit and download a website's content to your disk.

SuckIT Logo

Features

  • Vacuums the entirety of a website recursively
  • Uses multithreading
  • Writes the website's content to your disk
  • Enables offline navigation
  • Offers random delays to avoid IP banning
  • Saves application state on CTRL-C for later pickup

Options

Option Behavior
-h, --help Displays help information
-v, --verbose Activate Verbose output
-d, --depth Specify the level of depth to go to when visiting the website
-j, --jobs Number of threads to use
-o, --output Output directory where the downloaded files are written
-t, --tries Number of times to retry when the downloading of a page fails
-u, --user-agent User agent to be used for sending requests
-i, --include Specify a regex to include pages that match this pattern
-e, --exclude Specify a regex to exclude pages that match this pattern
-a, --auth Provide usernames and passwords for the downloader to use
--dry-run Do everything without saving the files to the disk

Example

A common use case could be the following:

suckit http://books.toscrape.com -j 8 -o /path/to/downloaded/pages/

asciicast

Installation

As of right now, SuckIT does not work on Windows.

To install it, you need to have Rust installed.

  • Check out this link for instructions on how to install Rust.

  • If you just want to install the suckit executable, you can simply run cargo install --git https://github.com/skallwar/suckit

  • Now, run it from anywhere with the suckit command.

Arch Linux

suckit can be installed from available AUR packages using an AUR helper. For example,

yay -S suckit

Want to contribute ? Feel free to open an issue or submit a PR !

License

SuckIT is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0)

See LICENSE-APACHE and LICENSE-MIT for details.