scraping/skallwar.suckit
1
0
Fork 0
mirror of https://github.com/Skallwar/suckit.git synced 2026-09-10 07:16:19 -04:00
Code Issues Actions Packages Projects Releases Wiki Activity
Tool for downloading websites.
245 commits 5 branches 4 tags 1.2 MiB
Rust 87.1%
Python 11%
Nix 1%
Shell 0.9%
Go to file
dependabot[bot] f78632c7ee
robo(deps): bump reqwest from 0.11.6 to 0.11.8 (#163) 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.6 to 0.11.8.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.6...v0.11.8)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-12-21 11:24:30 +01:00
.github robo(ci): bump actions/upload-artifact from 1 to 2.2.4 2021-09-23 20:56:28 +02:00
media Stops freezing Firefoxes looking into the repo (#139) 2021-07-07 19:20:24 +02:00
src Split html charset tests 2021-10-04 20:56:03 +02:00
tests Add tests for external download 2021-09-23 20:43:54 +02:00
.gitignore Use local webserver for speed testing (#108) 2021-01-06 11:10:38 +01:00
Cargo.lock robo(deps): bump reqwest from 0.11.6 to 0.11.8 (#163) 2021-12-21 11:24:30 +01:00
Cargo.toml misc: Enable lto 2021-11-18 12:32:07 +01:00
LICENSE-APACHE Rename LICENSE to LICENSE-APACHE 2020-04-29 23:51:14 +02:00
LICENSE-MIT Create LICENSE-MIT 2020-04-29 23:53:17 +02:00
README.md Add external site downloading support 2021-09-23 20:43:54 +02:00

README.md

Build and test codecov Crates.io Docs Deps License License MSRV

SuckIT

SuckIT allows you to recursively visit and download a website's content to your disk.

SuckIT Logo

Features

  • Vacuums the entirety of a website recursively
  • Uses multithreading
  • Writes the website's content to your disk
  • Enables offline navigation
  • Offers random delays to avoid IP banning
  • Saves application state on CTRL-C for later pickup

Options

Option Behavior
-h, --help Displays help information
-v, --verbose Activate Verbose output
-d, --depth Specify the level of depth to go to when visiting the website. Default is -1 (infinity)
--ext-depth Specify the level of depth to go to when visiting websites that have a different domain name. Default is 0 (ignore external links), -1 is infinity
-j, --jobs Number of threads to use
-o, --output Output directory where the downloaded files are written
-t, --tries Number of times to retry when the downloading of a page fails
-u, --user-agent User agent to be used for sending requests
-i, --include Specify a regex to include pages that match this pattern
-e, --exclude Specify a regex to exclude pages that match this pattern
-a, --auth Provide usernames and passwords for the downloader to use
--dry-run Do everything without saving the files to the disk

Example

A common use case could be the following:

suckit http://books.toscrape.com -j 8 -o /path/to/downloaded/pages/

asciicast

Installation

As of right now, SuckIT does not work on Windows.

To install it, you need to have Rust installed.

  • Check out this link for instructions on how to install Rust.

  • If you just want to install the suckit executable, you can simply run cargo install --git https://github.com/skallwar/suckit

  • Now, run it from anywhere with the suckit command.

Arch Linux

suckit can be installed from available AUR packages using an AUR helper. For example,

yay -S suckit

Want to contribute ? Feel free to open an issue or submit a PR !

License

SuckIT is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0)

See LICENSE-APACHE and LICENSE-MIT for details.