scraping/skallwar.suckit
1
0
Fork 0
mirror of https://github.com/Skallwar/suckit.git synced 2026-09-10 07:16:19 -04:00
Code Issues Actions Packages Projects Releases Wiki Activity
287 commits 5 branches 4 tags 1.2 MiB
Commit graph

287 commits

This branch
This branch
All branches
Author SHA1 Message Date
dependabot[bot] 4c2a255e9f robo(ci): bump bruceadams/get-release from 1.2.3 to 1.3.2 
Bumps [bruceadams/get-release](https://github.com/bruceadams/get-release) from 1.2.3 to 1.3.2.
- [Release notes](https://github.com/bruceadams/get-release/releases)
- [Commits](https://github.com/bruceadams/get-release/compare/v1.2.3...v1.3.2)

---
updated-dependencies:
- dependency-name: bruceadams/get-release
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-10-28 15:14:18 +02:00
Esteban Skallwar Blanc e9d4bb8dba Add CLI argument to disable SSL checks 2022-10-04 15:23:59 +02:00
Like0x 9282b5dee5 Update downloader.rs for Disable SSL checks 2022-10-04 15:23:59 +02:00
Esteban Skallwar Blanc ad4b89d18d misc: Update MSRV 2022-10-03 16:15:42 +02:00
Esteban Skallwar Blanc 62f17a236e misc: Update dependencies 2022-10-03 16:15:42 +02:00
Esteban Blanc 3fca2753b5 misc: Update MSRV 2022-08-01 22:40:05 +02:00
Esteban Blanc 20c4aa1346 misc: Update dependencies 2022-08-01 01:51:43 +02:00
Esteban Blanc 43c383a008 ci: Fix cross compilation for release workflow 2022-05-13 16:59:01 +02:00
Esteban Blanc 1b47539c71 ci: Add riscv64 build 2022-05-13 14:12:39 +02:00
Esteban Blanc 1f5e53f5eb misc: Revert to openssl 
We switched some time ago to rustls in order to cross compile more
easilly but it depends on `ring` which in full assembly and does not the
future: RISC-V

This revert to openssl while making sure the CI still cross-compile
correctly to aarch64
 2022-05-12 21:56:42 +02:00
Esteban Blanc c151757af6 ci: Build for aarch64 2022-05-11 10:47:57 +02:00
Orhun Parmaksız 3fb66dbffe ci: Build with --locked flag 2022-04-30 21:33:00 +02:00
Orhun Parmaksız 5e58e50a8e chore: Update Cargo.lock 2022-04-30 18:20:00 +02:00
Esteban Blanc 13e9d9f45d ci: Adjust MSRV 2022-04-29 20:51:35 +02:00
Esteban Blanc 18d16bffbd tls: Switch from openssl to rustls 2022-04-29 20:51:35 +02:00
Esteban Blanc 79cecf73d2 ci: Continue on error for release 
The build is fails because of op
enssl-sys crate when cross compiling
 2022-04-28 16:09:13 +02:00
Esteban Blanc fb124a5afb misc: Prepare for release 
Bump crate version
Update README (MSRV and usage)
 2022-04-28 13:27:05 +02:00
Esteban Blanc b467ee8a5e ci: Add riscv64 and aarch64 for release 2022-04-25 20:39:44 +02:00
Esteban Blanc c57fa1081c misc: Update dependencies in Cargo.lock 2022-04-25 17:27:08 +02:00
Esteban Blanc b09c9b487d misc: Fix all clippy warnings 2022-04-24 22:20:32 +02:00
dependabot[bot] 0171c62324 robo(ci): bump codecov/codecov-action from 2 to 3 
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 2 to 3.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-04-24 19:27:40 +02:00
raphTec a576ad9a9e Fix symlinks (relative link, argument order, folder creation) 
Was completely broken basically:
- No parent directories were created
- The link target was wrong and not a local path
- The symlink function was called with source and target swapped (this is a documentation
  bug in the symlink crate tho)
 2022-04-24 19:00:48 +02:00
raphTec 97e3a16f49 Add visit filter tests 2022-04-24 18:38:03 +02:00
raphTec 0d75eb68b9 Empty test directories before running tests 
Otherwise a failed test leaves the directory populated, which fails future test runs.
 2022-04-24 18:38:03 +02:00
raphTec ed56e9a078 Rename existing tests 2022-04-24 18:38:03 +02:00
raphTec 1ca0dcead8 Increase sleep duration 2022-04-24 18:38:03 +02:00
raphTec 62cb075e6c Add support for distinct dowload and visit regex 
This allows fine-grained control whether a page is visited, that means its links analyzed,
and saved to disk.
The decoupling of download and visit filter means the complete website may still be
explored while only downloading some files. To speed up scraping, irrelevant links can be
easily excluded from visiting.
 2022-04-24 18:38:03 +02:00
raphTec c6538e561d Fix lint warning (remove brackets) 2022-04-24 18:38:03 +02:00
raphTec 3a3f4784d2 Filter links before downloading / adding to the queue 
This commit speeds up scraping for scenarios where pages have a high branch factor, that
is many links and a majority of these links is excluded by the --exclude / --include
rules.
This also improves memory usage in these scenarios since the links are not stored.
Also the network traffic is reduced by not downloading these links in the first place.
 2022-04-24 18:38:03 +02:00
Esteban Blanc 84276b9631 ci: Stop dependabot PRs to the lockfile 2022-03-17 00:24:19 +01:00
dependabot[bot] 79f9614952
robo(deps): bump reqwest from 0.11.9 to 0.11.10 (#179) 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.9 to 0.11.10.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.9...v0.11.10)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-15 08:48:24 +01:00
dependabot[bot] c4cd36b98f
robo(deps): bump regex from 1.5.4 to 1.5.5 (#174) 
Bumps [regex](https://github.com/rust-lang/regex) from 1.5.4 to 1.5.5.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.5.4...1.5.5)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-03-09 09:08:50 +01:00
dependabot[bot] 2d2e747db8 robo(ci): bump actions/upload-artifact from 2.3.1 to 3 
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 2.3.1 to 3.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v2.3.1...v3)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-03-05 11:16:26 +01:00
dependabot[bot] a24000f900 robo(ci): bump actions/checkout from 2 to 3 
Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-03-04 21:37:25 +01:00
Esteban Blanc 23576977ad
Fix clippy warnings and add clippy ci check (#168) 
* misc: Run clippy and fixes all warnings

* ci: Add clippy check
 2022-02-28 09:51:23 +01:00
dependabot[bot] 422b810a78
robo(deps): bump rand from 0.8.4 to 0.8.5 (#169) 
Bumps [rand](https://github.com/rust-random/rand) from 0.8.4 to 0.8.5.
- [Release notes](https://github.com/rust-random/rand/releases)
- [Changelog](https://github.com/rust-random/rand/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-random/rand/compare/0.8.4...0.8.5)

---
updated-dependencies:
- dependency-name: rand
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-28 09:32:35 +01:00
dependabot[bot] f6072a9bee
robo(deps): bump tiny_http from 0.10.0 to 0.11.0 (#171) 
Bumps [tiny_http](https://github.com/tiny-http/tiny-http) from 0.10.0 to 0.11.0.
- [Release notes](https://github.com/tiny-http/tiny-http/releases)
- [Changelog](https://github.com/tiny-http/tiny-http/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tiny-http/tiny-http/compare/v0.10.0...0.11.0)

---
updated-dependencies:
- dependency-name: tiny_http
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-02-28 09:08:37 +01:00
dependabot[bot] 931f1e5131 robo(deps): bump structopt from 0.3.25 to 0.3.26 
Bumps [structopt](https://github.com/TeXitoi/structopt) from 0.3.25 to 0.3.26.
- [Release notes](https://github.com/TeXitoi/structopt/releases)
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md)
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.25...v0.3.26)

---
updated-dependencies:
- dependency-name: structopt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-19 11:54:31 +01:00
dependabot[bot] a999137d31
robo(deps): bump tiny_http from 0.9.0 to 0.10.0 (#165) 
Bumps [tiny_http](https://github.com/tiny-http/tiny-http) from 0.9.0 to 0.10.0.
- [Release notes](https://github.com/tiny-http/tiny-http/releases)
- [Changelog](https://github.com/tiny-http/tiny-http/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tiny-http/tiny-http/compare/0.9.0...v0.10.0)

---
updated-dependencies:
- dependency-name: tiny_http
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-13 13:08:04 +01:00
dependabot[bot] 52d372372e
robo(deps): bump encoding_rs from 0.8.29 to 0.8.30 (#160) 
Bumps [encoding_rs](https://github.com/hsivonen/encoding_rs) from 0.8.29 to 0.8.30.
- [Release notes](https://github.com/hsivonen/encoding_rs/releases)
- [Commits](https://github.com/hsivonen/encoding_rs/compare/v0.8.29...v0.8.30)

---
updated-dependencies:
- dependency-name: encoding_rs
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-12 10:03:55 +01:00
dependabot[bot] 9330231522
robo(deps): bump reqwest from 0.11.8 to 0.11.9 (#164) 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.8 to 0.11.9.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.8...v0.11.9)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-11 16:02:17 +01:00
dependabot[bot] ed119f15dd robo(ci): bump actions/upload-artifact from 2.2.4 to 2.3.1 
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 2.2.4 to 2.3.1.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v2.2.4...v2.3.1)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-12-21 16:46:35 +01:00
dependabot[bot] f78632c7ee
robo(deps): bump reqwest from 0.11.6 to 0.11.8 (#163) 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.6 to 0.11.8.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.6...v0.11.8)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-21 11:24:30 +01:00
Esteban Blanc 2f684793be misc: Enable lto 2021-11-18 12:32:07 +01:00
Esteban Blanc 7f6eb7ca37 misc: Update dependencies 2021-10-19 12:06:09 +02:00
dependabot[bot] 1f3a6cbcf7
robo(deps): bump tiny_http from 0.8.2 to 0.9.0 (#154) 
Bumps [tiny_http](https://github.com/tiny-http/tiny-http) from 0.8.2 to 0.9.0.
- [Release notes](https://github.com/tiny-http/tiny-http/releases)
- [Changelog](https://github.com/tiny-http/tiny-http/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tiny-http/tiny-http/compare/0.8.2...0.9.0)

---
updated-dependencies:
- dependency-name: tiny_http
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-10-15 12:43:16 +02:00
dependabot[bot] 399351261c robo(deps): bump reqwest from 0.11.4 to 0.11.5 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.4 to 0.11.5.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.4...v0.11.5)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-08 10:17:43 +02:00
dependabot[bot] 1386a4b66b robo(deps): bump pathdiff from 0.2.0 to 0.2.1 
Bumps [pathdiff](https://github.com/Manishearth/pathdiff) from 0.2.0 to 0.2.1.
- [Release notes](https://github.com/Manishearth/pathdiff/releases)
- [Commits](https://github.com/Manishearth/pathdiff/commits)

---
updated-dependencies:
- dependency-name: pathdiff
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-04 21:39:24 +02:00
Lucas eebbf5647d Split html charset tests 2021-10-04 20:56:03 +02:00
Lucas 4255ef29c2 Add tests 2021-10-04 20:56:03 +02:00