scraping/skallwar.suckit
1
0
Fork 0
mirror of https://github.com/Skallwar/suckit.git synced 2026-09-10 15:26:24 -04:00
Code Issues Actions Packages Projects Releases Wiki Activity
251 commits 5 branches 4 tags 1.2 MiB
Commit graph

251 commits

This branch
This branch
All branches
Author SHA1 Message Date
Esteban Blanc f59c9245c8 FIXME 2022-01-23 13:16:09 +01:00
dependabot[bot] 931f1e5131 robo(deps): bump structopt from 0.3.25 to 0.3.26 
Bumps [structopt](https://github.com/TeXitoi/structopt) from 0.3.25 to 0.3.26.
- [Release notes](https://github.com/TeXitoi/structopt/releases)
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md)
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.25...v0.3.26)

---
updated-dependencies:
- dependency-name: structopt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-19 11:54:31 +01:00
dependabot[bot] a999137d31
robo(deps): bump tiny_http from 0.9.0 to 0.10.0 (#165) 
Bumps [tiny_http](https://github.com/tiny-http/tiny-http) from 0.9.0 to 0.10.0.
- [Release notes](https://github.com/tiny-http/tiny-http/releases)
- [Changelog](https://github.com/tiny-http/tiny-http/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tiny-http/tiny-http/compare/0.9.0...v0.10.0)

---
updated-dependencies:
- dependency-name: tiny_http
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-13 13:08:04 +01:00
dependabot[bot] 52d372372e
robo(deps): bump encoding_rs from 0.8.29 to 0.8.30 (#160) 
Bumps [encoding_rs](https://github.com/hsivonen/encoding_rs) from 0.8.29 to 0.8.30.
- [Release notes](https://github.com/hsivonen/encoding_rs/releases)
- [Commits](https://github.com/hsivonen/encoding_rs/compare/v0.8.29...v0.8.30)

---
updated-dependencies:
- dependency-name: encoding_rs
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-12 10:03:55 +01:00
dependabot[bot] 9330231522
robo(deps): bump reqwest from 0.11.8 to 0.11.9 (#164) 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.8 to 0.11.9.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.8...v0.11.9)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-01-11 16:02:17 +01:00
dependabot[bot] ed119f15dd robo(ci): bump actions/upload-artifact from 2.2.4 to 2.3.1 
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 2.2.4 to 2.3.1.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v2.2.4...v2.3.1)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-12-21 16:46:35 +01:00
dependabot[bot] f78632c7ee
robo(deps): bump reqwest from 0.11.6 to 0.11.8 (#163) 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.6 to 0.11.8.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.6...v0.11.8)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-12-21 11:24:30 +01:00
Esteban Blanc 2f684793be misc: Enable lto 2021-11-18 12:32:07 +01:00
Esteban Blanc 7f6eb7ca37 misc: Update dependencies 2021-10-19 12:06:09 +02:00
dependabot[bot] 1f3a6cbcf7
robo(deps): bump tiny_http from 0.8.2 to 0.9.0 (#154) 
Bumps [tiny_http](https://github.com/tiny-http/tiny-http) from 0.8.2 to 0.9.0.
- [Release notes](https://github.com/tiny-http/tiny-http/releases)
- [Changelog](https://github.com/tiny-http/tiny-http/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tiny-http/tiny-http/compare/0.8.2...0.9.0)

---
updated-dependencies:
- dependency-name: tiny_http
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-10-15 12:43:16 +02:00
dependabot[bot] 399351261c robo(deps): bump reqwest from 0.11.4 to 0.11.5 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.4 to 0.11.5.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.4...v0.11.5)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-08 10:17:43 +02:00
dependabot[bot] 1386a4b66b robo(deps): bump pathdiff from 0.2.0 to 0.2.1 
Bumps [pathdiff](https://github.com/Manishearth/pathdiff) from 0.2.0 to 0.2.1.
- [Release notes](https://github.com/Manishearth/pathdiff/releases)
- [Commits](https://github.com/Manishearth/pathdiff/commits)

---
updated-dependencies:
- dependency-name: pathdiff
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-04 21:39:24 +02:00
Lucas eebbf5647d Split html charset tests 2021-10-04 20:56:03 +02:00
Lucas 4255ef29c2 Add tests 2021-10-04 20:56:03 +02:00
lhvy 034f286638 Fix <meta charset> with single quotes 2021-10-04 20:56:03 +02:00
dependabot[bot] c14cb0827f robo(ci): bump actions/upload-artifact from 1 to 2.2.4 
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 1 to 2.2.4.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v1...v2.2.4)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-09-23 20:56:28 +02:00
Esteban Blanc 04edd27550 Add tests for external download 2021-09-23 20:43:54 +02:00
marchellodev ef79cc2318 Add external site downloading support 
Introduces --edepth flag for external domain depth
Transforms urls that start with `//` into full https links for more accurate detection of external links
Fixes panic when url starts with `///`
Readme: Added explanation of the `--edepth` flag
--edepth -> --ext-depth
Refactors code to avoid repetition (normalizing urls)
Improved code documentation (url normalization)
README.md upd: --ext-depth
Minor docs changes (--depth)
rustfmt coding style
Updated `--ext_depth` command docs
Small code documentation improvements

Co-authored-by: CohenArthur <cohenarthur.dev@gmail.com>
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
 2021-09-23 20:43:54 +02:00
Esteban Blanc 1be1f856e9 ci: Add codecov 2021-09-10 14:39:21 +02:00
dependabot[bot] caa0dedc00
robo(ci): bump bruceadams/get-release from 1.2.2 to 1.2.3 (#147) 
Bumps [bruceadams/get-release](https://github.com/bruceadams/get-release) from 1.2.2 to 1.2.3.
- [Release notes](https://github.com/bruceadams/get-release/releases)
- [Commits](https://github.com/bruceadams/get-release/compare/v1.2.2...v1.2.3)

---
updated-dependencies:
- dependency-name: bruceadams/get-release
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-09-09 11:55:21 +02:00
dependabot[bot] 9cb0b81286
robo(deps): bump structopt from 0.3.22 to 0.3.23 (#148) 
Bumps [structopt](https://github.com/TeXitoi/structopt) from 0.3.22 to 0.3.23.
- [Release notes](https://github.com/TeXitoi/structopt/releases)
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md)
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.22...v0.3.23)

---
updated-dependencies:
- dependency-name: structopt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-31 23:12:46 +02:00
pinkforest(she/her) 729c737c50
Lib bin initial (#141) 
* Renames main binary to new lib-bin tree

* Initialises base lib-bin structure
 2021-07-07 23:39:32 +02:00
Esteban Blanc 7c05a181e0 Bump version for v0.1.2 release (#138) 2021-07-07 19:20:24 +02:00
mr-bo-jangles 0b1a556ce1 Lets make this non-unix specific for our windows based friends (#134) 
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
 2021-07-07 19:20:24 +02:00
Esteban Blanc e25e358f35 misc: Add MSRV (#137) 2021-07-07 19:20:24 +02:00
pinkforest(she/her) f87f6bf078 Stops freezing Firefoxes looking into the repo (#139) 
* Fixes medias for release

* SVG Change from 15 rows to 40 rows with 100ms frames
 2021-07-07 19:20:24 +02:00
Esteban Blanc de543179c0 ci: Fix same id issue in release workflow (#140) 2021-07-07 19:20:16 +02:00
dependabot[bot] 2c08ae00f4
robo(deps): bump structopt from 0.3.21 to 0.3.22 (#131) 
Bumps [structopt](https://github.com/TeXitoi/structopt) from 0.3.21 to 0.3.22.
- [Release notes](https://github.com/TeXitoi/structopt/releases)
- [Changelog](https://github.com/TeXitoi/structopt/blob/master/CHANGELOG.md)
- [Commits](https://github.com/TeXitoi/structopt/compare/v0.3.21...v0.3.22)

---
updated-dependencies:
- dependency-name: structopt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-05 09:33:37 +02:00
dependabot[bot] d388b3c3f2
robo(deps): bump crossbeam from 0.8.0 to 0.8.1 (#126) 
Bumps [crossbeam](https://github.com/crossbeam-rs/crossbeam) from 0.8.0 to 0.8.1.
- [Release notes](https://github.com/crossbeam-rs/crossbeam/releases)
- [Changelog](https://github.com/crossbeam-rs/crossbeam/blob/master/CHANGELOG.md)
- [Commits](https://github.com/crossbeam-rs/crossbeam/compare/crossbeam-0.8.0...crossbeam-0.8.1)

---
updated-dependencies:
- dependency-name: crossbeam
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-02 11:06:08 +02:00
dependabot[bot] ef8518e877
robo(deps): bump regex from 1.4.6 to 1.5.4 (#128) 
Bumps [regex](https://github.com/rust-lang/regex) from 1.4.6 to 1.5.4.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.4.6...1.5.4)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-02 10:53:17 +02:00
dependabot[bot] ed7a22876a
robo(deps): bump url from 2.2.0 to 2.2.2 (#127) 
Bumps [url](https://github.com/servo/rust-url) from 2.2.0 to 2.2.2.
- [Release notes](https://github.com/servo/rust-url/releases)
- [Commits](https://github.com/servo/rust-url/compare/v2.2.0...v2.2.2)

---
updated-dependencies:
- dependency-name: url
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-02 10:29:53 +02:00
dependabot[bot] f0c5448d1d
robo(deps): bump reqwest from 0.11.0 to 0.11.4 (#125) 
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.0 to 0.11.4.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.0...v0.11.4)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-02 10:19:53 +02:00
pinkforest(she/her) a7f4334910
Adds badges (#122) 
* * Adds badges

* Fixes mistake on badge
 2021-07-02 09:47:44 +02:00
pinkforest(she/her) 1e034011a7
Updated dependencies and dependabot (#123) 
* Create dependabot.yml

* robo(deps): bump rand from 0.8.1 to 0.8.4

Bumps [rand](https://github.com/rust-random/rand) from 0.8.1 to 0.8.4.
- [Release notes](https://github.com/rust-random/rand/releases)
- [Changelog](https://github.com/rust-random/rand/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-random/rand/compare/0.8.1...0.8.4)

---
updated-dependencies:
- dependency-name: rand
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* robo(deps): bump subprocess from 0.2.6 to 0.2.7

Bumps [subprocess](https://github.com/hniksic/rust-subprocess) from 0.2.6 to 0.2.7.
- [Release notes](https://github.com/hniksic/rust-subprocess/releases)
- [Commits](https://github.com/hniksic/rust-subprocess/compare/release/0.2.6...release/0.2.7)

---
updated-dependencies:
- dependency-name: subprocess
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* robo(deps): bump encoding_rs from 0.8.26 to 0.8.28

Bumps [encoding_rs](https://github.com/hsivonen/encoding_rs) from 0.8.26 to 0.8.28.
- [Release notes](https://github.com/hsivonen/encoding_rs/releases)
- [Commits](https://github.com/hsivonen/encoding_rs/compare/v0.8.26...v0.8.28)

---
updated-dependencies:
- dependency-name: encoding_rs
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* robo(deps): bump tiny_http from 0.7.0 to 0.8.2

Bumps [tiny_http](https://github.com/tiny-http/tiny-http) from 0.7.0 to 0.8.2.
- [Release notes](https://github.com/tiny-http/tiny-http/releases)
- [Changelog](https://github.com/tiny-http/tiny-http/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tiny-http/tiny-http/compare/0.7.0...0.8.2)

---
updated-dependencies:
- dependency-name: tiny_http
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* robo(deps): bump regex from 1.4.3 to 1.4.6

Bumps [regex](https://github.com/rust-lang/regex) from 1.4.3 to 1.4.6.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.4.3...1.4.6)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-07-02 09:19:30 +02:00
Markus Unterwaditzer 9a194d784e
Fix scraping on URLs that have fragments (#120) 2021-05-25 21:47:55 +02:00
Markus Unterwaditzer dcb5080fca
Fix flipped continue_on_error check (#118) 2021-05-24 14:03:58 +02:00
Markus Unterwaditzer 422534f146
Fix querystring support (#117) 
* Fix querystring support

* While technically valid filenames, having ? in filenames means that the directory structure cannot be served by nginx as the resulting filenames cannot be addressed via URLs unless the ? is percent-encoded in every link.

* Query strings can contain trailing slashes. If that happens, suckit will attempt to create a directory, then create a same-named file, which fails

* fix fmt
 2021-05-24 10:10:45 +02:00
Bastien Germond 330829913e
logger: fix panic! changes for Rust 2021 (#115) 2021-05-20 21:49:36 +02:00
Esteban Blanc 94bbc9986d
Minor Fixes (#112) 
* scraper: Fix charset regex

* scraper: Add unsuported charset warning

* scraper: Log error or warn on multiple download failed attempts
 2021-01-14 22:11:36 +01:00
Esteban Blanc 1bdb086651
Patches for #110 (#111) 
* scrapper: Log an error if an url is not in the path map

* url_helper: Fix domain/1/ vs domain/1 file or dir conflict

* url_helper: Fix to_path()
 2021-01-14 14:08:35 +01:00
Esteban Blanc 6090e4a538 misc: Bump version number 2021-01-11 21:58:35 +01:00
Esteban Blanc 96e1f2bf52
Respect website architecture (#110) 
* scraper: Stop flat archive creation

* scrapper: Fix_domtree create only relative path

* tests: Fix to_path() tests

* tests: Fix tests

* scraper: Fix doc

Co-authored-by: CohenArthur <arthur.cohen@epita.fr>

* misc: Cleanup

* Update tests/fixtures/mod.rs

Co-authored-by: CohenArthur <arthur.cohen@epita.fr>

Co-authored-by: CohenArthur <arthur.cohen@epita.fr>
 2021-01-10 22:55:37 +01:00
CohenArthur 299247a107
Use local webserver for speed testing (#108) 
* tests: Add script to set up local server

* tests: Remove old results from speed.csv since test methodology changes

* tests: Change speed_regression so it downloads locally hosted webserver

* local_server_setup: Remove need of sudo since suckit understands ports

* speed_regression: Start local web server in script

Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
 2021-01-06 11:10:38 +01:00
Esteban Blanc 56c3970edd Merge branch 'master' of github.com:Skallwar/suckit 2021-01-05 12:02:09 +01:00
Esteban Blanc 07df03fd07 ci: Add sha512 checksum 2021-01-05 12:01:45 +01:00
Esteban Blanc 1819a20ed5
Update README.md 2021-01-05 11:04:33 +01:00
Esteban Blanc db3c933423
Support other charset (#105) 
* scraper: Initial work for other charset support

Not working

* scraper: Get charset from html instead of http headers

Not working

* scraper: Charset working

* scraper: Fix regex ('"' after equals)

* scraper: Use lazy_static for charset regex

* misc: Refactor pr

* tests: Refactor fixtures

* tests: Add html charset support tests

* tests: Charset, split tests

* downloader: Get charset from http headers

* misc: Clarify iterator first

* Apply suggestions from CohenArthur

Co-authored-by: CohenArthur <arthur.cohen@epita.fr>
 2021-01-05 09:57:12 +01:00
Esteban Blanc d26647c17d ci: Use action for release info 2020-12-22 11:17:48 +01:00
Esteban Blanc 1ddbe1702e ci: Add release build (#106) 
* ci: Build target aware

* ci: Add release.yml
 2020-12-22 10:56:13 +01:00
Esteban Blanc 09af371589
misc: Update dependencies (#103) 2020-12-19 16:38:26 +01:00