CohenArthur
299247a107
Use local webserver for speed testing ( #108 )
...
* tests: Add script to set up local server
* tests: Remove old results from speed.csv since test methodology changes
* tests: Change speed_regression so it downloads locally hosted webserver
* local_server_setup: Remove need of sudo since suckit understands ports
* speed_regression: Start local web server in script
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
2021-01-06 11:10:38 +01:00
Esteban Blanc
56c3970edd
Merge branch 'master' of github.com:Skallwar/suckit
2021-01-05 12:02:09 +01:00
Esteban Blanc
07df03fd07
ci: Add sha512 checksum
2021-01-05 12:01:45 +01:00
Esteban Blanc
1819a20ed5
Update README.md
2021-01-05 11:04:33 +01:00
Esteban Blanc
db3c933423
Support other charset ( #105 )
...
* scraper: Initial work for other charset support
Not working
* scraper: Get charset from html instead of http headers
Not working
* scraper: Charset working
* scraper: Fix regex ('"' after equals)
* scraper: Use lazy_static for charset regex
* misc: Refactor pr
* tests: Refactor fixtures
* tests: Add html charset support tests
* tests: Charset, split tests
* downloader: Get charset from http headers
* misc: Clarify iterator first
* Apply suggestions from CohenArthur
Co-authored-by: CohenArthur <arthur.cohen@epita.fr>
2021-01-05 09:57:12 +01:00
Esteban Blanc
d26647c17d
ci: Use action for release info
2020-12-22 11:17:48 +01:00
Esteban Blanc
1ddbe1702e
ci: Add release build ( #106 )
...
* ci: Build target aware
* ci: Add release.yml
2020-12-22 10:56:13 +01:00
Esteban Blanc
09af371589
misc: Update dependencies ( #103 )
2020-12-19 16:38:26 +01:00
orhun
9adfe8bc1f
readme: Update README about installation from AUR ( #101 )
2020-10-12 20:56:20 +02:00
Patrick Sier
fa549f2fed
Remove hash fragment from scraped URLs ( #100 )
...
* refactor: remove hash fragment from scraped urls
* refactor: handle fragments in download paths
* test: split url_to_path tests
2020-10-12 16:17:50 +02:00
orhun
d9c17d9ec2
fix: Use CARGO_BIN_EXE instead of hardcoded path for tests ( #98 )
2020-10-12 13:43:10 +02:00
CohenArthur
5705175132
readme: Update README with latest additions ( #97 )
...
The README lacked the latest features added to suckit, such as random delays between downloads or user authentication
2020-10-12 10:14:46 +02:00
Patrick Sier
f3f19fd1e8
Add support for HTTP basic auth args ( #96 )
...
* feat: add arg for HTTP basic auth
* feat: chunk space-separated auth args
* refactor: update auth hashmap, host handling
Require host for auth HashMap, defaulting to the origin URL's host value
if not provided otherwise.
* refactor: auth help string, constant updates
* test: add auth functional tests
2020-10-12 09:17:44 +02:00
Jaslo Ziska
794d8ddd53
Add dry run flag ( #95 )
...
Add --dry-run flag to stop files from being saved, as mentioned in
https://github.com/Skallwar/suckit/issues/94
2020-10-03 22:36:35 +02:00
Esteban Blanc
ffec62254d
cargo: Add caret requirements ( #93 )
2020-08-31 16:52:40 +02:00
Esteban Blanc
16dc20519d
misc: Add info for crates.io ( #92 )
2020-08-16 17:22:05 +02:00
Esteban Blanc
e65560beb4
ci: Check before build, build needs lint ( #90 )
2020-07-02 15:56:18 +02:00
Esteban Blanc
809bdb5e0e
ci: Fix steps names ( #89 )
2020-07-02 15:46:23 +02:00
Jakub Szymański
8e6a5ebbb1
Add error handling for persistent errors ( #88 )
...
* Add error handling for persistent errors
* lint file and add init for new parameter
* Update src/args.rs
Co-authored-by: CohenArthur <arthur.cohen@epita.fr>
* misc: Run cargo fmt
Co-authored-by: CohenArthur <arthur.cohen@epita.fr>
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
2020-07-02 15:24:42 +02:00
Rahul Sharma
397315b5f3
Add support for including excluding pages based on regex patterns ( #83 )
2020-05-17 14:21:03 +02:00
Atul Bhosale
8a311c2526
Add cargo fmt to github-actions config ( #85 )
2020-05-14 21:48:52 +02:00
Atul Bhosale
ae20a2713e
Format code using 'cargo fmt' ( #84 )
2020-05-13 21:02:14 +02:00
CohenArthur
c5bed43ee5
aciinema: Add new public recording ( #80 )
2020-05-08 19:19:59 +02:00
Rahul Sharma
16a5ed9cf7
Feature: Custom user agent as a CLI argument ( #76 )
...
* Allow passing user agent via cli args
* user-agent: Short and long CLI arguments
Co-authored-by: creativcoder <rahul.sharma@atherenergy.com>
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
2020-05-06 19:29:39 +02:00
SpyrosRoum
c857a2c9dd
Feature: Add unlimited download depth ( #73 )
...
* Add -1 for depth to act as unlimited download and default for depth
* Use static instead of magic number for infinite depth
2020-05-06 17:53:32 +02:00
SpyrosRoum
e11ed48b46
feature: Add random ( #66 )
...
* Add a random delay of 2 - 7 seconds between downloads
* Add command line args for delay and move delay to method
* Ignore .idea folder
* Update test for scraper to work with new low and high arguments
* Add a comment
* Remove unnecessary `&` for method call to self
* Rename `low` and `high` to `--delay` and `--random-range`
* Test different values for new arguments
* rename variable for consistency
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
* change help for clarity purposes
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
* Some whitespace
Co-authored-by: CohenArthur <arthur.cohen@epita.fr>
* Some whitespace
Co-authored-by: CohenArthur <arthur.cohen@epita.fr>
* rename variable for consistency
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
* Fix variable names
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
Co-authored-by: CohenArthur <arthur.cohen@epita.fr>
2020-05-06 14:44:04 +02:00
CohenArthur
1d122621ae
readme: Add cargo install --git instead of cloning ( #65 )
...
* Update README.md
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
* readme: Add `cargo install --git` instead of cloning
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
2020-05-05 20:52:14 +02:00
CohenArthur
f9f6bc6c47
speed_regression: Change from 5 retries to 10 ( #61 )
...
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
2020-05-05 17:31:02 +02:00
CohenArthur
dc57581baa
readme: Add installation instructions ( #62 )
2020-05-05 17:28:53 +02:00
Esteban Blanc
8dc6945c3c
Update README.md
2020-05-05 14:33:28 +02:00
CohenArthur
6e93171185
Better help for CLI arguments ( #56 )
...
* args: Add 'help' section
* benchmark: Add more data
* Update src/args.rs
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
* Update src/args.rs
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
2020-05-01 13:48:39 +02:00
Esteban Blanc
5233aecbc1
doc: Add documentation ( #54 )
...
* doc: Add documentation part 1
* doc: More doc
* doc: src/args.rs
Co-authored-by: CohenArthur <arthur.cohen@epita.fr>
* doc: src/downloader.rs
Co-authored-by: CohenArthur <arthur.cohen@epita.fr>
* doc: src/url_helper.rs
Co-authored-by: CohenArthur <arthur.cohen@epita.fr>
* doc: src/url_helper.rs
Co-authored-by: CohenArthur <arthur.cohen@epita.fr>
Co-authored-by: CohenArthur <arthur.cohen@epita.fr>
2020-04-30 16:25:53 +02:00
Esteban Blanc
fa28eb13ea
Refactor whole project ( #53 )
...
* refactor: Move `Response` into its own module
* coding-style: Use field shorthand
Use `tries` instead of `tries: tries`
* cleanup: Refactor whole project part 1
* cleanup: Refactor whole project part 2
Co-authored-by: cohenarthur <arthur.cohen@epita.fr>
2020-04-30 13:31:11 +02:00
Esteban Blanc
fa32b55c20
Update README.md
2020-04-30 00:07:31 +02:00
CohenArthur
f685b2284b
Add asciinema embed link ( #47 )
...
* readme: Add asciinema
* readme: Embed asciinema
* readme: Add script for asciinema
* readme: Embed instead of script
2020-04-30 00:04:51 +02:00
Esteban Blanc
e318cfba59
Merge pull request #46 from Skallwar/update_licenses
...
Update Licenses
2020-04-29 23:56:44 +02:00
Esteban Blanc
5f2819ea59
Create LICENSE-MIT
2020-04-29 23:53:17 +02:00
Esteban Blanc
520a278f3f
Rename LICENSE to LICENSE-APACHE
2020-04-29 23:51:14 +02:00
Esteban Blanc
24cff8a3d0
Update README.md
2020-04-29 23:49:02 +02:00
Esteban Blanc
8b4bed5832
Merge pull request #44 from Skallwar/add-logo-readme
...
Add Suckit Logo
2020-04-29 22:51:22 +02:00
Esteban Blanc
1264745379
Merge branch 'master' into add-logo-readme
2020-04-29 22:50:33 +02:00
cohenarthur
1ec12c4042
logo: Uncrop logo
2020-04-29 23:21:45 +02:00
cohenarthur
469170e334
logo: Crop image
2020-04-29 23:21:06 +02:00
cohenarthur
840da51851
Merge branch 'master' into add-logo-readme
2020-04-29 23:16:15 +02:00
cohenarthur
7ebbc7a8ab
logo: Add suckit logo
2020-04-29 23:14:45 +02:00
CohenArthur
c247a60824
Merge pull request #45 from Skallwar/clippy_fmt
...
misc: Run clippy and rustfmt
2020-04-29 22:37:33 +02:00
Esteban Blanc
bc76cd3c9a
Merge branch 'master' into clippy_fmt
2020-04-29 22:30:05 +02:00
Esteban Blanc
c2cc1ba34c
misc: Remove dead code
2020-04-29 22:29:24 +02:00
Esteban Blanc
79c9a5d4bb
misc: Run clippy and rustfmt
2020-04-29 22:24:26 +02:00
Esteban Blanc
88a5f6cd9f
Merge pull request #43 from Skallwar/prettify-readme
...
Prettify readme
2020-04-29 20:27:55 +02:00
