* Fix querystring support
* While technically valid filenames, having ? in filenames means that the directory structure cannot be served by nginx as the resulting filenames cannot be addressed via URLs unless the ? is percent-encoded in every link.
* Query strings can contain trailing slashes. If that happens, suckit will attempt to create a directory, then create a same-named file, which fails
* fix fmt
* tests: Add script to set up local server
* tests: Remove old results from speed.csv since test methodology changes
* tests: Change speed_regression so it downloads locally hosted webserver
* local_server_setup: Remove need of sudo since suckit understands ports
* speed_regression: Start local web server in script
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
* scraper: Initial work for other charset support
Not working
* scraper: Get charset from html instead of http headers
Not working
* scraper: Charset working
* scraper: Fix regex ('"' after equals)
* scraper: Use lazy_static for charset regex
* misc: Refactor pr
* tests: Refactor fixtures
* tests: Add html charset support tests
* tests: Charset, split tests
* downloader: Get charset from http headers
* misc: Clarify iterator first
* Apply suggestions from CohenArthur
Co-authored-by: CohenArthur <arthur.cohen@epita.fr>
* Allow passing user agent via cli args
* user-agent: Short and long CLI arguments
Co-authored-by: creativcoder <rahul.sharma@atherenergy.com>
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
* Add a random delay of 2 - 7 seconds between downloads
* Add command line args for delay and move delay to method
* Ignore .idea folder
* Update test for scraper to work with new low and high arguments
* Add a comment
* Remove unnecessary `&` for method call to self
* Rename `low` and `high` to `--delay` and `--random-range`
* Test different values for new arguments
* rename variable for consistency
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
* change help for clarity purposes
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
* Some whitespace
Co-authored-by: CohenArthur <arthur.cohen@epita.fr>
* Some whitespace
Co-authored-by: CohenArthur <arthur.cohen@epita.fr>
* rename variable for consistency
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
* Fix variable names
Co-authored-by: Esteban Blanc <estblcsk@gmail.com>
Co-authored-by: CohenArthur <arthur.cohen@epita.fr>
* refactor: Move `Response` into its own module
* coding-style: Use field shorthand
Use `tries` instead of `tries: tries`
* cleanup: Refactor whole project part 1
* cleanup: Refactor whole project part 2
Co-authored-by: cohenarthur <arthur.cohen@epita.fr>
