scraping/skallwar.suckit
1
0
Fork 0
mirror of https://github.com/Skallwar/suckit.git synced 2026-09-10 07:16:19 -04:00
Code Issues Actions Packages Projects Releases Wiki Activity
315 commits 5 branches 4 tags 1.2 MiB
Commit graph

315 commits

This branch
This branch
All branches
Author SHA1 Message Date
cohenarthur 7ebbc7a8ab logo: Add suckit logo 2020-04-29 23:14:45 +02:00
CohenArthur c247a60824
Merge pull request #45 from Skallwar/clippy_fmt 
misc: Run clippy and rustfmt
 2020-04-29 22:37:33 +02:00
Esteban Blanc bc76cd3c9a
Merge branch 'master' into clippy_fmt 2020-04-29 22:30:05 +02:00
Esteban Blanc c2cc1ba34c misc: Remove dead code 2020-04-29 22:29:24 +02:00
Esteban Blanc 79c9a5d4bb misc: Run clippy and rustfmt 2020-04-29 22:24:26 +02:00
Esteban Blanc 88a5f6cd9f
Merge pull request #43 from Skallwar/prettify-readme 
Prettify readme
 2020-04-29 20:27:55 +02:00
cohenarthur 781befd3e0 Merge branch 'prettify-readme' of github.com:skallwar/suckit into prettify-readme 2020-04-29 20:37:53 +02:00
cohenarthur 8f6c32b632 readme: Switch from "|" separator to ", " 2020-04-29 20:36:50 +02:00
cohenarthur 6e0ac9527d readme: Add contributing links 2020-04-29 20:33:08 +02:00
cohenarthur a69ff75645 readme: Close unclosed backtick 2020-04-29 20:26:51 +02:00
cohenarthur a3b535ff85 readme: Escape pipe separators for short/long options 2020-04-29 20:26:05 +02:00
cohenarthur be1b6f07a6 readme: Add options and example 2020-04-29 20:25:06 +02:00
CohenArthur 0107f6559f
Merge branch 'master' into prettify-readme 2020-04-29 19:34:34 +02:00
Esteban Blanc 70edc12515
Update README.md 2020-04-29 19:18:33 +02:00
Esteban Blanc de68eae98d
Merge pull request #42 from Skallwar/split_ci 
ci: Split ci in two jobs
 2020-04-29 19:11:29 +02:00
Esteban Blanc 358b03af46 ci: Split ci in two jobs 2020-04-29 18:54:33 +02:00
CohenArthur e5baa004da
Merge pull request #41 from Skallwar/selective_html_parsing 
Feature: Selective html parsing
 2020-04-29 17:44:39 +02:00
CohenArthur bd2ea429a6
Merge branch 'master' into selective_html_parsing 2020-04-29 16:35:30 +02:00
Esteban Blanc 29ccb39cd4
Merge pull request #39 from Skallwar/fix-typo-encounter 
Fix encounter -> encountered
 2020-04-29 12:12:34 +02:00
Esteban Blanc dfb37b58b2 misc: Fix all warnings 2020-04-29 12:10:50 +02:00
Esteban Blanc 93c5241bf0 test: Remove unit test that need to be functional tests 2020-04-29 12:07:30 +02:00
Esteban Blanc 4115ba870d scraper: Selective parsing fully working 2020-04-29 11:24:24 +02:00
Esteban Blanc d2abf7722e scraper: Not working implementation of selective parsing 2020-04-28 15:48:45 +02:00
cohenarthur 7c0054cf7e typo: Fix encounter -> encountered 2020-04-27 23:51:05 +02:00
Esteban Blanc c730a93513 downloader: Create a Response type 
The Response type is used to encapsulate data needed in order to parse
(if necesary) and save a downloaded document inside the scraper
 2020-04-27 23:46:00 +02:00
Esteban Blanc 1f41bf709d scraper: Split handle_url and parse_html 2020-04-27 20:53:03 +02:00
Esteban Blanc fd0b7d4254 disk: Save from raw data 2020-04-27 16:59:55 +02:00
Esteban Blanc af4eb66c50
Merge pull request #38 from Skallwar/add-logging 
Add logging
 2020-04-18 16:42:03 +02:00
cohenarthur ed2f5ced95 Merge branch 'add-logging' of https://github.com/skallwar/suckit into add-logging 2020-04-18 16:31:25 +02:00
cohenarthur 3621c420c8 logger: Use reference instead of copy for write_log() 2020-04-18 16:30:18 +02:00
cohenarthur dc1d23f974 args: Change verbose to flag rather than option 2020-04-18 16:28:34 +02:00
cohenarthur 70f0acfca7 args: Change quiet to verbose since default behavior is quiet 2020-04-18 16:27:21 +02:00
cohenarthur fd44896c2b args: Fix typo defautl -> default 2020-04-18 16:25:12 +02:00
Esteban Blanc 478860b072
Update src/args.rs 2020-04-18 15:29:20 +02:00
CohenArthur 218861d7ff
scraper: Default behavior is quiet 
Co-Authored-By: Esteban Blanc <estblcsk@gmail.com>
 2020-04-18 15:23:37 +02:00
cohenarthur f5798e15d9 scraper: Add --quiet to unit tests 2020-04-18 15:58:09 +02:00
cohenarthur cc12b1fb6e logger: Use default time formatting instead 2020-04-18 15:56:56 +02:00
cohenarthur 2b84cb165a scraper: Use logger, add --quiet option 2020-04-18 15:52:36 +02:00
cohenarthur debe1eca64 Merge branch 'master' into add-logging 2020-04-18 15:45:05 +02:00
cohenarthur 9cba424acc logger: Add documentation 2020-04-18 15:43:11 +02:00
cohenarthur aa3b7e49ba logger: Add warn!, error! and info! 2020-04-18 15:30:33 +02:00
Esteban Blanc 182e77f99c
Merge pull request #37 from Skallwar/multithreaded-scraper 
Multithreaded scraper
 2020-04-18 14:36:39 +02:00
cohenarthur baade039db scraper: Tick multithreading box 2020-04-18 12:59:08 +02:00
cohenarthur f27ff19e43 scraper: Add depth test 2020-04-18 12:56:52 +02:00
cohenarthur 4534ed0237 scraper: Remove lazy_static 2020-04-18 12:53:34 +02:00
CohenArthur 25df65d24c
Merge pull request #36 from Skallwar/multithreaded-fix-tests 
Multithreaded fix tests
 2020-04-17 21:06:55 +02:00
Esteban Blanc 80d070eabc scraper: Reintegrate visited_urls and path_map inside Scraper 2020-04-17 20:58:54 +02:00
cohenarthur 780cde0073 scraper: Clear HashSet after every test 2020-04-17 12:21:42 +02:00
cohenarthur 7255a59f52 scraper-test: Clear VISITED_URLS between each test 2020-04-17 01:32:36 +02:00
cohenarthur c9ea9a5d11 scraper: WIP: Add HashSet for visited URLs 2020-04-17 01:20:10 +02:00