scraping/skallwar.suckit
1
0
Fork 0
mirror of https://github.com/Skallwar/suckit.git synced 2026-09-10 07:16:19 -04:00
Code Issues Actions Packages Projects Releases Wiki Activity
142 commits 5 branches 4 tags 1.2 MiB
Commit graph

142 commits

This branch
This branch
All branches
Author SHA1 Message Date
Esteban Blanc d2abf7722e scraper: Not working implementation of selective parsing 2020-04-28 15:48:45 +02:00
Esteban Blanc c730a93513 downloader: Create a Response type 
The Response type is used to encapsulate data needed in order to parse
(if necesary) and save a downloaded document inside the scraper
 2020-04-27 23:46:00 +02:00
Esteban Blanc 1f41bf709d scraper: Split handle_url and parse_html 2020-04-27 20:53:03 +02:00
Esteban Blanc fd0b7d4254 disk: Save from raw data 2020-04-27 16:59:55 +02:00
Esteban Blanc af4eb66c50
Merge pull request #38 from Skallwar/add-logging 
Add logging
 2020-04-18 16:42:03 +02:00
cohenarthur ed2f5ced95 Merge branch 'add-logging' of https://github.com/skallwar/suckit into add-logging 2020-04-18 16:31:25 +02:00
cohenarthur 3621c420c8 logger: Use reference instead of copy for write_log() 2020-04-18 16:30:18 +02:00
cohenarthur dc1d23f974 args: Change verbose to flag rather than option 2020-04-18 16:28:34 +02:00
cohenarthur 70f0acfca7 args: Change quiet to verbose since default behavior is quiet 2020-04-18 16:27:21 +02:00
cohenarthur fd44896c2b args: Fix typo defautl -> default 2020-04-18 16:25:12 +02:00
Esteban Blanc 478860b072
Update src/args.rs 2020-04-18 15:29:20 +02:00
CohenArthur 218861d7ff
scraper: Default behavior is quiet 
Co-Authored-By: Esteban Blanc <estblcsk@gmail.com>
 2020-04-18 15:23:37 +02:00
cohenarthur f5798e15d9 scraper: Add --quiet to unit tests 2020-04-18 15:58:09 +02:00
cohenarthur cc12b1fb6e logger: Use default time formatting instead 2020-04-18 15:56:56 +02:00
cohenarthur 2b84cb165a scraper: Use logger, add --quiet option 2020-04-18 15:52:36 +02:00
cohenarthur debe1eca64 Merge branch 'master' into add-logging 2020-04-18 15:45:05 +02:00
cohenarthur 9cba424acc logger: Add documentation 2020-04-18 15:43:11 +02:00
cohenarthur aa3b7e49ba logger: Add warn!, error! and info! 2020-04-18 15:30:33 +02:00
Esteban Blanc 182e77f99c
Merge pull request #37 from Skallwar/multithreaded-scraper 
Multithreaded scraper
 2020-04-18 14:36:39 +02:00
cohenarthur baade039db scraper: Tick multithreading box 2020-04-18 12:59:08 +02:00
cohenarthur f27ff19e43 scraper: Add depth test 2020-04-18 12:56:52 +02:00
cohenarthur 4534ed0237 scraper: Remove lazy_static 2020-04-18 12:53:34 +02:00
CohenArthur 25df65d24c
Merge pull request #36 from Skallwar/multithreaded-fix-tests 
Multithreaded fix tests
 2020-04-17 21:06:55 +02:00
Esteban Blanc 80d070eabc scraper: Reintegrate visited_urls and path_map inside Scraper 2020-04-17 20:58:54 +02:00
cohenarthur 780cde0073 scraper: Clear HashSet after every test 2020-04-17 12:21:42 +02:00
cohenarthur 7255a59f52 scraper-test: Clear VISITED_URLS between each test 2020-04-17 01:32:36 +02:00
cohenarthur c9ea9a5d11 scraper: WIP: Add HashSet for visited URLs 2020-04-17 01:20:10 +02:00
cohenarthur 74daf0bfcf Merge branch 'multithreaded-scraper' into multithreaded-fix-tests 2020-04-17 01:02:22 +02:00
cohenarthur f03b3e6198 WIP: Fix unit tests 2020-04-17 00:08:05 +02:00
Esteban Blanc ed84c8eba5
Merge pull request #35 from Skallwar/multithreaded-scraper-refactor 
Cleanup and Refactor scraper
 2020-04-17 00:02:06 +02:00
cohenarthur 82cfd340c8 scraper: Cleanup 2020-04-17 00:59:33 +02:00
Esteban Blanc 938999b7de
Merge pull request #33 from Skallwar/multithreaded-benchmark 
Get best result instead of previous result
 2020-04-16 19:59:40 +02:00
Esteban Blanc 6d6234c502 multithread: Readd depth level 2020-04-16 19:58:12 +02:00
cohenarthur 0f12dd5a6e benchmark: Get best result instead of previous result 2020-04-16 18:25:08 +02:00
Esteban Blanc 0e365db2b7
Merge pull request #32 from Skallwar/multithreaded-benchmark 
Multithreaded benchmark
 2020-04-16 17:05:15 +02:00
cohenarthur 203b6780ba typo: Remove fake results for testing 2020-04-16 16:58:22 +02:00
cohenarthur e66a217575 speed_regression: Add better output + mean computing 2020-04-16 16:46:51 +02:00
cohenarthur c1495bd802 scraper: Need to add 10 tries and mean to calculate time 2020-04-16 16:11:29 +02:00
cohenarthur b507ee5cdc speed_bench: Add work around benchmarks 2020-04-16 15:56:30 +02:00
Esteban Blanc dd9ee74a1d speed-test: Add print of improvement 2020-04-16 13:15:14 +02:00
cohenarthur d0890a920a speed_regression: WIP: Add base for speed benchmark 2020-04-16 13:25:52 +02:00
cohenarthur be42357caf scraper: Stop after 10 empty recv() 2020-04-16 13:21:00 +02:00
cohenarthur b6dba270d4 scraper: Refactor threaded code using thread::scope() 2020-04-16 12:48:14 +02:00
cohenarthur bee3cec831 multithreading: Add variable number of threads 2020-04-16 02:02:42 +02:00
cohenarthur 1e80a86e74 multithreading: Add framework for multithreading 
Add scraping with 2 threads instead of one !
 2020-04-16 01:15:33 +02:00
cohenarthur 7a97fd7564 scraper: WIP: start multithreading structure 2020-04-16 00:10:12 +02:00
cohenarthur c206cddba3 Merge branch 'master' into multithreaded-scraper 2020-04-15 22:53:11 +02:00
CohenArthur e39cd62719
Merge pull request #30 from Skallwar/depth_counter 
scraper: Add depth option
 2020-04-15 21:38:35 +02:00
cohenarthur 39d33d77ad scraper: Add unit test for recursive scraping 2020-04-15 21:38:10 +02:00
cohenarthur 3ffc171827 scraper: Separate run() into two 2020-04-15 21:31:53 +02:00