scraping/skallwar.suckit
1
0
Fork 0
mirror of https://github.com/Skallwar/suckit.git synced 2026-09-10 07:16:19 -04:00
Code Issues Actions Packages Projects Releases Wiki Activity
315 commits 5 branches 4 tags 1.2 MiB
Commit graph

315 commits

This branch
This branch
All branches
Author SHA1 Message Date
cohenarthur 74daf0bfcf Merge branch 'multithreaded-scraper' into multithreaded-fix-tests 2020-04-17 01:02:22 +02:00
cohenarthur f03b3e6198 WIP: Fix unit tests 2020-04-17 00:08:05 +02:00
Esteban Blanc ed84c8eba5
Merge pull request #35 from Skallwar/multithreaded-scraper-refactor 
Cleanup and Refactor scraper
 2020-04-17 00:02:06 +02:00
cohenarthur 82cfd340c8 scraper: Cleanup 2020-04-17 00:59:33 +02:00
Esteban Blanc 938999b7de
Merge pull request #33 from Skallwar/multithreaded-benchmark 
Get best result instead of previous result
 2020-04-16 19:59:40 +02:00
Esteban Blanc 6d6234c502 multithread: Readd depth level 2020-04-16 19:58:12 +02:00
cohenarthur 0f12dd5a6e benchmark: Get best result instead of previous result 2020-04-16 18:25:08 +02:00
Esteban Blanc 0e365db2b7
Merge pull request #32 from Skallwar/multithreaded-benchmark 
Multithreaded benchmark
 2020-04-16 17:05:15 +02:00
cohenarthur 203b6780ba typo: Remove fake results for testing 2020-04-16 16:58:22 +02:00
cohenarthur e66a217575 speed_regression: Add better output + mean computing 2020-04-16 16:46:51 +02:00
cohenarthur c1495bd802 scraper: Need to add 10 tries and mean to calculate time 2020-04-16 16:11:29 +02:00
cohenarthur b507ee5cdc speed_bench: Add work around benchmarks 2020-04-16 15:56:30 +02:00
Esteban Blanc dd9ee74a1d speed-test: Add print of improvement 2020-04-16 13:15:14 +02:00
cohenarthur d0890a920a speed_regression: WIP: Add base for speed benchmark 2020-04-16 13:25:52 +02:00
cohenarthur be42357caf scraper: Stop after 10 empty recv() 2020-04-16 13:21:00 +02:00
cohenarthur b6dba270d4 scraper: Refactor threaded code using thread::scope() 2020-04-16 12:48:14 +02:00
cohenarthur bee3cec831 multithreading: Add variable number of threads 2020-04-16 02:02:42 +02:00
cohenarthur 1e80a86e74 multithreading: Add framework for multithreading 
Add scraping with 2 threads instead of one !
 2020-04-16 01:15:33 +02:00
cohenarthur 7a97fd7564 scraper: WIP: start multithreading structure 2020-04-16 00:10:12 +02:00
cohenarthur c206cddba3 Merge branch 'master' into multithreaded-scraper 2020-04-15 22:53:11 +02:00
CohenArthur e39cd62719
Merge pull request #30 from Skallwar/depth_counter 
scraper: Add depth option
 2020-04-15 21:38:35 +02:00
cohenarthur 39d33d77ad scraper: Add unit test for recursive scraping 2020-04-15 21:38:10 +02:00
cohenarthur 3ffc171827 scraper: Separate run() into two 2020-04-15 21:31:53 +02:00
Esteban Blanc dbf1c74f10 scraper: Add depth option 2020-04-15 20:47:24 +02:00
Esteban Blanc 7f54c40315
Merge pull request #29 from Skallwar/incomplete_message_error 
downloader: Add a retry feature
 2020-04-14 22:54:05 +02:00
Esteban Blanc 863390805d downloader: Fix unit tests 2020-04-14 22:31:03 +02:00
Esteban Blanc 14e246c6fb downloader: Add a retry feature 2020-04-14 22:19:17 +02:00
Esteban Blanc a54e699e8f
Update README.md 2020-04-14 13:54:24 +02:00
Esteban Blanc 26ce62a84e
Merge pull request #26 from Skallwar/add-benchmark 
Add benchmarking 'framework'
 2020-04-14 13:53:44 +02:00
cohenarthur 18dad5d8d0 bench: Add URL arg and better default values 2020-04-14 12:41:00 +02:00
cohenarthur c6aed79591 Merge branch 'master' into add-benchmark 2020-04-14 12:18:53 +02:00
cohenarthur 942755d283 bench: Add option to give suckit path 2020-04-14 12:11:41 +02:00
cohenarthur ca2bf6e44e bench: Add argument parsing 2020-04-14 12:02:58 +02:00
cohenarthur a88a402dc4 bench: Clean directory afterwards 2020-04-14 11:50:45 +02:00
cohenarthur f5b2a2f434 bench: Add start of framework for benching suckit 2020-04-14 11:20:33 +02:00
Esteban Blanc 0c682541d0
Update README.md 2020-04-14 10:10:26 +02:00
Esteban Blanc a10cfbb813
Update README.md 2020-04-14 00:35:11 +02:00
Esteban Blanc d253cd9fc4
Merge pull request #22 from Skallwar/domtree_modif_url_to_path 
Dom: Modify link for offline navigation
 2020-04-14 00:34:47 +02:00
Esteban Blanc 54ec77d963
Merge branch 'master' into domtree_modif_url_to_path 2020-04-14 00:28:46 +02:00
Esteban Blanc 89335911ea
Update README.md 2020-04-14 00:27:55 +02:00
Esteban Blanc 1122a021e5 scrapper: Change domtree inplace and serialize it 2020-04-14 00:18:46 +02:00
Esteban Blanc 7051f311b3 dom: Add a serialize method 2020-04-14 00:13:03 +02:00
CohenArthur e3ceacf5ab
Merge pull request #17 from Skallwar/reference_domtree 
Dom returns &mut String
 2020-04-13 23:08:33 +02:00
Esteban Blanc 33a1975db8 parser: Remove parser, keep only Dom 2020-04-13 22:41:41 +02:00
Esteban Blanc 71de267fc0 disk: Expose url_to_path 2020-04-13 22:23:32 +02:00
Esteban Blanc e6d5801289 dom: Return a vec of &mut String insted of String 2020-04-13 22:22:22 +02:00
Esteban Blanc 1b22619191
Merge pull request #14 from Skallwar/update-reqwest 
Update reqwest to 0.10
 2020-04-13 11:14:27 +02:00
cohenarthur 18b209ef4f Merge branch 'master' into update-reqwest 2020-04-12 23:48:55 +02:00
cohenarthur 580910e74f reqwest: Update to 0.10 
Due to the use introduction of async with 0.10, we need to use
`reqwest::blocking::get()` instead of `reqwest::get()`
 2020-04-12 23:44:27 +02:00
Esteban Blanc beccc7a5cc
Merge pull request #13 from Skallwar/one-client-downloader 
Add single client to handle requests
 2020-04-12 22:46:17 +02:00