tag:blogger.com,1999:blog-5853964016719400932.post1500388700692039740..comments2009-12-10T14:23:57.820-08:00The Official Cruxlux Bloghttp://www.blogger.com/profile/09369467704623503662noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-5853964016719400932.post-36652617419922740802009-02-02T17:27:31.526-08:002009-02-02T17:27:31.526-08:00Anonymous... <BR/><BR/>You are correct that cURL provides just the download mechanism. We essentially have a global queue that producers enqueue urls to crawl based on how often they are updated and workers that pull from that queue and do the actual download/parse/enqueue of new urls. As you mentioned, the implementation of this has a lot of interesting hurdles.The Official Cruxlux Bloghttp://www.blogger.com/profile/09369467704623503662noreply@blogger.comtag:blogger.com,1999:blog-5853964016719400932.post-2115397155374706172009-02-02T17:20:00.000-08:002009-02-02T17:20:00.000-08:00Nice information! The crawler itself, did you write it yourself? I know the crawler concept is quite simple, but some implementation details regarding robustness/efficiency/scalability can make it quite interesting...<BR/><BR/>p.s.: AFAIK cURL provides you just the "download" mechanism, right?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5853964016719400932.post-90471622491645860002008-10-15T09:26:37.362-07:002008-10-15T09:26:37.362-07:00intriguing stuff. libev is also worth looking at as an event loop.Anonymousnoreply@blogger.com