performance - Best solution to host a crawler? -

-

i have crawler crawl few different domains new posts/content. total amount of content hundred of thousands of pages, , there lot of new content added each day. able crawl through content, need crawler crawling 24/7.

currently host crawler script on same server site crawler adding content to, , i'm able run cronjob run script during nighttime, because when do, website stops working because load of script. in other words, pretty crappy solution.

so wonder best option kind of solution?

  • is possible keep running crawler same host, somehow balancing load script doesnt kill website?

  • what kind of host/server looking host crawler? there other specifications need normal web host?

  • the crawler saves images crawls. if host crawler on secondary server, how save images on server of site? guess dont want chmod 777 on uploads-folder , allow put files on server.

i decided choose amazon web services host crawler both have sqs queues auto scalable instances. have s3 can store images.

i decided rewrite whole crawler python instead of php more take advantage of things such queues , keep app going 100% of time, instead of using cronjobs.

so did, , means

  1. i set elastic beanstalk application crawler set "worker" , listening sqs store domains need crawled. sqs "queue" can save each domain needs crawled, , crawler listen queue , fetch 1 domain @ time until queue done. there no need "cronjobs" or that, queue data it, send crawler. meaning crawler 100% of time, 24/7.

  2. the application set auto scaling, meaning when have many domains in queue, set second, third, fourth etc... instance/crawler speed process. think very important point wants set crawler.

  3. all images saved on s3 instance. means images not saved on server of crawler , can fetched , worked with.

the results have been great. when had php crawler running on cronjobs every 15min, crawl 600 urls per hour. can without problems crawl 10'000+ urls per hour, more depending on how set auto scaling.


Comments

Post a Comment

Popular posts from this blog

c# - How to get the current UAC mode -

-
can check current uac type set in device before running application through c#. i got know approach check whether current user administrator or not. but, need know uac type set (i.e. default/never notify) internal logic. is possible? please help. this article self-elevation , checking level program runs in. due security reasons i'm afraid won't able retrieve uac level set. http://code.msdn.microsoft.com/windowsdesktop/csuacselfelevation-644673d3
Read more

postgresql - Lazarus + Postgres: incomplete startup packet -

-
i've been building lazarus pascal program using postgresql on back-end. used work fine following lines opening db. dbconn:= tpqconnection.create(nil); dbconn.hostname := 'localhost'; dbconn.databasename:= 'dbhrs'; dbconn.username:='mizk'; dbconn.password:='123'; dbconn.open; if dbconn.connected openhrsdb := true else openhrsdb := false; but moment change localhost server's ip (on lan), program stops. no errors or warnings. have no clue happening. here related details may narrow down problem. the program i'm running workstation on same lan server. both machines run ubuntu 12.04 desktop i can access postgres db on server using pgadmin iii, guess it's not firewall issue. i have allowed postgresql ufw , ufw disabled. strangely, when firewall disabled, can ssh server terminal using workstation. when it's enabled, can't my biggest concern right why pascal program not working when i...
Read more

javascript - Ajax jqXHR.status==0 fix error -

-
$.ajax({ url: urlstring, datatype: "json", type: "get", success: function (data) { alert(data); }, error: function (jqxhr, exception) { if (jqxhr.status === 0) { alert('not connect.\n verify network.'); } else if (jqxhr.status == 404) { alert('requested page not found. [404]'); } else if (jqxhr.status == 500) { alert('internal server error [500].'); } else if (exception === 'parsererror') { alert('requested json parse failed.'); } else if (exception === 'timeout') { alert('time out error.'); } else if (exception === 'abort') { alert('ajax request aborted.'); } else { alert('uncaught error.\n' + jqxhr.responsete...
Read more