asp.net - PYTHON: Submitting queries in APSX, and scraping results from aspx pages -

-

i want scrap info people in "http://www.ratsit.se/bc/searchperson.aspx", doing following code written:

import urllib bs4 import beautifulsoup  headers = { 'accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'origin': 'http://www.ratsit.se', 'user-agent': 'mozilla/5.0 (windows nt 6.1) applewebkit/537.17 (khtml, gecko)  chrome/24.0.1312.57 safari/537.17', 'content-type': 'application/x-www-form-urlencoded', 'referer': 'http://www.ratsit.se/', 'accept-encoding': 'gzip,deflate,sdch', 'accept-language': 'en-us,en;q=0.8', 'accept-charset': 'iso-8859-1,utf-8;q=0.7,*;q=0.3' }  class myopener(urllib.fancyurlopener): version = 'mozilla/5.0 (windows nt 6.1) applewebkit/537.17 (khtml, gecko)     chrome/24.0.1312.57 safari/537.17'  myopener = myopener() url = 'http://www.ratsit.se/bc/searchperson.aspx' # first http request without form data f = myopener.open(url) soup = beautifulsoup(f) # parse , retrieve 2 vital form values viewstate = soup.select("#__viewstate")[0]['value'] #eventvalidation = soup.select("#__eventvalidation")[0]['value']  formdata = (  ('__lastfocus',''), ('__eventtarget',''), ('__eventargument',''), #('__eventvalidation', eventvalidation), ('__viewstate', viewstate), ('ctl00$cphmain$txtfirstname', 'name'),  ('ctl00$cphmain$txtlastname', ''),   ('ctl00$cphmain$txtbirthdate', ''),                                                          # etc. (not listed) ('ctl00$cphmain$txtaddress', ''),    ('ctl00$cphmain$txtzipcode', ''),   ('ctl00$cphmain$txtcity', ''),   ('ctl00$cphmain$txtkommun',''), #('btnsearchajax','sök'), )  encodedfields = urllib.urlencode(formdata)  # second http request form data f = myopener.open(url, encodedfields)  try:  # we'd better use beautifulsoup once again  # retrieve results(instead of writing out whole html file)  # besides, since result split multipages,  # need send more http requests  fout = open('tmp.html', 'w') except:  print('could not open output file\n')  fout.writelines(f.readlines())  fout.close()

i getting response server "my ip block" not true cause when i'm doing browser working... suggest going wrong..

thanks

your code not work.

  file "/users/florianoswald/git/webscraper/scrape2.py", line 16   version = 'mozilla/5.0 (windows nt 6.1) applewebkit/537.17 (khtml, gecko)     chrome/24.0.1312.57 safari/537.17'       ^   indentationerror: expected indented block

is supposed class definition? why need myopener class anyway? works well:

myopener = urllib.fancyurlopener() my.open("http://www.google.com") <addinfourl @ 4411860752 fp = <socket._fileobject object @ 0x106ed1c50>>

Comments

Post a Comment

Popular posts from this blog

c# - How to get the current UAC mode -

-
can check current uac type set in device before running application through c#. i got know approach check whether current user administrator or not. but, need know uac type set (i.e. default/never notify) internal logic. is possible? please help. this article self-elevation , checking level program runs in. due security reasons i'm afraid won't able retrieve uac level set. http://code.msdn.microsoft.com/windowsdesktop/csuacselfelevation-644673d3
Read more

postgresql - Lazarus + Postgres: incomplete startup packet -

-
i've been building lazarus pascal program using postgresql on back-end. used work fine following lines opening db. dbconn:= tpqconnection.create(nil); dbconn.hostname := 'localhost'; dbconn.databasename:= 'dbhrs'; dbconn.username:='mizk'; dbconn.password:='123'; dbconn.open; if dbconn.connected openhrsdb := true else openhrsdb := false; but moment change localhost server's ip (on lan), program stops. no errors or warnings. have no clue happening. here related details may narrow down problem. the program i'm running workstation on same lan server. both machines run ubuntu 12.04 desktop i can access postgres db on server using pgadmin iii, guess it's not firewall issue. i have allowed postgresql ufw , ufw disabled. strangely, when firewall disabled, can ssh server terminal using workstation. when it's enabled, can't my biggest concern right why pascal program not working when i...
Read more

javascript - Ajax jqXHR.status==0 fix error -

-
$.ajax({ url: urlstring, datatype: "json", type: "get", success: function (data) { alert(data); }, error: function (jqxhr, exception) { if (jqxhr.status === 0) { alert('not connect.\n verify network.'); } else if (jqxhr.status == 404) { alert('requested page not found. [404]'); } else if (jqxhr.status == 500) { alert('internal server error [500].'); } else if (exception === 'parsererror') { alert('requested json parse failed.'); } else if (exception === 'timeout') { alert('time out error.'); } else if (exception === 'abort') { alert('ajax request aborted.'); } else { alert('uncaught error.\n' + jqxhr.responsete...
Read more