Python scraping with Mechanize, cookies (login) and proxies -

-

i scraping bad designed , managed government agricultural website application data requires me login , kicks me out , temporarily blocks ip (se asia, don't ask!)... trying use python , mechanize along proxy list ( http://proxy-hunter.blogspot.sg/2013/01/01-01-13-l1l2l3-http-proxies-1502.html ) work of scraping permits. however, can script login doesnot seem use proxies in set_proxies list... can why , suggest can fix it? found answer on stackoverflow answer "don't use mechanize"... well, got working script minus proxy aspect, isn't of helpful answer.

current code (reading ip site test whether ip changes or uses proxies - doesn't prints real ip):

import mechanize import cookielib beautifulsoup import beautifulsoup import html2text import urllib2  # mechanize browser/cookie stuff  br = mechanize.browser() cj = cookielib.lwpcookiejar() br.set_cookiejar(cj)   br.set_handle_equiv(true) br.set_handle_gzip(true) br.set_handle_redirect(true) br.set_handle_referer(true) br.set_handle_robots(false) br.set_handle_refresh(mechanize._http.httprefreshprocessor(), max_time=1) br.addheaders = [('user-agent', 'firefox')]  # proxies use  br.set_proxies({"84.2.35.44:80": "http", "1.62.68.201:6675": "http", "119.254.90.18:8080": "http"})  # testing whether proxy works checking ip  page = br.open('http://whatismyipaddress.com/') soup = beautifulsoup(br.response().read()) ip = soup.findall('div', {'style':'text-align:center;padding-top:4px;'}) print ip

so works in terminal comes real ip, rather of 3 proxies set.

finally, please consider in answer logging website in actual code, using sth like:

br.open('http://www.terriblegovernmentsite.hk/login.php') br.select_form(nr=3) br.form[login'user'] = 'researcher01' br.form[login'pass'] = 'mypassword' br.submit()

however, bit irrelevant since can't proxy work simple reading of static webpage, never mind whilst handling cookies.

thanks in advance help, appreciated.


Comments

Post a Comment

Popular posts from this blog

wireshark - USB mapping with python -

-
while reading ctf write-ups came across script #!/usr/bin/env python import struct import image import dpkt init_x, init_y = 100, 400 def print_map(pcap, device): picture = image.new("rgb", (1200, 500), "white") pixels = picture.load() x, y = init_x, init_y ts, buf in pcap: device_id, = struct.unpack("b", buf[0x0b]) if device_id != device: continue data = struct.unpack("bbbb", buf[-4:]) status = data[0] x = x + data[1] y = y + data[2] if (status == 1): in range(-5, 5): j in range(-5, 5): pixels[x + , y + j] = (0, 0, 0, 0) else: pixels[x, y] = (255, 0, 0, 0) picture.save("riverside-map.png", "png") if __name__ == "__main__": f = open("usb.pcap", "rb") pcap = dpkt.pcap.reader(f) print_map(pcap, 5) f.close() and ...
Read more

c++ - nodejs socket.io closes connection before upgrading to websocket -

-
i'm trying write c++ client talk websocket server written using nodejs , socket.io. can't c++ client establish websocket connection; server drops connection right after client requests http connection upgraded websocket. i used tcpdump watch packets going , forth. when use websocket server browser, works great , tcpdump output shows packets going , forth. when try connect server using c++ client, websocket connection never gets established. output tcpdump shows upgrade request being sent server followed server dropping connection. i tried replicating sequence of events using nc . connected server (running on port 8080) , copy-pasted exact set of headers sent browser, captured tcpdump. duplicate of working headers, connection dropped. there no output on server end indicating why connection gets dropped. here headers sent: get /socket.io/?type=player&eio=3&transport=websocket&sid=7mdclht_8ctsd6f4aaac http/1.1 host: localhost:8080 connection: upgrade prag...
Read more

Deploying Qt Application on Android is really slow? -

-
as may know there 3 ways deploy qt application on android : use ministro service install qt deploy local qt libraries temporary directory bundle qt libraries in apk the first method takes 30 seconds , needs install apk . ministro . second takes 1 minute me ! , anytime try run program qt creator pushes qt libraries device. third 1 makes .apk file big , again takes 1 minute me. think situation that's not reasonable develop android application using qt. there way make deploying process faster? almost full year since op , things have not changed @ all. deployment of 7 mb apk takes on minute , half project compiles in 5 seconds. reason answering not problem got resolved, offer alternative solution. i've implemented "workaround" consisting of 2 applications work in tandem - 1 on pc , 1 on device - created compile files remotely, turned out faster alternative deployment well. on host create application launches compilation in separate process, when do...
Read more