Python scraping with Mechanize, cookies (login) and proxies -

-

i scraping bad designed , managed government agricultural website application data requires me login , kicks me out , temporarily blocks ip (se asia, don't ask!)... trying use python , mechanize along proxy list ( http://proxy-hunter.blogspot.sg/2013/01/01-01-13-l1l2l3-http-proxies-1502.html ) work of scraping permits. however, can script login doesnot seem use proxies in set_proxies list... can why , suggest can fix it? found answer on stackoverflow answer "don't use mechanize"... well, got working script minus proxy aspect, isn't of helpful answer.

current code (reading ip site test whether ip changes or uses proxies - doesn't prints real ip):

import mechanize import cookielib beautifulsoup import beautifulsoup import html2text import urllib2  # mechanize browser/cookie stuff  br = mechanize.browser() cj = cookielib.lwpcookiejar() br.set_cookiejar(cj)   br.set_handle_equiv(true) br.set_handle_gzip(true) br.set_handle_redirect(true) br.set_handle_referer(true) br.set_handle_robots(false) br.set_handle_refresh(mechanize._http.httprefreshprocessor(), max_time=1) br.addheaders = [('user-agent', 'firefox')]  # proxies use  br.set_proxies({"84.2.35.44:80": "http", "1.62.68.201:6675": "http", "119.254.90.18:8080": "http"})  # testing whether proxy works checking ip  page = br.open('http://whatismyipaddress.com/') soup = beautifulsoup(br.response().read()) ip = soup.findall('div', {'style':'text-align:center;padding-top:4px;'}) print ip

so works in terminal comes real ip, rather of 3 proxies set.

finally, please consider in answer logging website in actual code, using sth like:

br.open('http://www.terriblegovernmentsite.hk/login.php') br.select_form(nr=3) br.form[login'user'] = 'researcher01' br.form[login'pass'] = 'mypassword' br.submit()

however, bit irrelevant since can't proxy work simple reading of static webpage, never mind whilst handling cookies.

thanks in advance help, appreciated.


Comments

Post a Comment

Popular posts from this blog

javascript - Laravel datatable invalid JSON response -

-
everything work fine on code below fetch user logs: js: <script> $(document).ready(function () { var table =$('#systemlogs').datatable({ responsive: true, processing: true, serverside: true, "language": { "url": "/datatables/media/plug-in/persian.json" }, ajax: '{!! url('/admin/systemlogs/data/systemlogsdatatable') !!}', columns: [ { data: 'name', name: 'name' }, { data: 'message_text', name: 'message_text' }, { data: 'remote_addr', name: 'remote_addr' }, { data: 'log_created_at', name: 'management_logs.created_at' }, { data: 'log_updated_at', name: 'management_logs.updated_at' }, ] }); }); </script> routes: route::get('/systemlogs/data/systemlogsdatatable','systemlogscontrolle...
Read more

java - Exception in thread "main" org.springframework.context.ApplicationContextException: Unable to start embedded container; -

-
i getting error when run spring application: exception in thread "main" org.springframework.context.applicationcontextexception: unable start embedded container; nested exception org.springframework.context.applicationcontextexception: unable start embeddedwebapplicationcontext due missing embeddedservletcontainerfactory bean. @ org.springframework.boot.context.embedded.embeddedwebapplicationcontext.onrefresh(embeddedwebapplicationcontext.java:124) @ org.springframework.context.support.abstractapplicationcontext.refresh(abstractapplicationcontext.java:476) @ org.springframework.boot.context.embedded.embeddedwebapplicationcontext.refresh(embeddedwebapplicationcontext.java:109) @ org.springframework.boot.springapplication.refresh(springapplication.java:691) @ org.springframework.boot.springapplication.run(springapplication.java:320) @ org.springframework.boot.springapplication.run(springapplication.java:952) @ org.springframework.boot.springa...
Read more

sql server 2008 - My Sql Code Get An Error Of Msg 245, Level 16, State 1, Line 1 Conversion failed when converting the varchar value '8:45 AM' to data type int -

-
msg 245, level 16, state 1, line 1 conversion failed when converting varchar value '8:45 am' data type int. script declare @specimenid bigint = 0, @script varchar(max); declare @starttime int = (select convert(int,substring(start_time,0,charindex(':', start_time)) - 12) appt appt._id=112601) --select @starttime set @script = 'select distinct ptlast_name + '', '' + pt.first_name patient_name, dbo.formatdate(pt.dob,''mm/dd/yyyy'') dob, (convert(varchar, (datediff(yy, dob, appt_date) - case when (month(dob) > month(appt_date)) or (month(dob) = month(appt_date) , day(dob) > day(appt_date)) 1 else 0 end)) + ''/'' + convert(varchar, (datediff(m, dateadd(yy, (datediff(yy, dob, appt_date) - case when (month(dob) > month(appt_date)) or (month(dob) = month(appt_date) , day(dob) > day(appt_date)) 1 else 0 end), d...
Read more