multithreading - IP switching over PHP crawler -


i looking solution switch ips automatically when running php crawler. have customized crawler runs 100 threads due throttle limits blocked. since php doesn't support multi-threading have set windows scheduler run php application in parallel.

i assign each thread different ip address , welcome suggestion overcome issue.

multi-threading possible in php

some might call trivial ...

<?php define('log', mutex::create());  /* make output when writing stdout thread safe (so, readable) */ function slog($message, $args = []) {     $args = func_get_args();     if (($message = array_shift($args))) {         mutex::lock(log);         echo vsprintf($message, $args);         mutex::unlock(log);     } }  class webcrawler extends thread {      public function __construct($interface) {         $this->interface = $interface;     }      public function run() {         slog("thread %lu using %s\n",              $this->getthreadid(), $this->getinterface());     }      public function getinterface() {          return $this->interface;      }      protected $interface; }  $interfaces = [     "192.168.0.1",     "192.168.0.2",     "192.168.0.3",     "192.168.0.4",     "192.168.0.5" ];  $threads = []; $thread = 0; while (count($threads) < count($interfaces)) {     $threads[$thread] = new webcrawler($interfaces[$thread]);     $threads[$thread]->start();     $thread++; }  foreach ($threads $thread)     $thread->join();  mutex::destroy(log); ?>

the code above gives interface per thread pre-defined list of interfaces. configure clients use interface set thread using curlopt_interface or other magic.

further reading:


Comments

Post a Comment

Popular posts from this blog

Why can rails not find a route created by a helper? -

i created member route in rails 4: resources :line_items post 'decrement', on: :member end and gave matching method in line_items controller: def decrement @cart = current_cart @line_item = @cart.line_items.find_by_id(params[:id]) @line_item.decrement_quantity respond_to |format| if @line_item.save format.html { redirect_to shop_path, notice: 'line item updated.' } format.js {@current_item = @line_item} format.json { head :ok } else format.html { render action: "edit" } format.js {@current_item = @line_item} format.json { render json: @line_item.errors, status: :unprocessable_entity } end end end but when try make button: <%= button_to 'x', decrement_line_item_path(item) %> i error: no route matches [post] "/carts/25" what gives? your error message is: no route matches [post] "/carts/25" but expectin...
Read more

javascript - jquery or ashx not working -

i using ajax file upload control. it creating html: <form action="/pages/handlers/asyncfileuploadhandler.ashx" method="post" enctype="multipart/form-data" target="ajaxuploader-iframe-1397144545"> <input type="hidden" name="type" value="mindvoice"> <input type="file" id="imagefile" style="display: block; width: 80%; margin-right: auto; margin-left: auto;"> </form> and target is: <iframe width="0" height="0" style="display:none;" name="ajaxuploader-iframe-1397144545" id="ajaxuploader-iframe-1397144545"></iframe> the form submitted using .submit() . my ashx handler: <%@ webhandler language="c#" class="asyncfileuploadhandler" %> using system; using system.web; using nettpals.core; public class asyncfileuploadhandler : ihttphandler { public void processre...
Read more

opencv - DataType<cv::detail::deriv_type>::depth what is it used for -

i wondering why line used in lucas kanade tracker in opencv: datatype<cv::detail::deriv_type>::depth can explain me? in opencv, depth of mat refers type of data contained in mat 's data buffer. represented integer values correspond given data type. these integers commonly abstracted appropriate macro definition (e.g. uchar data represented macro cv_8u ). cv::datatype type-traits class provides method obtain corresponding integer value without having memorize macro means data type. there few cases user code needs use datatype::depth . more common datatype::type . a simple example shows 1 possible use of datatype::depth : cv::mat uchar_data = cv::mat::ones(3, 3, cv_8uc1); cv::mat float_data; uchar_data.convertto(float_data, cv::datatype<float>::depth); // ^^ equivalently replaced // cv_32f macro float_data.at<float>(0,1) += 0.5f; std::cout << float_data << std::endl;...
Read more