java - Why does Spark Streaming stop working when I send two input streams? -


i develping spark streaming application in need use input streams 2 servers in python, each sending json message per second spark context.

my problem is, if perform operations on 1 stream, works well. if have 2 streams different servers, spark freezes before can print anything, , starts working again when both servers have sent json messages had send (when detects 'sockettextstream not receiving data.

here code:

    javareceiverinputdstream<string> streamdata1 = ssc.sockettextstream("localhost",996,             storagelevels.memory_and_disk_ser);      javareceiverinputdstream<string> streamdata2 = ssc.sockettextstream("localhost", 9995,storagelevels.memory_and_disk_ser);      javapairdstream<integer, string> datastream1= streamdata1.maptopair(new pairfunction<string, integer, string>() {         public tuple2<integer, string> call(string stream) throws exception {               tuple2<integer,string> streampair= new tuple2<integer, string>(1, stream);              return streampair;         }     });      javapairdstream<integer, string> datastream2= streamdata2.maptopair(new pairfunction<string, integer, string>() {         public tuple2<integer, string> call(string stream) throws exception {               tuple2<integer,string> streampair= new tuple2<integer, string>(2, stream);              return streampair;         }     });  datastream2.print(); //for example 

notice there no error messages, spark simple freezes after starting context, , while json messages ports doesn't show anything.

thank much.

take @ these caveats spark streaming documentation , see if apply:

points remember

  • when running spark streaming program locally, not use “local” or “local1” master url. either of these means 1 thread used running tasks locally. if using input dstream based on receiver (e.g. sockets, kafka, flume, etc.), single thread used run receiver, leaving no thread processing received data. hence, when running locally, use “local[n]” master url, n > number of receivers run (see spark properties information on how set master).
  • extending logic running on cluster, number of cores allocated spark streaming application must more number of receivers. otherwise system receive data, not able process it.

Comments

Popular posts from this blog

javascript - Laravel datatable invalid JSON response -

java - Exception in thread "main" org.springframework.context.ApplicationContextException: Unable to start embedded container; -

sql server 2008 - My Sql Code Get An Error Of Msg 245, Level 16, State 1, Line 1 Conversion failed when converting the varchar value '8:45 AM' to data type int -