Mike Pone

United States

Age: 37

Big Data Engineer working in Spark, Pig, HBASE on the Cloudera stack. I also have experience in Enterprise Integration using OSGI, Apache Camel and Apache Active MQ. In my spare time, I chase my 5 kids around the house and take evening classes towards a Master's in Software Engineering.

2d
reviewed Approve suggested edit on I have written a program to login into yahoomail. but its giving me an error
2d
reviewed Approve suggested edit on Difference between ClassLoader.getSystemResourceAsStream and getClass().getResourceAsStream()
Feb
3
answered How to load multiline JSON in spark with Java
Feb
3
comment How to load multiline JSON in spark with Java
This requires fiddling with the original source which I was hoping not to have to do.
Feb
2
revised How to load multiline JSON in spark with Java
added example
Feb
2
comment How to load multiline JSON in spark with Java
@Dennis your definition of multiline is accurate. There are more than one json objects in this file
Feb
2
comment How to load multiline JSON in spark with Java
I have seen that if you have multiline JSON files, you will instead have to load the whole file and then parse each file. Does that mean parsing the file as JSON using some library and then getting those JSON records into an RDD somehow?
Feb
2
asked How to load multiline JSON in spark with Java
Dec
30
awarded Yearling
Dec
13
awarded Famous Question
Dec
1
accepted How to create RDD from within Task?
Dec
1
comment How to create RDD from within Task?
That answers my original question well, now a followup question. I am using JavaSparkContext.wholeTextFiles to read individual files from a directory. I need each of those files as their own JavaRDD<STring> to feed into spark ml algorithms, but wholeTextFiles returns JavaPairRDD<String, String> of (filename, content) and I want JavaPairRDD<String, JavaRDD<String>>. I thought it would be super easy to just create a new RDD in a map function, but alas, no... Any ideas on how to approach this?
Dec
1
asked How to create RDD from within Task?
Nov
23
awarded Constituent
Nov
23
awarded Caucus
Nov
10
answered Setting up eclipse for Pig
Nov
10
revised The output of my piglatin function is all messed up
not related to apache pig
Nov
10
comment Setting up eclipse for Pig
what error is eclipse giving you?
Nov
2
comment Apache PIG: Issue with UNION
Have you verified the counts of LOGS_BOOK_COL1 and LOGS_BOOK_COL2 just before the union?
Oct
30
answered How to know pig mode in grunt shell?
1 2 3 4 5