Passionate about programming, computer science, data-mining and new technologies in general. Always eager to learn, and happy to teach what I know.

Software engineer with a background in CS and AI, I'm currently working in the big data space with technologies such as Hadoop. I've joined the StackExchange network in an effort to improve my knowledge and also help others get the answers they're looking for.

2d
awarded Nice Answer
Aug
24
awarded Notable Question
Aug
21
awarded Popular Question
Aug
12
awarded Notable Question
Jul
30
awarded Good Question
Jul
21
awarded Popular Question
Jul
19
comment Reading SequenceFiles from S3 with PySpark on EMR causes RACK_LOCAL locality
This was because I didn't specify the -x option when installing Spark on EMR, so EMR doesn't configure anything for Spark if you don't specify that option somehow. The weird thing is that this option is not even indicated in their official doc ! But works perfectly after specifying it.
Jul
18
comment Reading SequenceFiles from S3 with PySpark on EMR causes RACK_LOCAL locality
@ChristopherB Oh, I was somehow under the impression that when setting up Spark on EMR it takes care of configuring the right number of executors in the config... I'll try setting up manually and see what happens, thanks for the suggestion !
Jul
18
comment Reading SequenceFiles from S3 with PySpark on EMR causes RACK_LOCAL locality
Also worth noting, I ran pyspark with --master yarn-client
Jul
18
revised Reading SequenceFiles from S3 with PySpark on EMR causes RACK_LOCAL locality
Added more details
Jul
18
asked Reading SequenceFiles from S3 with PySpark on EMR causes RACK_LOCAL locality
Jul
8
awarded Enlightened
Jun
27
awarded Guru
Jun
15
awarded Famous Question
May
22
accepted Python pandas - grouping by and summarizing on a field
May
22
comment Python pandas - grouping by and summarizing on a field
This is actually pretty cool, didn't know there was such an elegant way to do this, thanks !
May
22
asked Python pandas - grouping by and summarizing on a field
May
18
awarded Famous Question
May
18
awarded Popular Question
May
15
awarded Famous Question
1 2 3 4 5