Eran Medan

United States

linkedin.com/in/eranmedan

Age: 37

Tech Lead at NICE Actimize

  • Java, Scala, JavaScript, and all that is web
  • Data mining and fraud detection
14h
comment Understanding Spark's caching
Yep, seems you are on it. This is a little unfortunate, I wish "cache" would have been converted to a DAG operation and not just add the RDD ID to a map... there are plenty of cases where you want to cache something intermediately, create a new RDD, then drop the old one. Perhaps there are good theoretical reasons on why this is not a good idea though... in any case, the LRU (I assume) ordering of the cache means that the old unused rdd1 will be evicted if rdd2 and rdd3 need that space for caching...
14h
accepted Understanding Spark's caching
16h
asked Understanding Spark's caching
1d
awarded Popular Question
1d
comment Can I add class fields dynamically using macros?
Yes, this is what I actually use now, but I want it to be used in auto complete. also in dynamic I won't have a compile time error if the user of the API specifies a missing property on x.
2d
asked Can I add class fields dynamically using macros?
Apr
22
comment How do I upload my uber-jar to a spark cluster created using the spark-ec2 scripts?
Yes, I know :) but the jar is on s3, and I don't want to configure the master's AWS keys every time... I found out though that I can use sc.addJar and provide an hdfs or s3 location! but now my issue is that --copy-aws-credentials stopped working!
Apr
22
asked How do I upload my uber-jar to a spark cluster created using the spark-ec2 scripts?
Apr
20
asked Idiomatic way to create a spark context with a default master
Apr
20
awarded Nice Answer
Apr
20
awarded Notable Question
Apr
19
awarded Announcer
Apr
16
awarded Popular Question
Apr
2
awarded Supporter
Mar
31
awarded Good Question
Mar
30
awarded Nice Question
Mar
27
asked What is the right way to represent an "Any" type in Spark SQL?
Mar
27
asked What is the right Date/Datetime format in JSON for Spark SQL to automatically infer the schema for it?
Mar
27
awarded Famous Question
Mar
25
awarded Good Answer
1 2 3 4 5