Matti Lyra

Berlin, Germany

Age: 34

I am a PhD student at the University of Sussex studying Computational Linguistics. My research topic is text classification in imbalanced data streams.

1d
comment How to load Spark Cassandra Connector in the shell?
You can actually also just provide the connection host using --conf spark.cassandra.connection.host=127.0.0.1 when launching the shell spark.apache.org/docs/latest/…
1d
comment How to load Spark Cassandra Connector in the shell?
please see @chris-batey answer below - it's the only way to use sc.broadcast in the shell. If you stop the SparkContext and create a new one, you will get a NotSerialisableException if you try to use sc.broadcast.
1d
comment How to load Spark Cassandra Connector in the shell?
bravo - this is actually the only correct way I've found of connecting to cassandra. All of the above result in the sc not being serialisable and broadcast variables not working in the spark-shell.
2d
awarded Yearling
May
2
comment scala companion object templates (Iterator.tabulate)
ah, I knew it had to be something simple like that. I thought the O and T were just showing it's both an object and a trait. Didn't even try clicking them, not the best UI design ever. Also just noticed that the top right corner has related documentation.
May
2
accepted scala companion object templates (Iterator.tabulate)
May
2
comment What's the best tool to speed up a CPU bound computation in my Python code?
My approach has usually been to go for process level parallelism if/when the arrays can be split into discreet portions per process, and cython if they can't. You can get rid of a lot of unnecessary type checks and python VM calls with cython so I would look at there.
May
2
comment What's the best tool to speed up a CPU bound computation in my Python code?
No knowing what exactly f does it's difficult to say, but assuming f is the bottleneck I would look at cython. Another slightly easier route would be to try to vectorize the operation using numpy but that really depends on the internals of f
May
2
asked scala companion object templates (Iterator.tabulate)
May
2
comment Understanding GenericTraversableTemplate and other Scala collection internals
Great answer. This somehow reminds me of python's duck typing and what you need to implement that but with an actual type system.
May
2
revised Understanding GenericTraversableTemplate and other Scala collection internals
added backticks for readability
May
2
comment RowMatrix, MLlib, Java Spark
I think the API has changed, toBreeze seems to have been moved to linalg.Matrices and made private to that package. I can't find a way to convert RowMatrix to a breeze matrix.
Apr
20
awarded Notable Question
Apr
5
revised Efficient way to clone a generic object in Java preferably without a factory
deleted 16 characters in body
Mar
17
awarded Popular Question
Feb
1
comment "Missing number, treated as zero" error adding subfigures
I've just discovered that there's a package called nag that will do that for you ctan.org/tex-archive/macros/latex/contrib/nag
Dec
13
awarded Famous Question
Nov
24
comment Apply Number formatting to Pandas HTML CSS Styling
stick to using just the formatters for now - I don't see a way to work around this issue for now - I guess the .bar and highlight_min, highlight_max etc. should all take kwargs for the column data formats, could you file an issue for that on github.com/pydata/pandas so that the maintainers can track it, maybe link the PR from my answer to the issue as well.
Nov
24
answered Apply Number formatting to Pandas HTML CSS Styling
Oct
31
comment Pandas: change data type of columns
.convert_objects is depracated since 0.17 - use df.to_numeric instead
1 2 3 4 5