Data dodania Pytanie
2017-01-25 16:01 Dropping empty DataFrame partitions in Apache Spark »
I try to repartition a DataFrame according to a columnm the the DataFrame has N (let say N=3) different values in the partition-column x, e.g: val my...
(0) odpowiedzi
2017-01-25 15:01 Spark rdd.count() yields inconsistent results »
I'm a bit baffled. A simple rdd.count() gives different results when run multiple times. Here is the code i run: val inputRdd = sc.newAPIHadoopRDD(...
(0) odpowiedzi
2017-01-24 14:01 Spark DataFrame - drop null values from column »
Given a dataframe : val df = sc.parallelize(Seq(("foo", ArrayBuffer(null,"bar",null)), ("bar", ArrayBuffer("one","two",null)))).toDF("key", "valu...
(2) odpowiedzi
2017-01-24 12:01 Scala anonymous function syntax and return type »
I have found few kinds of anonymous function syntax in scala: val m5_1 = { (n: Int) => n * 5 } val m5_2 = (n: Int) => { n * 5 } : Int val m5_...
(3) odpowiedzi
2017-01-24 04:01 Derived classes as class constructor parameters »
[EDITED BELOW] I have a class hierarchy that is, right now, roughly as follows: sealed trait Foo { def method: Any } case class Bar extends Foo ...
(2) odpowiedzi
2017-01-23 20:01 Why do implicit value classes have an extra method invocation? »
I was inspecting the byte-code generated by implicit classes and wanted to compare to what is generated when they extend AnyVal. Without implicit: o...
(1) odpowiedzi
2017-01-23 17:01 Why can you not flatMap a Set when the elements are wildcarded but you can flatMap a Seq? »
Why won't the Scala compiler let you flatMap a Set[Set[_ <: SomeType]] but will let you flatMap a Seq[Seq[_ <: SomeType]]? This behaviour is dem...
(1) odpowiedzi
2017-01-23 13:01 Passing a map with struct-type key into a Spark UDF »
I want to write an Spark 1.6 UDF which takes the following map: case class MyRow(mapping: Map[(Int, Int), Double]) val data = Seq( MyRow(Map((1, 1...
(1) odpowiedzi
2017-01-23 01:01 Select column by name with multiple aggregate columns after pivot with Spark Scala »
I am trying to aggregate multitple columns after a pivot in Scala Spark 2.0.1: scala> val df = List((1, 2, 3, None), (1, 3, 4, Some(1))).toDF("a",...
(1) odpowiedzi
2017-01-22 03:01 Value Class w/ String Inner Type? »
Given the following function: def prefixDr(firstName: String, lastName: String): String = "Dr. " + firstName + " " + lastName Let's say I deci...
(2) odpowiedzi
2017-01-22 01:01 Call-by-name in Scala vs lazy evaluation in Haskell? »
Haskell's lazy evaluation will never take more evaluation steps than the eager evaluation. On the other hand, Scala's call-by-name evaluation may req...
(3) odpowiedzi
2017-01-22 01:01 Predef.StringCanBuildFrom's behaviour »
I stumbled accross Predef.StringCanBuildFrom surprising implementation that breaks assumptions I was making on CanBuildFrom in my code. Here's the imp...
(1) odpowiedzi
2017-01-21 15:01 Cats Writer Vector is empty »
I wrote this simple program in my attempt to learn how Cats Writer works import cats.data.Writer import cats.syntax.applicative._ import cats.syntax....
(2) odpowiedzi
2017-01-21 14:01 Why does scala return a Any(Val/Ref) when the function is obviously called with wrong types »
I have this function: def unpack[T] = (x:Option[T], y:T) => x match { case Some(z) => z; case None => y } when I call it like this: unpa...
(1) odpowiedzi
2017-01-21 00:01 Where is Scala's += defined in the context of Int? »
Just starting out with Scala var c = 0 c += 1 works c.+= gives me error: value += is not a member of Int Where is the += defined? ...
(2) odpowiedzi
2017-01-20 21:01 Chaining implicit conversions of function to a generic class »
I have the following code which was supposed to take a function A => Boolean (generic on the input type) and convert it to a generic trait Y[A] thr...
(1) odpowiedzi
2017-01-20 13:01 Scala Stream vs Java Stream Laziness Difference »
I'm new to the concept of lazy evaluation. When I execute this line of code in Scala; "12334".grouped(1).toStream.filter{n => println("n:" +n ); n...
(1) odpowiedzi
2017-01-20 11:01 Scala: why is there no sortBy on set? »
I'm wondering why there is no sortBy method on Scala Set as there is for Seq or List since it extends Iterable as well... ...
(1) odpowiedzi
2017-01-19 17:01 How to match type of instance in ScalaTest »
I was trying to assert exception type in a test using Matchers (don't ask why) and the solution that I got is this: exception.getClass shouldBe class...
(2) odpowiedzi
2017-01-19 16:01 Scala: merge two arrays in one single structure »
I have two arrays: val diceProbDist = new Array[Double](2 * DICE + 1) and val diceExpDist = new Array[Double](2 * DICE + 1) and I want to merge ...
(3) odpowiedzi
2017-01-19 15:01 Akka - worse performance with more actors »
I'm trying out some parallel programming with Scala and Akka, which I'm new to. I've got a pretty simple Monte Carlo Pi application (approximates pi i...
(3) odpowiedzi
2017-01-18 23:01 With Circe Json why is implicit resolution slower at runtime »
Why is Circe Json slower with implicit decoder lookup compared to saving the implicit decoder to a val. I would expect these to be the same because i...
(1) odpowiedzi
2017-01-18 21:01 Including null values in an Apache Spark Join »
I would like to include null values in an Apache Spark join. Spark doesn't include rows with null by default. Here is the default Spark behavior. v...
(2) odpowiedzi
2017-01-18 19:01 Why do abstract variables require type annotation in scala? »
When I have an abstract class with variables as below abstract class Book { val id : Int val name : String val book : Long } Declaring them...
(1) odpowiedzi
2017-01-17 22:01 symmetric equality when making a sanitized string type (using scala.Proxy) »
I have a scala (2.10.4) application where email addresses are passed around a lot, and I'd like to implement an abstraction that's called at IO to "sa...
(0) odpowiedzi
2017-01-17 21:01 Why do Scala function types conform to the FunctionN traits? »
In other words, what purpose do the FunctionN traits serve? Do they exist to enable the compiler to know how to treat functions as first class values?...
(2) odpowiedzi
2017-01-17 19:01 Use flatMap on List[Either] in Scala »
Either is right-biased ince Scala 2.12 which allows it to be used in for/yield blocks without projection just like Option. But apparently this isn't e...
(2) odpowiedzi
2017-01-17 10:01 Scala higher kinded type variance »
I am dipping my toes in higher kinded types, exploring a very basic Scala example: trait Mappable[F[_]] { def map[A, B](fa: F[A])(f: A => B): F[...
(2) odpowiedzi
2017-01-16 21:01 An object with unapply working in middle of a case statement »
scala> object Test { def unapply[L,R](v: (L, R)) = Some(v) } defined object Test scala> (1, 2) match { case 1 Test 2 =...
(2) odpowiedzi
2017-01-16 14:01 use implicits to infer an unknown type »
I am trying to get the compiler to infer some type based on 2 other type parameters that are known. here's the example: trait ReturnCount trait Retur...
(1) odpowiedzi
2017-01-16 09:01 Scala Compilation Error : Value += is not member of Int »
I am at a beginner level in Scala, I am trying the following code: var i: Int = 0 for (i <- 0 to 10) { if (i == 2) { i += 1 } p...
(3) odpowiedzi
2017-01-15 20:01 How is Predef in Scala inherited? »
Comment in the source for the Predef states: The Predef object provides definitions that are accessible in all Scala compilation units without e...
(1) odpowiedzi
2017-01-15 15:01 Left flatMap on EitherT »
Let's say I have functions which return Future[Either[_, _] and I want to apply some of these functions in case of failures, that means apply them onl...
(1) odpowiedzi
2017-01-15 00:01 package object type members and inheritence don't work together »
The following code compiles perfectly in Scala 2.11: trait A { type B def f(x: B): Unit } /*package*/ object C extends A { trait Z type B = ...
(1) odpowiedzi
2017-01-14 12:01 Override Java method with arg Class<?> in Scala »
I am trying to subclass a class (Y) from an external library that can not be changed. It has a method as such: protected Object doSomething(Class<...
(1) odpowiedzi