Sagar Ingale

Big Data Lake in the AWS Cloud

2018-03-17T19:38:49+01:00

In this post I try to talk about my experience of building a (Big) Data Lake on the AWS Cloud using S3 as a data store, and solutions to common problems that we faced.

I had worked with Philips Health Tech to build the data lake, however most of the problems we faced are universal and can be applied to almost any large scale data engineering product built these days.

Moving into a cloud first world

As the size of data collected by businesses increased day by day, the cost required to manage these kept going up, to a point when businesses seriously started to think of cost effective ways to store and process the data.

Hadoop turned out to be a good bet for quite a few years, and almost every company had tried to use this to solve their problems. It did a great job but it didn’t solve the Ops part of the story. Hadoop as a tool required managing and maintaining data centers, replacing old servers and disks as they wear out, making sure everything is running from day to day, etc. With Hadoop you didn’t have to worry about data security, data loss, data availability, but you were always required to have an Operations team to manage the infrastructure and keep it up and running. However the landscape of Big Data changed rapidly with technologies getting outdated faster than ever and new processing frameworks solving newer problems every other day. So this meant upgrading clusters and up-skill teams, which required effort, experimentation and risks.

The cloud first world aims to solve these problems with the use of managed services and decoupling everything. A Data Lake in the cloud aims to be a true No-Ops data store which is virtually of infinite scale, and doesn’t require managing separate teams monitoring or maintaining clusters. This is achieved by storing your data in a highly secure, durable and available shared data center. Also you have separate services for storing your data, and computing. With almost all kinds of services available for storing, caching, fast look ups, archived storage, on demand processing available, you can process almost anything without having to do anything about managing operations.

Huge data in On-Premise servers and the shift to Public Cloud

Before we could trust on cloud, software wasn’t really a service but a product you buy and keep in your office. This meant having space to stack up servers and maintaining and managing them. With data growing faster than ever, technologies becoming obsolete sooner than you think of, companies had to get into the business of managing, developing and maintaining huge data centers. It made all the sense for companies like Google, Facebook which operate at hyper scale, and face problems which are almost always unsolved. But not all business can or need to have separate potentially huge data center operations running with the primary motive being to manage servers. This is the time, when going to a shared data center model, provides huge benefits in terms of costs, scalability and operations, and this is where the cloud steps in.

This can be thought of huge data centers around the world, which you can rent from time to time when you need and pay exactly for what you use. Other than saving costs, you also save all the idle time your servers were consuming and you were paying. Also this looks more environment friendly, because as a small business you would be having (say) a 100 servers which you won’t switch off on the weekend. On the other hand, in a cloud platform with a service based model, you pay for only the seconds you use a product. And this is from a pool of tens of thousands of servers running in a data center shared by all. On the weekend when the load is low, they turn these down to low power mode. Or at night, someone from the other side of the world gets to use the same compute power when you sleep. With these simple optimizations the companies offering the shared data centers to you, can lower the cost, without you noticing and you save your money and the environment (our mother Earth, not dev-prod) is happy.

The world of HDFS

With HDFS being a standard of data storage when it came to Big Data, lets try and understand what it had to offer and how it solved most problems.

It is a File System and has all characteristics of a File System.
Provides good durability which can be controlled based on the replication you configure.
Is consistent. You would know if a write is successful in the same request you are writing the file.
Can run in high availability mode, with fail over masters.
Reliable - Has a good reputation of being capable of storing peta bytes of data without corrupting it.
Listing files is a trivial operation.
Uses blocks underneath to store files. Though the block size is configured, its wasteful for files which are very small (few bytes / mega bytes)
Moving files is an atomic operation.
Organizes files into hierarchical folders.

The world of Object Stores

The most popular data stores in the cloud are not files systems but object stores. An object is a set of bytes with some metadata, and files are stored as objects. Data Storage for long term and sometime short term in the cloud is mostly based on Object Stores. There are certain properties about object stores which makes it a very efficient solution for a large number of use cases while keeping costs low and only slightly compromising on certain characteristics.

Some of the common properties (ref. AWS S3):

Store data of any size up to a fixed upper limit (usually 5 TB)
Good for very small files as well as large.
Durability guarantees which is practically “forever”.
Cost based on the exact amount of data stored.
Very highly available.
Options to create instances in the same region to process data in the same data center.
Listing files is a non-trivial operation, and becomes costly for buckets with large number of files.
Moving files is non atomic.
It does not have directory (folder) based hierarchical structure.
Keys can have similar prefixes with a ‘/’ delimiter to make it look like folders, but there is no concept of folders.

The amazing S3 from AWS

S3 is the offering from Amazon as part of the AWS and is probably the largest and most used Object Store in the internet today. So much so, that a small outage can take down literally half the internet.

The use of S3 is virtually unlimited and I have noticed it being successfully used to:

store user generated content (pdfs, file scans, images, thumbnails, videos etc)
store music or any form of digital media
host of static websites
store files and use CDN over S3 to deliver content
store web logs
store big data
as a data lake behind external tables of data warehouses (Hive)

However for companies running in Hyper Scale, and using object stores for data lake did face problems which required specialized solutions before being successful.

The Caveats of S3

The most important things to keep in mind while using S3 is that:

It is an eventually consistent data store. (and there is no guarantee on when this eventual becomes consistent)
It is not good for storing temporary files (files written incrementally in between stages in a job / pipelines)
It not ready to use out of the box for use cases which involve multi step writes of data.

Note: There is Native S3, which solves a few more problems but is slow and has limitations on the file size. There is S3A, which is more suitable for a data lake however still has issues with consistency.

What I learnt from using S3 as a Data Lake

It is super reliable and durable and truly no-ops. For using as a Big Data Lake, where you would have potentially hundreds of pipelines reading and writing data, which are more often than not connected to each other, you would face the consistency problem. Non Atomic move / copy operation becomes an issue when used with Map Reduce, Spark, or other popular big data processing frameworks.

To get in detail, most data processing frameworks like Map Reduce and Spark first write data temporarily to the associated file system, and when the job is successfully completed, it moves the data to the correct folder, and creates a ‘_SUCCESS’ file. Since the move operation is built as atomic in HDFS, it runs as one call from one thread. In the Object Store world, this operation essentially means copy and delete. And copy here means copying the data by reading in memory and streaming to another file in S3. It is not a managed operation by S3, and hence is handled by the client. This also makes the operation run on the master which is performing the final commit operation and is therefore not distributed. For large data sets written directly from a Spark job or a MR job to S3 results in unusually slow commit times and also is prone to network failure.

One option to solve this problem is to use HDFS as a temporary store and finally have another job to copy the data to S3 (distcp). The other options is to somehow have the listing and discovery of files handled separately and not use S3’s features to list files. This is what Netflix did (S3mper) and partly Amazon does in the implementation of S3 consistent view in an EMRFS cluster.

GCS from Google and how it compares

In the Google Cloud Platform world, it offers Google Cloud Storage (GCS) which is also an object store competing with S3 in the market. However in terms of architecture, implementation and use, it is vastly different from Amazon S3. For instance it provides solutions to a lot more problems in the consistency aspect of a data store when compared with Amazon S3:

GCS provides strong global consistency for read after write / delete vs eventually consistency for S3
Bucket and Object Listing are consistent GCS and not in S3
Atomic moves are not supported by GCS or S3

The first point means that in S3, you may not be able to list / read an object just milliseconds after having successfully written it. And you don’t know your write was a success until you have read it, because replication is in process. This also means that after delete, stale reads are possible. S3 offers no guarantees when a write or delete operation would be complete, and there is also no way to track that.

This makes using S3 as a data store to write files and make atomic commits very difficult. To over come this Netflix uses DyanamoDB to store a secondary index for managing files in the S3 bucket. The tool is S3mper and is open sourced.

Conclusion

S3 is probably one of the best products solving a very wide variety of use cases. However before using this as the best alternative to HDFS there are certain things which should be kept in mind. In this article I try to uncover some of the things I learnt while using S3 as a Data Lake.

References

https://cloud.google.com/storage/docs/consistency
https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel
https://github.com/Netflix/s3mper

Akka Actors vs Streams for Rest APIs

2017-10-09T20:38:49+02:00

This post is about a comparison between Spray IO and Akka Http, and specifically one of the major changes in the way we design code when using actor based frameworks. Also a small comparison of choosing between Akka Actors and Akka Streams for development of applications and what to use where.

Background

Before going deeper and comparing the advantages and disadvantages of the two frameworks and when to use what, let us first try and understand a few concepts from definitions. This section is by no means exhaustive and might require additional reading before one can gain a complete understanding.

Concurrency

No it doesn’t mean things should run in parallel. It means that if you were to break a program into smallest of fragments and execute these out of order, the results would be same.

This implies that individual operations should be stateless. (From the point of REST api, we can say that each request in a very simple service should not change the state (or behavior) of the service itself. - It can make changes to the database or some other external service though, which makes the overall application stateful)

This is different from parallel execution though, which means that two operations happen at the same time, in different CPUs. Concurrency means that the operations doesn’t necessarily execute in parallel (at the same time). Wiki

Scala Futures

Futures are a way of defining a particular set of operations which will produce an output at a later stage after it has been executed as part of a different thread. The calling thread is not blocked during the execution of the Future. Futures allows us to define call backs where we can pass functions which will get executed after a future has been completed.

For a future to be executed we must define an execution context (or have it as an implicit). This is an abstraction over creating a Thread and running it. It offers a way to write cleaner code which abstracts out the creation and termination of Threads from the user.

Akka Actors

Actor model is a programming model which is used to implement concurrency. Actors mean the isolation of state into different components and each component communicating with others using message passing over a queue. In the world of Akka and Scala, we must keep the following properties in mind:

Actors have a mutable state stored inside the actor.
The state is mutated by sending messages. Thus an actor has a mailbox where messages come, and by default it is a java ConcurrentLinkedQueue. There can be other mailboxes as well.
The order of messages decides the order of execution. Remember, you are queueing your messages in a ConcurrentLinkedQueue for your Actor.
Processing of messages for one instance of the actor does not happen in parallel. It is concurrent by definition, not parallel. (This is the default behavior. Note that one mailbox can be shared by multiple instances of the same actor if configured - but that is not the default - and this needs a routing actor, which routes the messages to various instances of the actor, and this scheduling can be configured too.)
There is no shared mutable state across (different instances of) actors. Actors are self contained in their states.
Actors have hierarchy. Execution and state of actors are supervised by a supervisor actor.
Consumption of a message by an actor is determined by when the actor executes, this is in turn governed by the dispatcher or the execution context.
When an actor is scheduled to run, it gets a thread from the dispatcher and the thread for an actor is not reserved.
Akka doesn’t provide guarantees on messages being consistent, and messages can be dropped if the mailbox is full.

Thus we define actor as a state store that can be mutated by passing messages to it. It is definitely not some container for a deterministic stateless function which should be called for a response.

Why? Because that is what an object or a static function is used for. Since the function is deterministic, it doesn’t change the state of an Actor, and can be called in parallel from different threads, it doesn’t need to be concurrent.

Akka Streams

Streams are a model of programming which is very functional in nature. It starts with having an input and making it flow through a series of operators or functions on the stream and producing an output. One example of streaming is the Unix | (pipe) operator.

In most Http Servers that we build, we essentially write the function which maps requests to the responses. In most frameworks like Spring Boot or other MVC frameworks, one thread used to apply the transformation from a HttpRequest to a HttpResponse for every request to the server.

Akka Streams provide an abstraction over the Akka Actor model specifically in a way where we write the functions to be chained and executed on an input to produce an output.

With this we are abstracted out of how the messages should be passed between actors, buffering, back pressure control and other components which largely remain same for most streaming applications using actors. This also helps us define complex transformations expressed as a directed acyclic graph, and not merely a simple function. Akka Streams API allows us to cleanly define async boundaries in a set of operations and create flows without worrying much about concurrency and execution.

Spray IO

Simply put, Spray IO is a framework where you create REST API to talk and interact with Akka Actors. Your Http Routing is always contained within an Actor.

Now since actors execute messages one at a time, it is very good for situations involving reads and writes where there is a state change that needs to be maintained and passed on to the next request.

For situations which need only reads or only writes where each request is truly concurrent, we don’t need Actors.

Akka Http

This is a framework for building Http Servers and Clients based on Akka Streams and Akka Actors.

The primary design change when moving from Spray to Akka Http is that Akka-http is based on Akka Streams and not directly on Akka Actors. Akka Streams internally use Akka Actors but that is an implementation level detail for akka streams.

This essentially means that akka http and spray behave differently even with the same user facing code.

The REST API

With both Spray and Akka Http, defining REST apis is fairly simple. We define the route and then bind it to the port. It then starts accepting connections and responds. But there are a lot of differences in its implementations.

Spray

We define the “route” as a trait extending HttpService of Spray Routing. This route is evaluated for each request to Spray. As an example lets see the following:

Source code is here

For the purpose of examples in this post we would use a server which takes in a String and returns all possible permutations of the characters in the String.

I choose this function for this example because it has a time complexity of O(n!) which increases to a very high value with a small change in the input.

// Spray Routing trait
trait RestService extends HttpService with JsonSupport {
  implicit val log: LoggingAdapter
  implicit val executionContext: ExecutionContext

  val route =
    pathSingleSlash {
      get {
        complete {
          ServiceStatus("OK", "Welcome to Spray IO Rest Service")
        }
      }
    } ~ pathPrefix("sync") {
      pathPrefix("permutations") {
        get {
          parameters('string.as[String]) { (inputString) =>
            complete {
              new Permutations(inputString)
            }
          }
        }

      }
    }
}

Now we need to define our Router Actor, and override the receive method, to reply with the route that we have defined in our Service trait.

Thus the route is what gets executed when requests come to the actor. Where is the state of the actor? In this simple example, there is none. However, you can have the state inside the Router actor and this state can be updated concurrently with each request if the need arises.

// Spray Router Actor
class RestRouter(implicit val system: ActorSystem) extends Actor with RestService {
  implicit val executionContext = system.dispatcher
  implicit val log = Logging(system, getClass)

  // the HttpService trait defines only one abstract member, which
  // connects the services environment to the enclosing actor or test
  val actorRefFactory = context

  // Here we override the receive function of the Actor to execute our Rest Route
  val receive = runRoute(route)
}

Now that we have our service defined, we create the actors, and start the server by sending the Bind command to the Http extension.

This would start a HttpListener actor for a given port. When a request comes from client, it will first spawn a new HttpServerConnection actor. This actor is responsible for accepting the requests and sending them to our router actor which knows how to respond to a HttpRequest with a HttpResponse.

Now lets execute the code. Please refer to this GitHub Repo for the code, if you want to execute yourself

Things to note. All requests are executed sequentially but in different threads. Remember your actors had a mailbox which is a Queue. And there is only one instance of the actor defined. We didn’t define a set of actor instances and scheduler actor to give the received requests in parallel to different instances of the same actor.

Here it is easy to notice that the calls become slow as the characters in string increase, which is expected for the permutation function. However also note that requests are blocking in nature. Even though processing of requests can be made concurrent (We can process one request independent of another), the calls are blocking the server, and it cannot respond to another request concurrently.

Where can this work: For cases where requests must be processed in order, where you do not need things to run in parallel. Cases where each requests modifies the state of the service and the modification should be visible to the subsequent request. Eg: You have an API with which you want to submit a task, and check after sometime the status of it. However after submitting the task you want the server to start processing and the request to start should complete. There should be only one instance of the task running. So with a submit, you modify the state of the actor and then you query the actor using REST api for the status of the task. An Implementation of this use case is there with Akka Http later on.

Where will this be an anti pattern: If all your requests are stateless, and each request doesn’t change the state of the service, this is an anti pattern. You can possibly run these requests in parallel, but with an Actor pattern and one instance (ActorRef) of the Actor defined, you can’t run requests in different threads at the same time.

Spray IO and responding HttpRequests with Futures

Is there a workaround? Say for cases where there will be a high throughput required, all requests are concurrent in nature, and might take a few seconds to execute. It should not block the caller while it is executing.

This is where we use Futures inside an actor. It should be noted that actors are not meant for running things in parallel. It is meant to store state, but in these cases our operations would be defined in a multi threaded way and not using actors, i.e., we won’t have multiple actors which processes the request.

The route looks something like this:

pathPrefix("async") {
  pathPrefix("permutations") {
    get {
      parameters('string.as[String]) { (inputString) =>

        val respFut = Future {
          log.info("Processing request for {} in Future", inputString)
          new Permutations(inputString)
        }

        onComplete(respFut) {
          case Success(successPermutations: Permutations) => complete(successPermutations)
          case Failure(ex) =>
            log.error("Exception while processing : {}", ex.getMessage)
            complete(StatusCodes.InternalServerError, ServiceStatus("ERROR", "Some error in service"))
        }
      }
    }
  }
}

The code here takes an input string and returns all permutations by calculating that in a different thread.

We use the callback onComplete which takes in a Future. The router actor is no longer blocked while the Future is getting executed.

This makes requests to the server concurrent and non-blocking in nature.

Akka Http

An Http Layer running on top of Akka Streams is much different from what we saw earlier. Even though the Routing remains the same, the whole semantics of execution changes.

For this particular case itself we see that Akka Streams running requests concurrently just out of the box, without any coding with Futures.

This is because Akka Streams defines the operations as a DAG, with a source and sink. Actors has been abstracted here.

The code is exactly the same in terms of routing.

The code to start the server however changes. Here we don’t have any actors and we don’t instantiate any. We create an ActorSystem and an ActorMaterializer.

We call the Http().bindAndHandle(route, host, port) and this gives us a binding future which will terminate once the server terminates.

Actors are used and created underneath for the server to work. For user facing code and implementations, we can have Actors, and akka http will work well that as well.

Here is an example where an user defined actor is used to store the state.

Akka Http with Actors

Akka Http works perfectly well with actors defined by us. In the following example we create a Task Manager.

It is responsible for creating and managing Tasks (these can be server side jobs). It is implemented using Actors, which stores the state of each Task.

Task itself is a simple class which starts and does some operation. Its state is stored inside the instance of the class and gets updated as the Task progresses.

The TaskManager is an Actor which keeps the instances of Task and can query them to understand their state. This actor runs the requests concurrently and avoids all problems which would have arose because of multiple requests coming at the same time. All TaskCreation / Querying happens one at a time because the state of all the tasks is stored in the Actor.

Original Code in Github Repo

References

https://stackoverflow.com/questions/11675422/actor-pattern-what-it-exactly-constitutes
http://blog.michaelhamrah.com/2015/01/a-gentle-introduction-to-akka-streams/
https://www.chrisstucchio.com/blog/2013/actors_vs_futures.html

Implementing Statistical Mode in Apache Spark

2017-07-13T20:38:49+02:00

In this post we would discuss how we can practically optimize the statistical function of Mode or the most common value(s) in Apache Spark by using UDAFs and the concept of monoid.

What is mode?

In statistics, mode is defined as the value that appears most often in a set of data. Essentially it is the most common value in a given set of data.

Finding the mode in SQL

This might sound simple and we might be hoping that an aggregate function is already available. However in most cases it is not available because it is pretty complex to implement the theoretical definition of mode using SQL.

For example, lets say we have a data set, where all the values in a particular field is same, or there are multiple values having the same frequency. This boils down to the fact that the function should not give us one value as an aggregate, but a set of values as the output. This implies that the output should be an array. However in most real world use cases, where one would need to find mode, the attributes are usually nominal having a varied distribution and it can be assumed that there is only one value with the highest frequency. Thus for practical purposes we may assume that the output is a single value and not a set.

Now, let us consider a data set such as trip_id, user_id, timestamp, city, airline, hotel_name.

We want to find out the most visited city for each visitor and tag that as his home city, find the airline he uses the most, and the hotel he stays the most in, and then use this data for further computations. Now lets try and express this use case.

Intuitively we would want something like this:

SELECT user_id, mcv(city) as home_city, mcv(airline) as fav_airline, mcv(hotel_name) as fav_hotel
FROM <trips>
GROUP BY user_id

This is simple to express, easy to read, but the catch is that we don’t have a function to calculate the most common value (mcv) for a particular group of data.

Without having this function, here is how it looks like:

home_city =
SELECT user_id, city as home_city, ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY cityFreq) as rn
FROM (
    SELECT user_id, city, count(city) as cityFreq
    FROM <trips>
    GROUP BY user_id, city
) WHERE rn = 1 

Using Rank would give the true definition of mode, as there can be multiple values holding the same max frequency. But then would have to introduce another GROUP BY to convert that into a SET, and keep it in one row.

So now we have home_city, we write the same thing again for airline, and again for hotel, and again for 100 other features that we may have. And we have ended up with multiple tables, so join these based on user_id.

Lets see how this would work: First it would do a grouping operation then a ROW_NUMBER or a RANK on an ordered set. Ordering the data is not the best thing we want to do especially because we would be ignoring every but the one with row number 1. Can we do a max? Yes, but that would need another join, and all this to calculate the home city. That’s not the end, we will need one join for each feature, before we can have the final data set. Thus for 3 features that is 2 joins.

So we have ended up doing 3 grouping operations, 3 partitioning and 3 ordering operations, and 2 joins to find the most common value for 3 attributes. Pretty costly.

An UDAF is what helps us in these cases. Lets understand that now.

Defining custom aggregate functions

Analysing our problem statement, we can say that our UDAF would be run for each user, and the input for one run would be all city names for that user. The output should be the most common value from that set.

This sounds simple, and the steps for this function may be defined as follows:

Find the frequency of each element in the given input data.
Find the element with the max frequency.
Send that as the output.

For practical use cases, we can assume that there is only one value which is the most frequent, and thus we would return a single value. This also means that the complexity of our function would now be O(n). (If we wanted to find all the most common values - theoretical mode - then we would have to sort based on frequencies, and it would be O(n log n) )

Monoids

This might sound completely random to come in as a section here, but lets recollect the definition of Monoids. It would help us in writing much more optimized codes and understand how we can create UDAFs in Spark (or other SQL)

By definition, we can say:

Given a type T, a binary operation Op:(T,T) => T, and an instance Zero: T, with the properties that will be specified below, the triple (T, Op, Zero) is called a monoid. 
Here are the properties:

Neutral element: Zero Op a == a Op Zero == a
Associativity: (a Op b) Op c == a Op (b Op c)

Satisfying the associative property helps us in randomly grouping the given inputs and run these groups them in parallel and merge the outputs into one.

If we want to calculate avg, the neutral element is the average itself, but average is not associative. However average can be expressed in terms of sum and count, which are associative. We can count elements in parallel and then add them up to find the total count. Similarly we can also add up elements and then add the results to find total sum. Using this information we can find the average at the end. Thus expressing average as a monoid helps us in calculating the average in parallel. The neutral element for sum is 0 and for count is null.

It is easy to understand how we can find sum and average in parallel at the same time.

Now we can see that computation of functions which can be expressed as monoids can run in parallel very easily. This helps us in writing very efficient map reduce code.

Spark UDAF

Going by the user guide on Spark UDAF its clear that the above algorithm can’t be used as is, since we don’t even have a function that we can override where the input parameter contains the whole list of elements. To run our most common value function in parallel we can see that if we divide a given set of data into multiple sets and calculate the most common value in each set, and subsequently find the most common value, we wont end up with the actual most common value for the whole set.

This is where we try to define our function in terms of monoid, and see how it fits in the functions defined in the user guide.

Considering that for practical use cases the cardinality of the attribute wont be very high, (in most cases where you would logically want to find the mode) we can say that if we know the frequency, find the element with max frequency can happen in one node, and doesn’t need to be distributed. This leaves us with the first part: finding the frequencies. Can this be distributed? Let us try by writing a monoid which returns the frequency of each distinct element in a given set of values.

The computation would run in parallel, lets say we have n partitions.
The frequencies would essentially be of type Map[Object, Long] with the item and its frequency.
Initially the Map would be empty, in each of the partitions. (This is the neutral element)
Add each element to the already maintained Map. (Update the map with one element, by increasing its frequency)
Every partition is now reduced to a Map[Object, Long]
Merge the output of all the partitions till we get one output. (The actual function which is the monoid, this is where associativity is used)
Find out the max frequency, from the merged Map. (Some other operation at the end)

Thus finding the frequencies in each partition and then merging them again to find the total set of frequencies, can happen in parallel at the same time.

Now let us write these logic as part of the Spark UDAF.

We are given a buffer for our aggregate function to store the intermediate results Its Schema can be defined as follows. (Considering that we are calculating the mode on a field with elements of StringType)

override def bufferSchema: StructType = StructType(
  StructField("frequencyMap", DataTypes.createMapType(StringType, LongType)) :: Nil
)

Now for calculating we need to initialize the MapType for each partition.

override def initialize(buffer: MutableAggregationBuffer): Unit = {
  buffer(0) = Map[String, Long]()
}

The buffer is where we store intermediate results.

Now when a new value comes in, we need to add it to our frequency map.

override def update(buffer: MutableAggregationBuffer, input: Row): Unit = {
  buffer(0) = buffer.getAs[Map[String, Long]](0) |+| Map(input.getAs[String](0) -> 1L)
}

Note, the function |+| is from Scalaz and adds the frequencies for the same string. We also need to import scalaz.Scalaz._ This is a SemiGroup operator from the package. This function is associative and hence we use to merge the frequency maps.

After we get all the frequency maps from the parallel tasks, we need to merge them. This merges two aggregation buffers.

override def merge(buffer1: MutableAggregationBuffer, buffer2: Row): Unit = {
  buffer1(0) = buffer1.getAs[Map[String, Long]](0) |+| buffer2.getAs[Map[String, Long]](0)
}

Finally find the most common element from the merged Map.

override def evaluate(buffer: Row): String = {
  buffer.getAs[Map[String, Long]](0).maxBy(_._2)._1
}

With the above our Spark Code now becomes:

val mostCommonValue = new MostCommonValue
df.groupBy("user_id")
  .agg(mostCommonValue(col("airline")).as("fav_airline"), mostCommonValue(col("city")).as("home_city"))

Simple, clean and fast.

The full class for the UDAF is available here as a gist

Cheers

Analyzing Java Garbage Collection Logs for debugging and optimizing Apache Spark jobs

2017-05-14T20:38:49+02:00

Recently while trying to make peace between Apache Parquet, Apache Spark and Amazon S3, to write data from Spark jobs, we were running into recurring issues. In this post I would describe identifying and analyzing a Java OutOfMemory issue that we faced while writing Parquet files from Spark.

What we have

A fairly simple Spark job processing a few months of data and saving to S3 in Parquet format from Spark, intended to be used further for several purposes. To improve access, we got the data partitioned based on the date of generation. So far so good and in development our systems ran just fine to move forward. However the moment we started loading actual data, of approx 150 GB as a test, our jobs became slower and slower to a point when it started throwing Out of Memory Exception in Spark.

What the error looked like

One of the interesting errors was a java OutOfMemoryError with GC Overhead limit exceeded.

17/05/12 10:48:36 WARN NioEventLoop: Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: GC overhead limit exceeded
...
...
17/05/12 10:48:52 ERROR InsertIntoHadoopFsRelation: Aborting job.
java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
17/05/12 10:48:52 ERROR ErrorMonitor: Uncaught fatal error from thread [sparkDriver-akka.remote.default-remote-dispatcher-5] shutting down ActorSystem [sparkDriver]
java.lang.OutOfMemoryError: GC overhead limit exceeded

Reproducing the error

To reproduce the error I wrote a small piece of code after doing the full analysis to understand what is wrong. It is available as a gist here. I wrote just as much as it is needed to create this issue primarily for writing this blog. Submit the code as follows after making suitable changes to reproduce the error. I had used Spark 1.5.2 with Scala 2.10.4

nohup spark-submit --packages com.github.scopt:scopt_2.10:3.3.0 \
--master yarn-client --executor-memory 6g --executor-cores 1 --num-executors 40 \
--driver-memory 2g \
--conf spark.storage.memoryFraction=0.4 \
--class org.anish.spark.gcissue.SparkParquetGcOutOfMemory --verbose JarFileName.jar \
--outputTableName feed.temporary \
--outputTablePath s3a://your-bucket-name/junkData/ \
--outputTableFormat parquet \
--outputTableSaveMode append \
--rows 1000000 \
--cols 500 \
--repeat 20 \
--parquetEnableSummaryMetadata true &

This creates a data frame with random integers having 500 columns and 1000000 rows. And the repeat specifies how many times it should append to the final location where it is trying to write this data.

Debugging

A bit of background on the complexity of the logic in the Spark Job and data size:

The job we were having was fairly simple with a couple of transformations and filters processing around 100GB data. There were no joins or aggregations going on. In short all the transformations can be expressed as a single stage map operation and there was absolutely no need for any data to be cached in memory. We can very well stream the input and write to output, after adding partition information. So a OutOfMemory Exception is fairly unexpected. We also do not have any collect/take/show/etc in our job. The computation is triggered by the df.write.save api on the data frame, so the data should not be coming to the driver anytime.

A quick Google search with the error and some skimming through JVM docs says that “the garbage collector is taking an excessive amount of time (by default 98% of all CPU time of the process) and recovers very little memory in each run (by default 2% of the heap).” But we are yet to understand which JVM is throwing this error. There are multiple executors running and our dearest driver.

To debug, the fist option was to play around with the executor memory (because the data is being processed there), which proved futile after a few attempts. Checking a few executor logs managed by YARN also doesn’t reveal any error. This seems to be a dead end.

Enabling GC logging at the executors

The next approach is to understand what is there in the JVM heap that triggers this error. To understand this further we turn on GC logging so that we get to know what is going on in the JVM. Spark provides us with the configuration spark.executor.extraJavaOptions with which we can pass options to the JVM. To start this we add the following to the spark-submit

--conf spark.executor.extraJavaOptions='-XX:+PrintFlagsFinal -XX:+PrintReferenceGC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintAdaptiveSizePolicy -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark'

The above would print the GC logs to the executors stdout. These logs would be really big in size. We had around 100MB logs, with INFO log level for Spark. Now take out a few executor logs and compress them bring those out of the cluster to analyze further.

$ yarn logs -appOwner <your-user-name> -applicationId <yarn-application-id> -containerId <container-id> --nodeAddress <executor.node:port> -logFiles stdout >executor1stdout.txt

Here the container id, and node address can be seen from the Spark UI executors tab. This would give us the file which after compressing would be small enough to analyze further.

Here I used a free GC log analyzer: GC Easy

A few stats from the job is interesting and here as follows:

The analysis shows that there is no Memory Leak going on.

The Heap Usage after GC looks like a constant value, doesn’t look like there is any problem.

The amount of time spent by the Garbage Collector.

Stats calculated from the GC logs

From the above graphs, its clear that things are running smoothly at the executors. GC isn’t taking a long time and is in milliseconds. Most of the garbage is collected by Young GC, and Full GC doesn’t do a lot of work. (You might be interested in reading and understanding a bit about Java GC before interpreting these graphs. This Spark related post from Databricks has some excellent explanations)

Enabling GC logging at the driver

Now the above action was a waste of time. In fact if I had analyzed the Spark log a bit carefully I would have identified the issue faster. It tells us two things (in the last 3 lines of the log at the beginning of the post):

A class that was able to see this error and act: It is ERROR InsertIntoHadoopFsRelation: Aborting job. So this means that the job was aborted due to an error somewhere by the InsertIntoHadoopFsRelation class. This implies that the data was fine and was processed without exceptions being thrown, but it wasn’t able to insert into a HadoopFsRelation.
The thread which threw this uncaught exception: This thread is named [sparkDriver-akka.remote.default-remote-dispatcher-5] and after the error the sparkDriver ActorSystem was shut down. A brief idea about how Spark is implemented and how Akka Actors communicate with each other helps us infer that this error is thrown in the sparkDriver Actor System which is in our driver JVM.

I did try a random shortcut to increase the driver memory from the default 1g using --driver-memory 2g in my spark-submit, just to see if it resolves the issue or there is really something nasty going underneath, but this doesn’t help. However after this, I don’t want to try with 32g driver because I know I am not collecting any data to the driver and something is wrong here.

Thus as the next step we enable GC logging in the driver.

--conf spark.driver.extraJavaOptions='-XX:+PrintFlagsFinal -XX:+PrintReferenceGC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintAdaptiveSizePolicy -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark'

These logs have been printed in the stdout (I was running with yarn-client as a master). Once again uploading these to GC Easy shows us the following results:

The analysis shows that there is Memory Leak in the driver JVM.

The Heap Usage after GC. Towards the end it remains constant.

The amount of time spent by the Garbage Collector. It is very high towards the end.

Stats calculated from the GC logs showing high amount of time taken for Full GC

Now we can understand the following things:

It clearly tells us that there is a Memory Leak.
Towards the end of processing, we can see that there are multiple Full GC runs with no change in Heap Size. (This looks apt with the description in JVM docs about the Exception being thrown)
The max time of Full GC is 7 sec.
The number of Full GC is quite high towards the end before it throws the error and shuts down.

We are sure that the driver is the culprit here. Now the next question would be about the objects that are being created in the driver JVM which needs so much memory. So my next step is to analyze the JVM Heap Memory using Heap Dumps.

Analyzing the JVM Heap Dump

Along with all the other driver JVM options we add -XX:+HeapDumpOnOutOfMemoryError.

This would create a Heap Dump in the working directory. Since we are running in yarn-client, the driver home directory is same as the one from which we are submitting the job. The final Spark driver JVM options looks as follows:

--conf spark.driver.extraJavaOptions='-XX:+PrintFlagsFinal -XX:+PrintReferenceGC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintAdaptiveSizePolicy -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:+HeapDumpOnOutOfMemoryError'

Now since we were running with 2g driver memory, we have a nice 2.4gb heap dump created.

To analyze what is there in the Java Heap, I used a tool called Memory Analyzer which is an Eclipse plugin.

Now we need to run Eclipse on a JVM which is having more than 2g memory to analyze a 2g heap dump. Here I had change the JVM options of Eclipse so that it can analyze this dump. I changed the VM option to -Xmx4096m in Contents/Eclipse/eclipse.ini inside the application package of the Eclipse app for Mac OS X.

Now after loading the heap dump and a bit of clicking around leads us to some interesting things.

1.5g of total 2g is used by Thread main
This also tells us leak suspects. Going there we try to see the Thread Stack to understand which function goes on allocating memory. This stack is the key to finding the root cause. More on this later
The dominator tree tells us which are these objects. We see org.apache.parquet.format.RowGroup which also stores org.apache.parquet.format.ColumnChunk which is internally having org.apache.parquet.format.ColumnMetaData and then org.apache.parquet.format.Statistics

These information about the object names along with the call stack points us to the root cause described below.

The overview of the Heap. We see that one thread has 1.5g of the total 2g of memory.

The call stack of the thread which is taking up most of the memory.

The dominator tree showing us the objects which are consuming the most space in memory.

The problematic call stack

main
  at org.apache.parquet.bytes.BytesUtils.intToBytes(I)[B (BytesUtils.java:221)
  at org.apache.parquet.column.statistics.IntStatistics.getMaxBytes()[B (IntStatistics.java:56)
  at org.apache.parquet.format.converter.ParquetMetadataConverter.toParquetStatistics(Lorg/apache/parquet/column/statistics/Statistics;)Lorg/apache/parquet/format/Statistics; (ParquetMetadataConverter.java:242)
  at org.apache.parquet.format.converter.ParquetMetadataConverter.addRowGroup(Lorg/apache/parquet/hadoop/metadata/ParquetMetadata;Ljava/util/List;Lorg/apache/parquet/hadoop/metadata/BlockMetaData;)V (ParquetMetadataConverter.java:167)
  at org.apache.parquet.format.converter.ParquetMetadataConverter.toParquetMetadata(ILorg/apache/parquet/hadoop/metadata/ParquetMetadata;)Lorg/apache/parquet/format/FileMetaData; (ParquetMetadataConverter.java:79)
  at org.apache.parquet.hadoop.ParquetFileWriter.serializeFooter(Lorg/apache/parquet/hadoop/metadata/ParquetMetadata;Lorg/apache/hadoop/fs/FSDataOutputStream;)V (ParquetFileWriter.java:405)
  at org.apache.parquet.hadoop.ParquetFileWriter.writeMetadataFile(Lorg/apache/hadoop/fs/Path;Lorg/apache/parquet/hadoop/metadata/ParquetMetadata;Lorg/apache/hadoop/fs/FileSystem;Ljava/lang/String;)V (ParquetFileWriter.java:433)

In the third line of the above stack it is clear that Parquet is trying to calculate Column Statistics and the problem comes up when writing the MetadataFile (last line above). Also from the objects in the dominator tree, it seems that it is trying to compute column stats for each file. A quick Google Search about Parquet and how it works helps in understanding that it creates the _common_metadata file which stores metadata information from each of the file footers. And this whole merging of metadata from each of the files’ footers to the common metadata file is happening in the driver and on the main thread itself. Thus it is trying to download the whole metadata and trying to combine that.

Problem analysis and Solution

Does Spark really need the common metadata? Apparently while reading, it tries to create it and initializing reads from Parquet files is also slow. Is this really necessary? No. Things would work just fine (Read here). It would read the metadata while reading the files anyway. The common metadata file helps when there are multiple schemas and there are multiple nested sub directories. With our use case we were not having that at the current moment.

Thus a probable candidate for fixing this issue is to disable this metadata generation. To do this we add the following to our code, and this has to be with the hadoopConfiguration:

sparkContext.hadoopConfiguration.set("parquet.enable.summary-metadata", "false")

While reading the data we also disable merging of metadata by using the following in spark-submit

--conf spark.sql.parquet.mergeSchema=false

Now we run the code again. And yes, things work fine.

If we really need to generate the MetaData, we can use the following code:

Other optimization

One more optimization to take note of is to use DirectParquetOutputCommitter which commits directly without creating a _temporary folder which speeds up processing considerably when working with Blob stores like S3. But more on this on a different post.

sparkContext.hadoopConfiguration.set("spark.sql.parquet.output.committer.class", "org.apache.spark.sql.parquet.DirectParquetOutputCommitter")

Notes:

Spark version used for this is 1.5.2
JVM version is 1.8.0_77 and is a Java HotSpot 64-Bit Server VM.
Scala compiler version is 2.10.4
The JVM options and tools for Heap Dump analysis may vary from different JVM distributions.

References

https://plumbr.eu/outofmemoryerror/gc-overhead-limit-exceeded
http://stackoverflow.com/questions/542979/using-heapdumponoutofmemoryerror-parameter-for-heap-dump-for-jboss
https://dzone.com/articles/java-heap-dump-are-you-task
https://crunchify.com/increase-eclipse-memory-size-to-avoid-oom-on-startup/
http://javaeesupportpatterns.blogspot.in/2011/11/hprof-memory-leak-analysis-tutorial.html
https://forums.databricks.com/questions/6494/do-you-need-a-metadata-file-in-a-parquet-folder-to.html
http://stackoverflow.com/questions/37327626/generate-metadata-for-parquet-files
http://stackoverflow.com/questions/34935393/append-new-data-to-partitioned-parquet-files
https://www.appsflyer.com/blog/the-bleeding-edge-spark-parquet-and-s3/

Using HashSet based indexes in Apache Spark

2017-05-13T20:38:49+02:00

This post is about de-duplication of data while loading to tables using HashSet based indexes in Apache Spark.

Introduction

Indexes in R-DBMS systems are used to improve query performance and access times of the data stored. But while using systems like Spark which is primarily an in-memory computation engine, we don’t have indexes. Data sources on the other hand implement indexing and in Spark we can do predicate push down to take advantage of this indexing. Loading data to indexed tables take more time since it rebuilds the index. In most cases indexes are also not optimized to load data faster, but to enhance faster reads. Here I try an use case of incremental loading of data, where I eliminate duplicates without using a join or a full table scan of the existing data set.

The problem statement

A table exists in Hive or any destination system and data is loaded every hour (or day) to this table. As per the source systems, it can generate duplicate data. However the final table should de-duplicate the data while loading. We assume that data is immutable, but can be delivered more than once, and we need a logic to filter these duplicates before appending the data to our master store.

Assumption: We have the data partitioned and data doesn’t get repeated across partition. This just makes the problem a simpler to optimize as it would help in reducing shuffles. For other use cases we can very well consider the whole data to be in one partition.

Sample use case: I used this on click stream data with an at least once delivery guarantee. However since the data is immutable, the source timestamp doesn’t change, and I partition the data based on this source timestamp.

SQL based solution

A typical query to do this require a join of the existing and new data sets. Indexes do speed up joins in R-DBMS systems.

A simple sql based solution of doing a de-duplicated insert is often found in data bases which appends new data from a source table to a destination table. However to realize this insert in a SQL query, it essentially means to do an outer join of the two data sets and then choose only the records that are actually new and discard the old. Also we must make sure we take only a distinct set of records from the new data, which means a distinct before joining or a group by after joining to completely remove duplicates.

The same query in expressed in terms of the map reduce paradigm, transforms to a very costly join and multiple shuffles. The distinct or a group by to eliminate the duplicates from the new data set also means more complex reduce or shuffle and reduce operation respectively. Over all the whole query is not the most optimized way of doing things.

The data size

With a very high data size, traditional data bases don’t scale well, so we go for a map reduce based solution, but we don’t have any indexes here. Spark cannot take advantage of indexes to optimize the join even if those are defined in the data sources.

A Spark native solution

Here I will discuss a solution for managing and maintaining a HashSet of the composite keys for each partition using which we can eliminate a full table scan of the existing data to calculate the subset of the new data that should be appended after removing the duplicates. The whole code base is here and can be downloaded and tested. The main starting point of this etl is the runEtl() method in DeDuplicateMergeData class.

The Logic

When the data is being loaded, create and maintain another pair RDD where the key is the partition and the value is one single HashSet of the composite keys present in that partition of the data.

While loading the data first we convert it into a pair RDD and partition the incoming data in the same way as the existing data. Then for each partition we de-duplicate the data using a HashSet for each partition. (Explained with code later)

In the first run, there is no data present, so we go ahead and load this de-duplicated data and generate the index to be used for the next load.

If there is data existing, but the HashSet index is not available, as a recovery option, we create the HashSet of the existing data and then proceed.

Now we co-group the new data along with the HashSet index and then filter the duplicate records from the new data based on the HashSet. This lookup now happens in O(1) time since we are having a HashSet present.

After having the filtered data, append it to the existing data and merge the index of the new data with the index of the old data. Write the existing data first and then if successful, save the index to a file system.

Now let us look at the individual steps in detail to understand further.

The HashSet based index

The HashSet based index is essentially another RDD which stores a HashSet of the composite keys for each partition of the data that has already been loaded. Since a Sequence of multiple values form the composite key, we maintain one HashSet of these keys. So the type is HashSet[Seq[String]], and since this HashSet is for only one partition, our pair RDD for the whole data is of type (Int, HashSet[Seq[String]]) where the data is partitioned based on a column of type Int.

We create the HashSet index RDD from a partitioned pair RDD. (The partitioning strategy is discussed after this)

private def createHashSetIndex(partitionedPairRDD: RDD[(Int, Row)],
                                 partitioner: Partitioner,
                                 primaryKeys: Seq[String]): RDD[(Int, HashSet[Seq[String]])] = {
    // Create a HashSet (or a Bloom Filter) index for each partition
    val createCombiner = (row: Row) => HashSet(SerializableHelpers.getKeysFromRow(row.asInstanceOf[GenericRowWithSchema], primaryKeys))
    val mergeValue = (C: HashSet[Seq[String]], V: Row) => C + SerializableHelpers.getKeysFromRow(V.asInstanceOf[GenericRowWithSchema], primaryKeys)
    val mergeCombiners = (C1: HashSet[Seq[String]], C2: HashSet[Seq[String]]) => C1 ++ C2

    // Pass our partitioner to prevent repartitioning / shuffle
    partitionedPairRDD.combineByKey(
      createCombiner = createCombiner,
      mergeValue = mergeValue,
      mergeCombiners = mergeCombiners,
      partitioner = partitioner)
  }

Here the function getKeysFromRow is defined as follows and is used to extract and make a sequence of the composite keys from one Spark Row object.

def getKeysFromRow(row: GenericRowWithSchema, primaryKeys: Seq[String]): Seq[String] =
    primaryKeys.map(fieldName => row.getAs[String](fieldName))

The createCombiner function creates a HashSet from the row object. The mergeValues function is used to combine a Row into a HashSet. The mergeCombiners are used to merge two HashSets. Using these three functions, Spark runs the operation of creating the HashSets for each partition in parallel. The resulting output of rdd.combineByKey is one pair RDD where the key is the partition column value and the value is a HashSet of the composite keys in that partition.

Custom Partitioner

To make sure that every step in creation of the index or other co-group operation doesn’t not trigger a new Shuffle, we use the same partitioner everywhere and have most operations as a OneOnOneDependency on the previous RDD rather than a ShuffleDependency. Our idea is to make sure that the internal partitioning logic of RDDs and the partitioning that we want to maintain is the same. For example if we want our data to be partitioned by date, the RDD should internally keep values of the same date in the same partition. We can increase or decrease the granularity of the partitioning column to control the size of the data in one partition. By default Spark uses HashPartitioner which takes the data and puts to a constant number of partitions based on the hashcode of the key, in case of a key value pair RDD. This means data with multiple values of the key can end up with the same value of nonNegativeMod(key.hashcode, numPartitions) which means that these would be present in the same partition. We can decide on the number of partitions by using estimators described later, but it would still result in irregular partitions.

For this we implement an exact value based partitioner. The would make sure that the data from one key value would only go to one partition. (One trick to implement this is to use RangePartitioner in Spark, with a very high value for the number of partitions, but this does a sampling underneath and scans a part of the data which is not necessary for our purpose.) Our implentation is to simple do a nonNegativeMod(key, numPartitions) and make sure that we are using an Integer or a Double value as the partition key. If we were to partition based on geographical regions, the HashPartitioner with a high value in the numPartitions would probably be good enough.

override def getPartition(key: Any): Int = key match {
    case null => 0
    case key: Int => nonNegativeMod(key.asInstanceOf[Int], numPartitions)
    case key: Double => nonNegativeMod(key.asInstanceOf[Double].toInt, numPartitions)
    // Long is not defined as numPartitions should be Int as per Spark Partitioner class
    case _ =>
      throw new RuntimeException("Key is not Int or Double or it is not supported yet")
  }

In this partitioner implementation all records having the same partition column value ends up in the same partition of the RDD. This helps us in more optimized filtering after we co-group with the HashSet RDD.

Partition size estimators

Our next step is to find out what should be our number of partitions. We can either to a group by or a distinct (in Spark v2.0 and later, distincts are optimized to use groupBy internally) and then count the values of the partitioning column. Or we can use a high enough number and have some empty partitions which gets ignore while processing producing no output, and is not really a huge overhead.

The other option is to get this information from the metadata of the source. In this case I used it with weblogs and extracted the date from the log file name. The idea was to keep the data originating in all the log files in one day in a single partition. Estimating from dates in the file name somewhat looks like this in the implementation here:

lazy val estimate: Int = {
    val datePatternRegex = datePattern.r
    val datesInFiles = sparkContext
      .wholeTextFiles(inputFilePath)
      .map(_._1)
      .map {
        case datePatternRegex(date) => date
        case _ => println("ERROR : did not match given regex")
      }
      .distinct
      .collect
      .toList

    datesInFiles.length
  }

datePattern is a regex which has the date pattern in the file name. This however doesn’t look very optimized for blob stores like S3 and can be optimized further to uses FileStatus of Hadoop.

Finding distinct using HashSets

Given a pair RDD which is partitioned using the above partitioning techniques, we are sure that data of the same key is always there in only one partition and one partition has values with only one key. Finding the distinct records for each partition now becomes very easy as we don’t need a shuffle to find the distinct. Our aim is to find the distinct values in one given partition only. This means that the partition information is not needed and the logic is stateless and can run independently for each partition. Think of this as data from China can never be duplicate in India, if the data is partitioned by countries. And if the date is generated at source, the data generated on one day may be send multiple times on the same day but never on different days, because the data is immutable and we have partitioned based on the time the data was generated. Data generation time (not when a server logs it) would always be immutable.

The above operation can be expressed as mapPartitions on our new data RDD. However I though of implementing this as a new type of RDD extending the actual RDD implementation. Here we directly define the way we want to compute the data from the previous RDD. The core logic looks as follows as is implemented here:

@DeveloperApi
override def compute(split: Partition, context: TaskContext): Iterator[(K, V)] = {
    val iterList = firstParent[(K, V)].iterator(split, context).toList

    if (iterList.nonEmpty) {
      val hashSetForPartition: MHashSet[Seq[Any]] = MHashSet()

      iterList.filter(record => {
        // Cast to GenericRowWithSchema to enable access by field name
        val keysInRecord = getKeysFromRow(record._2.asInstanceOf[GenericRowWithSchema])
        if (hashSetForPartition.contains(keysInRecord)) {
          false
        }
        else {
          hashSetForPartition.+=(keysInRecord)
          true
        }
      }).toIterator
    }
    else firstParent[(K, V)].iterator(split, context) // The partition is empty
}

This is however a DeveloperAPI is Spark and should be used with caution. The method getKeysFromRow is same as the one described previously.

Merging data based on Hash Set index

Now comes the most interesting part where we skip the full table scan but still join using the index that we had created and filter the data that was already present.

private def filterOldRecordsUsingHashSet(deDupedNewData: RDD[(Int, Row)],
                                           existingHashSetRDD: RDD[(Int, HashSet[Seq[String]])],
                                           newDataPartitioner: Partitioner,
                                           primaryKeys: Seq[String]): RDD[(Int, Row)] = {
    // Step 3 - Merge - this is an incremental load, and old data is already available.
    // Step 3.1 - Cogroup the data. Passing the partitioner same as new data should prevent repartitioning the data
    val coGroupedRDD = deDupedNewData.cogroup(existingHashSetRDD, partitioner = newDataPartitioner)

    // Step 3.2 - Remove duplicates using old data HashSet - the actual merge operation
    coGroupedRDD.flatMapValues {
      case (vs, Seq()) => // This is a new partition and this wasn't present in old data
        vs.iterator
      case (vs, ws) => // This is a partition which is there in old data as well as new data
        val newRecordsIterator = vs.iterator
        val existingHashSet = ws.iterator.toList.headOption.getOrElse(HashSet()) // We expect only one HashSet
        newRecordsIterator.filter({ newRecord =>
          // Filter already existing data
          !existingHashSet.contains(SerializableHelpers.getKeysFromRow(newRecord.asInstanceOf[GenericRowWithSchema], primaryKeys))
        })
      // Ignore the case for only old partition with no new partition present -> case (Seq(), ws)
    }
}

This is implemented here. We first co-group the new data and the index to bring the HashSet to the partition we are working on. Now we use rdd.flatMapValues to implement our actual filter operation. This is much like a left outer join operation which internally calls the cogroup and flatMapValues API of RDDs. While doing the flatMap we check if the new data is already present in the HashSet of the old data for that partition. Note that we are only using the composite keys to check this, much like the join operation would be on some primary keys in any relation.

Merge HashSets

This is a another operation we need because we also have to merge the new HashSet index that was added to the existing HashSet index. We use the following implented here to achieve this:

private def mergeHashSetIndex(rdds: Seq[RDD[(Int, HashSet[Seq[String]])]],
                                partitioner: Partitioner): RDD[(Int, HashSet[Seq[String]])] = {
    // Apply reduce to union all the given rdds into one rdd.
    // This would forget the previous partitioning and simply increase the number of partitions.
    val unionedRDD = rdds.reduce(_.union(_))

    // The following function for merging two HashSets would work for merging values as well as Combiners
    val mergeVaulesAndCombiners = (C1: HashSet[Seq[String]], C2: HashSet[Seq[String]]) => C1 ++ C2
    // Because after co grouping we expect only one HashSet

    unionedRDD.combineByKey(createCombiner = (row: HashSet[Seq[String]]) => row,
      mergeValue = mergeVaulesAndCombiners,
      mergeCombiners = mergeVaulesAndCombiners,
      partitioner = partitioner) // This is required to force repartitioning as union would have likely increased the partitions
}

This is fairly simple because we first do a union of all the RDDs and then partition it to bring all the HashSets to the same partition as the key, and then merge the HashSets. The function used for merging values and combiners are the same in this case.

Storing and reading the Hash Set index

We also need to store and read this RDD of HashSets. For this we use rdd.saveAsObjectFile and sparkContext.objectFile[(Int, HashSet[Seq[String]])](hashSetIndexLocation) to save and read the index. We also built some fault tolerance and around the reading to handle cases like first execution when no index would be present and cases when the we are applying the logic for the first time and data is already existing, so the index needs to be rebuilt. This logic is implemented here.

Building the complete ETL and handling failures

Now we try and implement all the above logic into one ETL which should solve our initial problem. The steps can be described as follows:

Partitioning and de-duplicating new data
1. Prepare (make a pair rdd) and partition (spark memory partition) the new data.
2. Remove duplicates using Hash table for new data. (Remove duplicates present in incoming raw data)
3. Create a HashSet of the new data.
4. Check if it is the first load, then return the de-duplicated data and the HashSet. End of story
Get ready to merge with old data. -> We know it is not a first load
1. Load HashSet of Old Data. If this doesn’t exist it should return empty. This might have got deleted.
2. Check if existing HashSet is present, rebuild HashSet.
Merge
1. If no old data is available, this is the fist load, load the new data as is.
2. Co group the two data sets. Remove data that was already loaded before, based on available HashSet.
3. Now we have the actual data that needs to be appended
Create/Update HashSet index
1. Create the HashSet Index, for the incoming data set.
2. If this is the first run, store this HashSet index.
3. If HashSet already existed for old data, update and overwrite the new HashSet.
4. Handle failures for writing and storing this HashSet.

Output:

The de-duplicated data that should be appended
The total HashSet index that would be valid after appending the data.

This is implemented in the runEtl method here

Conclusion

The overall aim of this project was to speed up duplicated data loading to a master table from where data could be reported or used for various purposes. An optimization which decreases the time complexity from a full table scan to an O(1) look up is fairly complex to think, code and explain in a blog post. It would take quite a few iterations to go through this and understand the overall optimization and how it benefits by using less computation.

References

http://blog.madhukaraphatak.com/extending-spark-api/
http://rahulkavale.github.io/blog/2014/12/01/extending-rdd-for-fun-and-profit/
http://codingjunkie.net/spark-combine-by-key/
http://abshinn.github.io/python/apache-spark/2014/10/11/using-combinebykey-in-apache-spark/
http://apachesparkbook.blogspot.in/2015/12/cogroup-example.html

Exception Handling in Spark Data Frames

2017-05-04T17:38:49+02:00

General Exception Handling

Handling exceptions in imperative programming in easy with a try-catch block. Though these exist in Scala, using this in Spark to find out the exact invalid record is a little different where computations are distributed and run across clusters. A simple try catch block at a place where an exception can occur would not point us to the actual invalid data, because the execution happens in executors which runs in different nodes and all transformations in Spark are lazily evaluated and optimized by the Catalyst framework before actual computation.

For most processing and transformations, with Spark Data Frames, we usually end up writing business logic as custom udfs which are serialized and then executed in the executors. Without exception handling we end up with Runtime Exceptions. Sometimes it is difficult to anticipate these exceptions because our data sets are large and it takes long to understand the data completely. Also in real time applications data might come in corrupted and without proper checks it would result in failing the whole Spark job.

The Runtime Exception

Lets take an example where we are converting a column from String to Integer (which can throw NumberFormatException). Our idea is to tackle this so that the Spark job completes successfully.

Now this can be different in case of RDD[String] or Dataset[String] as compared to Dataframes. In most use cases while working with structured data, we encounter DataFrames. So our type here is a Row.

We define our function to work on Row object as follows without exception handling. This can however be any custom function throwing any Exception.:

val df: DataFrame = ...

// Convert using a map function on the internal RDD and keep it as a new column
val simpleCastRDD: RDD[Row] = df.rdd.map { row: Row =>
  val memberIdStr = row.getAs[String]("member_id")
  val memberIdInt = memberIdStr.toInt
  Row.fromSeq(row.toSeq.toList :+ memberIdInt)
}

// Create schema for the new Column
val simpleCastSchema = StructType(df.schema.fields ++ Array(StructField("member_id_int", IntegerType, true)))
println("Simple Casting")
val simpleCastDF: Dataset[Row] = sparkSession.sqlContext.createDataFrame(simpleCastRDD, simpleCastSchema)

simpleCastDF.printSchema()
simpleCastDF.show()

The above can also be achieved with UDF, but when we implement exception handling, Spark wont support Either / Try / Exception classes as return types and would make our code more complex. Since the map was called on the RDD and it created a new rdd, we have to create a Data Frame on top of the RDD with a new schema derived from the old schema.

The above code works fine with good data where the column member_id is having numbers in the data frame and is of type String. When an invalid value arrives, say “**” or “,” or a character “aa” the code would throw a java.lang.NumberFormatException in the executor and terminate the application. Worse, it throws the exception after an hour of computation till it encounters the corrupt record.

Handling Exceptions

Here I will discuss two ways to handle exceptions. One using an accumulator to gather all the exceptions and report it after the computations are over. The second option is to have the exceptions as a separate column in the data frame stored as String, which can be later analysed or filtered, by other transformations.

If the number of exceptions that can occur are minimal compared to success cases, using an accumulator is a good option, however for large number of failed cases, an accumulator would be slower.

Using Collection Accumulator

Spark Accumulators

Spark provides accumulators which can be used as counters or to accumulate values across executors. Accumulators have a few drawbacks and hence we should be very careful while using it.

The accumulator is stored locally in all executors, and can be updated from executors.
Only the driver can read from an accumulator.
The accumulators are updated once a task completes successfully.
If a stage fails, for a node getting lost, then it is updated more than once.
If an accumulator is used in a transformation in Spark, then the values might not be reliable. If multiple actions use the transformed data frame, they would trigger multiple tasks (if it is not cached) which would lead to multiple updates to the accumulator for the same task.
In cases of speculative execution, Spark might update more than once.
The values from different executors are brought to the driver and accumulated at the end of the job. Thus there are no distributed locks on updating the value of the accumulator.
If the data is huge, and doesn’t fit in memory, then parts of might be recomputed when required, which might lead to multiple updates to the accumulator.

Keeping the above properties in mind, we can still use Accumulators safely for our case considering that we immediately trigger an action after calling the accumulator. This prevents multiple updates. In Spark 2.1.0, we can have the following code, which would handle the exceptions and append them to our accumulator. We use Try - Success/Failure in the Scala way of handling exceptions.

First we define our exception accumulator and register with the Spark Context

val exceptionAccumulator: CollectionAccumulator[(Any, Throwable)] = sparkSession.sparkContext.collectionAccumulator[(Any, Throwable)]("exceptions")

We cannot have Try[Int] as a type in our DataFrame, thus we would have to handle the exceptions and add them to the accumulator. Finally our code returns null for exceptions. While storing in the accumulator, we keep the column name and original value as an element along with the exception. This would help in understanding the data issues later.

val df: DataFrame = ...

val errorHandledRdd: RDD[Row] = df.rdd.map { row: Row =>
  val memberIdStr = row.getAs[String]("member_id")
  val memberIdInt = Try(memberIdStr.toInt) match {
    case Success(integer) => integer
    case Failure(ex) => exceptionAccumulator.add((("member_id", memberIdStr), ex))
      null // Because other boxed types are not supported
  }
  Row.fromSeq(row.toSeq.toList :+ memberIdInt)
}

val errorHandledSchema = StructType(df.schema.fields ++ Array(StructField("member_id_int", IntegerType, true)))

val castDf: Dataset[Row] = sparkSession.sqlContext.createDataFrame(errorHandledRdd, errorHandledSchema) 
// Note: Ideally we must call cache on the above df, and have sufficient space in memory so that this is not recomputed.
// Everytime the above map is computed, exceptions are added to the accumulators resulting in duplicates in the accumulator. 

castDf.printSchema()
castDf.show()

This works fine, and loads a null for invalid input.

To see the exceptions, I borrowed this utility function:

def printExceptions(acc: CollectionAccumulator[(Any, Throwable)]): Unit = {
  acc.value.toArray.foreach { case (i: Any, e: Throwable) =>
    // using org.apache.commons.lang3.exception.ExceptionUtils
    println(s"--- Exception on input: $i : ${ExceptionUtils.getRootCauseMessage(e)}") // ExceptionUtils.getStackTrace(e) for full stack trace
  }
}

// calling the above to print the exceptions
printExceptions(exceptionAccumulator)

This looks good, for the example. But say we are caching or calling multiple actions on this error handled df. This would result in invalid states in the accumulator. To demonstrate this lets analyse the following code:

castDf.show()

println("Show has been called once, the exceptions are : ")
printExceptions(exceptionAccumulator)

println("Cache and show the df again")
castDf.cache().show()
println("Now the contents of the accumulator are : ")
printExceptions(exceptionAccumulator)

The output is as follows:

+---------+-------------+
|member_id|member_id_int|
+---------+-------------+
|      981|          981|
|        a|         null|
+---------+-------------+

Show has been called once, the exceptions are : 
--- Exception on input: (member_id,a) :  NumberFormatException: For input string: "a"
Cache and show the df again
+---------+-------------+
|member_id|member_id_int|
+---------+-------------+
|      981|          981|
|        a|         null|
+---------+-------------+

Now the contents of the accumulator are : 
--- Exception on input: (member_id,a) :  NumberFormatException: For input string: "a"
--- Exception on input: (member_id,a) :  NumberFormatException: For input string: "a"

Process finished with exit code 0

It is clear that for multiple actions, accumulators are not reliable and should be using only with actions or call actions right after using the function. When a cached data is being taken, at that time it doesn’t recalculate and hence doesn’t update the accumulator.

Log Exceptions to a new Column

Another interesting way of solving this is to log all the exceptions in another column in the data frame, and later analyse or filter the data based on this column. In the following code, we create two extra columns, one for output and one for the exception.

val df: DataFrame = ...

val rddWithExcep = df.rdd.map { row: Row =>
  val memberIdStr = row.getAs[String]("member_id")
  val memberIdInt = Try(memberIdStr.toInt) match {
    case Success(integer) => List(integer, null)
    case Failure(ex) => List(null, ex.toString)
  }
  Row.fromSeq(row.toSeq.toList ++ memberIdInt)
}

val castWithExceptionSchema = StructType(df.schema.fields ++ Array(StructField("member_id_int", IntegerType, true)
  , StructField("exceptions", StringType, true)))

val castExcepDf = sparkSession.sqlContext.createDataFrame(rddWithExcep, castWithExceptionSchema)

castExcepDf.printSchema()
castExcepDf.show()

Now we have the data as follows, which can be easily filtered for the exceptions and processed accordingly

+---------+-------------+------------------------------------------------------+
|member_id|member_id_int|exceptions                                            |
+---------+-------------+------------------------------------------------------+
|981      |981          |null                                                  |
|a        |null         |java.lang.NumberFormatException: For input string: "a"|
+---------+-------------+------------------------------------------------------+

Would love to hear more ideas about improving on these.

References

http://danielwestheide.com/blog/2012/12/26/the-neophytes-guide-to-scala-part-6-error-handling-with-try.html
https://www.nicolaferraro.me/2016/02/18/exception-handling-in-apache-spark/
http://rcardin.github.io/big-data/apache-spark/scala/programming/2016/09/25/try-again-apache-spark.html
http://stackoverflow.com/questions/29494452/when-are-accumulators-truly-reliable

SQL Analytical Functions - II - Window Frames, ROWS and RANGE

2017-04-29T17:38:49+02:00

A follow up post about specifying window frames to SQL analytical functions. This assumes you have already read my previous post where I described the use of OVER(), PARTITION BY and ORDER BY clause.

ROWS and RANGE clause

The Over clause also allows us to specify window frames on which the analytical functions get applied. This is useful in calculating things like moving average and cumulative sum using SQL. The partition by clause allows us to group the data set before the analytical function is applied. However it doesn’t allow us to choose different sets records for each input record to apply a function. Consider we have a rainfall data set for a city for 1 month. It has a schema like “date, precipitation”. We want to find the out the total rainfall for the past 5 dates for each day of the month. For arguments sake, lets visually ignore the first 5 values, since they wont be having data for the last 5 days. Here we see that to find sum, we can use the SUM function but we need only the last 5 records and not all the records, for each row. Lets start with a simpler problem, and find the sum of all records and have that as a column.

SELECT date, precipitation,
       SUM(precipitation) OVER () as total_precipitation
FROM rainfall

Next we would apply the window on the input data to the SUM function, so that only the last 5 days are taken. For this we would first sort the data and then add the window as follows

SELECT date, precipitation,
       SUM(precipitation) OVER (ORDER BY date ROWS BETWEEN 1 PRECEDING AND 5 PRECEDING) as total_precipitation
FROM rainfall

This query first orders the data based on date, and then selects the window frame for each row, which consists of the previous 5 rows based on the ordering specified. This set of records changes for every record. The analytical function is then applied on the selected window. This is particularly useful for moving averages.

ROWS or RANGE requires that the ORDER BY clause be specified.

Practice problem Calculate Moving Average on HackerEarth

Using PARTITION BY, ORDER BY and ROWS/RANGE together

Consider the above example but lets say now the data set contains data for multiple cities. Now the precipitation for each city would be different and the sum would also have to be calculated differently. Here we have the data set as “city, date, precipitation”. We would patition the data by city in the Over clause.

SELECT city, date, precipitation,
       SUM(precipitation) OVER (PARTITION BY city ORDER BY date ROWS BETWEEN 1 PRECEDING AND 5 PRECEDING) as total_precipitation
FROM rainfall

When applied with partition by clause, the window frame is applied separately for each partition.

Other keywords

The window frame can be specified using (but not limited to) the following:

ROWS BETWEEN 1 PRECEDING AND CURRENT ROW -> the current value and one previous
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW -> all records from start to current row in the given ordering. This is useful for cumulative sum calculation
ROWS BETWEEN 2 FOLLOWING AND 10 FOLLOWING -> records after the current row as per the number specified.
ROWS 5 PRECEDING -> This is equivalent to ROWS BETWEEN 5 PRECEDING AND CURRENT ROW

Summary

This is very useful for statistical analysis which involves calculation where the input set record for a particular analytical function changes for every record. Applications of this technique include calculation of Year to Date sales, seasonal precipitation analysis, etc.

SQL Analytical Functions - I - Overview, PARTITION BY and ORDER BY

2017-04-27T17:38:49+02:00

For a long time I had faced a lot of problems while working with data bases and SQL where in order to get a better understanding of the available data, simple aggregations using group by and joins were not enough. For deep diving into data, advanced SQL features were necessarry.

Introduction

Aggregate functions are good with the Group By clause for most simple data related calculations. However things like having the average of a group as a column or calculating the deviation from average for each record becomes increasingly complex to express using group by and self join in a sql query. Often while learning SQL, the group by and join are probably enough to gain a basic understanding but very difficult for most analytical operations where the aim is to understand the data.

A few features of the SQL dialect available in most RDBMS or SQL interface which help in better understanding are

the OVER() clause
the analytical functions, for example AVG, LAG, LEAD, PERCENTILE_RANK
the PARTITION BY clause, for example PARTITION BY dept, job
the ORDER BY clause, for example order by job nulls last
the ROW or RANGE clause for the window frame extent, for example RANGE UNBOUNDED PRECEDING or ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING

Over the course of a series of posts, I will try to explain the use of these in SQL along with solutions to few problems from HackerEarth for practice and better understanding.

Recap

The aggregate function and GROUP BY: This post assumes that the reader has good knowledge of group by and finding aggregates for a given data set. As a quick example, suppose we want to find the average marks scored by students from a table having “id, name, subject, marks” we can run

SELECT AVG(marks) 
FROM marks

The above query gives one record as output which is the average of the whole data. Now to find AVG of marks for each subject, we can use a group by clause:

SELECT subject, AVG(marks) 
FROM marks 
GROUP BY subject

Now I have applied a group by clause and specified the column to group the data. This has produced as many groups as there are subjects, and calculated avg marks for each subject. This has returned me records exactly equal to the number of distinct subject values in the table.

OVER() clause

The Over clause is used to define the partition, ordering and window extent about how a particular analytical function would act on the data set. Analytical functions that are used can be AVG, SUM (which are aggregate functions) or RANK, ROW_NUMBER which are purely analytical functions. While using the analytical version of these functions, we don’t need a group by clause and the number of records produced are same as the input records. Lets us take a simple example of AVG, and apply it to a dataset of “id, name, subject, marks”

SELECT id, name, subject, marks,
       AVG(marks) OVER() AS avg_marks
FROM marks

Here the OVER clause is blank and it doesn’t partition / order / window the data before applying the AVG function. Thus the output has an extra column with the avg marks of all the records as a new column. The real application of the over clause is when we use it to specify how to handle the data before applying the AVG function and what should act as inputs to each function calls.

PARTITION BY clause

Lets say our aim was to find out how much each student’s marks deviates from the mean for that subject. For this we know that we don’t want to aggregate the data and want to keep the same number of rows as the input. An easy way to approach this problem is to find the average of marks for each subject and then have that as a column and find the difference for each student and subject mark. Now suppose we wanted to have avg marks for each subject as a column, without the over clause, would have meant grouping the data by subject, and then applying the AVG function for each group. This would produce records equal to the number of subjects. To get the avg as another column we would have to do a self join and then find the deviation for each student making things increasingly complex.

The OVER clause allows us to apply a function over a specified subset of input and have that aggregate or output as a column. For our example, we need to find average for each subject, which means we can partition the data based on each subject and then apply our function on each partition, have the output as a column. Lets try the query

SELECT id, name, subject, marks,
       AVG(marks) OVER (PARTITION BY subject) AS avg_subject_marks
FROM marks

The above query applies the AVG of marks after partitioning the data by subject. So average is calculated for each subject, and is returned as a column. Our initial problem to find the difference of each student from subject average is now simple and can be expressed as follows:

SELECT id, name, subject, marks,
       ABS(marks - AVG(marks) OVER (PARTITION BY subject)) AS mean_subject_diff
FROM marks

ORDER BY clause

We all know using the Order by clause to sort the query output. But inside the Over Clause the order by is used to specify the order in which the data would be provided to the analytical function as input and processed. This is particularly useful with ranking and percentile functions which work on ordered datasets. Consider we have to find the rank of each student. For this we create a temporary table which has data for each student.

CREATE TABLE total_marks AS
SELECT id, name, SUM(marks) AS total_marks
FROM marks
GROUP BY id, name

Thus our input data set has 3 columns, and total records equal to the total number of students. To find the rank, we simply order the data descending by total_marks, and put a rank against it.

SELECT id, name, RANK() OVER (ORDER BY total_marks DESC) as overall_rank
FROM total_marks

Here the OVER clause specified how to order the data before passing it over to the rank function which assigns a new value for each input record. Yes, these to queries can be combined using a sub query or a CTE, but to keep things simple I made it as a table.

Using PARTITION BY along with ORDER BY

Consider we have to find the rank of each student for each subject. For this we partition the data for each subject and then order the students based on their ranks. Now we can easily put a number and have a rank for each student for each subject. The query looks like

SELECT id, name, subject, marks,
       RANK() OVER (PARTITION BY subject ORDER BY marks DESC) AS subject_rank
FROM marks

This query is pretty readable and self explanatory. It says to partition the data first and then order it descending based on marks, and finally put a rank for each record.

Summary

The above techniques cover most requirements for simple understandings. Coupled with functions like RANK(), DENSE_RANK(), ROW_NUMBER(), NTILE() which we would soon visit and try to understand, these techniques give a lot of simple ways to explore data. In the future blog posts I would write about using ROWS and RANGE to specify window frames to apply analytical functions.

Geo Location Batch Search in Spark

2017-04-22T17:38:49+02:00

A module written in Scala for Apache Spark v2.0.0 to batch process mapping of Geo Locations in two skewed data sets. Link to code: https://github.com/anish749/geo-search-spark

Introduction

Apache Spark application written in Scala to map given latitude longitude values to nearest latitude longitude values in a given set using functional programming simulating a map side join of two data sets. This takes two sets of data as input. A master set of latitude longitude values (which are considered constant), and a set of latitude longitudes which are to be searched and mapped to this master set. This is expected to change on every run of the ETL/ join code.

Assumption:

The master data set can be cached in memory, thus not very large. This is the index being used, and only has the available pairs of latitude and longitudes. An id can also be attached to this to track each point. This data set is already sorted (or can be sorted using the code provided) based on latitudes and then optionally on longitudes.
The master set has all unique coordinates.

Mapping logic

The two sets of coordinates are mapped based on the distance calculated by using the Haversine formula. The max limits considered as near can be specified while using the utility:

maxDistMiles - maximum distance that is considered as near (in miles) [used 30 here]
latLimit, lonLimit - To increase efficiency, the area in which the actual distances are calculated is narrowed down by using these limits. Eg: 0.5 latLimit means the distance of points beyond searchLat +- 0.5 are ignored, since 0.5 degree means 34.5 miles approx, the mapped lat lon values would not lie beyond this range. Similarly for longitude, till 83 deg latitude the values for 30 miles is 3.9 degree

Geosearch in databases and SQL queries (if written in Hive) use theta joins which are not supported in Hive, thus forcing cross products. This version using broadcasting of a sorted index to reduce the need for a cross product using a map side join.

A map reduce version of the code with a map side theta join implementation is available at https://github.com/anish749/geo-search-mapreduce

Build

The project is written in Scala 2.11.6, and dependencies are managed using Maven 3.3.9. This is written in Apache Spark v2.0.0

$ mvn clean install

Usage

Adding this package as a dependency

Once installed via maven, this package can be added as a dependency as follows:

<dependency>
    <groupId>com.anish.spark</groupId>
    <artifactId>geosearch</artifactId>
    <version>1.0-SNAPSHOT</version>
</dependency>

Note: This is not available as a package in Maven Central repository.

As a stand alone application

Create index

The first step is to create an index (a sorted array available to all executors during mapping) using the data for available latitude longitudes. This is provided by the com.anish.spark.geosearch.Indexer class. The path where the index would be stored and where the available latitude longitudes are present are configured as variables in this class. (Should be changed / taken as arguments before usage in projects) This class reads the files in the ‘input/availableLatLon’ folder which are configured to be ‘;’ separated pairs of latitude longitudes to which data would be mapped. It then creates an index out of this data and saves it as files in the index Path given after sorting this based on latitudes and then on longitudes.

Run batch search

This is the actual batch job which searches for nearest latitude longitudes and maps them. This is implemented in the class named com.anish.spark.geosearch.Search. This can be run as follows:

$ java -jar target/geosearch-1.0-SNAPSHOT.jar

It would automatically set the following configurations for demo as described in the Mapping Logic section:

val latLimit = 0.5
val lonLimit = 4.0
val maxDistMiles = 30.0

It would take the unknownLatLons as input and map them to the nearest available lat lon as read from the index and write the output in the folder configured.

Batch Search in Spark applications (Scala API)

From a Scala/Java Application where the Spark Session is not available, we can call the following function to execute the search:

runSparkBatchSearch(indexFilePath, unknownLatLonPath, outputPath, latLimit, lonLimit, maxDistMiles)

If the Spark Session is available, then the actual search logic is implemented in the following function and can be called directly:

/**
 * @param sparkSession  The spark session. This is used for importing spark implicits
 * @param indexBc       Available latitude longitude Spark Broadcast variable
 * @param unknownLatLon A spark Dataset of the Coordinates that are to be mapped.
 * @return A DataSet of Row, which has all given Coordinates mapped to available Coordinates. (null for those which were not mapped)
 */

search(sparkSession: SparkSession, indexBc: Broadcast[List[Coordinate]], unknownLatLon: Dataset[Coordinate], latLimit: Double, lonLimit: Double, maxDistMiles: Double): Dataset[Row]