While I do believe, I have a good grasp over pythonic concepts, I don’t want to mess up my interview and make sure that I do rather well. So, I am going through the textbook:

• Fluent Python, 2nd edition

I just want to get a hang of bits of textbook definitions to make sure that I don’t speak garbage or don’t lose out on words.

1. What is ‘self’ in python?

   1. Self refers to an instance of a class. It is used to access variables, attributes and methods of a class.

2. What is ‘__init__’ in python?

   1. ‘__init__’ is an instance method in a class used to initialise objects in python. It is a constructor basically.

3. What are special methods in Python?

   1. The Python interpreter invokes special methods to perform basic object operations, often triggered by special syntax. The special method names are always written with leading and trailing double underscores. For example, the syntax obj[key] is supported by the __getitem__ special method. In order to evaluate my_collection[key], the interpreter calls my_collection.__getitem__(key).

   2. The first thing to know about special methods is that they are meant to be called by the Python interpreter, and not by you. You don’t write my_object.__len__(). You write len(my_object) and, if my_object is an instance of a user-defined class, then Python calls the __len__ method you implemented.

4. Why len () Is Not a Method ? [Copied from the book]

   I asked this question to core developer Raymond Hettinger in 2013, and the key to his answer was a quote from “The Zen of Python”: “practicality beats purity.” In “How Special Methods Are Used”, I described how len(x) runs very fast when x is an instance of a built-in type. No method is called for the built-in objects of CPython: the length is simply read from a field in a C struct. Getting the number of items in a collection is a common operation and must work efficiently for such basic and diverse types as str, list, memoryview, and so on.

   In other words, len is not called as a method because it gets special treatment as part of the Python Data Model, just like abs. But thanks to the special method __len__, you can also make len work with your own custom objects. This is a fair compromise between the need for efficient built-in objects and the consistency of the language. Also from “The Zen of Python”: “Special cases aren’t special enough to break the rules.”

I think I had a good time reading the 1st chapter, and definitely enjoyed it.

For any table, Schema defines the table columns and the data type of the columns. The schema for any dataframe can be seen by using the printSchema() method.

spark = SparkSession.builder.appName("demo").getOrCreate()

df = spark.read.json("data/flight-data/json/2015-summary.json")
df.printSchema()

myManualSchema = StructType([
  StructField("DEST_COUNTRY_NAME", StringType(), True),
  StructField("ORIGIN_COUNTRY_NAME", StringType(), True),
  StructField("count", LongType(), False, metadata={"hello":"world"})
])
df = spark.read.format("json").schema(myManualSchema)\
  .load("data/flight-data/json/2015-summary.json")

The schema datatype is referred to as StructType and is composed of fields called StructFields. These fields consist of the name of the column, datatype and a Boolean flag which indicates whether the column is containing any null value or not. Users can also add any associated metadata with the column.

Columns and Expressions

Columns

Columns in spark only exist in the context of rows which further exist only in the context of dataframes. So, the upshot is that there is no equivalent pd.Series in Spark.

So what is the use of col() function in spark?

This is explained well by a stack overflow answer, available here. The upshot is to refer columns only using strings, the col() function can be used.

Expressions

An expression is a set of transformations on one or more values in a record/row in a DataFrame. Expressions are evaluated by forming a logical tree which specifies the order of operations. For example:

(((col("someCol") + 5) * 200) - 6) < col("otherCol")

has a logical tree which looks like this:

Records (rows)

Each row in a Spark Dataframe is a single record, representing arrays of bytes. The Row objects are manipulated using column expressions. To access the first row of any dataframe in spark, the following can be executed:

df.first()

Always note that transformations are always performed on a dataframe. They cannot be applied on singular rows or singular columns without any dataframe context.

So, Dataframes are the key objects in Spark to work with. They are composed of records and these records are manipulated by specifying transformations on the columns.

This is all for this piece. I am consistently going to update this Spark Section on the blog. If you are a beginner enthusiast like me and want to learn Apache spark, feel free to reach out and we can do it together!

Spark follows a rather simple plan to execute user written code. The agents involved in this execution are:

• Catalog

• Catalyst Optimizer

• Resilient Distributed Datasets

There are two major steps in executing user code:

1. Converting user code to a logical plan.

2. Converting the logical plan to a physical plan.

User code to logical plan

1. The user code is first converted into a logical plan, which is only a set of abstract transformations. No executors or drivers are called yet. The plan is unresolved because even a correct code, might have tables or columns that might not exist.

2. The Catalog, which acts as the repository of all tables, resolves the logical plan. The logical plan then passes through the catalyst optimizer. The catalyst optimizer is a collection of rules to optimize the logical plan by optimizing transformations.

Logical plan to physical plan

1. The physical plan directly involves the driver and executors as it dictates how the code is executed on the cluster. Different physical plans are ideated and they are compared through a cost model.

2. Spark then runs the best physical plan further optimizing it at runtime, by generating native Java bytecode. The results are then returned to the user.

This is all for this piece. I am consistently going to update this Spark Section on the blog. If you are a beginner enthusiast like me and want to learn Apache spark, feel free to reach out and we can do it together!

Earlier, we saw that Apache Spark was a unified computing engine with a set of libraries for parallel data processing on computer clusters. Now, the natural questions that arise are:

• How does Apache Spark handle parallel processing?

• What are the processes that work underneath the higher level Python and SQL calls which make it possible to compute and transform large amounts of data?

Today, we will be taking a sneak-peek on some of those behind the scene management performed by Spark. I am using the textbook Spark: The Definitive Guide as my reference focusing on the 2nd chapter, A Gentle Introduction to Spark

Core Spark Architecture

At the heart of all Spark Applications are two sets of processes:

1. Driver Process

   1. maintaining information about the spark application

   2. responding to a user’s program or input

   3. analyzing, distributing and scheduling work across the executor

2. Executor Process

   1. executing code assigned by the driver

   2. reporting the state of the computation back to the driver node

There is a single driver process and multiple executor processes, with the driver process occupying one node of the cluster whereas the executor occupies the other nodes. The cluster manager only allocates resources to complete the required work. The SparkSession is generally the driver process in Spark applications.

The core spark architecture is exceedingly simple and elegant which allows it to scale to petabytes of data. The allocations of tasks by the driver is carried out by creating partitions in the data. Now we discuss the ideas of partitions, transformations, actions and lazy evaluation.

Actions, Transformations, Partitions and
Lazy Evaluations

Partitions

To work in parallel and use the available executors, Spark breaks up the data into chunks called partitions. Each partition sit separately on one physical machine in the cluster. The number of partitions and the available executors control the degree of parallelism that Spark offers. The user can set the number of partitions but does not manipulate how the data is partitioned. The user only has to specify the high level transformations on the data.

Transformations

Any action or modifications on the data is performed through Transformations. Note that transformations are not immediately applied as in pandas but are only abstractly stored. These transformations (and only the necessary ones) are performed only when an Action is specified.

There are generally two kinds of transformation we deal with, narrow transformations and wide transformations. As evident in the figure, under narrow transformations one input partition contributes to only one output partition whereas under wide transformations one input partition contributes to many output partitions. map(), filter() are some easy examples of narrow transformations whereas groupby(), aggregate() are some examples of wide transformations.

Lazy Evaluations

Transformations on the data are only specified and not applied immediately onto the data. This is because Spark follows a lazy evaluation plan, and waits until the last moment to execute code. The goal here is to provide a streamlined physical plan for the list of transformations which will run as efficiently as possible across the cluster.

Actions

Actions are the logical end-step where the computation specified by the list of transformations is triggered. So for any actual computation, an action needs to be performed. There are generally three types of action:

• View data in the console

• Collect data to native objects

• write to output data sources

This is all for this piece. I am consistently going to update this Spark Section on the blog. If you are a beginner enthusiast like me and want to learn Apache spark, feel free to reach out and we can do it together!

I am in the software/data job market. I know the market is slumpy, opportunities are scarce and one needs to apply to as many jobs and reach out to as many people in this period. I have good expertise in programming via Python, R; understanding of data structures and algorithms and have been developing my own Python package for a bioinformatics workflow. I am looking to get a foot-hold in the Software/Tech domain, and am documenting my job-hunt process. Learning something relevant daily and documenting seems like one way to attract employers and get some interviews. So, everyday I want to learn something mentioned in the job profiles and document it. For today it is:

What is Apache Spark?

The above explanation was provided by the developers of Spark: Bill Chambers & Matei Zaharia in Spark: The Definitive Guide. The key themes motivating the founders of Spark were:

1. Having a single (unified) tool to deal with a range of analytical tasks encompassing simple data load to SQL queries to machine learning and graph processing.

2. Focus on computing rather than storage. Spark works hard to deal with all kind of data sources and is primarily a computing engine: a data processing tool.

3. Provide unified API for common data analysis tasks. Spark libraries serve that purpose, with different libraries providing different functionalities. Spark SQL is associated to SQL and structured data, MLlib with Machine Learning and GraphX with Graph Analytics.

References:

1. Spark: The Definitive Guide by Bill Chambers & Matei Zaharia