We will do this with the containers running on the same machine in the first instance. I also assume that you have at least basic experience with a cloud provider and as such are able to set up a computing instance on your preferred platform.įirst we need to get to grips with some basic Docker networking. For a full drawn out description of the architecture and a more sequential walk through of the process I direct the reader to my github repo. With considerations of brevity in mind this article will intentionally leave out much of the detail of what is happening. Therefore, I do not recommend this article if either of these two technologies are new to you. I assume knowledge of Docker commands and terms as well as Apache Spark concepts. Apache Spark providing the analytics engine to crunch the numbers and Docker providing fast, scalable deployment coupled with a consistent environment. With the rise of Big Data these two technologies are a matched made in heaven. Finally, Docker provides an abstraction layer called the Docker Engine that guarantees compatibility between machines that can run Docker solving the age-old headache of “it works on my machine, I don’t know why it doesn’t on yours”. Furthermore, due to its use of linux containers users are able to develop Docker containers that can run be run simultaneously on a single server whilst remaining isolated from each other. Docker provides users the ability to define minimal specifications of environments meaning you can easily develop, ship, and scale applications. In 2014 Spark won the Gray Sort Benchmark test in which they sorted 100TB of data 3x faster using 10x fewer machines then a Hadoop cluster previously did.ĭocker on the other hand has seen widespread adoption in a variety of situations. It’s adoption has been steadily increasing in the last few years due to its speed when compared to other distributed technologies such as Hadoop.

Two technologies that have risen in popularity over the last few years are Apache Spark and Docker.Īpache Spark provides users with a way of performing CPU intensive tasks in a distributed manner.

Running kafka in docker on mac