This is the first post in a series of articles about modern tooling and techniques for building distributed systems. In this post, I will show how to use Docker for Windows to set up an ELK (Elasticsearch, Kibana, Logstash a.k.a. Elastic) server that we are going to use in future articles. The series is particularly geared toward traditional .NET developers.
64Kb Was Good Enough To Put a Man on The Moon
Before we talk about containers, I want to talk a little about computers.
As someone who cut his teeth on Microsoft with 5.25 inch floppy disks and MS-DOS 3.1 on a 6Mhz IBM tower that probably weighed 75 lbs, let’s just say I’ve seen a lot of technology come and go. Even though I grew up with a command line interface, I never fell in love with Linux like some of my peers. For me, while there was a lot of power you could wield, it always got in the way of productivity as I had to “own” too much of its care and feeding.
Microsoft by comparison has always favored more of an “easy button” approach, with panels on the side so you can feel free to void the warranty if you want to, but you don’t have to. This is what people either love or hate about Microsoft. Some want even less of that ability to void the warranty and move to an OSX platform. None of the choices are wrong, none are right. They just all fill different needs that individuals favor.
The flavor of OS that we choose comes with it a great deal of tribal knowledge about that platform. For example, there are parts of Windows 10 that date all the way back to 1982 and some of the first editions of DOS. Most developers now won’t have to learn much of the deep stuff about how their computer works because the low level bits have stabilized. I remember a time where I had to manually set interrupt values for individual programs to be able to bus signal from the CPU to the audio card and display processor at the same time without colliding. This still happens, except with many more physical hard-lines and automation that is a black box to the end user.
Divergent Tribes
What does any of that have to do with Docker? Great question. Over the years Linux and Windows diverged in one primary way: Heterogeny vs. Monogeny. As Windows grew and frameworks came out like .NET, the goal of those frameworks were to drive the development of software libraries closer to uniformity. Alternatively, the open nature of Linux led toward a more heterogeneous environment.
In the 90s, you basically had C++ or VB in Windows as a primary language. You had Python, Java, Haskell, Delphi, Lua, Ruby. While of course most all of those are portable, they didn’t ever take off in windows environments with the same fever pitch they did in Linux, with the exception of Java, Python, and PHP. But from a market-share standpoint on Windows Servers, you didn’t see much code in those languages except Java. Part of this was due to the nature of how Windows services worked under the hood.
You “could” use other languages but it was usually tedious. So Windows services got more and more self-similar and Linux services drifter further and further apart.
Fast forward 20-30 years, post-2010. Data ingestion is exploding, people are carrying ultra-powerful computing devices with wireless broadband capability in their pockets. Every company is a tech company (whether they realize it or not). We are living in the future.
If you are running multiple stacks in Linux, all of that activity and divergence can get complex: colliding environment variables, the need for very specific sets of frameworks that can’t co-exist, the need for different machine configurations that can’t co-exist. That’s difficult to navigate. Containers have been a huge boon to solving those problems (which you can read about in our Docker article by Zach Gardner), but I never got the warm and fuzzies about them for Windows development.
By & large, .NET has remained backwards-compatible (with the exception of .NET core), and environment conflicts don’t happen nearly as frequently in a Windows. Most of the articles I had been reading and press I had been seeing felt more like “Everyone seems on board with this and we better ride the hype train.” Not because there wasn’t a need, but because no one was making compelling arguments for Windows environments. Again, it made complete sense to me for Linux environments.
There are a great number of amazing tools that are already developed in Linux and have containers to go along with them. This is my core use case for Microsoft developers and containers. As professional engineers we have a ton to get done every day, which is why we like that easy button. Every easy button we can press instead of having to spend hours putting things together allows us to vet and explore faster, and with less resources spent. Containers let us stand up infrastructural tools in a few minutes that otherwise might take hours.
Today we are going to look at an example of this by using Docker for Windows to set up an ELK (Elasticsearch, Kibana, Logstash) server that we are going to use in future articles. Normally we would have to set up 3 sets of software. Depending on how familiar we already were with Linux this might take anywhere from hours to days. We are going to do it much faster.
Setting Up Your Environment
The first thing we’ll need to do is get Docker set up and running. You will need to have a computer that has either Windows 10 professional or Windows Server 2016 with the ability to run virtualization. For best results I recommend using Hyper-V.
Enable Hyper-V
- Go to your control panel.
- Open up Programs & Features
- Select Turn Windows feature on or off
- Click the check next to Hyper-V
- Hit OK
- Restart the machine
I wonder why Internet Explorer isn’t under Legacy Components now that they have built Edge…
Install Docker For Windows
Go here: https://docs.docker.com/docker-for-windows/install/
I’m not going to copy the instructions since you have to go there for the link anyways, but I will show you some needed setup options.
Because later in this series we are going to learn about persistence of container data, go ahead and enable a disk for use with Docker persistence.
Next, add at least 3 GB of RAM (If you don’t, ELK won’t start).
Install Kitematic
I’m going to not take you through the powershell version of installing and running containers. I want you to feel the ease of this so we are going to use the Kitematic which is a powerful GUI for your container environment. I find it to be especially useful for local workloads in development environments when you want to browse for something try it out and dump it quickly.
Go here: https://kitematic.com/
Again, I’m not going to copy the install instructions.
Add An ELK Container
Open up Kitematic and search for ELK. The one I recommend is by sebp (The one with 2million downloads) then click create.
On my machine it takes about 10-15 seconds for it to start up completely.
Docker will automatically assign local ports for you to map to the internal ports of the container. We can see and edit these by clicking on the Settings tab in the top-left, below the close button, and then clicking the Ports tab.
The one that we are most interested in at the moment is 5601, as this is the port for Kibana. On my machine it is mapped to 32782. It will likely be different on your machine.
Let’s test it out by going to http://localhost:32782.
SUCCESS!!
That’s all for this demo. Now you can play with the ELK stack to your heart’s desire without having to go stand up a bunch of VMs or messing with a plethora of frameworks and configurations. Now if you want to, you sure can, but that is outside of the scope of this article.
On the next article we look at how we can leverage ELK in a .NET application to get better insight into our applications and aggregate our logs into a single place. Stay tuned!
