Something every great engineer should knowPosted on by Iain McDonald
Skyscanner’s core values are helpful in encouraging the right choices to maintain the positive culture we’ve built up over the years. One such value is Master, Teach, Learn. It encourages us to become subject matter experts, seek opportunities to learn new things, and ensure we pass on what we’ve learned to colleagues. An opportunity recently presented itself to share what I knew about declarative programming.
Apologies for the potato like quality, but this is a video still from the fireside chat Joel Spolsky (right) had with our CTO, Bryan Dove (left), in our Edinburgh office earlier this summer. We were excited to have one of the earliest engineering bloggers, and someone responsible for Stack Overflow and Trello, share his experience with us in person.
Near the end of the chat, Bryan asked a great question and Joel’s answer surprised me.
Bryan: Are there things that you believe every great engineer should know, that they commonly don’t?
Joel: The idea of a function as a first class object. That was something I was surprised at how few programmers knew.
Joel goes on to describe how this observation was the basis for a blog post back in 2006 called Can Your Programming Language Do This? This surprised me not because I thought declarative programming was a common practice – in all the interviews where I’ve asked a question requiring iteration over a collection, I’ve yet to see a single answer that doesn’t use a for loop. I was surprised because I’d written a blog post along similar lines the previous year.
To help us with the Master, Teach, Learn core value, we have an internal training platform called Skyscanner University. This allows us to attend training courses from external presenters, but crucially, it also allows anyone – any employee – to create their own course and teach to whomever wants to attend. In my time here I’ve learned about Docker containers, coaching, leadership… even home brewing – thanks Raymond 😉
I decided to find out by asking in the #engineering channel in Slack, if anyone would be interested in learning more about declarative programming. I was happy to find a dozen or more interested engineers, so I created a short course (with apologies to Mike, one of our resident functional gurus).
Higher Order Functions
Ganglia showing all nodes in the EMR cluster with CPU at 100%
Sharing understanding of higher order functions first occurred when I was a Data Scientist creating Elastic MapReduce clusters in AWS to run large Spark jobs to transform and aggregate loosely structured data – a key constituent of Machine Learning projects. I wouldn’t have known where to begin if it hadn’t been for Jon Skeet and his reimplementation of LINQ, which I worked through several years earlier.
Enabling parallelisation with the Spark framework is just one advantage of knowing how to code in a declarative style. Declarative code is an evaluation of expressions, whereas imperative code is a series of statements that mutate state. The reduction of dependence on state creates code that’s easier to test in an automated manner because less mocking is required; code that’s more stable to ongoing change because it has less dependence on state side effects; code with greater separation of concerns which aids readability and conceptual understanding.
> const square = x => x * x;
But why is the ability to store a function in a variable so valuable? First class functions enable the use of higher order functions i.e. functions that take functions as parameters, either stored in an intermediate variable or written as an anonymous function, and these function parameters are called during the processing of the higher order function. Or as Xzibit puts it…
Knowledge & Understanding versus Problem Solving
It’s not enough to know what a higher order function is, or even what a few specific ones do. The skill in declarative programming comes from practicing solving problems. Mastering declarative programming takes both knowledge and understanding and problem solving. The slide deck I presented in the Skyscanner University course was only half the story. The fun came from solving a small problem using only higher order functions in a stateless manner.
Declarative Programming Repo Problem – can you solve it?