Introduction

Convolutional neural networks (or convnets for short) are used in situations where data can be expressed as a “map” wherein the proximity between two data points indicates how related they are. An image is such a map, which is why you so often hear of convnets in the context of image analysis. If you take an image and randomly rearrange all of its pixels, it is no longer recognizable. The relative position of the pixels to one another, that is, the order, is significant.

Convnets are commonly used to categorize things in images, so that’s the context in which we’ll discuss them.

A convnet takes an image expressed as an array of numbers, applies a series of operations to that array and, at the end, returns the probability that an object in the image belongs to a particular class of objects. For instance, a convnet can let you know the probability that a photo you took contains a building or a horse or what have you. It might be used to distinguish between very similar instances of something. For example, you could use a convnet to go through a collection of images of skin lesions and classify the lesions as benign or malignant.

Convnets contain one or more of each of the following layers:

• convolution layer

• ReLU (rectified linear units) layer

• pooling layer

• fully connected layer

• loss layer (during the training process)

We could also consider the initial inputs i.e. the pixel values of the image, to be a layer. However, since no operation occurs at this point, I’ve excluded it from the list.

Convolution Layer

The architecture of a convnet is modeled after the mammalian visual cortex, the part of the brain where visual input is processed. Within the visual cortex, specific neurons fire only when particular phenomena are in the field of vision. One neuron might fire only when you are looking at a left-sloping diagonal line and another only when a horizontal line is in view.

Our brains process images in layers of increasing complexity. The first layer distinguishes basic attributes like lines and curves. At higher levels, the brain recognizes that a configuration of edges and colors is, for instance, a house or a bird.

In a similar fashion, a convnet processes an image using a matrix of weights called filters (or features) that detect specific attributes such as diagonal edges, vertical edges, etc. Moreover, as the image progresses through each layer, the filters are able to recognize more complex attributes.

To a computer, an image is just an array of numbers. An image using the full spectrum of colors is represented using a 3-dimensional matrix of numbers.

The depth of the matrix corresponds to RGB values. For the sake of simplicity, we’ll initially consider only grayscale images. Each pixel in a grayscale image can be represented using a single value that indicates the intensity of the pixel. These values lie between 0 and 255, where 0 is black and 255 is white.

The convolution layer is always the first step in a convnet. Let’s say we have a 10 x 10 pixel image, here represented by a 10 x 10 x 1 matrix of numbers:

I’ve stuck to zeroes and ones to make the math simpler. You’ll see what I’m talking about in a bit.

The output of a convolution layer is something called a feature map (or activation map).

In order to generate a feature map, we take an array of weights (which is just an array of numbers) and slide it over the image, taking the dot product of the smaller array and the pixel values of the image as we go. This operation is called convolution. The array of weights is referred to as a filter or a feature. Below, we have a 3 x 3 filter (as with the image, we’ve used 1s and 0s for simplicity).

We then use the filter to generate a feature map

The image shows the calculation for the first dot product taken. Imagine the filter overlaid on the upper left-hand corner of the image. Each weight in the filter is multiplied by the number beneath it. We then sum those products and put them in the upper left hand corner of what will be the feature map. We slide the filter to the right by 1 pixel and repeat the operation, placing the sum of the products in the next slot in the feature map. When we reach the end of the row, we shift the filter down by 1 pixel and repeat.

Filters are used to find where in an image details such as horizontal lines, curves, colors, etc. occur. The filter above finds right-sloping diagonal lines.

Note that higher values in the feature map are in roughly the same location as the diagonal lines in the image. Regardless of where a feature appears in an image, the convolution layer will detect it. If you know that your convnet can identify the letter ‘A’ when it is in the center of an image, then you know that it can also find it when it is moved to the right-hand side of the image. Any shift in the ‘A’ is reflected in the feature maps. This property of the convolution layer is called translation equivariance.

Thus far, we’ve been discussing the convolutional layer in the context of a grayscale image with a 2-dimensional filter. It’s also possible that you would be dealing with a color image, the representation of which is a 3-dimensional matrix, as well as multiple filters per convolutional layer, each of which is also 3-dimensional. This does require a little more calculation, but the math is still basically the same.

You’ll see that we’re still taking the dot product, it’s just that this time around we need to add the products along the depth dimension as well.

In the above example, we convolved two 3 x 3 x 3 filters with a 6 x 6 x 3 image and the result was a 4 x 4 x 2 feature map. If we had used three filters, the feature map would have been 4 x 4 x 3. If there were 4, then the size would have been 4 x 4 x 4, and so on.

To get the dimensions of the feature map is fairly straightforward.

Our input image has a width of 6, a height of 6, and a depth of 3, i.e. wi= 6, hi = 6, di = 3. Each of our filters has a width of 3, a height of 3 and a depth of 3 i.e. wf = 3, hf = 3, df = 3.

The stride is the number of pixels you move the filter between each dot product operation. In our example, we would take the dot product, move the filter over by one pixel, and repeat. When we get to the edge of the image, we move the filter down by one pixel. Hence, the stride in our example convolution layer is 1.

The width and height of the feature map are calculated like so:

where s is the stride.

Let’s break that down a bit. If you have a filter that’s 2 x 2 and your image is 10 x 10, then when you overlay your filter on your image, you’ve already taken up 2 spaces across, i.e. along the width. That counts for 1 spot in the feature map (that’s the ‘+1’ part). To get the number of spots left, you subtract the filter’s width from the total width. In this case, the result is 8. If your stride is 1, you have 8 spots left, i.e. 8 more dot products to take until you get to the edge of the image. If your stride is 2, then you have 4 more dot products to take.

The depth of the feature map is always equal to the number of filters used; in our case, 2.

ReLU Layer

The ReLU (short for rectified linear units) layer commonly follows the convolution layer. The addition of the ReLU layer allows the neural network to account for non-linear relationships, i.e. the ReLU layer allows the convnet to account for situations in which the relationship between the pixel value inputs and the convnet output is not linear. Note that the convolution operation is a linear one. The output in the feature map is just the result of multiplying the weights of a given filter by the pixel values of the input and adding them up:

where w is a weight value and x is a pixel value.

The ReLU function takes a value x and returns 0 if x is negative and x if x is positive.

As you can see from the graph, the ReLU function is nonlinear.

In this layer, the ReLU function is applied to each point in the feature map. The result is a feature map without negative values.

Other functions such as tanh or the sigmoid function can be used to add non-linearity to the network, but ReLU generally works better in practice.

Pooling Layer

The pooling layer also contributes towards the ability of the convnet to locate features regardless of where they are in the image. In particular, the pooling layer makes the convnet less sensitive to small changes in the location of a feature, i.e. it gives the convnet the property of translational invariance in that the output of the pooling layer remains the same even when a feature is moved a little. Pooling also reduces the size of the feature map, thus simplifying computation in later layers.

There are a number of ways to implement pooling, but the most effective in practice is max pooling. To perform max pooling, imagine a window sliding across the feature map. As the window moves across the map, we grab the largest value in the window and discard the rest.

As mentioned earlier, the output indicates the general region where a feature is present, as opposed to the precise location, which isn’t really important. In the above diagram, the result of the pooling operation indicates that the feature can be found in the upper left-hand corner of the feature map, and, thus, in the upper left hand corner of the image. We don’t need to know that the feature is exactly, say 100 pixels down and 50 pixels to the right relative to the top-left corner.

As an example, if we’re trying to discern if an image contains a dog, we don’t care if one of the dog’s ears is flopped slightly to the right.

The most common implementation of max pooling, and the one used in the example image, uses a 2 x 2 pixel window and a stride of 2, i.e. we take the largest value in the window, move the window over by 2 pixels, and repeat.

The operation is basically the same for 3D feature maps as well. The dimensions of a 3D feature map are only reduced along the x and y axes. The depth of the pooling layer output is equal to the depth of the feature map.

The Fully-Connected and Loss Layers

The fully-connected layer is where the final “decision” is made. At this layer, the convnet returns the probability that an object in a photo is of a certain type.

The convolutional neural networks we’ve been discussing implement something called supervised learning. In supervised learning, a neural network is provided with labeled training data from which to learn. Let’s say you want your convnet to tell you if an image is of a cat or of a dog. You would provide your network with a large set of pictures of cats and dogs, where pictures of cats are labeled ‘cat’ and pictures of dogs are labeled ‘dog’. This is called the training set. Then, based on the difference between its guesses and the actual values, the network adjusts itself such that it becomes more accurate each time you run a test image through it.

You confirm that your network is in fact able to properly classify photos of cats and dogs in general (as opposed to just being able to classify photos in the training set you provided) by running it against an unlabeled collection of images. This collection is called the test set.

In this example, the fully-connected layer might return an output like “0.92 dog, 0.08 cat” for a specific image, indicating that the image likely contains a dog.

The fully-connected layer has at least 3 parts - an input layer, a hidden layer, and an output layer. The input layer is the output of the preceding layer, which is just an array of values.

You’ll note in the image, there are lines extending from the inputs

to nodes

that represent the hidden layer (so called because they’re sandwiched between the input and output layers and, thus, “invisible”). The input values are assigned different weights

per connection to a node in the hidden layer (in the image, only the weights for the value

are labeled). Each of the circles in the hidden layer is an instance of computation. Such instances are often called neurons. Each neuron applies a function to the sum of the product of a weight and its associated input value.

The neurons in the output layer correspond to each of the possible classes the convnet is looking for. Similar to the interaction between the input and hidden layer, the output layer takes in values (and their corresponding weights) from the hidden layer, applies a function and puts out the result. In the example above, there are two classes under consideration - cats and dogs.

Following the fully-connected layer is the loss layer, which manages the adjustments of weights across the network. Before the training of the network begins, the weights in the convolution and fully-connected layers are given random values. Then during training, the loss layer continually checks the fully-connected layer’s guesses against the actual values with the goal of minimizing the difference between the guess and the real value as much as possible. The loss layer does this by adjusting the weights in both the convolution and fully-connected layers.

Hyperparameters

It should be said at this point that each of these layers (with the exception of the loss layer) can be multiply stacked on one another. You may very well have a network that looks like this:

convolutional layer >> reLU >> pooling >> convolutional layer >> pooling >> fully-connected layer >> convolutional layer >> fully-connected layer >> loss layer

There are other parameters in addition to the order and number of the layers of a convnet that an engineer can modify. The parameters that are adjusted by a human agent are referred to as hyperparameters. This is where an engineer gets to be creative.

Other hyperparameters include the size and number of filters in the convolution layer and the size of the window used in the max pooling layer.

Overview

So, let’s run through an entire convolutional neural network from start to finish just to clarify things.

We start with an untrained convnet in which we have determined all of the hyperparameters. We initialize the weights in the convolution and fully-connected layers with random inputs. We then feed it images from our training set. Let’s say we have one instance of each layer in an example network. Each image is processed first in the convolution layer, then in the ReLu layer, then in the pooling layer. The fully-connected layer receives inputs from the pooling layer and uses these inputs to return a guess as to the contents of the image. The loss layer compares the guess to the actual value and figures out by how much to adjust the weights to bring the guess closer to the actual value. For instance, if the convnet returns that there is an 87% chance that an image contains a dog, and the image does indeed contain a dog, the guess is off by 13% and the weights are adjusted to bring that guess closer to 100%.

Other Resources

• Understanding Neural Networks Through Deep Visualization by Jason Yosinksi, Jeff Clune, Ang Nguyen, et al.

• How Do Convolutional Neural Networks Work? by Brandon Rohrer

• A Quick Introduction to Neural Networks by Ujjwal Karn

• Convolutional Neural Networks for Visual Recognition by Andrej Karpathy

On a few occasions, I’ve started work on a brownfield project and found that, either due to poor programming practices or because there was no elegant way to implement a feature, it’s been necessary to create documentation beyond what is normally included in a README. This documentation often took the form of glossaries, diagrams, or additional exposition on the codebase.

Your typical README contains all or some of the following: a short introductory paragraph, a list of dependencies, instructions on how to install the app, configuration, how to run the test suite, database creation and migration, services, and how to deploy the app. If it’s an open source project, there will likely be sections devoted to listing authors, rules around contributing, the license under which it can be used, as well as additional documentation around how to use the package.

There are contexts in which this is inadequate. Let’s describe such a situation in detail.

You’ve recently been assigned to a brownfield project. You’re looking at the app for the first time, and you notice fairly quickly that the codebase looks like a Jackson Pollock painting.

For starters, the company pivoted a couple years ago, so large portions of the app remain that aren’t relevant anymore. Perhaps entities are named in a way that no longer accurately describes them. Maybe there are references to features that have long since been abandoned.

Obscured by all of this cruft are the operative parts of the app, which aren’t much to look at, either. There are models named Article, Post, and Essay as well as models named Writer, Scribe, and Author. For business logic reasons, an Article can belong to an Author or a Writer, but not a Scribe, a Post can belong to all three, and an Essay can belong only to an Author.

There’s a “God” model that encompasses multiple concepts, another model that should really implement a state machine, etc.

In short, this app needs a serious overhaul, but maybe you’ve been brought on to implement just a few features or the client simply doesn’t have the budget to pay for an overhaul right now or the powers-that-be don’t see the value in well-crafted software and aren’t willing to pay you to refactor the code, no matter how many times you explain that substandard code actually costs more in the long run. Heck, maybe the client thinks the code is just fine and not in need of refactoring at all.

For whatever reason, refactoring this code is not something you’ll be able to do right now and you suspect that the technical debt that’s accumulated will only increase for the foreseeable future, making the app ever more brittle and difficult to add features to.

In an ideal world, this codebase would have been cleaned up ages ago. In the meantime, you still have to help your client.

Unfortunately, the mental overhead of working on this app makes development a grind. You’re constantly having to remind yourself of the subtle differences between an Author, a Writer, and a Scribe and how they relate to the other models in the application.

What’s needed is documentation that will reduce the mental overhead that comes with working with certain kinds of apps; basically, an augmented README. For simplicity, I’ll refer to this ‘superREADME’ as a VELLUM (’cause it’s fancy).* I’ll also stop screaming README(!) and VELLUM(!) at you from here on out.

A vellum and a readme are quite similar (really, the former could be thought of as a ‘superset’ of the latter). The difference is that a vellum adheres to a longer, more expository style of documentation than is typically found in readmes and there is an expectation that the author(s) of the vellum will reference it with some frequency. A vellum is the place where you document your confusion.

A vellum can serve as a guide through less-than-ideal codebases, but it also has a place in well-architected projects (brownfield or greenfield) that have complex domains and require a bit of explication. You would use the vellum as a way to note difficult or subtle concepts around the domain or the codebase that any developer coming into the project should know before starting work.

The remainder of this blog post will cover what should go into a thorough vellum file and some of the pitfalls of over-reliance on such documentation. It will be written from the perspective of a consultant working on a client application, but that isn’t to say that vellums can’t be carried over to open-source contexts. They can certainly be used to convey ideas that will help other or new contributors better understand the codebase.

Introduction

The first part of the vellum should consist of an introduction to the application including its purpose, its audience, and details around the business itself, such as how the app makes money. At this point, you’re providing context.

In the example below, we write an introduction to an application that helps users manage their stock portfolios.

This application is called SmartMoney. It enables the casual trader to manage their portfolio of stocks. The app makes money by charging traders a flat per month fee of $5, in contrast to the payment structure of other services where users are charged per trade. The app allows users to make a wide variety of orders. Namely:

• market orders

• limit orders

• stop-loss orders

• stop-limit orders

• bracketed orders

• trailing stop-loss orders

The app also provides a simple interface for shorting stocks. Some other key features of the app include the ability to set up notifications for when a particular stock’s worth has dropped below or risen to a certain value. The app also has a recommendation engine that suggests stocks to invest in that are “hot” and in an industry in which the trader has shown interest.

When development first began, the app was geared towards only sending notifications based on changes in stock prices. However, the company decided to make a more fully featured app as customers wanted an experience that integrated setting up notifications with making orders.

You’ll note that the introduction also lays out some terms (the types of orders that are available in the app) that the next developer should understand before writing any code (more about that soon). It also mentions the app’s history, which helps to explain any cruft that might be left behind.

Your goal is to convey enough salient information about the app that any developer could just open the vellum and know who this app is for, why certain quirks of the app are present, etc. You don’t want to go into excruciating detail. Save that for storyboarding.

Glossary

You should follow this introductory paragraph with a glossary of terms. The glossary should include any terms you think a developer stepping into this project should be aware of, but reasonably might not know.

...

• ETF (Exchange Traded Fund): a collection of assets (stocks, bonds, etc.) that can be traded like a common stock, i.e. partial ownership in a corporation.

• P/E Ratio (Price-Earnings Ratio): measures the amount of money traders are willing to pay for each dollar of earnings from a single share. It is equal to the stock price divided by the earnings per share.

• PEG Ratio (Price-Earnings to Growth Ratio): It is similar to the P/E ratio, except that it takes the growth of the company into account. It is equal to the P/E ratio divided by the growth rate of the company.

• Stop-Loss Order: an order to sell a stock if its price drops below a specified value

...

Pitfalls

Your vellum file should also contain a section devoted to potentially confusing aspects of the app. You want to provide clarification on things like

• complex models or relations between models

• confusing or unusual model names

• distinctions between similar or closely-related models

• non-obvious relations between what a developer is dealing with on the backend versus what the user sees.

Bracketed Order Model: This model corresponds to bracketed orders. On the backend, it is a bracketed order, but for the user, it is a ‘One-Cancels-the-Other Order’.

Architectural Decisions

You should also take the time to justify non-obvious architectural decisions. This may include talking about design patterns you used:

We opted for using the Factory design pattern to handle the creation of order objects. Order types are quite similar and we want the client to be order-agnostic.

README

The final section should discuss the topics you would cover in a readme, including things like configuration, how to run the test suite, database creation and migration, services, and how to deploy the app, etc. I assume you’ve already written your fair share of readmes. If not, read this brief tutorial on how to write a kick ass readme.

Caveats

You don’t want to over-rely on your vellum file to the extent that you abandon good consulting and programming practices.

If you’re walking into a codebase badly in need of a refactor, this should be conveyed to the client. In such a situation, a vellum is stopgap documentation, i.e. explanation that makes up for code that is not self-explanatory.

If you’re writing the application from scratch, a vellum is used to elucidate complexity. It’s not an excuse to write opaque code or inexact stories.

Certainly, not every project requires a vellum file or all of the parts of a vellum detailed here. Simpler applications will generally not require one as there isn’t enough mental overhead to justify having this extra, rather involved document you have to maintain.

A good rule of thumb is that if you don’t have to consult the vellum very often, then you can go ahead and delete it. Conversely, if you constantly find yourself checking your notes while coding, your notes likely contain information that should be extracted into some kind of documentation.

Of course, all of the topics I described could simply be appended to a readme file. The important thing is to remember to reduce confusion for yourself and subsequent developers.

* Vellum is a high-quality parchment made from the skin of calves

Related articles:

Acceptable Technical Debt

A view is a stored query the results of which can be treated like a table. Note that it is the query that is saved and not the results of the query. Each time you use a view, its associated query is executed.

A related concept is that of the common table expression or CTE. A CTE can be thought of as a temporary view; you can only use it within the query in which it appears (you can refer to it multiple times, however).

Let’s say that you’re the SQL-savvy owner of a store that sells kits for building robots. On your site, customers are guided through the process of selecting all of the components needed to build a quadrupedal robot, i.e. getting the right motors, microcontrollers, sensors, batteries, grippers, etc.

Robots have lots of component parts and you have to make sure that the correct components for each order arrive or your customers will leave mean comments on social media.

Right now you have the following tables:

And those tables contain the following data:

You want to see a list of all the purchases made and the components in those purchases. You might write a query like the following:

Just a quick explanation in case anything in the query is unclear. We’re joining the tables purchases, component_purchases, and components and then grouping the rows by purchase_id. SQLFiddle here.

For each of the components in a purchase, the sum function adds the multiple of the number of units of a component ordered and the price of a component. The array_agg function composes an array of the details of the purchase. Notice that the sum and array_agg functions act across rows, but the concat function acts across columns.

The result of our query is thus:

Check out the SQLFiddle here.

It’s not an exceptionally long query, but you suspect that you’ll be using it quite a lot and you’d prefer not to have to write the whole thing out all of the time.

That’s where views come in:

Now you have a simple way to reuse this query. Let’s say that you want to know all of the purchases made within the last week. Now you can just write the following:

SQLFiddle here

From within the PostgreSQL interactive terminal, you can use the command \dv to see a list of all the views in the database.

Dropping a view is as simple as dropping a table:

Note that if you drop a table using cascade, any views dependent on that table will be dropped as well.

It’s also possible to perform operations on the data in the underlying tables of some views. Updatable views are views where data can be edited, inserted, or deleted. To be updatable, a view must be really basic. Think something along the lines of:

Updatable views can only select rows from a single table (or updatable view). They also cannot contain ‘group_by’, ‘limit’, or ‘with’ clauses.

You can see all of the relevant restrictions in the Postgres docs here.

If you do have an updatable view, you can do something like

A new row for ‘Anne Droid’ has been inserted into the customer table.

Note that the customer_info view only includes customers whose names begin with letters after ‘E’. If you were to run the following query:

Anne’s name would not appear in the results because her last name begins with a ‘D’. A new row has been inserted into the customers table using the customer_info view, but that row is not accessible via the view. SQLFiddle here.

Note that you cannot insert values for columns that are present in the underlying table, but not in the view. The following query would result in an error:

So, that’s views in a nutshell. Let’s move on to CTEs.

To start with, CTE stands for ‘common table expression’. They’re quite similar to views except that once the query is executed, a CTE ceases to be.

In the view purchase_information, we were able to get all of the components in a purchase, as well as the price of the purchase. Let’s say I wanted a query that would provide me with purchase information, but also the customers to whom the purchases belonged. This can be achieved pretty easily using a common table expression.

You start a CTE using the keyword with followed by the name of the CTE and the query itself.

SQLFiddle here

The purchases CTE can be referred to like a table. Of course, once the query is executed, purchases doesn’t exist.

The results of this query are:

Phew!

Now you can send the correct robot parts to your customers.

But wait! There’s some complexity to the orders that you’ve forgotten. Some components need to be combined with other components in order to work. For the sake of simplicity, let’s say that any given component requires one or fewer other components to function.

Drawing out the relationship between our components, we see that we get a tree graph.

To describe this in the database, we’ll need to add another column to the component table. The subcomponent_id field will point to the component upon which the current component depends.

Let’s say that a customer orders the component with id 4. In order to get all of the parts the customer needs, we’ll need to traverse the tree all the way down.

Luckily, Postgres has just the solution for this - recursive common table expressions. Recursive CTEs provide a way to allow queries to perform some action with a table, and then to perform the same action on the results of the previous action.

That’s probably kind of confusing.

Below is the structure of a recursive common table expression:

The recursive CTE begins much like a non-recursive CTE. You’ll notice that the modifier recursive is required after with. Immediately after recursive is the name of the CTE.

Above the union operator is a query that initializes the working table. The working table is the result set upon which the recursive term is currently acting. For instance, you might put select id, name from robots where name=’Cindi Mayweather’. Since there is one and only one of the illustrious Cindi Mayweather, this query will return one row.

Note that this query is not executed again.

We’ll come back to what purpose the union operator serves in a bit. For now, let’s talk about the recursive query. The recursive query will make a reference to the CTE name. Wherever you see the CTE name in the query, that is a reference to the working table.

Given a single component that needs to be shipped, we need to fetch the ids of all of its subcomponents, and their subcomponents, etc. To get all of the subcomponents for the component with an id of 4, we use the following query:

That’s pretty gross-looking, so let’s go through what’s happening step by step.

Execution of the CTE starts with the non-recursive query. Running that query returns the following:

This is the first working table.

We see that an inner join of the working table and the components table produces the following:

From this, we get the next working table.

The working table is substituted for purchase_components in the recursive query, with the following result:

And so on...

The recursion ends when the working table has no rows, i.e. when the values for the rows are all null.

Now we come to what union is doing. The union operator acts on all of the working tables that have been generated, combining them all into one table. To ensure that there are no duplicate rows, you use union all.

Hence the result of running the query is

SQLFiddle here.

Boom! Nailed it.

Kinda.

If you recall, up top we have a list of purchases with the associated components and customer information. Well, that component list isn’t complete; we have to include subcomponents as well.

SQLFiddle [here](http://sqlfiddle.com/#!17/03571/6)

In this query, we have two CTEs. When you want to use multiple CTEs in a query, you place with at the start of the group and separate each CTE with a comma. Notice that, even though only one of the CTEs is recursive, the recursive modifier is placed directly after with, as opposed to directly before the recursive CTE. Each CTE is executed normally. Another nifty thing about using recursive is that its presence means your CTEs can be in any order, i.e. one CTE can make reference to a CTE that is defined later. If you’re wondering how the component_id alias showed up in the final results, remember that aliases are preserved in the results of a union.

To understand how this query works, we first need to understand what the results of the purchases_made query are.

SQLFiddle [here](http://sqlfiddle.com/#!17/03571/7)

This is our first working table.

Following a process similar to the one in the earlier recursive query, the first inner join results in:

The results of the purchase_components CTE can be found in this SQLFiddle.

I would recommend drawing out tables in a notebook to really hammer home some of the ideas presented in this blog post. And feel free to play around in the SQLFiddle examples provided. It’s a pretty great tool.

Related articles

• Tree Traversal Algorithms

• Graph Algorithms in a Database: Recursive CTEs and Topological Sort in Postgres

• Understanding Common Table Expressions with FizzBuzz

Optionals: The Basics

In Swift, the optional type is used to indicate that a variable (or constant) might not hold a value, i.e. that it might equal nil. Based on this, you may suspect that a non-optional variable cannot be assigned a value of nil, and you would be correct.

You might be wondering if you can make use of type inference and simply assign nil to a variable for which no type has been assigned. Nice try but, alas:

The compiler is trying to infer the type here, but you haven’t really given it one. You’ve only indicated that stillCannotBeNil is an optional, not if it’s an optional String or Int or what have you.

Anyhow, let’s go ahead and declare an optional variable:

We could also make the declaration like so:

Optional declarations can be formally declared with Optional<TypeName> as well as the provided syntactic sugar in the form of TypeName?. Either way is acceptable, but be aware that the latter is more common.

You’ll note in the example above that canTotesBeNil can totes be nil and, in fact, is nil. When you declare an optional variable, its default value is nil.

Let’s assign a value to canTotesBeNil and print it out.

Gross.

What’s happening here? If an optional is equal to nil, then you can go ahead and treat it as you’d expect. However, if an optional has a non-nil value, you have to unwrap it using the force-unwrap operator (!) to get access to its value.

Boom!

This works great when canTotesBeNil is totes not nil. If canTotesBeNil is totes nil, there’s some awfulness.

One way to get around this would be to check if canTotesBeNil is equal to nil before unwrapping it.

There’s a simpler way to do this that we’ll talk about later.

At this point, you’re probably wondering about optional constants and when in the heck you’d ever use them.

I mean, you’d likely never do something like this:

You would more likely use them in cases where a method returns an optional value and you have no intention of changing said value.

So that’s optionals in a nutshell.

Right now optionals probably seem like some pain-in-the-ass BS, what with having to constantly check if they have a value and the insulting error messages they give you. Fortunately, Swift provides a number of ways to avoid peppering your code with if myOptional != nil.

Optionals and Control Flow

So, if you want to run one bit of code when your optional has a value, and another when it doesn’t, Swift has you covered with optional binding. Optional binding allows you to check for and assign the value of an optional

You see this mess?

This is lame and you shouldn’t do it. Instead, use optional binding:

Notice anything special about the print statement on line 2? We didn’t have to force unwrap the argument to print. If methodThatReturnsAnOptionalString() returns a non-nil optional, then textToPrint is set to that value. Otherwise, the code in the else branch is executed.

There is one gotcha, however. The variable declaration after the if is not a normal declaration. textToPrint is only available in the first branch of the conditional. If you attempt to use it in the else branch or outside of the conditional, you will get an error.

Optional binding also allows for variables, meaning you can do this:

Note, however, that if textToPrint is never mutated you will get an error from the compiler asking you to change it to a constant.

You can also make use of optional binding in guard statements.

Optional binding in the context of a guard statement is a little different. Unlike the if conditional above, textToPrint is only available outside of the guard statement. Hence, we’re able to print it.

Nil-Coalescing Operator

Let’s say you would like to set a variable to the value of an optional if the optional is non-nil or to a default value if it’s nil. You can accomplish this using the ternary operator.

Alternatively, you could make use of the nil-coalescing operator.

If the optional is not nil, the nil-coalescing operator will assign its value to getStuff. If the optional is nil, getStuff is assigned the provided default. One potential gotcha lies in a type mismatch between the optional and the default. The default must be of the same type as the optional. For instance, you couldn’t do something like this:

Ouch.

Optional Chaining

What about situations in which you want to call a property or method on an optional? You might, for instance, want to get the size of an optional array. In order to do that, however, you’d have to force unwrap it, potentially triggering an ego-obliterating error.

Optional chaining lets you call whatever properties or methods you would like to on an optional and to get back nil if the optional is empty.

In contrast to the force-unwrap operator, which has no chill, the question mark operator loves and accepts you for who you are and does not trigger an error.

One important thing to note is that optional chaining always returns an optional value, given that its result could be nil.

I imagine some part of you wants to write something like this to get ahold of the value of count directly:

I see what you’re trying to do, but the above will bring the wrath of the compiler upon you. What you’re actually aiming for is

Implicitly Unwrapped Optionals

It’s likely you won’t make much use of implicitly unwrapped optionals outside of class initialization. I don’t want to get too deep into classes, but we’ll talk about them briefly here.

Implicitly unwrapped optionals are used whenever you are absolutely, positively sure that your application will not make any attempt to get an optional’s value before it has been set. Implicitly unwrapped optionals are kinda unsafe, so you want to be dead certain that they’re what you want. If you’re not sure, just use a plain old optional.

Note the absence of the force-unwrap operator in the print statement. That’s because dangerString has been unwrapped for you.

Keep in mind, though, that if you were to try to print dangerString without first giving it a value (or if, like some kind of adrenaline-junkie, you set it to nil later), you would get an error. That’s what makes implicit unwrapping so dangerous. So, you know, be careful.

Optionals in the Wild

Try to keep in mind that optionals will always be used whenever it is possible that a value will not be present. That’s not simply a guideline for how you should write your own code, but a general rule for what you should expect when interacting with Swift objects. For instance, if you try to receive a value from a dictionary, you will get an optional.

Similarly, if you attempt to convert a String to an Int using the Int initializer, you get an optional.

Optionals. Optionals everywhere.

Get Swifty

Great. So…get out there and use some optionals. Be super inspired. Be the best. Dream big. Swift hard. Good night.

For those not familiar with the TIL format, it’s a short blog post wherein you describe something you learned recently. It doesn’t have to be anything terribly impressive. It could be some obscure Ruby method you discovered or a neat thing you can do with Git. It’s a fun way to catalogue the small facts you learn everyday.

Today I Learned: Squeaky Clean Anaconda Environments

originally published 7/24/2017 on the Hashrocket TIL blog

To delete packages, caches, and other files that aren’t being used in any of your environments run:

conda clean -a -y

The -a flag is to delete all unused files and the -y option runs the clean command without asking for confirmation.

If you want to know what files would be deleted before actually deleting them, run:

conda clean -a --dry-run

Today I Learned: Keep Your Brews Bubbly

originally published 8/25/2017 on the Hashrocket TIL blog

Life is too short to have Homebrew problems. Run these commands to keep your brews bubbly.

Checks your system to make sure that future installs go smoothly:

brew doctor

Upgrades Homebrew to the latest version:

brew update

Gets a list of what packages are outdated:

brew outdated

Looks through your installed packages and deletes any old versions that may still be hanging around:

brew cleanup

You could alternatively add the --dry-run flag to cleanup to see all the outdated packages that would be removed.

Deletes old symlinks:

brew prune

Updates packages to the latest version:

brew upgrade

You can add the --cleanup flag to delete older versions of the packages you are updating.

Today I Learned: Viewing the Git Leaderboard

Originally published 8/31/2017 on the Hashrocket TIL blog

Everyone knows that writing code is a competition in which she with the most commits wins. So, obviously, you want to periodically check the leaderboard to see who the 10x developer on the team is.

How do you do that?

git shortlog

This will give you a list of all the contributors to the repository and the number of commits they have made. You can add the -s flag (for ‘summary’) to just see names and numbers.

Once you have acquired the evidence that you are indeed the mythical 10x developer, demand a raise from your employer and threaten to go work for Google if you don’t get it.

If you already work for Google, threaten to...I don’t know, get a job at Space X or something.

Today I Learned: Ruby Binding Class

originally published 10/16/2017 on the Hashrocket TIL blog

Ruby’s Binding class allows you to access classes, variables, and methods outside of your current scope.

class Foo

	def bar

		my_var = 20

		binding()

	end

end

Normally, if you made reference to my_var outside of that method, you’d get an error.

Binding objects allow you to do this:

my_foo = Foo.new

foo_bar = my_foo.bar

puts foo_bar.eval(’my_var’)

# 20

Today I Learned: Hi, Sierra

originally published 11/25/2017 on the Hashrocket TIL blog

I recently upgraded my Mac to High Sierra and got this gross message when I tried to use Git:

xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

My machine was expecting Git to be implemented as an Xcode command line tool. To fix the problem, I simply installed Xcode.

xcode-select --install

Today I Learned: Aliasing an Elixir Module Within Itself

originally published 1/26/2018 on the Hashrocket TIL blog

I was attempting to compile a Phoenix application and I got this error:

Post.__struct__/0 is undefined, cannot expand struct Post

The issue was in a function I defined in the module.

def changeset(%Post{}= post, attrs \\ %{}) do

    ...

end

I assumed that you would get references to a module within said module for free. That’s not the case.

There are two ways to fix the error. One is to use the full module name in the parameter list.

def changeset(%Forum.Post{}= post, attrs \\ %{}) do

    ...

end

Alternatively, I can alias the module within itself.

alias Forum.Post

Today I Learned: Prying Open Your Phoenix App

originally published 1/29/2018 on the Hashrocket TIL blog

I wanted to do some debugging in a Phoenix app, so I threw require IEx and IEx.pry() into my code, only to receive the following error:

Cannot pry
...

Is an IEx shell running?

In order to pry into your code, you’ll have to start the server from within the Elixir shell.

iex -S mix phx.server

Today I Learned: Phoenix Select Form Helper

originally published 1/30/2018 on the Hashrocket TIL blog

The select form helper allows you to easily add a select input to your forms.

= form_for @changeset, resource_path(@conn, :create), fn f ->

  = select f, :book_id, @books

  = submit “Save Post”, class: “btn”

Among the types of arguments that the select helper can accept are two-item tuples. The first item in the tuple is used as the label for the option and the second item is used as the value for the option.

Media.list_books returns a list of structs representing all of the books in the database. We’ll need to get from a list of structs to a list of tuples.

To do that, we’ll pipe the list of Book structs to Enum.map and use an anonymous function to generate two-value tuples from those structs.

def new(conn, _params)

  ...

  books =

    Media.list_books

      |> Enum.map(&{”#{&1.title} by #{&1.author}”, &1.id})

  render conn, “new.html”, changeset: changeset, books: books

end

Today I Learned: Little Bits

originally published 4/25/2018 on the Hashrocket TIL blog

4 bits (or half a byte) is referred to as a “nibble”.

Today I Learned: Truncating Floats in Elixir

originally published 4/26/2018 on the Hashrocket TIL blog

To remove everything after the decimal point in a floating-point number, use Kernel.trunc/1.

iex> Kernel.trunc(3.141)

3

iex> Kernel.trunc(-3.141)

-3

Alternatively, you can simply type trunc

iex> trunc(3.141)

3

Today I Learned: Elixir String Manipulation

originally published 4/26/2018 on the Hashrocket TIL blog

To get the list of strings that compose a longer string, use String.codepoints/1

iex> cdp =  String.codepoints(”abcdefg”)

[”a”, “b”, “c”, “d”, “e”, “f”, “g”]

iex> Enum.at(cdp, 0)

“a”

To get the list of codepoints that represent each letter in the string, use String.to_charlist/1

iex> chr = String.to_charlist(”abcdefg”)

‘abcdefg’

iex> Enum.at(chr, 0)

97

Today I Learned: Capturing Frames from Video Files

originally published 8/6/2018

First head over here to download FFMPEG. If you happen to be a Mac user, you can use Homebrew:

brew install ffmpeg

I would check out the Homebrew page for all of the options available (of which, there are quite a few).

To extract frames from a video file, you use the following command:

ffmpeg -i my_vid.webm -r 10 -frames:v 30 -f image2 foo-%03d.jpeg

The above command extracts 10 frames per second and saves them as jpeg files with names ‘foo-001.jpeg’, ‘foo-002.jpeg’, and so on. The maximum number of images that will be created is 30. `image2` refers to the demuxer used.

You can read more about this in the documentation.

I talk about tech, comedy, martial arts, and other things.