Buzzword Engineering

The buzzword.engineering Tech Stack

Sat, 26 Dec 2020 00:00:00 GMT

Welcome!

I've been meaning to get around to setting up a blog for a long time. In the past, I've gotten as far as getting halfway through trying out different static site generators before getting depressed about my lack of frontend design chops and given up.

The perfect storm finally came:

I took two weeks off. After recharging my batteries for a few days, I was ready for a little side project.
I recently discovered Obsidian, which is a dope AF note-taking app.
I've tried a decent number of static site generators to build documentation for various projects and wanted to take a deeper dive into Gatsby.
I recently discovered Pipedream and wanted to use it for something.

I wanted to see if i could use Obsidian as a content management system (CMS) for a tech blog and Pipedream to automate tweeting out new blog posts.

Spoiler Alert

It was, in fact, possible.

Anyway, here's Buzzword Engineering's inaugural blog post. If you like it, go give me a github star on the blog repo or something. It's a nice dopamine boost and fuels my self-worth.

Tech Stack

Let's dive in. Here's a digram for those visual learners out there.

Excalidraw is Dope

If you haven't heard of it, stop reading this and go play with Excalidraw and then come back. It's the tool I used to sketch out this diagram.

Obsidian

I'll keep this short. Maybe a future blog post will talk about Obsidian in a lot more detail. For now, let me just say that I've tried to get into taking notes for... a long time. I could never do it in college. I struggle to do it for work. I've always found that taking notes takes away from my ability to absorb the content in the moment and make meaningful contributions.

Obsidian was the first app that actually made me want to take notes. The general idea is that all your notes are written in markdown. Jumping around is super easy with CMD + O (which also will create pages for you if they don't exist). Linking between pages to build connections is really easy as can be with a syntax like [[this]]. Obsidian builds a visual graph of the relationships between pages (I'm a sucker for graphs). And finally, you can build templates and insert them with CMD + T. Templates dramatically simplified the boiler plate needed to capture who's attending a meeting, agenda, the date, etc.

Long story short, try it out. (Or don't, whatever.) I'm a fan and thought that maybe if I can use it as the interface for writing blog posts that I might actually write some.

Automated Backups

Obsidian has some 3rd-party plugins that do nifty things. One of these plugins is called Obsidian Git which can automatically backup your notes to a Git repository.

I figured that had to be a way to fetch markdown content from a remote github repository and use it as a content source for Gatsby. There was.

Defining Post Information

Blog post information is defined through the markdown frontmatter. For example, the frontmatter for this blog post:

---
title: The buzzword.engineering Tech Stack
date: "12/26/2020"
publish: true
template: "post"
slug: blog-tech-stack
description: "I finally got around to putting a blog together that uses Obsidian, Gatsby, and automates tweeting out new posts with Pipedream."
---

Gatsby

I think it's important to start here by saying that I'm not a frontend developer. Well, let's rephrase that. I'm writing a blog post that has Gatsby in it. So it's probably more accurate to say that I'm a very junior frontend developer.

My mental model for Gatsby so far is that it's a framework for building static site generators. There might be a couple frontend purists or gatsby enthusiasts out there who take issue with that definition, please let me know if you've got a better one down in the comments.

There are two main components of Gatsby that drew me to it:

It uses React, which is a lot more powerful to me over something like handlebars or go-based html templating.
Gatsby is extensible with a rich plugin ecosystem that contribute to a shared GraphQL data layer. When developing your site, you can query the data layer to fetch content for particular pages/components.

I like React and I think Gatsby's extensibility framework and GraphQL data layer is brilliant.

The Starter

Another great thing about Gatsby is their concept of Starters. For this blog, I kicked things off with the gatsby-starter-lumen.

Fetching Content

The first thing I had to customize was content sources. The Lumen starter fetches content from the same repository as the blog itself. Thankfully, there's a Gatsby plugin called gatsby-source-git that allows you to fetch content from a remote Git repository.

During development, I wanted to be able to fetch content from the local copy of the Obsidian backup repository. Gatsby plugins are done by exporting a javascript object from a file called gatsby-config.js.

Here, I toggle between using the gatsby-source-git plugin and the [gatsby-source-filesystem] based on whether a GATSBY_PREVIEW environment variable is set.

if(process.env.GATSBY_PREVIEW == "true"){
  console.log(`using local vault path: ${siteConfig.obsidian.vaultPath}`)
  config.plugins.unshift({
    resolve: 'gatsby-source-filesystem',
    options: {
      path: siteConfig.obsidian.vaultPath,
      name: 'local_obsidian',
      ignore: [ "**/.git/**/*", "**/.obsidian/**/*", "**/Templates/**/*" ]
    }
  })
} else{
  console.log("fetching from remote repo: ", siteConfig.obsidian.repo)
  config.plugins.unshift({
    resolve: `gatsby-source-git`,
    options: {
      name: `obsidian`,
      remote: siteConfig.obsidian.repo,
      patterns: [ "!**/Templates/**/*", "**/*" ]
    }
  })
}

Comments

Comments on blog posts are made possible through a nifty tool called utteranc.es. It's a GitHub Application that uses GitHub Issue threads per blog post to track comments.

Post Filtering

In the spirit of premature optimization, I wanted to integrate a way to filter blog posts with fuzzy-searching. To accomplish this, I integrated Fuse.js and added a new Filter component to the blog.

Most of the logic for how this was accomplished can be seen in the Index Template.

caution

I wanted to insert a gif of the filtering taking place. Apparently that's easier said than done with Gatsby andgatsby-transform-remark. I'll update this post once I get gifs working 🙄.

Automation

With the site actually working how I wanted it to, I got to focus on the side of things I'm actually good at: digital duct tape. The goal is for changes in markdown content in the Obsidian backup repository to trigger a deployment of the site and if there is a new blog post, to send out a tweet letting you all know about it.

Step 1: GitHub Action on the Obsidian Backup Repo

First things first, the content repository needs to trigger a deployment of the site. The easiest way I could think to accomplish this would be to a GitHub Action on the blog post repository that does the build/deploy logic.

This meant that I needed a way to invoke a GitHub Action on one repository as part of the execution of an Action on another repository. This is where the repository_dispatch event comes in handy. Basically, it means that you can use the GitHub API to trigger an Action.

Here's what the GitHub Action workflow looks like for the obsidian repository:

name: Trigger Build
on:
  # trigger this action on changes to the main branch
  push:
    branches: [ main ]
  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

jobs:
  trigger:
    runs-on: ubuntu-latest
    steps:
     - name: Trigger Upstream Blog Action
        run: |
          curl -XPOST \
          -u "${{ secrets.PAT_USERNAME}}:${{secrets.PAT_TOKEN}}" \
          -H "Accept: application/vnd.github.everest-preview+json" \
          -H "Content-Type: application/json" \
          https://api.github.com/repos/steven-terrana/steven-terrana.github.io/dispatches \
          --data '{"event_type": "blog"}'

Step 2: GitHub Action on the Blog Repo

Sweet. Now commits to the Obsidian backup repository will trigger actions on the blog repository.

The next step was to automate the build and deployment steps using a GitHub Action on the blog repository. Here's what that action looks like:

name: Build and Publish
on:
  repository_dispatch:
  workflow_dispatch:

jobs:
  build-deploy-notify:
    runs-on: ubuntu-latest
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - name: Checkout Code 🛎
        uses: actions/checkout@v2
        with: 
          persist-credentials: false
       - name: Install & Build 🔧
        run: |
          npm ci
          npm run build
          echo "buzzword.engineering" > public/CNAME
        env: 
          PAT_USER: ${{ secrets.PAT_USER }}
          PAT_TOKEN: ${{ secrets.PAT_TOKEN }}
      - uses: peaceiris/actions-gh-pages@v3
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: public
          force_orphan: true

This blog is hosted using GitHub Pages, so you'll notice a few things:

I add a custom CNAME file to the public directory so that GitHub Pages knows the custom domain for this blog. (I should definitely incorporate this into an inherit part of the build of the site using the onPostBuild Gatsby Node API method or something).
I use the peaceiris/actions-gh-pages action to publish the site.

All in all, this was a pretty painless setup.

Step 3: Automating Tweets

So at this point, we've got content changes automatically getting deployed to GitHub Pages. The whole process takes about three minutes from commit to publish.

The last piece was to automate letting all of you know about the whatever new insightful thing I had to say!

I had stumbled on Pipedream before through targeted ads and sort of ignored it until I saw David McKay talk about how much he loves it on rawkode.live. Here's a link to the stream!

I went into this adventure thinking I was going to have to do all kinds of fancy logic and scripting to make this possible. I was wrong.

After setting up a Pipedream account, starting looking at what event sources were available to trigger a workflow. Well, the Lumen gatsby starter had already integrated an RSS feed using the gatsby-plugin-feed plugin. Pipedream had an event source for monitoring RSS feeds. Done.

That was easy enough. Next up was automating how to send the tweet. I figured that would mean doing some research on a Twitter Rest API or SDK.

Nope! Pipedream also had a step for automating the sending off a tweet and even made it really simple to query the event that triggered the workflow to template the tweet content with the new blog post information. Here's what this whole setup looks like in Pipedream:

This whole workflow took about 1 minute to figure out and configure. I was almost disappointed in how easy it was. In the end, it was pretty boring. And that's probably the best compliment I can give Pipedream.

A Groovy Alternative to Ordinals With Annotation-Driven Directed Acyclic Graphs

Sat, 02 Jan 2021 00:00:00 GMT

Some Background Context

If you know anything about me, it's probably that I'm the lead maintainer of the Jenkins Templating Engine (JTE). While working on JTE 2.0, we refactored how pipeline initialization works.

Aggregated Pipeline Configurations: The Jenkins folder structure is used to create a configuration hierarchy. Pipeline configurations are aggregated across the hierarchy into an Aggregated Pipeline Configuration.
Interpret Pipeline Configuration: JTE has what are called Pipeline Primitives. They're just objects that get created by parsing the pipeline configuration and instantiating objects defined to be stored in a common Groovy binding.
Determine and Execute the Pipeline Template: The configured pipeline template is executed just like any other Jenkinsfile, just with the common binding that's been injected with primitives so that the templated aspects actually do something.

The benefit of this design is that all 3 phases of initialization are totally decoupled. The pipeline configuration files are a custom configuration DSL that get compiled into a PipelineConfigurationObject. Each Pipeline Primitive has a corresponding TemplatePrimitiveInjector that implements an extension point. We then dynamically fetch all the different Injectors and iterate over them 3 times. The first time is to invoke an optional interface on each injector to validate the pipeline configuration. The second time is to invoke an optional interface on each injector to instantiate a primitive and store it in the binding. The third time is to validate the completed binding.

This all happens from a common entry point that has no idea which Injectors exist. This makes it possible for 3rd party plugins to extend JTE's functionality through adding custom Injectors.

The challenge is now: How can certain Injectors ensure that specific other injectors run before them?

Here's an example required iteration order:

Learn More About JTE

To learn more about the Jenkins Templating Engine, I'd recommend checking out this Jenkins Online Meetup and the docs

The Common Approach: Ordinals

Typically, the way I've seen this problem solved is through a pattern called Ordinals. Ordinals are defined, mathematically, as an integer or character indicating a position in a sequence.

When this pattern is implemented, an annotation is defined and optionally applied to elements in the sequence. Prior to iterating, elements are sorted according to their self-defined ordinal values.

Okay? So What's the Problem?

Every time I've ever seen an Ordinal declared, it's either trying to be first (with a value of 0) or last (with an absurdly high number). I have literally never seen an example where Ordinals are used to reasonably create an order of iteration.

Hot Take

Ordinals are bad.

A Better Approach

The thing about that iteration example above is that it looks pretty familiar. It's an example of a pretty classic computer science data structure, the Directed Acyclic Graph.

The most natural way to solve this problem is not ordinals. The most natural way to solve this problem is to:

Build a direct graph where the vertices of the graph are TemplatePrimitiveInjectors with edges going from a dependent Injector to the Injector that must proceed it.
Ensure the dependency graph is actually acyclic -- meaning that there are no circular dependencies between Injectors.
Perform a topological sorting of the graph
Iterate over the Injectors according to the sorted order

Immediately after thinking up with this approach, my tenured engineering mind started yelling at me:

Cool. Neat idea, Steven. Is it really worth the level of effort to program this fancy approach just because it's "better"? Just hard code the ordering or use Ordinals like everyone else. The (Return on Investment) / (Level of Effort) Ratio is way too low.

I'm glad I didn't immediately cave. My favorite part of all of this is just how easy it was to implement.

Step 1: Build the Graph

JGraphT did most of the work for me on this one!

Finding the Vertices

We said before that the graph's vertices would be the different TemplatePrimitiveInjector's. Thankfully, the Jenkins plugin ecosystem is built on the idea of extending and creating ExtensionPoint's, so looking them up shouldn't be too hard.

abstract class TemplatePrimitiveInjector implements ExtensionPoint{

  /**
   * fetches all registered TemplatePrimitiveInjectors
   *   
   * @return list of all TemplatePrimitiveInjectors
  */
  static ExtensionList<TemplatePrimitiveInjector> all(){
    return Jenkins.get().getExtensionList(TemplatePrimitiveInjector)
  }
  
  // shorthand for the interface methods that 
  // we're trying to define the order of execution for
  void validateConfiguration(...){}
  void injectPrimitives(...){}
  void validateBinding(...){}
  
}

Invoking the static method TemplatePrimitiveInjector.all() returns the full list of registered Injectors.

Identifying the Edges

Next up, we need a mechanism for identifying the dependencies.

Functional Goals:

The TemplatePrimitiveInjector interface has 3 stages and each stage may have its own dependencies, so we can't just define the dependencies on a class-wide basis.
Injectors should be able to define this dependency with minimal effort
Injectors should be able to define this dependency with minimal coupling to the rest of the code.

It turns out, a method-based annotation works perfect for this.

Let's define a @RunAfter annotation that accepts as a parameter the class names of other Injectors that must be executed first.

@Retention(RUNTIME)         // be discoverable at runtime 
@Target(ElementType.METHOD) // annotate methods (not class-wide)
@interface RunAfter{
  
  /** define a list of other Injectors */ 
  Class<? extends TemplatePrimitiveInjector>[] value();

}

Through this annotation, we can now easily construct dependencies. Here's a short hand example:

class A extends TemplatePrimitiveInjector{
  void validateConfiguration(...){}
  void injectPrimitives(...){}
  void validateBinding(...){}
}

class B extends TemplatePrimitiveInjector{
  void validateConfiguration(...){}
  
  @RunAfter(A)
  void injectPrimitives(...){}
 
  void validateBinding(...){}
}

class C extends TemplatePrimitiveInjector{
  void validateConfiguration(...){}
  void injectPrimitives(...){}
  
  @RunAfter([A, B])
  void validateBinding(...){}
}

Now we just need a way to look up a method's dependency's:

/**
 * returns the dependencies listed in the RunAfter annotation if present
 * on the given Injector's methodname
 */
private static List<Class<? extends TemplatePrimitiveInjector>> getPrerequisites(TemplatePrimitiveInjector injector, String methodName, Object... methodArgs){
  List<TemplatePrimitiveInjector> prereqs = []
  MetaMethod metaMethod = injector.metaClass.pickMethod(methodName, methodArgs*.class as Class[])
  if(metaMethod instanceof CachedMethod) {
    Method method = metaMethod.getCachedMethod()
    RunAfter annotation = method.getAnnotation(RunAfter)
    if (annotation) {
      prereqs = [annotation.value()].flatten() as List<Class<? extends TemplatePrimitiveInjector>>
    }
  }
  return prereqs
}

Creating the Graph

private static Graph<Class<? extends TemplatePrimitiveInjector>, DefaultEdge> createGraph(String methodName, Object... methodArgs){

  Graph<Class<? extends TemplatePrimitiveInjector>, DefaultEdge> graph = new DefaultDirectedGraph<>(DefaultEdge)
  ExtensionList<TemplatePrimitiveInjector> injectors = TemplatePrimitiveInjector.all()

  // add each injector as a node in the graph
  injectors.each{ injector -> graph.addVertex(injector.class) }

  // for each injector, connect edges
  injectors.each{ injector ->
    List<Class<? extends TemplatePrimitiveInjector>> prereqs = getPrerequisites(injector, methodName, methodArgs)    
    prereqs.each{ req ->
      graph.addEdge(req, injector.class)
    }
  }

  // check for infinite loops
  CycleDetector detector = new CycleDetector(graph)
  Set<TemplatePrimitiveInjector> cycles = detector.findCycles()
  if(cycles){
    throw new JTEException("There are cyclic dependencies preventing initialization: ${cycles}")
  }

  return graph

}

Iterating over the Graph

Finally, we can create a method that takes a phase (either validateConfiguration, injectPrimitives, or validateBinding) and method arguments and ties all of this together!

private static void invoke(String phase, Object... args){
  
  List<Class<? extends TemplatePrimitiveInjector>> failedInjectors = []
  Graph<Class<? extends TemplatePrimitiveInjector>, DefaultEdge> graph = createGraph(phase, args)
  AggregateException errors = new AggregateException()

  new TopologicalOrderIterator<>(graph).each{ injectorClazz ->
    TemplatePrimitiveInjector injector = injectorClazz.getDeclaredConstructor().newInstance()
    try{
      // check if a dependent injector has failed, if so, don't execute
      if(!(getPrerequisites(injector, phase, args).intersect(failedInjectors))){
        injector.invokeMethod(phase, args)
      }
    } catch(any){
      errors.add(any)
      failedInjectors << injectorClazz
    }
  }
  
  if(errors.size()) { // this phase failed throw an exception
    throw errors
  }
  
}

Conclusion

I like to say that, "the line between brilliant and hacky is often thin". Let me know which side you think this falls on in the comments :).

Feel free to check out the actual code for this within the Jenkins Templating Engine: TemplatePrimitiveInjector, RunAfter, TemplateBindingFactory