Lover of micro-interations, Elixir and cooking.
Maker of apps of all shapes and sizes.
Author of  

Building a user insights tool releasing soon

Iterating on a community for indiehackers building in public 

Github

Twitter

Twitch

YouTube

LinkedIn

Instagram

Substack

Medium

Photography

Two people cycling under a bridge in Valencia

A woman with her face in her hands on a table

Here's the thing about PDFs - they're complex beasts that require quite a bit of thinking to properly parse - they come in all shapes and sizes, and they can contain a lot of different types of data and formatting. 90% of the time, we just want to extract the text from the file, but that's not always easy - for the remaining 10%, well we won't be covering that in this blog post.

If you've been in the Elixir world for long enough, you'll probably have tried to parse a PDF file and realised that it's not as easy as it seems. A quick look on the [Elixir Forum](https://elixirforum.com/t/parsing-pdf-file/23287) will quickly show you that there is no simple way to do it.

Most people will tell you to upload the file to S3 and use a Lambda to handle the contents. Offloading to AWS Lambda might seem elegant at first ("Look, Ma, no dependencies!"), but it comes with its own baggage:

- You're adding network latency to what should be a simple operation
- AWS costs can spiral if you're processing lots of PDFs
- You're now dependent on external services for core functionality
- Debugging becomes a distributed systems problem

These aren't ideal solutions - and software engineering is already made more complicated than it needs to be at times - we don't need to add more complexity to the mix.

We need a robust, native solution that plays nicely with the BEAM. So how do we do that?

## Enter the crabs!

Elixir is my favourite language, but it can't do everything - web services, background jobs, and more are easy but sometimes we need a little help from our friends closer to the hardware for some of the tasks Elixir doesn't have a native solution for. That's where Rust and NIFs come in!

Rust is a systems programming language that is fast, safe, and easy to use. It's a great language for writing code that needs to be performant and reliable.

But Rust isn't just fast - it's "zero-cost abstractions" fast. What does that mean? You get high-level, ergonomic code that compiles down to something as efficient as hand-written C. For PDF parsing, where you're dealing with complex file formats and potentially large documents, this performance is a game-changer.

> What is a NIF?
> A NIF (Native Implemented Function) is a way to call Rust code from Elixir - it's the BEAMs method of allowing processes to directly call native functions. It allows you to write code in Rust that can be called directly from Elixir, giving you the performance benefits of Rust without sacrificing the ease of use of Elixir.

For this blog post, we're going to be using the [Extractous library](https://github.com/yobix-ai/extractous) which provides fast and efficient unstructured data extraction in Rust. This combined with the NIFs in Elixir gives us a powerful combination for parsing PDFs.

## The Setup

First things first, ensure you have Elixir and Rust installed on your machine.

Let's begin by creating a new LiveView Elixir application that will allow users to upload a PDF file and see a breakdown of the contents. We won't be needing any database functionality for this so we can use the `--no-ecto` flag to skip the database setup.

```elixir
mix phx.new elixir_pdf --no-ecto
```

We'll also need to add the `rustler` dependency to our `mix.exs` file so we can call Rust code from Elixir.

```elixir
defp deps do
  [
    {:rustler, "~> 0.27.0"}
  ]
end
```

Once we pull down our dependencies using `mix deps.get`, we can use `mix rustler.new` to generate our new Rust project in our code.

If you head to `lib/elixir_pdf/<name_of_your_rust_project>.ex`, you'll see that it's already generated a basic NIF for us. A default NIF implementation is provided for us, but we'll be implementing our own in the next step. I've named my Rust project `rustreader` for this example.

```elixir
defmodule RustReader do
  use Rustler, otp_app: :elixir_pdf, crate: "rustreader"

  # Define the function that will be implemented in Rust
  def extract_pdf(_path), do: :erlang.nif_error(:nif_not_loaded)
end

```

Now, let's grab the `extractous` library and add it to our `native/rustreader/Cargo.toml` file - this will allow us to use the `extractous` library in our Rust code.

```rust
[dependencies]
rustler = "0.36.0"
extractous = "0.2.0"
```

With this in place, we can run `cargo build` to build our Rust code - this will also pull down the `extractous` library and any other dependencies.

## The fun part - writing some code

Next we need to actually write some Rust code to implement the `extract_pdf` function in our `native/rustreader/src/lib.rs` file.

```rust
use extractous::Extractor;
use rustler::{Encoder, Env, NifResult, Term};

#[rustler::nif(schedule = "DirtyCpu")]
fn extract_pdf(path: String) -> NifResult<(String, String)> {
    let extractor = Extractor::new();

    match extractor.extract_file_to_string(&path) {
        Ok((content, metadata)) => Ok((content, format!("{:?}", metadata))),
        Err(e) => Err(rustler::Error::Term(Box::new(format!("Extraction failed: {}", e))))
    }
}

rustler::init!("Elixir.RustReader", [extract_pdf]);
```

This code will define a new instance of the `Extractor` struct and use it to extract the contents of the PDF file. We'll then return the contents and the metadata as a tuple.

The magic of the `rustler::init!` macro is that it will automatically generate the necessary code to call the Rust function from Elixir.

Astute observers will note our use of the `DirtyCpu` schedule. This ingenious feature instructs Rustler and the BEAM to automatically schedule our task in a manner that prevents global blocking during execution. This functionality, known as a DirtyNif, significantly simplifies our work compared to the complexities of manual implementation in C.

<Divider />

Now we need to write a some simple LiveView Elixir code to allow users to upload a PDF file and then call our Rust function from the server.

```elixir
defmodule ElixirPdfWeb.HomeLive do
  use ElixirPdfWeb, :live_view

  @impl true
  def mount(_params, _session, socket) do
    {:ok,
     socket
     |> assign(:uploaded_files, [])
     |> allow_upload(:pdf,
       accept: ~w(.pdf),
       max_entries: 1,
       # 10MB limit
       max_file_size: 10_000_000,
       chunk_size: 64_000
     )}
  end

  @impl true
  def handle_event("validate", _params, socket) do
    {:noreply, socket}
  end

  @impl true
  def handle_event("save", _params, socket) do
    uploaded_files =
      consume_uploaded_entries(socket, :pdf, fn %{path: path}, _entry ->
        dest = Path.join(["priv", "static", "uploads", Path.basename(path)])
        File.cp!(path, dest)
        {:ok, dest}
      end)

    pdf_document =
      uploaded_files
      |> hd()

    {:noreply,
     socket
     |> assign(:pdf_document, pdf_document)
     |> update(:uploaded_files, &(&1 ++ uploaded_files))}
  end
end
```

Alongside this we need to add a little bit of code to our `router.ex` file to allow us to upload files.

```elixir
scope "/", ElixirPdfWeb do
  pipe_through :browser

  live "/", HomeLive
end
```

We also need a simple LiveView template to allow users to upload a PDF file and see the results.

```elixir
<div class="mx-auto max-w-2xl py-8">
  <div class="flex flex-col items-center justify-center">
    <h1 class="text-2xl font-bold mb-8">Upload PDF</h1>

    <form phx-submit="save" phx-change="validate" class="w-full">
      <div class="flex flex-col items-center space-y-4 w-full" phx-drop-target={@uploads.pdf.ref}>
        <div class="w-full border-2 border-dashed border-gray-300 rounded-lg p-12 text-center hover:border-gray-400 transition-colors">
          <div class="space-y-2">
            <div class="text-gray-600">
              Drag and drop your PDF here or
              <label class="cursor-pointer text-blue-500 hover:text-blue-600">
                browse <.live_file_input upload={@uploads.pdf} class="hidden" />
              </label>
            </div>
            <p class="text-xs text-gray-500">PDF files only, up to 10MB</p>
          </div>
        </div>

        <%= for entry <- @uploads.pdf.entries do %>
          <div class="w-full">
            <div class="flex items-center justify-between p-4 bg-gray-50 rounded">
              <div class="flex items-center space-x-2">
                <span class="font-medium">{entry.client_name}</span>
                <span class="text-sm text-gray-500">
                  ({entry.client_size}B)
                </span>
              </div>

              <button
                type="button"
                class="text-red-500 hover:text-red-700"
                phx-click="cancel-upload"
                phx-value-ref={entry.ref}
              >
                &times;
              </button>
            </div>

            <%= for err <- upload_errors(@uploads.pdf, entry) do %>
              <div class="text-red-500 text-sm">
                {err}
              </div>
            <% end %>
          </div>
        <% end %>

        <%= if length(@uploads.pdf.entries) > 0 do %>
          <button
            type="submit"
            class="px-4 py-2 bg-blue-500 text-white rounded hover:bg-blue-600 transition-colors"
          >
            Upload
          </button>
        <% end %>
      </div>
    </form>
  </div>
</div>
```

## Putting it all together

So we can upload a PDF file - but let's call our Rust function and see what it returns.

In our handle_event function, we can call our Rust function as simply as this:

```elixir
  @impl true
  def handle_event("save", _params, socket) do
    uploaded_files =
      consume_uploaded_entries(socket, :pdf, fn %{path: path}, _entry ->
        dest = Path.join(["priv", "static", "uploads", Path.basename(path)])
        File.cp!(path, dest)
        {:ok, dest}
      end)

    pdf_document =
      uploaded_files
      |> hd()
      |> RustReader.extract_pdf() ## This is where the magic happens!

    {:noreply,
     socket
     |> assign(:pdf_document, pdf_document)
     |> update(:uploaded_files, &(&1 ++ uploaded_files))}
  end
```

We're grabbing the first uploaded file and calling our Rust function. The result is a tuple containing the contents of the PDF and the metadata.

Let's try it out with the [LiveView Cookbook PDF](https://liveviewcookbook.com/).

![Unstructured](https://i.imgur.com/dpyzduq.png)

Success! We've now got a PDF parser that's fast, efficient, and written in Rust.

But that's quite hard to read so we're not done yet, let's make this a little nicer to work with.

Let's create a new module to handle the Jason encoding of the metadata.

```elixir
defmodule ElixirPdf.PdfDocument do
  @derive {Jason.Encoder, only: [:content, :metadata]}
  defstruct [:content, :metadata]

  def from_rustler({content, metadata_json}) do
    with {:ok, metadata} <- Jason.decode(metadata_json) do
      %__MODULE__{
        content: String.trim(content),
        metadata: metadata
      }
    end
  end
end
```

This will allow us to encode the metadata to JSON and decode it back to a struct in Elixir to make it easier to work with.

All we have to do is pipe the result of our Rust function through this module and we're done!

```elixir
...
RustReader.extract_pdf(@pdf_document)
|> ElixirPdf.PdfDocument.from_rustler()
...
```

Now when we upload a PDF file, we'll see the metadata results in a much more readable format.

![Structured](https://i.imgur.com/QvG2QUV.png)

Much better!

This approach is simple and effective - it's fast, efficient, and leverages the strengths of both Elixir and Rust to provide a robust solution for PDF parsing.

We're only talking about PDF files here but extractous supports a [wide range of file types](https://github.com/yobix-ai/extractous?tab=readme-ov-file#-supported-file-formats) - so keep that in mind if you need to extract data from other file types.

## What about deployment?

Keep in mind that this is a native extension and so you'll need to build the Rust code before deploying your application. This can be done in a CI/CD pipeline or manually.

If you're using Docker, you can update the `Dockerfile` to build the Rust code as part of the build process and update `config/prod.exs` to tell Rustler to skip compilation and load the compiled NIF from where it was built in the Docker image.

Check out the [Fly.io blog post](https://fly.io/phoenix-files/elixir-and-rust-is-a-good-mix/) for more information on how to deploy an Elixir application with Rust NIFs.

## Shoutouts

Some shoutouts are in order - firstly this blog post from [Fly.io's Phoenix Files](https://fly.io/phoenix-files/elixir-and-rust-is-a-good-mix/) outlining how to use Rust with NIFs in Elixir. It was a key inspiration for this approach and gave me the idea to use Rust in the first place. Also check out Fly in general for some great Elixir hosting options - I use them for all my Elixir applications.

Also a shoutout for the excellent [Extractous library](https://github.com/yobix-ai/extractous) which provides fast and efficient unstructured data extraction in Rust - it's also 25x faster than the very popular unstructured-io library.

Finally, a shoutout to the [Rustler](https://github.com/rusterlium/rustler) library for providing a simple way to call Rust code from Elixir!

<Divider />

All the code for this blog post can be [found here on my Github](https://github.com/chrisgreg/elixir_pdf_tutorial) if anyone wants to clone it and run it yourselves!

I hope you found this post useful, subscribe to my Substack below for similar content and  [follow me on Twitter](https://www.twitter.com/codestirring) and [Bluesky](https://bsky.app/profile/codestirring.bsky.social) for more Elixir (and general programming) tips.

If you're building a Phoenix project, I'd also encourage you to take a look at my open-source component library [Bloom](https://bloom-ui.fly.dev) to help you out even further or check out my [new voice to notes application to automatically tag and sync your voice to your calendar](https://mmbl.io).


Parsing PDFs (and more) in Elixir using Rust

## mix phx.gen.auth
If you’re an Elixir developer and you’ve created a new Phoenix LiveView application in the last few years then you’ve likely come across the excellent authentication generator that ships out of the box with Phoenix. 

For the uninitiated, a brand new Phoenix app can tack on a Postgres-backed, bcrypt-hashsed, session-storing, pre-built authentication system in about 30 seconds right out of the box. 

All you have to do is run `mix phx.gen.auth` and you’ll be asked if you want a LiveView or a <Sparkles>DeadView</Sparkles> version.

The best part is you get to fully customise the implementation since the generator locally installs everything into your source code, you’re not using someone else’s library or third-party system to manage auth, you’re doing it right in your app. Even if you want to opt for something like Auth0, this is easily the fastest way to add authentication to your new app - for what it’s worth I haven’t seen a faster way to add authentication in any stack.

## Why add Google Authentication then?
The truth is, today’s web users have come to expect lightning-fast sign-up journeys, and anything less can raise serious concerns about a website’s legitimacy. Without the familiar comfort of Google’s seamless login experience, you’re essentially inviting users to abandon ship. Let’s face it - friction is a major obstacle that can kill user engagement.

On one of my latest projects, [Uini](https://uini.io), adding Google Authentication massively improved conversion from visiting users to registered users. Check out Uini if you want to make gathering feedback from your users effortless.

All of this sounds useful and maybe scary, but don't let the intricacies of Google's site deter you! With the right approach, handling OAuth tokens, credentials, and client secrets becomes a seamless process. And I'm here to guide you through it. By harnessing the power of `mix phx.gen.auth`, you'll discover that integrating authentication with your Phoenix app is remarkably straightforward. 

So, let's dive in and tackle this challenge head-on!

## First things first - the Google Project
The thing you’ll definitely need here - is a Google account and project.

Head over to the [Google Cloud Platform](https://console.developers.google.com/) and authenticate through until you can see “API & Services Dashboard”. Run through the basic steps of setting up a project (I’d just choose the same name as the application you’re integrating with), until you have your app setup - this should take a minute or two. 

Once your Google Project has been created, you’ll want to navigate to the `OAuth consent` screen by clicking on the navigation menu item on the left-hand side.

Make your application `Public` and use the same name as the previous step. I’d leave the scopes to `email`, `profile` and `opened` as a minimum.

Now scroll down and hit save before navigating to the `Credentials` page. There you’ll want to create credentials, namely `OAuth client ID`.

From this new shiny page, select `Web application` give it a name, and you’ll come to the restrictions. Here’s where you tell Google which domains it’s allowed to redirect users back to, to ensure no one is hijacking your applications login. 

Add `https://localhost:4000` (or whatever port you use for local development) as well as your production environment (and any other) URLs.

Underneath you’ll also see `Authorised redirect URIs` - this is important and will be the exact path that users are redirected to once authenticated - this is the shiny bit that will bring all that delicious Google data straight to your application so you can store some credentials and authenticate your users.

Hit create.

Now you’ll be shown two values, your `client ID` and your `client secret`.

> Never show these to anyone but trusted developers working on your application. Do not commit them to git, do not keep them in your source code, do not put them anywhere that can be accessed unsafely. 

Okay, the annoying part is done, now you’re ready for some code.

## Integrating with ueberauth
We’re gonna use [ueberauth/ueberauth: An Elixir Authentication System for Plug-based Web Applications](https://github.com/ueberauth/ueberauth) to help take care of all the hard work authenticating with Google.

First we need the core library and the strategy that we want to use. So head into your `mix.exs` file and add the following to your dependency list:

```elixir
{:ueberauth_google, "~> 0.10"}
```
 
Now pull down your dependencies with `mix deps.get` and head into your `config.exs` file to put those earlier secret tokens to good use.

```elixir
config :ueberauth, Ueberauth,
  providers: [
    google: {Ueberauth.Strategy.Google, [default_scope: "email profile"]}
  ]

config :ueberauth, Ueberauth.Strategy.Google.OAuth,
  client_id: System.get_env("GOOGLE_CLIENT_ID"),
  client_secret: System.get_env("GOOGLE_CLIENT_SECRET")
```

> Again, ensure you store those previous secrets in environment variables on your machines for safety and not in your source code.

Now we need to make a simple controller to ensure all that user information gets sent straight to our database, so new users can have new accounts created or to let in existing users.

```elixir
defmodule MyAppWeb.GoogleAuthController do
  alias MyAppWeb.UserAuth
  alias MyApp.Accounts
  use MyAppWeb, :controller
  require Logger
	
	plug Ueberauth

  def request(conn, _params) do
     Phoenix.Controller.redirect(conn, to: Ueberauth.Strategy.Helpers.callback_url(conn))
  end

  def callback(%{assigns: %{ueberauth_auth: auth}} = conn, params) do

    email = auth.info.email

    case Accounts.get_user_by_email(email) do
      nil ->
        # User does not exist, so create a new user
        user_params = %{
          email: email,
          first_name: auth.info.first_name,
          last_name: auth.info.last_name
        }

        case Accounts.register_oauth_user(user_params) do
          {:ok, user} ->
            UserAuth.log_in_user(conn, user)

          {:error, changeset} ->
            Logger.error("Failed to create user #{inspect(changeset)}.")
            conn
            |> put_flash(:error, "Failed to create user.")
            |> redirect(to: ~p"/")
        end

      user ->
        # User exists, update session or other details if necessary
        UserAuth.log_in_user(conn, user)
    end
  end
end
```

This controller takes care of handling the request from Googles callback, checks for an existing user in the database with the email, lets them in or creates an account for them.

You may have noticed the `Accounts.register_oauth_user` which doesn’t ship out of the box from the generator, so we can simply add that too to `my_app/accounts/accounts.ex` :

```elixir
  def register_oauth_user(attrs) do
    %User{}
    |> User.oauth_registration_changeset(attrs)
    |> Repo.insert()
  end
```

We’ll also need to add the changeset function to our user schema too at `my_app/accounts/user.ex` so let’s add that and add an additional field to track that this user is oauth_based.

```elixir
 schema "users" do
    ...
    field :is_oauth_user, :boolean, default: false
    ...
 end

 def oauth_registration_changeset(user, attrs, opts \\ []) do
    user
    |> cast(attrs, [:email, :first_name, :last_name])
    |> validate_required([:first_name, :last_name, :email])
    |> validate_email(opts)
    |> put_change(:is_oauth_user, true)
  end
```

Alongside creating a migration for this new field, `mix phx.gen.auth` also requires a hashed password by default, so we need to change that on a database level.

So run `mix phx.gen.migration add_oauth_user` and ensure you add the following:

```elixir
  def up do
    alter table(:users) do
      add :is_oauth_user, :boolean, default: false
      modify :hashed_password, :string, null: true
    end
  end

  def down do
    alter table(:users) do
      remove :is_oauth_user
      modify :hashed_password, :string, null: false
    end
  end
```

Almost done!

Now we need to add our new controller to the `router.ex`:

```elixir
scope "/auth", MyAppWeb do
      get "/:provider", GoogleAuthController, :request
      get "/:provider/callback", GoogleAuthController, :callback
    end
```

And finally add a button to kick start the authentication journey. I like to add a button to `core_components.ex` like so:

```elixir
  def google_auth_button(assigns) do
    ~H"""
    <.link href={~p"/auth/google"}>
      <.button>Login with Google</.button>
    </.link>
    """
  end
```

<Sparkles>And that’s it!</Sparkles>

If you’ve followed along, you should have an end-to-end sign up flow with Google leveraging the excellent starting point provided by the Phoenix Auth generator.

Session management, database persistence, logging out, and more is all covered. 

<Divider />

I hope you found this post useful, subscribe to my Substack below for similar content and  [follow me on Twitter](https://www.twitter.com/codestirring) for more Elixir (and general programming) tips.

If you're building a Phoenix project, I'd also encourage you to take a look at my new open-source component library [Bloom](https://bloom-ui.fly.dev) to help you out even further.

Implementing Google Authentication in a LiveView Application

Erlang/OTP 26 has arrived, and it brings a plethora of new features and improvements that Elixir developers can benefit from. You can find the [original blog post here](https://www.erlang.org/blog/otp-26-highlights/) but I've summarised some of the most exciting highlights that are particularly relevant to Elixir developers below.

## 1. Shell Improvements: Enhanced Auto-Completion

One of the most significant enhancements in Erlang/OTP 26 is the improved auto-completion feature in the shell. This improvement makes it easier for developers to work with variables, record names, record fields names, map keys, function parameter types, and file names. With this update, developers can save time and reduce errors by quickly accessing the correct information without having to type out the entire name. This feature is especially useful for Elixir developers who frequently work with the interactive shell, as it streamlines the development process and boosts productivity.

## 2. Maps & Lists: Better Performance and Usability

Maps are a fundamental data structure in Elixir, and Erlang/OTP 26 brings enhancements that can improve the performance and usability of maps and lists in Elixir projects. 

In Erlang/OTP 26, the `maps:merge/2` function has been improved to try to reuse the key tuple from one of the input maps, resulting in better performance. This can be particularly useful when merging maps with atom keys. Additionally, Erlang/OTP 26 introduces the `maps:iterator/2` function, which supports iterating over the elements of a map in a more intuitive order. 

These improvements can lead to more efficient code and better overall performance for applications that rely heavily on maps. Hooray!

Who doesn't love a little performance buff?


## 3. Erlang Compiler and JIT: Performance Boost

This is a biggie.

Erlang/OTP 26 introduces enhancements to the Erlang compiler and Just-In-Time (JIT) compilation, which can lead to better performance and optimization for Elixir projects. 

These improvements can result in faster execution times and more efficient code. Anecdotally - the last JIT compiler performance boost helped bring our compilation down by almost 10% last time in our production stack.

I've never been one to complain about compilation times in Elixir but suping that up isn't a bad thing.

## 4. Dialyzer: Incremental Mode for Faster Analysis

Dialyzer, a static analysis tool for Erlang and Elixir, now supports an incremental mode in Erlang/OTP 26. 

This new mode allows Elixir developers to analyze their code more efficiently, as it only re-analyzes the parts of the code that have changed since the last analysis. This can save developers time and help them identify potential issues more quickly, leading to more robust and reliable Elixir applications.

## Conclusion

Ultimately, Erlang/OTP 26 brings a host of new features and improvements that can benefit us. From enhanced auto-completion in the shell to performance boosts in the compiler and JIT, these updates can help all of us write cleaner, more efficient code and build high-performance applications. 

I love incremental updates like these, let me know what your favourite feature is over on [Twitter](https://twitter.com/codestirring) and subscribe to my substack for similar content.

Erlang/OTP 26 Highlights

On Saturday morning, I woke up late, still satisfied from last night’s tasty Amatriciana, Tiramisu, and wine. I took a moment to enjoy the warm breeze that Rome’s 10am winds brought with them. The plan for the day was an exciting visit to the Colosseum, followed by an inspiring tour of the Vatican. However, everything changed when I checked my emails and found a message from Fly.io.

**My server for madepublic.io had crashed in the middle of the night.**

Odd I thought - there’s a dedicated VPS with 2GB of RAM running that Elixir application - what could have happened?

I tried to access the platform using my phone - I didn’t have my laptop with me, I decided I wasn’t going to be doing any work or indiehacking this weekend - and no response. 

I logged into fly.io to check the logs and saw a stream of 200 OK responses coming from my user registration controller action. Looks like someone was creating 240 user accounts with real email addresses with bitcoin related spam in their user details per second.

Credential stuffing attack while I’m on holiday - *great*.

![sad](https://media2.giphy.com/media/gfsQffBnuc6e096brx/200.gif)

I scaled the server up to 2GB to assess the damage. 

Madepubic has a real-time metadata visualisation on the homepage built with Phoenix PubSub that shows the number of registered users, projects, project views, etc. 

So, what was the damage?  150,000 new accounts (and counting - by scaling the server up, the script attacking me was back in action). It’s a good thing I didn’t have autoscaling on Fly.io turned on, or I would have been helping the attacker create false accounts.

I took a big sigh, told my girlfriend I’d be a little late for breakfast, and started thinking about what I could do without my personal laptop, access to my code, access to Fly.io, or access to anything I would normally use to stop this (and in time to meet all the scheduled plans for the day).

### Attempt 1
*Scale the server back down to less RAM to stop the script running.* 

Fly.io has a dashboard I could access from my phone, I scaled the server up, so I could scale it down - that would stop the accounts being created but I’d be turning off my entire platform for any real users - a bad look. 
No bueno.

### Attempt 2
*Use my girlfriends laptop to push a fix*

My girlfriend had her work laptop thanks to working abroad before I joined her! Excellent, I’ll download Git, my code and the fly command line tools.

Argh, Windows - never mind, I’ll press on, I thought.

I pulled down my code (after panic typing the Airbnbs 24 character Wi-Fi password incorrectly 4 times) and when I tried to push to GitHub I got an administrator error and required an Admin JumpCloud password. Great.

What else did I have near me? My flat mates laptop (also on holiday with his girlfriend with us)!

### Attempt 3
*Use my flatmates laptop to deploy a code change*

I called him at breakfast, I found his laptop, got his password, repeated half the steps from *Attempt 2* and was met with the need to insert an admin PIN (Why Windows?!). Okay cool, no problem, just a phone call away.

My flat mate uses his laptop entirely for Chrome and Microsoft Word - he’s never tried to install anything on it, so when I told him that I needed the PIN that his ex-girlfriend had evidently set up (as the laptop was shared between them during their relationship), I knew my morning wasn’t going to get better anytime soon. 

![awkward](https://media4.giphy.com/media/HP7mtfNa1E4CEqNbNL/200.gif)

After a bit of texting from his end, I finally had the PIN (thanks Paul). But there was a problem - I had to be across Rome in 30 minutes for a pre-paid tour of the coliseum and time was running out - I didn’t have the time to install Elixir, asdf, flyctl, git, an IDE (and figure out how to do all of that on an OS I’d not used in 10 years).

So what did I do?

The hackiest thing I’ve ever done on a production server: 

<Sparkles>I re-defined my user registration controller action while the server was still running.</Sparkles>

### Attempt 4
*Hot-code deploy an invalid controller action*

Elixir applications run as BEAM processes - you can access these in iEX sessions connected to your application. You know what you can also do as a result? Redefine your modules in real-time. 

I downloaded `flyctl`and authenticated myself using my phone, accessed my application and used the `fly ssh console`command to get straight into my server.

Once in I was inside my VPS so I had to find my application - after some digging I found what I was looking for in `/app/bin/madepublic` .

I got into an iEX session and started to think about what could I do in 5 minutes to stop the script - I was racing against the clock and I was determined not to give the bastard responsible any more of my holiday, but at the same time, I also didn’t want to let any of my users down.

So I thought if I could disable the ability to create accounts, existing users could still use the platform and no more accounts could be made.

In iEX, I redefined my module and skipped the changeset that was required to render the page in both the GET handler for the page, and the POST handler for submitting the form - just in case the script attacking me was a headless browser or not. Could I have just thrown an error instead? Probably, but I was stressed, a little hungover, full of last nights pasta and with a group of people I was probably starting to piss off.

```elixir
defmodule MadepublicWeb.Users.RegistrationController do
	use MadepublicWeb, :controller

	def new(conn, _params) do
		render(conn, "new.html")
	end

	def create(conn, %{"user" => user_params}) do
		render(conn, "new.html")
	end
end
```

I refreshed my page and boom, 500 error. 

It had worked. 

The server load went down so users could carry on using the platform as intended, and even though the malicious script kept going for another 5 hours, all it did now was give a 500 error that a changeset wasn’t available in a form.

*The colosseum tour was saved (as was my server bandwidth)!*

## So what did I learn?
* Secure your site from bot attacks before you enable auto-scaling. I was lucky that I didn’t have auto-scaling enabled or the damage would have actually been worse. I’m grateful that this ordeal only meant that my free plan for Mailjets email sending maxed out for the month and I didn’t end up having to pay for 150,000 confirmation emails.

Yay for not being so successful that I have a dedicated plan, I guess?


I initially thought about throwing Cloudflare over my side projects and implement google re-captcha going forward - a good suggestion to any indie hackers going forward that are rolling their own auth - but then I realised that I’m an indiehacker, I want to do the least amount possible!

So I’m using an Elixir based method to protect my registration form which was highlighted to me by Michael Lubas, the founder of [Paraxial](https://twitter.com/paraxialio) (a bot protection and security platform for Elixir and Phoenix) on the Elixir slack, shout out to this awesome community.

Michael suggested using PlugAttack which I’ll be writing about in an upcoming blog post if people are interested (let me know on Substack or Twitter).

* Bring your laptop with you on holiday if you have a platform in production that you care about being in production. If I had my laptop with me, I could have blown over the most stressful ordeal of this entire thing and had access to what I needed immediately. 
* Fly.io are awesome for letting me get into my server with a single terminal command - even better if I could do this from a web browser.
If you’re reading this Chris McCord, there’s a feature request for you.
* Elixir is awesome for giving me the (admittedly terrifying) capabilities to hot-update my code in production - BEAM ❤️.

<Divider />

I hope you’ve found this cautionary tale of Elixir production debugging while in Rome useful.

Be sure to subscribe to my Substack below for more content like this, and [follow me on Twitter](https://www.twitter.com/codestirring) for more indiehacking tips and insights.

Elixir hot code swapping to the rescue

If you're an indiehacker, chances are you've experienced the highs and lows of working independently. It's a rollercoaster ride, with the freedom to create and innovate on one hand, and the constant pressure to succeed on the other. This pressure can sometimes lead to the dreaded phenomenon known as <Sparkles>burnout</Sparkles>. In this post, we'll explore what burnout is, why indiehacking is hard, and some tips to keep you from getting discouraged and steering clear of burnout.

<Divider />

<Highlight title={'What is Burnout?'}>
A state of chronic physical and emotional exhaustion, often accompanied by feelings of cynicism, detachment, and a decreased sense of accomplishment.
</Highlight>

Burnout can creep up on you silently, leaving you feeling drained, disinterested, and disillusioned with your work. As an indiehacker, you're especially susceptible to burnout due to the high-stakes nature of your work and the constant need to stay ahead of the curve.

## Why is indiehacking so hard?
There are several reasons why indiehacking can be a challenge:

### Seeing your peers on Twitter report their successes
It's hard not to compare yourself to others, especially when you see their incredible monthly recurring revenue (MMR) reports and success stories. This can lead to feelings of inadequacy and envy, further fueling your burnout.

### Lack of structure
Working independently means you're solely responsible for setting your goals, managing your time, and staying disciplined. This can be both liberating and overwhelming, often leading to burnout.

### High expectations
The pressure to succeed and constantly innovate can be immense, leading you to work tirelessly to meet these expectations. However, this can also result in neglecting your personal needs and well-being, ultimately leading to burnout.

<Divider />

So how can you avoid burnout as an indiehacker? Here are some tips to help you stay motivated and energised:

### Set realistic goals
Imagine you're embarking on a long road trip. You wouldn't expect to reach your destination in a single day, would you? The same principle applies to indiehacking. Break down your goals into smaller, achievable milestones and celebrate each accomplishment along the way, no matter how small. Remember, success is a journey, not a destination. Every step you take brings you closer to your ultimate goal, so take pride in your progress and keep moving forward.

### Take breaks and prioritise self-care
Think of yourself as a high-performance vehicle – you wouldn't run it non-stop without refueling and maintenance, right? The same goes for your mind and body. Make sure to take regular breaks throughout the day to recharge your batteries, and schedule time for hobbies, exercise, and relaxation. Your mental and physical well-being should always come first. By investing in self-care, you'll be more efficient and productive in the long run.

### Connect with other indiehackers
Indiehacking can often feel like navigating through a dense forest with no map. By surrounding yourself with like-minded individuals, you'll find support, encouragement, and advice to help light your path. Join online communities, attend meetups or find a mentor to be your guiding star on this journey. Sharing your experiences and learning from others can make the indiehacking process more enjoyable and rewarding.

### Limit your exposure to comparison
Constantly comparing yourself to others can feel like running on a never-ending treadmill – exhausting and discouraging. While it's important to stay informed and connected, try to limit your exposure to others' success stories. This one is actually quite difficult given that a core part of indiehacking is building and relying on the community - what's one thing the communiy loves to do? Talk about their successes on Twitter. 
Remember, everyone's path is different, and comparing your journey to someone else's is like comparing apples to oranges. Focus on your own progress, and celebrate each milestone you achieve.

### Remember your 'why'
When the going gets tough, it's easy to lose sight of the reasons you started indiehacking in the first place. Revisit your passion and let it fuel your motivation when times get tough. Think back to the spark that ignited your indiehacking journey – whether it was the desire for freedom, creativity, or the thrill of building something from scratch. By keeping your 'why' in mind, you'll be better equipped to persevere through challenging moments and stay on course.

<Divider />

Burnout is a real and ever-present danger for indiehackers, but by recognissing the signs and implementing strategies to stay motivated and energised, you can avoid succumbing to this dreaded condition.

I hope you found this post helpful! Be sure to subscribe to my Substack below for more content like this, and [follow me on Twitter](https://www.twitter.com/codestirring) for more indiehacking tips and insights.

If you're looking for a community of fellow indiehackers, I'd encourage you to check out [madepublic.io](https://madepublic.io)!

Burnout as an Indiehacker

If you're an Elixir developer, I can guarentee that you'll be spending a lot of time looking at Hexdocs - where Hex packages host their documentation. 

And if you're anything like me, you'll usually google your way to the answer by searching for the package name followed by `elixir` or `hex` or `phoenix`.

Allow me to revolutionise the way your Elixir development journey by introducing you to a lesser known Google Chrome in-built feature called <Sparkles>custom search engines</Sparkles>.

<Divider />

<Highlight title={'Custom Search Engines?'}>
A function of Google Chrome which lets you setup shortcuts to web addresses by using @ as a delimiter followed by an interpolated value.
</Highlight>

There are a few default custom search engines that Chrome ships with, an example being the ability to search your history by typing: `@history <search_term>` straight into your address bar to find something in your Chrome history.

![Mindblown](https://media1.giphy.com/media/l0NwHXQy3kUSfFF60/giphy.gif?cid=ecf05e4776phke4c7aaf7xjg5i140nm6it9j1cxf4p7mmpp0&rid=giphy.gif&ct=g)

Did I just blow your mind?

## What does this have to do with Elixir packages?
The other required chunk of knowledge is knowing that hexdoc urls all follow the following pattern:

```
https://hexdocs.pm/<NAME-OF-PACKAGE>/*
```

Do you see where we're going here?

I've set up my Chrome settings to allow me to type `@hd ecto` to get straight to the Ecto documentation - something I do several times a week!

## How do I do this?

* Manually navigate to [chrome://settings/searchEngines](chrome://settings/searchEngines)
* Under 'Site Search' add a new entry
* Name the engine what you like (I used `Hexdocs`)
* Add the shortcut of your choosing (I used `@hd`)
* Under URL enter `https://hexdocs.pm/%s`
* Hit Save

And you're done! 

Enjoy getting to Hex documentation without jumping through the Google hoop.

<Divider />

I hope you found this post useful, subscribe to my Substack below for similar content and  [follow me on Twitter](https://www.twitter.com/codestirring) for more Elixir (and general programming) tips.

If you're building a side project I'd also encourage you to take a look at [madepublic.io](https://madepublic.io)!

Setting up a Google Chrome shortcut to Elixir documentation in 30 seconds

## What is ETS?
ETS stands for Erlang Term Storage - it's an Erlang module that lets us store any Erlang term to an in-memory table. 

You can perform basic tasks like creating a table, adding and checking for records by key look-up and even create the tables in a way that they're shared across your application instances - ETS is cool. 

Think of it like [Redis](https://redis.io/) only it comes **built into the Elixir & Erlang language as part of OTP**.

<Sparkles>Ahh, Elixir 💖</Sparkles>

<Divider />

In this post we're gonna talk about why you might want to reach for ETS to speed up some tasks in your applications or processes. 

<Highlight title={'Premature caching warning'}>
⚠️ Never pre-maturely optimise your application - carefully analyse your application and figure out the bottlenecks and fix them after the system is working. That's not to say you should intentionally write your code in ways that are slow and cause long-running tasks, but don't do it too early - you never know how your system might change during implementation and ruin any pre-mature optimisations you may have done.
</Highlight>


## What are we backfilling?
As discussed in [a previous post](https://www.chriis.dev/opinion/how-to-custom-jason-encoding-for-structs-in-elixir), at work, I've been working on backfilling ~1m database rows from a legacy system into our Elixir platform. We're taking over a feature and we need to bring in all the existing user data - each of these records have relations to two other tables in our database and we need to transform the data as we go. The transformations are quite simple, converting epoch time into timestamps and so on. 

To stop our system from grinding to a halt in production, we're consuming our data (from a CSV file exported via `psql dump`) by iterating over it and firing a RabbitMQ message to our topic, which our system picks up and processes. 

For each entry, we need to fetch and make a few associations to other records describing the account that the user is linked to and a link to a category stored in another table. The category is simple enough, we have an `category_id` in our entry which acts as a foreign key to the `category` table - easy. 

**DB fetch count: 1**

The other association is a little bit trickier: our entry provides us with a `legacy_user_id` (from the old system), we need to fetch the user associated to the legacy id and grab the account record that the user correlates to - so two more DB fetches.

**DB fetch count: 3**

With 1m entries to process that's 3 million database reads 😬. It's also worth noting that each user has around 20 entries associated to them. 

## The un-optimised way

The original code to fetch the data looked like this:
```elixir
  defp category_by_legacy_id(legacy_category_id) do
    Repo.get_by(LogCategory, legacy_id: legacy_category_id)
  end

  defp account_by_user_id(user_id) do
    case Users.by_user_id(user_id) do
      {:ok, %{uuid: user_uuid}} -> Repo.get_by(Account, user_id: user_uuid)
      _ -> nil
    end
  end
```

We've got two functions, one that fetches the category by the `category_id`, and another that uses our `Users` context to look up the user (from a different database we don't have direct access to - hence why we can't just do a `Repo.get`), then fetches the account associated. The two records are then used to enrich the entry log changeset using `put_assoc`. 

This wasn't too bad - it worked, but a few benchmarks showed that importing the data took around **23 minutes** - a long-running database import with 1m user records is scary, things can go wrong, the faster they get sorted, the faster the entire thing is over. This import is also going to happen in the dead of night to ensure minimal user impact in case something bad does occur and I don't want anyone to be awake longer than they have to at 4 am. It's also a matter of pride, this code can definitely be improved upon. 

Let's take a look at how ETS comes into play.

## Optimisation Part I

Where to begin?

Well, considering these categories that we're associating were just a few simple Elixir structs with a name and an ID, why should we hit the database every time to fetch them? Storing them in memory is perfect for what we want and ETS has an option we want to take advantage of: `read_concurrency` - a way to tell our tables to optimise for concurrent read operations, [read more here](https://www.erlang.org/doc/man/ets.html#new_2_read_concurrency).

Let's start by creating our in-memory tables and priming that cache with the information it needs to be useful. 

```elixir
  defp fetch_and_cache_categories_in_ets do
    :ets.new(:categories, [:named_table, read_concurrency: true])

    # cache categories in ETS
    Repo.all(LogCategory)
    |> Enum.each(fn category ->
      :ets.insert(:categories, {category.legacy_id, category})
    end)
  end
```

Not bad, we've created a new ETS table, named it `:categories`, told it to optimise for concurrent read operations and then we've stuck our 10 or so categories in there right when the script for importing starts up.

We need to edit our code for actually fetching the data too now:
```elixir
  defp category_by_legacy_id(legacy_category_id) do
    [{_, category}] = :ets.lookup(:otj_categories, legacy_category_id)
    category
  end
```

A simple switch up - we know the categories will always be there because those are the only possible values so we don't need a fallback to the database.

What was the result?

**20 minutes** - we shaved off 3 minutes, not too bad for such a small fix right? We've just eliminated 1 million database fetches from our import with a few minor tweaks.

But we can do better.

## Optimisation Part II

The user lookup and account fetch are clearly what's slowing us down here, each call depends on the last database fetch to success and we're doing it across databases. That doesn't mean we can't use caching to make our lives easier though.

Let's start by making a new ETS table:
```elixir
  defp create_account_ets_cache do
    :ets.new(:otj_accounts, [:named_table, read_concurrency: true])
  end
```

We can't prime our cache before anything has started this time sadly, but we can check our ETS cache for a record and fallback to a database fetch if we don't get a hit - when we fetch from the database we can just pop it into our cache for the next time we come across the same `legacy_user_id`.


```elixir
  defp account_by_user_id(user_id, :db) do
    case Users.by_user_id(user_id) do
      {:ok, %{uuid: user_uuid}} ->
        account = Repo.get_by(Account, user_id: user_uuid)
        :ets.insert(:otj_accounts, {user_id, account})
        account

      _ ->
        nil
    end
  end

  defp account_by_user_id(user_id, :ets) do
    account = :ets.lookup(:otj_accounts, user_id)

    case account do
      [] ->
        account_by_user_id(user_id, :db)

      [{_, account}] ->
        account
    end
  end
```

In our import code, we use the `:ets` version of our function, which will call the `:db` version if it needs to.

So what has all of this done to our import?

## A slight tune-up - 278% faster
The new version of the code finishes importing all 1,200,000 records in 8 minutes 20 seconds - the average RabbitMQ message acknowledgements went from **870 ack/s to 3200 ack/s**.

We just increased the rate of our import by 278% with a git diff that comprises of about 35 lines of code. 

<Sparkles>Ahh, Elixir 💖</Sparkles>

<Divider />

I hope you found this post useful, subscribe to my Substack below for similar content and  [follow me on Twitter](https://www.twitter.com/codestirring) for more Elixir (and general programming) tips.

If you're building a side project I'd also encourage you to take a look at [madepublic.io](https://madepublic.io)!

Using ETS to speed up database imports by almost 300% 

## What is a feature flag?
Feature flags are a common technique used in software engineering to easily turn select functionality on or off during runtime. 

Ideally these systems don't require any deployment but for the purpose of this tutorial I'm going to be sharing the mechanism I built in about 3 minutes for [madepublic.io](https://madepublic.io).

If you need a full on solution I recommend checking out the excellent [Fun With Flags library](https://github.com/tompave/fun_with_flags) but for my purposes I needed something that I didn't need to configure, that would allow me to hide new functionality from production (that wasn't ready yet) and carry on developing them in dev and test runtimes. 

This mechanism requires deployment but since I'm deploying with [fly.io](https://fly.io), it takes all of 90 seconds to push up my changes which is great for me - I want to spend my time building useful features over tooling.

<Divider />

## On with the building!

We want something flexible so lets make sure this mechanism is flexible enough to handle other features I want to do this with in the future.

So I want to essentially not render certain templates or code if a feature is turned off - let's do that.

First, let's define a module that can read from our Application environment variables:
```elixir
# lib/features.ex
defmodule Madepublic.Features do
  def feature_available?(feature_key) do
    Application.get_env(:madepublic, :features)[feature_key]
  end
end
```

For this to work we need to add new variables to our config files, let's use a keyword list so we can define more in the future.

We also want to make sure that we have a default set - I want every environment other than production to have this feature turned on - for that we can use the `config.exs` file to specify a default, then change it on a more granular level for production by overriding it in `prod.exs`.

```elixir
# config/config.exs
config :madepublic, :features,
  auth: true

# config/runtime.exs
if config_env() == :prod do
	config :madepublic, :features,
		auth: false
end
```

Sweet - we have the feature but we want it to be available to every template we might have.

Let's hope into our `web.ex` file and include our new module in our `view_helpers`.

```elixir
# lib/madepublic_web.ex
defp view_helpers do
  quote do
    # all the original imports and aliases
    import Madepublic.Features # <- New addition
  end
end
```

Now let's give it a whirl:

```elixir
# In any of my templates or .heex files
<%= if feature_available?(:auth) do %>
    <%= render "_user_menu.html", assigns %>
<% end %>
```

<Divider />

### Localhost
![Locally](https://i.imgur.com/iZzaASs.png)

### Production
![Production](https://i.imgur.com/XLfTg5n.png)

<Divider />

It works!

We have a repeatable, simple feature flag system that we can use to slowly unveil features to our users.

I hope you found this tutorial useful, subscribe to my Substack below for similar content and  [follow me on Twitter](https://www.twitter.com/codestirring) for more Elixir (and general programming) tips.

If you're building a side project I'd also encourage you to take a look at [madepublic.io](https://madepublic.io)!


How to build the simplest feature flag system in Elixir & Phoenix apps

## Phoenix LiveView
Phoenix LiveView is my **favourite** way to create web applications these days - the PETAL stack is effortlessly fun to use and will (in my opinion) soon be a mainstream stack of choice for web developers looking to create real-time applications without having to worry about the present-day client-side worries that acompany todays most popular tools of choice. 

Chris McCord eloquently describes the power you gain alongside the god-send of being able to practically forget about 90% of the client-side code in his [Fly.io blog post of how LiveView came to be](https://fly.io/blog/liveview-its-alive/) - highly recommended reading.

I've built a few LiveView applications (and consider myself lucky enough to be able to use it at work) including:
* [6words.xyz](https://6words.xyz) - a wordle inspired web game
* [niceice.io](https://niceice.io) - a SaaS service for capturing feedback from your users as easily as possible

If you want to hear more about any of these projects, send me a message on Twitter and I'll be happy to dive into more about how they're built.	

<Highlight title={'PETAL stack?'}>
**P**hoenix, **E**lixir, **T**ailwind, **A**lpine & **L**iveView.
</Highlight>

## Why would you want to trigger front-end animations from the backend?
LiveView is great at building real-time applications - when I say real-time I mean instantly reactive across all users currently browsing the site. 

I'm currently working on a platform to enable the sharing of user-created fictional stories online when I managed to stumble across the basis of why this tutorial is needed whilst trying to bring a new feature to life.

As part of my new platform users can submit stories, chapters and produce content for users to read. Wanting to add some more pizzazz to my application - I figured it'd be cool to have a Live Global Statistics component on the front-page of my site so users could see how active the site was in real-time!

### So I made a simple LiveView component
![Component Screenshot](https://i.imgur.com/TKFCnqw.png)

```elixir
  def mount(_params, _session, socket) do
    socket =
      socket
      |> assign(:story_count, total_story_count())
      |> assign(:word_count, total_word_count())
      |> assign(:chapter_count, total_chapter_count())

    {:ok, socket}
  end
	
  def render(assigns) do
	~H"""
	 <p><span id="stories-count"><%= @stories_count %></span> stories submitted</p>
	 <p><span id="chapters-count"><%= @chapters_count %></span> chapters published</p>
	 <p><span id="word-count"><%= @word_count %></span> words written</p>
	"""
  end
```

So this is cool - I have a component that updates whenever a user access the page, but that's not very real-time is it?

Presently the information is only updated on `mount` - lets change that with the <Sparkles>magic</Sparkles> of the `Phoenix.PubSub` module that ships with Phoenix by default. 

To do so we need to create a topic for our PubSub to subscribe to (and enable PubSub in my applications supervisor tree):

```elixir
# MyApplication.Submissions

  @topic inspect(__MODULE__)

  def subscribe do
    PubSub.subscribe(MyApplication.PubSub, @topic)
  end

  defp notify_subscribers({:ok, result}, event) do
    PubSub.broadcast(MyApplication.PubSub, @topic, {__MODULE__, event, result})
    {:ok, result}
  end
```

I can now use this `notify_subscribers/2` function whenever I want to alert something subscribed to an update I'm interested in broadcasting like so:

```elixir
  def update_story(%Story{} = story, attrs) do
    story
    |> Story.update_changeset(attrs)
    |> Repo.update() 
    |> notify_subscribers([:story, :updated]) # ⬅️ the interesting bit
  end
```

Then we need to ensure that when our `live_component` mounts and connects to the websocket, it subscribes to the topic.

```elixir
  def mount(_params, _session, socket) do
	if connected?(socket) do
		MyApplication.Submissions.subscribe()
	end
	
	# and add an event listener to ensure our LiveView knows to react when it receives a message from our subscribed topic
	def handle_info({MyApplication.Submissions, [:story, _], _}, socket) do
		socket =
			socket
			|> assign(:story_count, total_story_count())

		{:noreply, socket}
	end
  end
```

<Divider />

Now when we update our stories - *notice I'm ignoring the second atom so I'll call my new assignment whenever any story change happens* - our front-end will update for all users!

**We have an issue though.**

There's no animation! This can be pretty jarring for users so let's get onto the real point of this post; triggering animations from the backend to really delight our readers.

## Animation time 🎨
<Highlight title={'A note on CSS'}>
For my example I'm using Tailwind (yay, PETAL 🌸 stack) but this will work with any CSS class so long as the animation and keyframe attributes have been set appropriately. 
</Highlight>

First let's define our animation in CSS (in our `tailwind.config.js`):
```javascript
theme: {
    extend: {
      keyframes: {
        wiggle: {
          '0%': { transform: 'translateY(0px) scale(1,1)' },
          '25%': { transform: 'translateY(-4px) scale(1.05,1.05)', background: 'aquamarine' },
          '100%': { transform: 'translateY(0px) scale(1,1)' },
        }
      },
      animation: {
        wiggle: 'wiggle 0.5s linear 1 forwards',
      }
    },
  },
```

All we're doing is making it jump a little; let's press on with actually integrating this.

<Divider />

### First attempt
At first I believed I could simply use the `LiveView.JS` library to add a class to the element in question from the backend and pass it to the front end like so:

```elixir
  def do_animation do
    JS.add_class("animate-wiggle", to: "#word-count")
  end
```

**Keep in mind** I was also testing this using a simple button with a click handler `phx-click={do_animation}` for ease of not having to actually trigger backend events each time - so I was using phx-click... 👀

This added the class and the animation did a little jump - great. 

I clicked it again and nothing happened, not great.

This is because the class lived on the element so adding it again meant nothing would happen - my animation wasn't repeatable. Whoops.

### Second attempt
Let's remove the class after the class has been added.

```elixir
  def do_animation do
    JS.add_class("animate-wiggle", to: "#word-count")
	send(self(), JS.remove_class("animate-wiggle", to: "#word-count"))
  end
```

This didn't work because the class was being removed as it was being added. I could've added a timeout but that seems far too hacky.

### Third attempt
```elixir
  def animate_wiggle(element_id) do
    JS.transition(%JS{}, "animate-wiggle", to: element_id, time: 500)
  end
```

`JS.transition/2` to the rescue! The LiveView team built a specific function for triggering transitions repeatedly. ❤️

**But there was an issue** - `LiveView.JS` functions simply generate JavaScript, so they have to be rendered in the page! 

So what do we do?

[RTFM](https://hexdocs.pm/phoenix_live_view/js-interop.html#handling-server-pushed-events) of course! Onwards!

### Fourth attempt
I had to push the event to the browser so that some JavaScript could execute the wiggle animation for me - so the flow goes like this:

* PubSub broadcasts event
* Each subscribed LiveView process listens to that event and triggers an event to their clients
* Client has a JavaScript event listener to pick up on `phx` events to react to them
* JavaScript fires a call to the client to trigger the animation
* `JS.transition/2` fires
* Wiggle wiggle 

Let's add the JS event listener in our `App.js`:
```javascript
window.addEventListener(`phx:wiggle`, (e) => {
  let el = document.getElementById(e.detail.id)
  if(el) {
    liveSocket.execJS(el, el.getAttribute("data-wiggle"))
  }
})

```

Let's update our event handler when to push the event to the client:

```elixir
def handle_info({MyApplication.Submissions, [:story, _], _}, socket) do
	socket =
		socket
		|> assign(:story_count, total_story_count())
		|> push_event("wiggle", %{id: "stories-count"}) # ⬅️ the new addition
			
	{:noreply, socket}
end
```

We also need to ensure we add an id and a data attribute to the element we want to wiggle so our JavaScript can find it and know what to do with it:
```elixir
<p><span id="stories-count" data-wiggle={animate_wiggle("#stories-count")}><%= @stories_count %></span> stories submitted</p>
```

### Result
![Wiggle](https://i.imgur.com/49aY4NH.gif)

What you can't see is I have another window triggering the aforementioned events. 

<Divider />

We're done! 🚀

We've successfully triggered repeatable front-end animations from live events coming from other users of our application with minimal code (and literally 6 lines of JavaScript).

I love LiveView and I hope this post has given you a flavour of why. 

Subscribe to my Substack below for similar content and [follow me on Twitter](https://twitter.com/codestirring) for more LiveView, Elixir, and general programming tutorials & tips.




Triggering repeatable animations from the server in LiveView & Elixir

## Jason?
[Jason](https://github.com/michalmuskala/jason)  is an actively maintained JSON parser and generator in Elixir - if you think of handling JSON structures to send to your clients or just shepherding data around your Elixir application you’re likely to reach for this or [Poison](https://github.com/devinus/poison) - of which the contents of this post are relevant to too. 

## Purpose & Use-case
<Highlight title={'Disclaimer'}>
	The following isn’t required - feel free to skip to the content below for the tutorial proper. Reading this, however, will give you an understanding and real-life example of when you might want to use custom Jason encoding.
</Highlight>

At Multiverse we’re working on developing a new version of an existing third-party integration and taking it in-house as part of our core platform. The feature in question isn’t important, it’s just important to know that there’s *a lot* of data - think millions of records.

One problem is that we don’t have direct access to the database of the existing system - what we do have is the ability to export the data into a CSV! Thankfully there are only 8 or so fields for each record meaning we can resort to a good, old-fashioned CSV import to bring the data over.

Simple right? *Wrong*. 

Importing 1m+ rows of data in PSQL isn’t *too* bad but it will slow down and potentially lock up the rest of our application while we do the import - and that’s bad! We also need to transform the data into a more reasonable data structure for our new implementation - the old version had some inconsistencies and type issues we’d like to do away with - for example dates represented in epoch time and un-required data fields.  

<Highlight title={'Flash lesson ⚡️ - Epoch'}>
	Epoch time (a.k.a. Unix time) is a system for describing a point in time by counting the number of seconds (excluding leap seconds) that have elapsed since the /Unix epoch/ - the Unix epoch being *00:00:00 UTC on 1-1-1970*.
</Highlight>

So now we need to transform and import each record, something that’ll take time. To mitigate this, we’re going to use [RabbitMQ, an open source message broker](https://www.rabbitmq.com/) by packaging each of the rows from our CSV into a JSON message, asynchronously firing them off as messages and let our RabbitMQ consumer deal with them as they come in concurrently - *if you’re interesting in learning more about Rabbit, let me know!*

<Divider />

So where does Jason encoding come in?

Remember that we want to transform our data - the issue with exporting everything into a CSV file is that when we read it into our application we’ll be reading everything as a string. Converting our date fields now involve us manually calling `String.to_integer/2`  which is something we don’t want to have to do manually - imagine we had 100 fields to import! 

<Sparkles>A good programmer is a lazy programmer.</Sparkles>

### Defining our struct 
```elixir
defmodule LegacyLog do
  defstruct [:id, :user_id, :legacy_log_id, :time, :notes, :date, :inserted_at, :updated_at]
end
```

All we have here is a simple Elixir struct with fields for each of the records we want to handle and parse from the CSV import.

Using it in our CSV import:
```elixir
  defp serialise(data) do
    data
    |> Enum.map(fn row ->
      [id, user_id, logtype_id, time, _target, notes, date, timecreated, timemodified, _, _] = row

      %LegacyLog{
        id: id,
        user_id: user_id,
        legacy_log_id: logtype_id,
        time: time,
        notes: notes,
        date: date,
        inserted_at: timecreated,
        updated_at: timemodified
      }
    end)
  end
```

Yes, we could transform the data here as we import it but good RabbitMQ practise involves pretending that our messages are coming from somewhere external we don’t have direct access to - even though we’re producing and consuming them ourselves in this instance.

### Custom encoding the JSON
*The bit you’re really here for.*

We need to take advantage of the way `Jason` has defined `Protocols` to define a custom implementation on how to handle our encoding.

Following on from our previously defined struct, we’re going to add a `defimpl` for the struct itself and specify what we want to do:

```elixir
defmodule LegacyLog do
  defstruct [:id, :user_id, :legacy_log_id, :time, :notes, :date, :inserted_at, :updated_at]

  defimpl Jason.Encoder, for: LegacyLog do
    @impl Jason.Encoder
    def encode(value, opts) do
      {notes, remaining_log} = Map.pop(value, :notes)

      remaining_log
      |> Map.from_struct()
      |> Map.new(fn {k, v} -> {k, String.to_integer(v)} end)
      |> Map.put(:notes, notes)
      |> Jason.Encode.map(opts)
    end
  end
end
```

In the above code I’m defining an implementation for the type `LegacyLog` meaning when a `Jason.Encode/3` call comes it’s way, it knows to call my custom code - any other encoding function call will use the default implementation (which is fine for most use cases).

The custom code itself is:
* Popping the one field I want to remain a string out of the Struct and keeping the rest
* Converting the remaining  `Struct` into a `Map`
* Iterating through each field and changing them into integer values
* Putting the notes back
* Doing the iodata generation using `Jason.Encode` which guarantees valid JSON.

<Highlight title={'Flash lesson ⚡️ - Protocols'}>
Protocols are a mechanism to achieve polymorphism in Elixir when you want behaviour to vary depending on the data type. It can be pretty powerful.
Again, reach out if you’d like to hear more about Protocols.
</Highlight>

<Divider />

We’re done!

 Subscribe to my Substack below for similar content and  [follow me on Twitter](https://www.twitter.com/codestirring)  for more Elixir (and general programming) tips.



Custom JSON encoding for structs in Elixir with Jason

Elixirs popularity has exploded in the last year - and for good reason! 

Stack Overflows 2022 Developer Survey lists Phoenix as _the most [most loved Web framework](https://survey.stackoverflow.co/2022/#section-most-loved-dreaded-and-wanted-web-frameworks-and-technologies)_ and Elixir is now [the sixth highest paid language](https://survey.stackoverflow.co/2022/#section-most-loved-dreaded-and-wanted-web-frameworks-and-technologies) 👀 - it might be time to have a look if you haven't already.

<Divider />

## What is Elixir?

It's a dynamic, functional programming language with an emphasis on fault tollerance, scalability, and developer experience. **It's gorgeous.**

I'm fortunate enough to write Elixir daily (when I'm not in meetings) as part of my job as the core tech stack and have done so for over 3 years. As such I've encountered some of the gotchas you're likely to experience if you're new to the language and amassed a bank of helpful tips to ensure you're really getting the most out of the language.


## 1. Get used to pattern matching
In Elixir, the `=` operator is actually the match operator. You can use it to assign variables just as you might expect but it's hidden power comes from it's ability to destructure complex data types.

**A quick example from the official language guide:**

```Elixir
iex> {a, b, c} = {:hello, "world", 42}
{:hello, "world", 42}
iex> a
:hello
iex> b
"world"
```

Now this is all well and good but some of the real fun comes from pattern matching on function heads to:

* Replace pesky `if` statements

```Elixir
defp greet(%{first_name: first_name} = user), do: "Greetings, #{first_name}!"
defp greet(_), do: "Greetings, stranger!"
```

* Handle different outcomes in your code declaritively 

```Elixir
defp split_traffic(%{age: users_age}) when is_even(users_age), do: # redirect to path A
defp split_traffic(%{age: users_age}) when is_odd(users_age), do: # redirect to path B
```


## 2. Append to a list the correct way
Lists in Elixir are effectively linked lists - they are internally represented in pairs containing the head and the tail of a list. 

Meaning both of these approaches are valid:
```Elixir
list = [1 | [2 | [3 | []]]]
# [1, 2, 3]
list = [1,2,3]
# [1, 2, 3]
```

Because of their linked list nature, prepending to a list is always faster (as it's constant time vs having to internally traverse each list til the end and append - linear time).

You can add to them by concatenating lists together using `Kernal.++/2` or you can use `|` to do the same thing *faster*.

Have a look at the example below (which is using the erlang timer module to time the function call).

```Elixir
iex(19)> :timer.tc(fn -> [1,2,3 | 4] end)
{6, [1, 2, 3 | 4]}
iex(20)> :timer.tc(fn -> [1,2,3 | 4] end)
{6, [1, 2, 3 | 4]}
iex(21)> :timer.tc(fn -> [1,2,3 | 4] end)
{6, [1, 2, 3 | 4]}
iex(22)> :timer.tc(fn -> [1,2,3] ++ [4] end)
{13, [1, 2, 3, 4]}
iex(23)> :timer.tc(fn -> [1,2,3] ++ [4] end)
{14, [1, 2, 3, 4]}
iex(24)> :timer.tc(fn -> [1,2,3] ++ [4] end)
{12, [1, 2, 3, 4]}
```

## 3. Finding N slowest tests in mix application
You'll be writing tests in your Elixir code (*right?* 🙏🏼), and once your application grows to a large size you may find your tests take a long time to execute (not too long, Elixir's pretty speedy). 

You can narrow the culprits down with the following:

```Elixir
# prints timing information for the N slowest tests
mix test --slowest N  
```

## 4. Read the docs (especially Enum)
This might be an obvious one to many of you but you'd be surprised how often people don't [RTFM](https://en.wikipedia.org/wiki/RTFM). I actually discovered that last tip by reading the official docs 🤯 - Elixir's documentation and standard libraries are stellar. 

I can guarentee that you'll be using the `Enum`, `Map`, `List`, and  `Kernal` modules often - get used to reading their documentation. 

You'll find examples like:
* `Enum.chunk_while/2` for chunking lists when certain conditions are met
* `Enum.into/2` for converting enumerables into collectables (lists into maps)
* `Map.put_new_lazy/3` for putting the result of a function into a key if it's not already there

Any time you think there's a more functional way to do something - there probably is, get reading, chances are the standard librarys got you.

This leads me on to my next point quite nicely...

## 5. Embrace Doctests
It's impossible to talk about how good Elixirs documentation is without talking about the <Sparkles>magic</Sparkles> of doctests. I've spoken about these before in my [Elixirs Hidden Potions post](https://chrisgregori.substack.com/p/elixirs-hidden-potions-664814640969) (which you should also read) but I have to re-iterate just how powerful this can be.

Doctests let you annotate your functions with example usage (including expected output) and have them run as part of your test suite. 

**Yes**, they're as amazing as they sound.

Taken from the official [getting started guide](https://elixir-lang.org/getting-started/mix-otp/docs-tests-and-with.html):

```elixir
defmodule KVServer.Command do
  @doc ~S"""
  Parses the given `line` into a command.

  ## Examples

      iex> KVServer.Command.parse("CREATE shopping\r\n")
      {:ok, {:create, "shopping"}}

  """
  def parse(_line) do
    :not_implemented
  end
end

# Running the tests...

1) test doc at KVServer.Command.parse/1 (1) (KVServer.CommandTest)
     test/kv_server/command_test.exs:3
     Doctest failed
     code: KVServer.Command.parse "CREATE shopping\r\n" === {:ok, {:create, "shopping"}}
     lhs:  :not_implemented
     stacktrace:
       lib/kv_server/command.ex:7: KVServer.Command (module)
```

## 6. Atoms are not garbage collected
Be careful about converting user supplied parameters and converting them into atoms. Atoms are meant to be used as named constants - do not take parameters from your users and convert them into atoms. 

**Atoms are not garbage collected.**

The reason for this is becase Atoms are represented as integers internally - when Beam (the Erlang VM) sees an atom for the first time, it inserts it into the atom table. 

Don't worry - when your code is litered with `:ok` atoms from tuples they're all referencing the same atom (the one referenced from your atom table).

If you really, for some reason, need to convert user generated data into atoms - reach for `String.to_existing_atom/1`.

Fun fact: the default maximum number of atoms is `1048576`.

## 7. Embrace the small, but mighty community
Elixir isn't the biggest language (yet  😏) but it has one of the most helpful communities I've ever stumbled across. 

### Slack
I'm consistently being assited by the creators of platforms, libraries, **the language itself** all on the Elixir Slack channel. Get involved and do your part answering questions you can help with as they pop up. 

### Twitter
Some people hate Twitter - I maintain that it's a treasure trove of great information if you're willing to curate your feed and block out the noise. 

Some of my favourite Elixir twitter accounts to follow include:
* [Jose Valim - the creator of Elixir](https://twitter.com/josevalim/)
* [Chris McCord - the creator of Phoenix & LiveView](https://twitter.com/chris_mccord)
* [Uku Täht - Founder of Plausible Analytics](https://twitter.com/ukutaht)
* [Chris Gregori 😉 - Founder of niceice.io & 6words.xyz](https://twitter.com/codestirring)
* [Johanna Larson](https://twitter.com/joladev)
* [Evadne W](https://twitter.com/evadne)

If I've left you out [send me a message on Twitter](https://www.twitter.com/codestirring) and I'll add you to the list!

## 8. Add labels to your IO.inspect/2 calls
Really you should be using Pry for debugging your applications (see more in my other article here), but sometimes you just need a quick and dirty way to print out and that's where `IO.inspect` comes in.

The thing is you may get lost in the sea of terminal print out when you're trying to debug lots of data that looks very similar. 

Consider the following pseudo-code:

```Elixir
customer_orders
|> Enum.map(&validate_purchases/1)
|> IO.inspect()
|> Enum.map(&exclude_late_orders/1)
|> IO.inspect()
|> Enum.map(&email_customers/1)
```

If your `customer_orders` represented a complicated schema with lots of nested data (and many of them) you'd struggle to decipher which are which in your printed logs after transformation - a simple option being passed to our IO.inspect/2 calls makes this inspection a hell of a lot more legible:

```Elixir
customer_orders
|> Enum.map(&validate_purchases/1)
|> IO.inspect(label: "Validated purchases")
|> Enum.map(&exclude_late_orders/1)
|> IO.inspect(label: "On-time orders")
|> Enum.map(&email_customers/1)
```

## 9. Try and keep your functions small and composable
This is a tip that applies to most software disciplines also.

When I first stumbled across functional programming, one thing that took some getting used to was creating lots of little functions. I used to write a lot of React - and thus a fair amount of boilerplate - which meant that I was used to large render functions. 

In Elixir, and all functional programming, try and create lots of small function calls to describe your code. There are plenty of benefits:

* It's more declarative so it's easier to read and reason about
* It's easier to refactor later on down the line
* It's easier to test
* It helps enforces the DRY principle
* If you structure your code nicely you'll find that you'll re-use plenty of functions often which inadvertently leads to better code structure. Better code structure also leads less circular dependency referencing which leads to faster compile times (but this is a topic for another blog post - let me know if you'd like to hear more!)


## 10. Learn about Macros but be cautious
Macros are powerful - with great power comes great ~responsibility~ chance of doing something invisible and horrendous to your codebase. Even [the official language docs](https://elixir-lang.org/getting-started/meta/macros.html) tell you to use them responsibly. 

You can use macros to extend module behaviour by adding functionality through wrappers - think along the lines of always logging when you call a function without having to declare it - or dynamically generating routes in your Phoenix application to warn a developer when they've not accounted for something they should've like localisation. 

It's good to be aware of Macros and what they can do for you - but seriously, triple check yourself every time you think about adding one. If you want a custom guard clause for your function heads pattern matching, look at `defguard` and `defguardp` instead.


<Divider />

There we have it.

10 tips to help you embark upon your Elixir journey. It's an incredible language, give it a chance and you'll slowly realise why I've been hooked for the past 3 years. Subscribe to my Substack below for similar content and [follow me on Twitter](https://www.twitter.com/codestirring) for more Elixir (and general programming) tips.



10 tips for new Elixir developers

Version control is <Sparkles>magical.</Sparkles>

I strongly believe everyone should learn how to use it - not just developers. How many times did you complete homework back at school and create arrays of filenames to denote just how final that piece of work was?

Today, in modern web development, Git is an absolute pre-requisite. But it's still shocking to see just how many experienced developers are just scratching the surface of what Git can do for you.

In this blog post I hope to unearth at least one useful Git command you perhaps haven't come across or would even think to use.

## git rebase
Git rebasing is boils down to moving or combining a sequence of commits into a new base commit. It essentially appears like you've created your branch from a different commit base - extremely useful if you've been working on a branch for a while, you've diverged off of the main branch for a while and you run into conflicts when you try and merge back in.

It works something like this:

```bash
git checkout main # change to main branch
git pull origin main 
git checkout - # go back to last branch (feature branch)
git rebase main
# fix conflicts
git add .
git rebase --continue # until all changes are in
```

You can also use `--interactive` mode for rebasing which will let you manually select which commits you want to ignore throughout - helpful if you've made a commit purely to add some debugging and forgotten before committing. 

Now, some of you might be thinking you can just `git pull origin main` in your branch and resolve the merge conflicts _in an even easier fashion_. While they are both similar there's one massive difference that you want to keep in mind here:
* `git pull` will create a new commit that combines the two branches together
* `git rebase` will move the commits of one branch on top of the other

You can also use `git pull --rebase` to pull using rebase which can be great and save you a lot of time. However

`git rebase` will essentially keep your git history clean. 

**Why is this important?** 

It means you can safely and effectively use our next arcane power: `git bisect`.

## git bisect

`git bisect` uses binary search to find a commit that introduced some new unwanted functionality (read: bug). 

It kind of works like this:

Imagine you're digging for the commit that broke a feature that we knew was working in commit `0cd429f02` (or a tagged commit), and somewhere along the way we broke something and a test is failing. 

We can use git bisect to narrow down the commits and help us identify the culprit like so:

```bash
git bisect start
git bisect bad
git bisect good 0cd429f02
```

**Note: You can use `git bisect reset` to jump out of this.**

Once you've specified a good and a bad commit, bisect will find a commit in the middle of that range, check it out and give you some information regarding how many possible steps are left to split through.

Now that you're in your checked out commit you should run your tests and assuming it works, you can inform bisect of that by typing `git bisect good`, or tell bisect that that particular version is failing with `git bisect bad`.

Repeat the above steps and eventually you'll run out of revisions to test - bisect will then spit out the description of the first bad commit with a nice little summary - and there's your buggy commit. ✨

## git stash
Stashing is used to take your uncommitted changes (including staged) and temporarily "stash" them away for later. 

Extremely useful when you're not quite ready to commit a change, but need to move to another branch for some reason - maybe there's a bug you need to investigate. 

You can stash your content like this:

```bash
git status

Changes not staged for commit:
	modified:   README.md
	modified:   mix.exs
	modified:   test/inspyre_web/live/story_live_test.exs

no changes added to commit (use "git add" and/or "git commit -a")

git stash

git status

On branch main
Your branch is up to date with 'origin/main'.
```

Like magic your changes are seemingly gone.. or are they?

If you run `git stash list` you'll see a log of previously stashed events alongside the branch you were operating on and the last commit in that log. 

Running `git stash apply` will take the last stashed set of changes and apply them to your current branch **but will not remove them from your stash list**. If you know you won't need them this one time stash you'll want to use `git stash pop` to apply the changes and pop them out of your stash list.

## git reset --soft HEAD^N
Have you ever accidentally thought you changed branch but really you were still in the last feature you were working on?

Have you committed multiple times?

**Whoops**. It's cool - we can fix that. 

Use `git reset --soft HEAD^N` where N is the number of commits you want to un-commit to bring them back to you. From here you can stash or move to another branch and commit them there.

**Be careful not to type `--hard` though. That'll trash unsaved changes for good.**

## git reflog
Gits secret weapon / time capsule. 

`git reflog` will give you a list of every single thing you've done in git across every single branch. 

If you've accidentally ended up in a state you weren't expecting to be in, deleted things, merged things accidentally or can't figure out what's happened, reach for `git reflog` (at least before you trash everything and re-clone your repo).

## git branch -M <Sparkles>BONUS</Sparkles>
This one is simple but effective - and one I know several devs resort to deleting and recreating a branch to solve!  🤯 

Renaming a branch!

Simply type `git branch -M <new_branch_name>` to set your new branch name for the branch your currently in.


<Divider />

That's all folks. 

Hopefully that was useful for some - subscribe to my Substack below for similar content and [follow me on Twitter](https://twitter.com/codestirring) for more git (and general programming) tips.


5 scary Git commands you'll eventually have use for

### What is Imposter Syndrome?
> The persistent inability to believe that one's success is deserved or has been legitimately achieved as a result of one's own efforts or skills. - Oxford Dictionary

<Divider />

A psychological occurrence in which someone (in the case of this article, a developer) doubts their skills, talents or accomplishments despite clear external evidence suggesting otherwise.

Almost every developer - irrespective of education or experience - has felt it at some stage of their career - and if they say they haven't, *I don't believe them.*

## Why is it so prevalent in software?
It's due to the very soul of the tech industry - a symptom of the open, free sharing, nature of the job. With so many different tech stacks, languages, paradigms, layers of logic and knowledge, it's literally impossible for a developer to know absolutely everything.

You _will_ come across someone that knows more about a topic than you do. 

Is it a language feature that'll set you off? A design pattern? Knowing how to host and distribute software? AI? Blockchain? Web3? SEO? Accessibility? Web design? Performance? Systems design? UX? 

We expose ourselves to as much as we can to make ourselves better in our roles - hell, the job itself is about constant learning to solve problems. **Our job is about learning things.** If a new ticket emerges tomorrow on your board and you don't know how to solve it, you're supposed to go and figure out how to do it. That's the job.

The modern web developer browses the internet daily for assistance, news and updates about their job - this culminates in more exposure to more things we don't know, and that can be a scary, scary thing for someone unsure of themselves in a professional capacity. Read something on Twitter? An article on Hackernews or Medium? Did someone just announce a new library on Reddit about something you feel like you _should_ know about?

Chances are, that can be the trigger for that anxiety and ultimately make you feel like an imposter, and that's before even considering how quickly things change around here. 

The rate that this industry moves is absolutely staggering. There is *always* going to be more to learn and it'll come at you at a pace that most humans can't keep up with. My team built a state-of-the-art marketing website with all the bells and whistles one could hope for around a year ago using some of the most bleeding edge, production ready tech available. Today, spending six hours on Twitter and Hackernews will tell you that your tech stack is archaic and that you should be building everything in [Deno](https://deno.com/deploy) and [Fresh](https://deno.com/blog/fresh-is-stable). 

## So, what is the cure?

Honestly, it never really goes away. **Sorry, but that's the truth**.

At the time of writing this, I'll have been working as a professional Software Engineer for over 9 years.

I have worked with engineers to create software that has won accolades, won hackathons against a plethora of talented engineers, been promoted several times across multiple organisations and now lead a team due to my _supposed_ knowledge. 

**But none of that will matter to my brain.** 

Every now and then (*read: weekly*) I'll see something online or talk to someone and feel like I'm a fraud. It happens. I know this is the case for other engineers I've spoken to as well, but even if there's no cure, there _is_ medicine to relieve the symptoms. 

<Divider />

### Embrace the opportunity
Re-frame your mindset. If you come across a person or an experience that triggers that feeling of not knowing, take the bull by the horns; ask questions, admit that you don't know something and take it as an opportunity to learn and get better. It's better to be humble and admit you don't know something if it presents a chance to fix it.

Do you find that there's an insurmountable number of things that you need to learn? *Prioritise*. Write them all down, score them according to their importance, or how much they interest you or perhaps which topic would be most relevant. 

Life is about learning - embrace the opportunity to learn something today that you didn't know yesterday.

### Embrace what that desire to learn really means
If not knowing something is upsetting you, it's likely that it's because you care and you want to improve. If you have the capacity and desire to be honest about what you know and don't, are able to learn new skills, tech stacks and paradigms then that's as much of a definition of the job as there'll ever be.

Recognise that and recognise that it's a better situation to be in than thinking you know everything. You don't. No one does.

### Rationalise the facts
Start listing the facts. 

**Out of the following scenarios, which is more likely?**
* That you scammed your way through all of your credentials, tricked every experienced interviewer you met and bamboozled a company into paying you a good salary to do a job you're not equipped to do

Or

* You know how to do your job, you've proved that through a series of rigorous tests which _are researched and designed by "the professional developers" to weed out candidates_, and met the criteria of a company willing to pay someone an agreed upon wage to do a job they believe you can do

Be really honest with yourself here. 


### Encourage your peers and promote collaboration
Software development is a team sport - almost every single product worth its merit - be that a website, an open source library you use, the circuit powering the device you're reading this on - has been built by a team of people with different skills all working together.

* Encourage teamwork 
* Recognise failure as an opportunity to learn and celebrate wins together 
* Pair program 
* Mob program 
* Write the code together
* Write the systems design document in a group
* Collaborate
* Teach each other
* Tackle open source issues
* Inspire and challenge each other.

You'll be a better developer for it and you'll learn _so_ much more.

<Divider />

Ultimately, just remember that there will always be a higher bar to compare yourself against - the breadth of possible topics of expertise in this industry makes people feel like they're not up to scratch if they come across something they don't know and that's completely natural.

Jimi Hendrix couldn't play the guitar until he could.

Everyone is a beginner until they're not. 

Be okay with the fact that you might come across something you know nothing about, but take solace in the fact that you have the capacity to; you've proven that before.



Why do developers experience so much imposter syndrome?

## 👋🏼

My name's Chris Gregori - I'm a full stack software engineer specialising in Elixir and React, currently leading a team at [Multiverse](https://www.multiverse.io) - <Sparkles>the UK's first [EdTech unicorn](https://www.uktech.news/education/euan-blair-multiverse-edtech-unicorn-20220609)!</Sparkles> (*Yes, we're hiring, reach out if you're interested in changing education for the better*). 

<Divider />

This is my new personal site - it's been coming for a while, I initially created it as a Next.js project about a year ago and life (and design fatigue and side project-itis) got in the way until about a week ago when I had a spurt of inspiration. So I scraped 90% of the repository (aside from some static data I had written which was still relevant), upgraded Next to v12.0 and hooked everything up to a new [Sanity.io](https://www.sanity.io/) project. 

## Why Next.js?
Next is a React.js framework. According to the [latest Stack Overflow survey](https://survey.stackoverflow.co/2022/#most-popular-technologies-webframe) it's getting fairly popular - I'm a huge fan personally. It comes out the box with hybrid client, server, and static rendering, route pre-fetching, an image component that does all the hardwork for you, TypeScript support and more out of the box. 

With static rendering, it means I can pre-fetch all my content at compile time and render it in the lightest way possible to my readers, so it's as fast as can be. Oh, and I'm hosting it on Vercels edge network (for free!) for world wide distribution. 

Personally I think it's the nicest way to get a React project going these days - I'll mention that while I used to specialise in front-end tech, I've been less involved in it recently so feel free to correct me! Remix was also a consideration but I stuck to the devil I knew this time around. 

## Why / What is Sanity?
Sanity is a CMS (content management system) that puts developers first. If you've ever used or integrated with one before you'll likely find that the structure of the content you have to integrate with is strict and uncomfortable and the flexibility is usually limited. Another issue is it's usually all defined in some external service with no source control or ability to enforce rules and validations as and when you need them. 

Sanity does away with that by letting you define all of your objects (think chunks of content) and documents (defined groups of content) in JSON which you control and manage the code for. This means you can create usable pieces, import them into your version of the Sanity editor (Sanity Studio) and deploy it! 

**It's nuts** - in the new world:

* All of your schema can live in source control
* It can be as granular as you like with validation, rules and documentation you surface to your users
* You can create and use community made, open source plugins in your editor
* You can self host it
* It's free

<Divider />

I had used Sanity in the last year when we built the Multiverse marketing site (which [we won a Webby award](https://winners.webbyawards.com/2022/websites-and-mobile-sites/general-websites-and-mobile-sites/employment/212761/multiverse--building-an-effective-marketing-site-and-scaling-the-brand-to-a-new-region-and) for!) from scratch using it with React and Next.js - the development experience was ridiculously nice so I wanted to encorporate some of that into my blogging and personal site. 

My previous personal sites had always been more of a manual affair. I had to do something around the realms of:
* Clone or update the repo depending on where I was accessing it from
* Re-teach myself how I'd stored the data
* Write some JS or HTML
* Commit the code
* Push the code to source control
* Re-deploy it manually or FTP it (*yes, that's right*)

Now with Next.js, Vercel hosting and Sanity, it looks more like this:
* Access my authorised, hosted Sanity Studio (for free)
* Change whatever I like in Markdown or alter dedicated fields (with validation!)
* Hit publish

I've setup webhooks so that everytime I publish from Sanity I trigger a new deployment in Vercel which statically generates all of my content, styling and markup at compile time and hosts it on Vercels edge network globally which means my platform is as performant as can be. Sanity and Next.js also take care of rendering images beautifully with webp, and blured placeholders. 

## Why not blog on a dedicated platform?
I used to love Medium - it's awesome for a bunch of reasons. The design is beautiful, it's given writers a way to grow an audience without having to do the leg work of a self-hosted blog, it's promoted content is normally quite good, the list goes on. 

But over the last few years I'd found myself becoming less and less enamoured with the platform. The introduction of the partner program seems to have bred a culture of writers creating un-original content with spammy titles just to get clicks, it's been YouTube-ified, and that's a real shame.

There's also the controversy that arose when [FreeCodeCamp stole everyones content without consent](https://dev.to/ben/i-m-concerned-with-the-move-that-freecodecamp-just-pulled-by-leaving-medium-io8). **Yikes.**

I've also found that I sometimes wanted more powerful editing features. 

**Seriously Medium, it's 2022, why do I still have to host code snippets in gists and embed them?**

That said, I still like the platform and I'll be cross-posting there too for increased exposure thanks to the power of cannonical URLs. 

## What's new and funky?

### MDX
This is one of my favourite bits - I'm using MDX for my blog posts which means I can write my content in markdown in Sanity and include JSX in the content itself which gets parsed and rendered out to the user. <Sparkles>It means I can do things like this and control it all from a markdown file I have hosted on the internet </Sparkles> - credit goes to [Josh Comeau](https://www.joshwcomeau.com/) for the awesome sparkles tutorial.

Since I can render React components (that my code knows about beforehand) I can do some pretty powerful stuff.

<CodeEditor code="<pre>Like render a JSX code editor on my blog</pre>" />

### Code Snippets with Prism
I've also added code snippets as you'd expect from any decent developer blog. This snippet is how I determine if I should add a sparkly "New" tag next to my blog post listings on the homepage.

```JavaScript
const NewTag = ({createdAt}) => {
  const oneWeekAgo = dayjs().subtract(1, 'week')
  const publishedDate = dayjs(createdAt)
  const isNew = publishedDate.isAfter(oneWeekAgo, 'week')

  return isNew && <Sparkles><Tag>New</Tag></Sparkles>
}
```

### Lottie
If you scroll to the end of this blog post (hopefully after reading all of this delightful content), you'll see a lottie animation fire off. How cool. 

The keen-eyed amongst you reading on desktop might have also noticed a progress indicator in the top left of the screen signaling how far you are through the post itself. This, and the lottie trigger are tracked using a `useEffect` hook alongside some magic from `framer-motion`.


```JavaScript
  const { scrollYProgress } = useViewportScroll();
  
  const yRange = useTransform(scrollYProgress, [0, 0.9], [0, 1]);
  const pathLength = useSpring(yRange, { stiffness: 400, damping: 90 });
  const lottieRef = useRef();

  useEffect(() => yRange.onChange((v) => {
    if (v >= 1) {
      setTimeout(() => lottieRef.current.play(), 500);
    }
  }), [yRange]);
```
	
##  Cool, what's going to be on here?
I'm going to be blogging a lot more this year I hope and on a vast range of topics, including:
* Elixir, Phoenix and LiveView 
* Front-end technologies like React, Svelte and Vue
* CSS tips 
* New language deep dives
* Architecture and macro-design
* My opinions on the technical landscape
* Coding tutorials and code snippets
* Lots, lots more

I also do a fair bit of photography if that interests you. 

## How do I keep up?
*Lots of ways.*

* Follow me on [Twitter](https://twitter.com/_ChrisGreg)
* Follow me on [Instagram if you're interested in my photography](https://www.instagram.com/chrisgregori__/)
* Follow me on [Medium](https://medium.com/@chrisgregori)
* Sign up to my [SubStack](https://chrisgregori.substack.com/) using the subscription box below to get my articles straight into your inbox

I'm always down for a chat so feel free to reach out!

Chris Gregori.

Creations

2025

mmbl

2024

Uini

Bloom UI

2023

Tiny Tales

Meal Mind

Sweet Talk

ROG Ally Games DB

2022

Play Date Video Game Calendar

6words

madepublic

2021

Nice Ice

Timebomb

2019

Triple Equals

Flagmojis

Football Tonight

2018

Seagull Trainer

Jott

Loggr

Opinions · RSS

Parsing PDFs (and more) in Elixir using Rust

Implementing Google Authentication in a LiveView Application

Erlang/OTP 26 Highlights

Elixir hot code swapping to the rescue

Burnout as an Indiehacker

Setting up a Google Chrome shortcut to Elixir documentation in 30 seconds

Using ETS to speed up database imports by almost 300%

How to build the simplest feature flag system in Elixir & Phoenix apps

Triggering repeatable animations from the server in LiveView & Elixir

Custom JSON encoding for structs in Elixir with Jason

10 tips for new Elixir developers

5 scary Git commands you'll eventually have use for

Why do developers experience so much imposter syndrome?

Embracing your own platform for blogging

How to become a better developer

Elm and why it's not quite ready yet

TIL - Git supports a global exclude file

"Why should I leave?"

talks

facts

Github

Twitter

Twitch

YouTube

LinkedIn

Instagram

Substack

Medium

Photography

My LiveView Book

currently

Photography