Daniel Agbemava

[Project A] Devlog #3: File uploads

Daniel Agbemava's photo
Daniel Agbemava
·

4 min read

Before we begin, if you're not sure what this is about you can check out the previous entry in this series right here.

Over the past couple of days, I have been trying to get file uploads working. The files we will be uploading are mainly csv / excel (well, for what I intend to do now anyway, other file types will come later). The idea is, we already have existing profile data stored in spreadsheets, so to make it so that the users do not have to enter the data manually, I decided to create a way for them to be able to upload the existing data into the app and have it processed for them so that from then on, they can just use the app to handle the new profiles.

In all likelihood, this method will see very little use and I considered just seeding the database with the existing data so that they don't have to do the upload in the first place, but I thought to myself: "What's the fun in that?". (Famous last words)

I looked around for packages I could use for processing csv & spreadsheets. I initially was going to use csv for the csvs but I found the API a bit confusing (I'm still new to elixir after all, haven't fully wrapped my head around the language quirks) so I later switched to nimble_csv. It seemed like it was more recommended because it was simpler (don't quote me on that). For spreadsheets, I was thinking xlsxir. (I haven't tested with this library yet)

I created a FileUpload module to handle the file uploads. I think I will rename this module a little later because currently, my vision is to have it do only csv and spreadsheet processing. After all, they share a few similar methods. I will probably use a different module for the other types of files, so that it's cleaner and easier to navigate through the files, but I digress.

The idea was to restrict the file types for the method I created so that only the csv and excel are supported. So I defined a method in the module like so

def save_file_data(%Plug.Upload{filename: _filename, content_type: content_type, path: path}) do
    case content_type do
      "text/csv" -> save_to_db(parse_csv(...))
      "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" -> save_to_db(parse_xlsx(...))
      _ -> {:error, "Unsupported format"}
    end
 end

So I get the content_type from the file and check if it's the type I want to handle, if it is I call the method required to process it, otherwise, I return a friendly message to the frontend. There are probably a few things I'm not handling at this point, but for now, this is ok. (The goal behind this series is incremental improvements after all.)

So at this point, everything was more or less like I imagined until I began testing the API. I created a separate resource in my router and exposed it to the frontend. The controller method itself was pretty straightforward

def bulk_upload(conn, %{"file" => upload}) do
    # Logger.info("file uploaded is: #{IO.inspect(upload)}")

    case FileUpload.save_file_data(upload) do
      {:error, message} -> render(conn, "upload_fail.json", message: message)
      _ -> render(conn, "upload_success.json")
    end
  end

The heavy lifting was offloaded to the save_file_data method.

When I began testing, I ran into a couple of issues (at this point, I am only testing CSV, spreadsheets will be done later) which in hindsight and I should have documented, but I can't seem to recall at the moment. I did a lot of running around and searching and chat-gpt prompting to get the solution that the current solution is like some Victor Frankestein's monster code.

defp parse_csv(body) do
    try do
    data = CSV.parse_stream(body)
      |> Enum.map(fn [...proplist] ->
        # ... do some small checks

        # return data here (omitted for brevity)
        %{}
      end)

      {:ok, data}
    rescue
      e in [RuntimeError] -> {:error, e.message}
    end
  end

It looks ok and it works, so I'm not going to worry too much about it for now, but I have noticed a few issues with the implementation as it is which I will fix when I am working on the spreadsheets (whenever that may be, I'll be leaving a todo in the source).

So to recap,

  • Profile uploads are working

  • We can do bulk uploads

  • Updates/retrievals and the likes are working for profiles

All around good progress I'd say. I may switch to the frontend for a couple of days because I need to start integrating with the backed. I may or may not write about it. I have set a hard deadline for myself that the profile and attendance features should be live by the 3rd of September so that I can start taking feedback, so we'll see how that goes.

Until next time, I bid you adieu.

ElixirPhoenix frameworkbackend